Compare commits

..
Author SHA1 Message Date
jochen 5955c4a767 Merge remote-tracking branch 'origin/main' into feat/mail-list 2026-10-23 11:53:29 +02:00
jochen 96b201920f Merge remote-tracking branch 'origin/main' into feat/mail-list 2026-10-23 01:07:15 +02:00
jochen d14ef647d6 issue 1859: mark the page after an act by a one-shot cookie, not its address
Review round 2 F3 (ADR 0517 §3, 1858/AC5): acted=1 stayed in the address, so a reload, a restored tab or a
bookmark of it recorded no page opened or opened. An act's 303 (and a mode chosen) now sets mail_acted for
10 seconds, HttpOnly and SameSite=Strict, Secure but on 127.0.0.1; the next list or pane it reaches records
nothing and clears it; the address carries no marker. The local run keeps the browser's other cookies.
2026-10-22 20:29:17 +02:00
jochen af89a69670 issue 1859: the page shown after an act records nothing, so each act leaves one line
Review round 1 F1 (ADR 0517 §3, §5): a form act went back to the list or the pane, which recorded page
opened or opened again, and the script's refresh did too, so last looked became the last act and opened
counts grew. The redirect after an act carries acted=1 and the refresh an X-Mail-Refresh header; that page
records nothing. The 1860 test now follows each redirect and counts every line, and reminds the task (F2,
1858/AC9), which the store's own integration test also reads back as a date.
2026-10-22 20:26:54 +02:00
jochen 4e68ec63c9 issue 1860: prove the operator's list end to end (integration, compiled only)
Both bundles start whole: mesh-mail-reader against a fresh database on the real store and an added event
on a real bus, mesh-mail-web's mail-reading asks carried to the reader's own tools/call as mesh-mail-web,
with stand-ins for mail-store and a fixed judgement. A task due today shows under today, ticked through
the page it is under handled today with one handled line holding no mail text, undo returns it, and a
correction from the reading pane keeps a lesson and one corrected line, within two minutes. pgx joins
mesh-mail-web's go.mod for the test's fresh database; the lead's try runs it.
2026-10-22 20:18:34 +02:00
jochen d8144b704d issue 1859: run the mail page locally with invented mail (1859.5)
So the operator can click through the page on their own machine before it is delivered: cmd/mail-page-dev
serves internal/web's real handlers on 127.0.0.1 alone, over invented mail in two accounts and an in-memory
mail reading that fills every section, with an always-signed-in stand-in; nothing reaches the bus, a
database, Keycloak or a mail server. -now fixes today. The README says how to run it and what is a stand-in.
2026-10-22 20:14:24 +02:00
jochen f554928ce1 issue 1859: modes by account, and the reading pane with its corrections; name a nameless sender (1859.4, 1850)
ADR 0515 §4, §5: the trusted setting modes names the accounts Work and Personal show, Everything shows every
granted account; /read shows one mail as escaped text with its tasks, promises and attachments as downloads,
records it opened, and offers its corrections (mark and group, not a task, not a promise, read later), each
calling correct. Issue 1850: a sender with no display name is named by its address, on the list, the pane and
All mail.
2026-10-22 20:09:29 +02:00
jochen ab152687e5 issue 1859: tick, undo and remind an item, and choose a mode, from the page (1859.3)
ADR 0517 §1, §2, §4, §5: /item and /mode are posts behind the session's token, each calling its one
mail-reading verb (handle, unhandle, remind, record) for the operator; the script gets done or failed with
the cause as JSON, a plain form goes back to the list or shows it again saying what failed and why.
2026-10-22 20:05:31 +02:00
jochen 7e83b99615 issue 1859: serve one script and one stylesheet from the page's own origin (1859.2)
ADR 0515 §6: /static/app.js and /static/app.css are built in and served with their types and nosniff; the
policy allows script and style from the origin alone and connect-src 'self'; no page carries an inline script,
a style or a handler. The script posts the page's forms without a reload and says a failure on the page.
2026-10-22 20:04:26 +02:00
jochen ad4f312263 issue 1859: open the mail page on the operator's list (1859.1)
ADR 0515 §1, §5: / greets the operator with what the mail signal told today and when they last looked, then
shows today, this week, later, the promises by kind, good to know, read later, handled today and the scam
warnings, from mail-reading.list; it records the page opened. The classical list moves to /all, as All mail.
2026-10-22 20:03:21 +02:00
jochen 4f3ebe370b issue 1859: answer the mail page's greeting and reading pane from mail reading
ADR 0515 §1, §5: list also answers the mails the mail signal told of today, and each line its mail's folder
and UID; entries takes an id and answers that one entry whole, with its sender's address, tasks, promises
and read-later, for the reading pane's corrections.
2026-10-22 20:00:44 +02:00
jochen f24a4d858a issue 1859: move the mail page into internal/web, so the local run can serve it
A program cannot import a main package; the page's code moves to modules/mesh-mail-web/internal/web, and
cmd/mesh-mail-web is a two-line program calling it, so cmd/mail-page-dev (1859.5) serves the same handlers.
Nothing the module does changes.
2026-10-22 19:58:43 +02:00
jochen 4071349078 issue 1858: prove each act and its line in the real store (integration, compiled only)
The lead's try runs it: an act and its line kept together, a failed act keeping neither, correct's three
writes together, the removal of old lines, and handling gone with its entry.
2026-10-22 19:56:31 +02:00
jochen 0e9004664c issue 1858: write correct's line in its own transaction, and guard who writes the log (1858.6)
ADR 0517 §5: correct keeps the entry's change, its reading lesson and one corrected line together, or none
of them; a test reads the source to prove only the acts write the action log, and each new verb refuses an
ungranted caller. The README and the callers setting name the twelve verbs.
2026-10-22 19:56:16 +02:00
jochen 0ed94e5466 issue 1858: remove action lines older than 365 days each night (1858.5)
ADR 0517 §9: the action log keeps a line one year; the removal runs at start and each night at 03:00
(Europe/Brussels) and says how many it removed. Handling goes with its entry by the schema's cascade.
2026-10-22 19:54:57 +02:00
jochen da8a6a6df4 issue 1858: record the acts that change no entry, and answer the action log (1858.4)
ADR 0517 §3 to §5: record writes one line for a mail opened, a mode chosen or the page opened (the last
looked list answers); every other line is its own verb's. actions answers the log newest first.
2026-10-22 19:54:28 +02:00
jochen fac091756e issue 1858: answer the operator's list for a mode (1858.3)
ADR 0517 §4, ADR 0515 §1: list gives tasks under today, this week and later by due date in the operator's
zone, promises by kind, good to know, read later, handled today with its undo and the scam warnings, for the
mode's accounts, and when the operator last looked.
2026-10-22 19:53:47 +02:00
jochen a5ffe4eea7 issue 1858: handle, unhandle and remind an item, each with its own line (1858.2)
ADR 0517 §1, §2, §4, §5: an item is a task, a promise or the mail itself; each act and its action line are
kept in one transaction, a refused or failed act writes none, an undo lasts until the day ends and a
reminder shows the item again the next day (Europe/Brussels). The seat's serves list in mesh-mail's
manifest names the new verbs.
2026-10-22 19:51:54 +02:00
jochen fdd56d2259 issue 1858: keep the operator's handling and the action log (1858.1)
ADR 0517 §1 to §3, §9 to §11: migration 0005 and schema.sql add item_handling (per item, gone with its
entry) and action_line (what, mail id, item, mode, when; never mail text); the store writes each act and its
line in one transaction.
2026-10-22 19:50:15 +02:00
jochen 75ce0e8a8c issue 1857: read the last 14 days of inbox mail once more (1857.5)
ADR 0516 §5: so the operator's list does not start empty, the inbox mail of the last 14 days of each granted
account is read again in ordinary batches, after any mail just heard; it is kept as done in its own table
(migration 0004), and the journal says its start, end and count. The reader now invokes mail-store.search.
2026-10-22 19:48:21 +02:00
jochen af54720907 issue 1857: let correct say not a task, remove a promise, or set read later (1857.4)
ADR 0516 §4: each is kept as a reading lesson, as a change of mark or mail group is, so later batches with a
mail of that sender follow it; an item number the entry does not hold is refused.
2026-10-22 19:42:04 +02:00
jochen 6b759f708e issue 1857: judge at most 6 mails a batch (1857.3)
ADR 0516 §3: the answer grows by each mail's tasks and promises, so a batch holds 6 mails, not 10, to stay
within its 40 s; a test proves the largest question stays under 64 KiB with every mail whole.
2026-10-22 19:41:17 +02:00
jochen c3b504e26e issue 1857: ask and check each mail's tasks, promises and read-later (1857.2)
ADR 0516 §1, §2: the answer schema asks at most 3 tasks and 3 promises in the model's own words and the
read-later flag; ParseAnswer refuses a 4th, an unknown kind, a line over 120 characters or a bad date, and a
line sharing a run of 8 words with its mail's text fails the batch, so nothing copied is ever kept.
2026-10-22 19:40:33 +02:00
jochen daff20d351 issue 1857: keep a mail entry's tasks, promises and read-later (1857.1)
ADR 0516 §1: an entry holds at most 3 tasks and 3 promises in the model's own words, and whether it is read
later. Migration 0004 and schema.sql add mail_task, mail_promise and read_later; the store writes and reads
them with their entry, in one transaction.
2026-10-22 19:39:15 +02:00
jochen a3626e7920 issue 1580: say no batch has run yet until one has, then when it succeeded
The health answered that the last batch judgement succeeded on a fresh database, before any batch ran.
reader_health keeps when a batch last succeeded (migration 0003, backfilled from the last judged entry),
and the health and status say it, or that no batch has run yet.
2026-10-22 19:37:20 +02:00
jochen 92dc8e313c issue 1849: keep one mail entry per message, by its Message-ID, else its folder and UID
Gmail files one message in INBOX and All Mail, each with its own UID; the reader kept an entry per folder.
waiting_mail and mail_entry are now unique per account and message key (migration 0002 and schema.sql), a
mail that already has its entry no longer waits again, and the test double keeps PgStore's own keys
(issue 1548). A schema model test proves schema.sql equals the migrations in order.
2026-10-22 19:36:19 +02:00
953 changed files with 36873 additions and 56565 deletions
-13
View File
@@ -14,7 +14,6 @@ modules/claude-code/claude-code
modules/mesh-watcher/mesh-watcher
modules/dnsmasq/dnsmasq
modules/mesh-issues/cmd/mesh-issues/mesh-issues
modules/mesh-issues/mesh-issues
modules/mounts/cmd/mounts/mounts
modules/mesh-knowledge/cmd/mesh-knowledge/mesh-knowledge
modules/mesh-knowledge/mesh-knowledge
@@ -29,15 +28,3 @@ modules/brother/cmd/brother/brother
modules/mesh-build-go/cmd/mesh-build-go/mesh-build-go
modules/mesh-config/mesh-config
modules/mesh-build-image/cmd/mesh-build-image/mesh-build-image
*.test
# binaries a local go build leaves in a module root
/checks/paths/paths
/modules/mesh-build-go/mesh-build-go
/modules/mesh-delivery-gate/mesh-delivery-gate
modules/mesh-config/cmd/mesh-config/mesh-config
# Built by their modules' build step (issue 1577), never committed.
modules/forticlient/cmd/forticlient-tools/forticlient-tools
modules/networkmanager/uplink-tools
checks/words/words
-5
View File
@@ -1,5 +0,0 @@
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# Check("../.."): every manifest.
modules/*/module.json
-4
View File
@@ -1,7 +1,3 @@
module git.novox.be/novox/mesh-catalog/checks/config
go 1.22
require git.novox.be/novox/mesh-catalog/checks/trunk v0.0.0
replace git.novox.be/novox/mesh-catalog/checks/trunk => ../trunk
+1 -1
View File
@@ -110,7 +110,7 @@ func Check(root string) (map[string][]string, error) {
// The modules taking a whole layer were found after the list was first written and before the trunk held a copy
// (issue 735), so they stand in it too.
var seed = map[string]bool{
"fail2ban": true, "influxdb": true, "keycloak": true, "mailu": true, "searxng": true,
"betterbird": true, "fail2ban": true, "influxdb": true, "keycloak": true, "mailu": true, "searxng": true,
"baserow": true, "confluence": true, "gitea": true, "gitlab": true, "grafana": true, "home-assistant": true,
"icecast": true, "jira": true, "letta": true, "matrix": true, "nodered": true, "supabase": true, "unifi": true,
}
-5
View File
@@ -1,5 +0,0 @@
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# Check("../.."): every manifest and the files it names.
modules
-33
View File
@@ -1,33 +0,0 @@
package main
import (
"path/filepath"
"testing"
"git.novox.be/novox/mesh-catalog/checks/trunk"
)
// The script's comparison of this list with the trunk's (novox/hq ADR 0529 clause 2): the list of modules not yet moved gains none
// the trunk's copy at the merge base does not hold. The catalogue itself is judged against the list by
// TestTheCatalogueSaysConfigInEveryMovedModule. Skipped, and said, where no trunk is beside.
func TestTheUnmovedListHoldsNoModuleTheTrunksDoesNot(t *testing.T) {
path, ok := trunk.Copy(t, "checks/config/unmoved")
found, err := Check(filepath.Join("..", ".."))
if err != nil {
t.Fatal(err)
}
unmoved, err := Unmoved("unmoved")
if err != nil {
t.Fatal(err)
}
base := seed
if ok {
if base, err = Unmoved(path); err != nil {
t.Fatal(err)
}
}
_, _, added := Judge(found, unmoved, base)
for _, m := range added {
t.Errorf("checks/config/unmoved gains %s, which the trunk's copy does not hold: the list only shrinks", m)
}
}
-5
View File
@@ -1,5 +0,0 @@
module checks/credentialform
go 1.26.0
require git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030001116-2da0de75c3f0
-2
View File
@@ -1,2 +0,0 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030001116-2da0de75c3f0 h1:lKU3HjNXGiVbLW2hUgFjU/TMTUYMlNnnmZxG5T7biQA=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030001116-2da0de75c3f0/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
-70
View File
@@ -1,70 +0,0 @@
// Package credentialform refuses an offer that does not say the form of its credential: an own secret, an issuer, or
// `form` pair or none (the lead, 2026-10-29). It reads every modules/*/module.json through mesh-sdk's
// CredentialFormProblems.
package credentialform
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"sort"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
// Problems is every offer of the manifests under root/modules that says no credential form, one line each.
func Problems(root string) ([]string, error) {
files, err := filepath.Glob(filepath.Join(root, "modules", "*", "module.json"))
if err != nil {
return nil, err
}
sort.Strings(files)
var out []string
reads := map[string]bool{}
var binders []string
for _, f := range files {
raw, err := os.ReadFile(f)
if err != nil {
return nil, err
}
var m catalogue.Manifest
if err := json.Unmarshal(raw, &m); err != nil {
return nil, fmt.Errorf("%s: %w", f, err)
}
out = append(out, catalogue.CredentialFormProblems(m)...)
var kept struct {
Reads []string `json:"reads"`
State []json.RawMessage `json:"state"`
}
_ = json.Unmarshal(raw, &kept) // reads may hold objects in another module; only the provisioner's are read
if m.Module == "mesh-provisioner" {
for _, r := range kept.Reads {
reads[r] = true
}
}
if len(m.Provides) > 0 && keepsBound(kept.State) {
binders = append(binders, m.Module)
}
}
for _, module := range binders {
if !reads[module+".bound"] {
out = append(out, fmt.Sprintf("%s provides and keeps a `bound` state, and mesh-provisioner's manifest does not read %s.bound", module, module))
}
}
return out, nil
}
// keepsBound says whether a manifest's `state` names `bound`, as a plain name or an object with a name.
func keepsBound(state []json.RawMessage) bool {
for _, s := range state {
var name string
var obj struct {
Name string `json:"name"`
}
if json.Unmarshal(s, &name) == nil && name == "bound" || json.Unmarshal(s, &obj) == nil && obj.Name == "bound" {
return true
}
}
return false
}
-49
View File
@@ -1,49 +0,0 @@
package credentialform
import (
"os"
"path/filepath"
"strings"
"testing"
)
func aCatalogue(t *testing.T, manifests map[string]string) string {
t.Helper()
root := t.TempDir()
for module, body := range manifests {
dir := filepath.Join(root, "modules", module)
if err := os.MkdirAll(dir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "module.json"), []byte(body), 0o644); err != nil {
t.Fatal(err)
}
}
return root
}
// An offer that says no form is refused and named; one that says pair, none or an own secret passes; and the real
// catalogue is clean.
func TestEveryOfferSaysTheFormOfItsCredential(t *testing.T) {
root := aCatalogue(t, map[string]string{
"silent": `{"module":"silent","provides":[{"name":"thing","scope":"mesh"}]}`,
"binder": `{"module":"binder","provides":[{"name":"d","credential":{"form":"pair"}}],"state":["bound"]}`,
"mesh-provisioner": `{"module":"mesh-provisioner","reads":["other.bound"]}`,
"says": `{"module":"says","provides":[{"name":"a","credential":{"form":"pair"}},{"name":"b","credential":{"form":"none"}},{"name":"c","credential":{"own":"jwt"}}]}`,
})
got, err := Problems(root)
if err != nil {
t.Fatal(err)
}
if len(got) != 2 || !strings.Contains(got[0], `silent provides "thing"`) ||
!strings.Contains(got[1], "binder") || !strings.Contains(got[1], "mesh-provisioner") {
t.Fatalf("refused %v, wanted the silent offer and the binder the provisioner does not read", got)
}
real, err := Problems(filepath.Join("..", ".."))
if err != nil {
t.Fatal(err)
}
if len(real) != 0 {
t.Fatalf("the catalogue has offers with no credential form:\n%s", strings.Join(real, "\n"))
}
}
-3
View File
@@ -1,3 +0,0 @@
# What these tests read outside their package, as paths from the repository's root (issue 1936).
# every manifest, for what each offer says of its credential's form.
modules/*/module.json
-128
View File
@@ -1,128 +0,0 @@
// Package failurekit refuses a build module whose tests do not run mesh-sdk's conformance kit (issue 1901, its thorough
// fix): a module that claims a node-build-* seat and serves a test step must hold a test that calls conformance.Run, so
// its parse of the test runner's output is held to the failure contract (a failing test named with its cause, never a
// passing line in its place). A build module with no test step (mesh-build-image builds images and runs no tests) is
// exempt from that, and said so. Every build module, test step or not, must also hold a test that calls
// conformance.LeftRunning with its own start sweep, so a step a previous process left running is ended at its start
// (issue 1890).
package failurekit
import (
"encoding/json"
"fmt"
"io/fs"
"os"
"path/filepath"
"sort"
"strings"
)
// Judged is what the check says of one build module: Refused says why it fails the check, "" when it passes; Exempt
// says why the kit does not apply.
type Judged struct {
Module, Refused, Exempt string
}
type manifest struct {
Claims []struct {
Name string `json:"name"`
} `json:"claims"`
Tools []json.RawMessage `json:"tools"`
}
// Judge reads every modules/<m>/module.json of the catalogue at root and judges each module that claims a node-build-*
// seat.
func Judge(root string) ([]Judged, error) {
manifests, err := filepath.Glob(filepath.Join(root, "modules", "*", "module.json"))
if err != nil {
return nil, err
}
sort.Strings(manifests)
var out []Judged
for _, p := range manifests {
raw, err := os.ReadFile(p)
if err != nil {
return nil, err
}
var m manifest
if err := json.Unmarshal(raw, &m); err != nil {
return nil, fmt.Errorf("%s: %w", p, err)
}
seat := ""
for _, c := range m.Claims {
if strings.HasPrefix(c.Name, "node-build-") {
seat = c.Name
}
}
if seat == "" {
continue
}
dir := filepath.Dir(p)
j := Judged{Module: filepath.Base(dir)}
var refused []string
test := testTool(m.Tools)
switch {
case test == "":
j.Exempt = fmt.Sprintf("it claims %s and serves no test step, so the failure contract's cases do not apply", seat)
default:
runs, err := runsTheKit(dir, "conformance.Run(")
if err != nil {
return nil, err
}
if !runs {
refused = append(refused, fmt.Sprintf("it claims %s and serves the test step %s, and no test of it calls conformance.Run "+
"(mesh-sdk go/buildstep/conformance): run the kit with the module's own parse path (issue 1901)", seat, test))
}
}
// Every build module keeps steps, so every one runs the left-running case, test step or not (issue 1890).
sweeps, err := runsTheKit(dir, "conformance.LeftRunning(")
if err != nil {
return nil, err
}
if !sweeps {
refused = append(refused, fmt.Sprintf("it claims %s, and no test of it calls conformance.LeftRunning "+
"(mesh-sdk go/buildstep/conformance): run the kit's left-running case with the module's own start sweep (issue 1890)", seat))
}
j.Refused = strings.Join(refused, "; ")
out = append(out, j)
}
return out, nil
}
// testTool is the name of the module's test step tool (<prefix>_test), or "".
func testTool(tools []json.RawMessage) string {
for _, raw := range tools {
var name string
if json.Unmarshal(raw, &name) != nil {
var t struct {
Name string `json:"name"`
}
if json.Unmarshal(raw, &t) != nil {
continue
}
name = t.Name
}
if strings.HasSuffix(name, "_test") {
return name
}
}
return ""
}
// runsTheKit says a _test.go file under dir calls the kit's function call (such as "conformance.Run(").
func runsTheKit(dir, call string) (bool, error) {
found := false
err := filepath.WalkDir(dir, func(p string, d fs.DirEntry, err error) error {
if err != nil || found || d.IsDir() || !strings.HasSuffix(p, "_test.go") {
return err
}
raw, err := os.ReadFile(p)
if err != nil {
return err
}
found = strings.Contains(string(raw), call) &&
strings.Contains(string(raw), `"git.novox.be/novox/mesh-sdk/go/buildstep/conformance"`)
return nil
})
return found, err
}
-77
View File
@@ -1,77 +0,0 @@
package failurekit
import (
"os"
"path/filepath"
"testing"
)
// 1901/AC7 and 1890/AC8: a build module with a test step and no kit is refused, one that runs the kit passes, one with no
// test step is exempt from the failure cases but not from the left-running case, a module that runs only the failure
// cases is refused, and a module that claims no node-build-* seat is not judged.
func TestABuildModuleWithATestStepMustRunTheKit(t *testing.T) {
root := t.TempDir()
const imp = "package main\n\nimport \"git.novox.be/novox/mesh-sdk/go/buildstep/conformance\"\n\n"
const sweep = "func TestSweep(t *testing.T) { conformance.LeftRunning(t, nil) }\n"
kit := imp + "func TestKit(t *testing.T) { conformance.Run(t, conformance.GoTest, nil) }\n" + sweep
for path, body := range map[string]string{
"modules/a/module.json": `{"claims":[{"name":"node-build-a"}],"tools":["build_a_fetch","build_a_test"]}`,
"modules/a/cmd/a/check_test.go": "package main\n",
"modules/b/module.json": `{"claims":[{"name":"node-build-b"}],"tools":[{"name":"build_b_test"}]}`,
"modules/b/cmd/b/kit_test.go": kit,
"modules/c/module.json": `{"claims":[{"name":"node-build-c"}],"tools":["build_c_build"]}`,
"modules/c/cmd/c/kit_test.go": imp + sweep,
"modules/f/module.json": `{"claims":[{"name":"node-build-f"}],"tools":["build_f_build"]}`,
"modules/g/module.json": `{"claims":[{"name":"node-build-g"}],"tools":["build_g_test"]}`,
"modules/g/cmd/g/kit_test.go": imp + "func TestKit(t *testing.T) { conformance.Run(t, conformance.GoTest, nil) }\n",
"modules/d/module.json": `{"claims":[{"name":"other"}],"tools":["d_test"]}`,
"modules/e/module.json": `{"claims":[{"name":"node-build-e"}],"tools":["build_e_test"]}`,
"modules/e/cmd/e/kit.go": kit, // not a test file
} {
p := filepath.Join(root, path)
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(p, []byte(body), 0o644); err != nil {
t.Fatal(err)
}
}
got, err := Judge(root)
if err != nil {
t.Fatal(err)
}
want := map[string]string{"a": "refused", "b": "passes", "c": "exempt", "e": "refused", "f": "refused", "g": "refused"}
if len(got) != len(want) {
t.Fatalf("judged %+v, want a, b, c, e, f and g", got)
}
for _, j := range got {
is := "passes"
if j.Refused != "" {
is = "refused"
} else if j.Exempt != "" {
is = "exempt"
}
if is != want[j.Module] {
t.Errorf("%s %s (%+v), want it %s", j.Module, is, j, want[j.Module])
}
}
}
// Every build module of this catalogue with a test step runs the kit.
func TestTheCataloguesBuildModulesRunTheKit(t *testing.T) {
got, err := Judge(filepath.Join("..", ".."))
if err != nil {
t.Fatal(err)
}
if len(got) == 0 {
t.Fatal("no module claims a node-build-* seat: the check reads the wrong tree")
}
for _, j := range got {
switch {
case j.Refused != "":
t.Errorf("%s: %s", j.Module, j.Refused)
case j.Exempt != "":
t.Logf("%s is exempt: %s", j.Module, j.Exempt)
}
}
}
-3
View File
@@ -1,3 +0,0 @@
module git.novox.be/novox/mesh-catalog/checks/failurekit
go 1.22
-7
View File
@@ -1,7 +0,0 @@
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# Judge("../.."): every manifest, and the directory of each module claiming a node-build-* seat, walked for a test that
# calls the kit. The build modules are the mesh-build-* directories; a build module named otherwise needs its line here.
modules/*/module.json
modules/mesh-build-*
-8
View File
@@ -1,8 +0,0 @@
# The calls of the controller's hand-act log a manifest of this catalogue may still invoke, each `<module> <verb> <why>`:
# the fallback of a recorder (mesh-sdk handacts.Recorder) while the seat mesh-hand-acts has no holder that answers, until
# the controller's lease is written to a holder that is no controller. The check of this directory (handacts) refuses any
# other, and an entry whose module no longer invokes it: the list only shrinks, and it is empty before the lease write
# (novox/hq ADR 0565 §11; issue 1953, issue 774).
mesh-config hand-act fallback until the lease write: its sets, clears and held values record through mesh-hand-acts first
mesh-config warranted fallback until the lease write: its warrants record through mesh-hand-acts first
mesh-healers drill fallback until the lease write: a drill records through mesh-hand-acts first
-3
View File
@@ -1,3 +0,0 @@
module git.novox.be/novox/mesh-catalog/checks/handacts
go 1.22
-109
View File
@@ -1,109 +0,0 @@
// Command handacts refuses a manifest that records into the controller's hand-act log (novox/hq ADR 0565 §11): every
// recorder records through the seat mesh-hand-acts from its delivery, so no manifest invokes
// `seat:mesh-controller.hand-act`, `.warranted` or `.drill`. The recorders' fallbacks to the controller's log while the
// seat has no holder are named in checks/handacts/fallbacks, each module and verb with why; that list only shrinks, and
// an entry whose module no longer invokes the verb is refused too.
//
// go run . <catalog root>
//
// Exits 1 on a finding.
package main
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
)
// Verbs are the controller's hand-act verbs, as a manifest invokes them, and Allowed the list of named fallbacks.
var Verbs = []string{"hand-act", "warranted", "drill"}
const (
invokePrefix = "seat:mesh-controller."
Allowed = "checks/handacts/fallbacks"
)
// Findings is every refusal in the catalog at root, sorted.
func Findings(root string) ([]string, error) {
allowed := map[string]bool{}
raw, err := os.ReadFile(filepath.Join(root, Allowed))
if err != nil && !os.IsNotExist(err) {
return nil, err
}
for _, line := range strings.Split(string(raw), "\n") {
if f := strings.Fields(line); len(f) >= 3 && !strings.HasPrefix(f[0], "#") {
allowed[f[0]+" "+f[1]] = true
} else if len(f) > 0 && !strings.HasPrefix(f[0], "#") {
return nil, fmt.Errorf("%s: %q is not `<module> <verb> <why>`", Allowed, line)
}
}
manifests, err := filepath.Glob(filepath.Join(root, "modules", "*", "module.json"))
if err != nil {
return nil, err
}
var out []string
invoked := map[string]bool{}
for _, path := range manifests {
raw, err := os.ReadFile(path)
if err != nil {
return nil, err
}
var m struct {
Module string `json:"module"`
Invokes []string `json:"invokes"`
}
if json.Unmarshal(raw, &m) != nil || m.Module == "mesh-controller" {
continue
}
for _, inv := range m.Invokes {
verb, ok := strings.CutPrefix(strings.TrimSpace(inv), invokePrefix)
if !ok || !contains(Verbs, verb) {
continue
}
invoked[m.Module+" "+verb] = true
if !allowed[m.Module+" "+verb] {
out = append(out, fmt.Sprintf("%s invokes %s%s: record through seat:mesh-hand-acts.%s (novox/hq ADR 0565 §11), "+
"or name the fallback in %s", m.Module, invokePrefix, verb, verb, Allowed))
}
}
}
for entry := range allowed {
if !invoked[entry] {
out = append(out, fmt.Sprintf("%s names %q, which is no longer invoked: take it off the list, which only shrinks",
Allowed, entry))
}
}
sort.Strings(out)
return out, nil
}
func contains(list []string, s string) bool {
for _, x := range list {
if x == s {
return true
}
}
return false
}
func main() {
if len(os.Args) != 2 {
fmt.Fprintln(os.Stderr, "usage: handacts <catalog root>")
os.Exit(2)
}
found, err := Findings(os.Args[1])
if err != nil {
fmt.Println(err)
os.Exit(1)
}
for _, f := range found {
fmt.Println(f)
}
if len(found) > 0 {
os.Exit(1)
}
fmt.Println("no manifest records into the controller's hand-act log but the fallbacks named")
}
-59
View File
@@ -1,59 +0,0 @@
package main
import (
"os"
"path/filepath"
"strings"
"testing"
)
// No manifest invokes the controller's hand-act verbs but a named fallback; a named fallback no longer invoked is
// refused (novox/hq ADR 0565 §11's catalog test).
func TestFindings(t *testing.T) {
for _, row := range []struct {
name string
files map[string]string
want string // empty: clean
}{
{"clean", map[string]string{"modules/a/module.json": `{"module":"a","invokes":["seat:mesh-hand-acts.hand-act"]}`}, ""},
{"an unnamed call", map[string]string{"modules/a/module.json": `{"module":"a","invokes":["seat:mesh-controller.warranted"]}`},
"a invokes seat:mesh-controller.warranted"},
{"a named fallback, and the controller's own", map[string]string{
"modules/a/module.json": `{"module":"a","invokes":["seat:mesh-controller.drill"]}`,
"modules/mesh-controller/module.json": `{"module":"mesh-controller","invokes":["seat:mesh-controller.hand-act"]}`,
Allowed: "# why\na drill fallback until the lease write\n"}, ""},
{"a fallback no longer invoked", map[string]string{Allowed: "gone hand-act fallback\n"}, `names "gone hand-act"`},
{"another verb of the controller", map[string]string{"modules/a/module.json": `{"module":"a","invokes":["seat:mesh-controller.conditions"]}`}, ""},
} {
t.Run(row.name, func(t *testing.T) {
root := t.TempDir()
for path, text := range row.files {
p := filepath.Join(root, path)
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(p, []byte(text), 0o644); err != nil {
t.Fatal(err)
}
}
got, err := Findings(root)
if err != nil {
t.Fatal(err)
}
if all := strings.Join(got, "\n"); (row.want == "") != (all == "") || !strings.Contains(all, row.want) {
t.Fatalf("findings = %q, want one containing %q", all, row.want)
}
})
}
}
// The run of this check over the real tree.
func TestTheCatalogRecordsOnlyThroughTheSeat(t *testing.T) {
got, err := Findings(filepath.Join("..", ".."))
if err != nil {
t.Fatal(err)
}
for _, f := range got {
t.Error(f)
}
}
-3
View File
@@ -1,3 +0,0 @@
# What these tests read outside their package, as paths from the repository's root (issue 1936).
# every manifest, for what it invokes.
modules/*/module.json
-5
View File
@@ -1,5 +0,0 @@
module checks/health
go 1.26.0
require git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030001116-2da0de75c3f0
-2
View File
@@ -1,2 +0,0 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030001116-2da0de75c3f0 h1:lKU3HjNXGiVbLW2hUgFjU/TMTUYMlNnnmZxG5T7biQA=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030001116-2da0de75c3f0/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
-70
View File
@@ -1,70 +0,0 @@
// Package health is the catalogue's count of long-running resources that do not say how they are ready (novox/hq ADR 0240 rule
// 8), kept in health-undeclared and judged by a test over the real tree (ADR 0529 clauses 2 and 4): it replaces the step of
// merge-check.sh that read it out of the controller's `module check`. Every manifest is read through mesh-sdk's catalogue
// package, the one the controller counts with.
package health
import (
"fmt"
"os"
"path/filepath"
"sort"
"strconv"
"strings"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
// Count is the long-running resources without `health` of every module under root/modules.
func Count(root string) (int, error) {
manifests, err := filepath.Glob(filepath.Join(root, "modules", "*", "module.json"))
if err != nil {
return 0, err
}
sort.Strings(manifests)
if len(manifests) == 0 {
return 0, fmt.Errorf("no manifest under %s: the count would be zero for any catalogue", filepath.Join(root, "modules"))
}
n := 0
for _, path := range manifests {
m, err := catalogue.ReadTreeManifest(root, filepath.Dir(path))
if err != nil {
return 0, fmt.Errorf("%s: %w", path, err)
}
n += len(catalogue.Undeclared(m))
}
return n, nil
}
// Kept is the number the file keeps: the first number of its first line.
func Kept(path string) (int, error) {
raw, err := os.ReadFile(path)
if err != nil {
return 0, err
}
first, _, _ := strings.Cut(string(raw), "\n")
digits := ""
for _, r := range strings.TrimSpace(first) {
if r < '0' || r > '9' {
break
}
digits += string(r)
}
n, err := strconv.Atoi(digits)
if err != nil {
return 0, fmt.Errorf("%s does not begin with the number it keeps: %q", path, first)
}
return n, nil
}
// Judge says what is wrong with a count against the kept number; empty when they agree. It only goes down: a rise is refused,
// and a fall asks the new number written in health-undeclared, so the count cannot rise again.
func Judge(kept, counted int) string {
switch {
case counted > kept:
return fmt.Sprintf("the long-running resources without health rose from %d to %d: declare how each new one is ready (ADR 0240 rule 8)", kept, counted)
case counted < kept:
return fmt.Sprintf("the long-running resources without health fell from %d to %d: write %d in health-undeclared, so the count cannot rise again", kept, counted, counted)
}
return ""
}
-88
View File
@@ -1,88 +0,0 @@
package health
import (
"os"
"path/filepath"
"strings"
"testing"
)
func aCatalogue(t *testing.T, manifests map[string]string) string {
t.Helper()
root := t.TempDir()
for module, body := range manifests {
dir := filepath.Join(root, "modules", module)
if err := os.MkdirAll(dir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "module.json"), []byte(body), 0o644); err != nil {
t.Fatal(err)
}
}
return root
}
// A service that stays up and does not say how it is ready is counted; one that says, and a file, are not.
func TestOnlyALongRunningResourceWithoutHealthIsCounted(t *testing.T) {
root := aCatalogue(t, map[string]string{
"quiet": `{"module":"quiet","resources":[{"id":"svc","type":"service","unit":"quiet.service","state":"running"}]}`,
"says": `{"module":"says","resources":[{"id":"svc","type":"service","unit":"says.service","state":"running",` +
`"health":{"kind":"unit"}}]}`,
"file": `{"module":"file","resources":[{"id":"cfg","type":"file","path":"/etc/file.conf","content":"x"}]}`,
})
counted, err := Count(root)
if err != nil {
t.Fatal(err)
}
if counted != 1 {
t.Fatalf("counted %d, wanted the one quiet service", counted)
}
}
func TestTheCountMayOnlyGoDown(t *testing.T) {
for _, c := range []struct {
kept, counted int
says string
}{
{58, 58, ""},
{58, 59, "rose from 58 to 59"},
{58, 57, "fell from 58 to 57"},
} {
got := Judge(c.kept, c.counted)
if (c.says == "") != (got == "") || !strings.Contains(got, c.says) {
t.Errorf("kept %d counted %d: %q, wanted %q", c.kept, c.counted, got, c.says)
}
}
}
func TestTheKeptCountIsTheFirstNumberOfTheFile(t *testing.T) {
path := filepath.Join(t.TempDir(), "health-undeclared")
if err := os.WriteFile(path, []byte("58\nThe long-running resources ...\n"), 0o644); err != nil {
t.Fatal(err)
}
if n, err := Kept(path); err != nil || n != 58 {
t.Fatalf("%d %v", n, err)
}
if err := os.WriteFile(path, []byte("many\n"), 0o644); err != nil {
t.Fatal(err)
}
if _, err := Kept(path); err == nil {
t.Fatal("a file with no number gave a count")
}
}
// The script's `module check` run over the real tree (novox/hq ADR 0240 rule 8, ADR 0529 clause 2 and 4): the long-running
// resources that do not say how they are ready, counted as the controller counts them, are the number health-undeclared keeps.
func TestTheCatalogueSaysHowItsLongRunningResourcesAreReady(t *testing.T) {
kept, err := Kept(filepath.Join("..", "..", "health-undeclared"))
if err != nil {
t.Fatal(err)
}
counted, err := Count(filepath.Join("..", ".."))
if err != nil {
t.Fatal(err)
}
if say := Judge(kept, counted); say != "" {
t.Fatal(say)
}
}
-10
View File
@@ -1,10 +0,0 @@
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# every manifest through mesh-sdk's ReadTreeManifest: the root's predefined keys and each templated module's
# templates (files/ by default, or the directory its manifest names); and the root's kept count of resources that do not say how they are ready.
modules/*/module.json
predefined.json
modules/*/files
modules/*/templates
health-undeclared
-5
View File
@@ -1,5 +0,0 @@
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# Check("../.."): every manifest.
modules/*/module.json
-5
View File
@@ -1,5 +0,0 @@
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# Check("../.."): every manifest, and whether the path each image artifact builds from is there.
modules
-12
View File
@@ -61,15 +61,3 @@ func TestAModuleThatIsNotTypeScriptIsNotJudged(t *testing.T) {
t.Fatalf("%v %v", got, err)
}
}
// The script's run of this check over the real tree (novox/hq ADR 0529 clause 2): every TypeScript module that names packages
// beyond the SDK commits its lock.
func TestTheCatalogueCommitsTheLocksItNeeds(t *testing.T) {
found, err := Check("../..")
if err != nil {
t.Fatal(err)
}
for _, f := range found {
t.Errorf("%s names packages beyond the SDK (%v) and has no package-lock.json: commit one made by the build module's toolchain (novox/hq ADR 0469 §5)", f.Module, f.Packages)
}
}
-7
View File
@@ -1,7 +0,0 @@
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# Check("../.."): every manifest, a TypeScript module's package.json and whether its lock is there.
modules/*/module.json
modules/*/package.json
modules/*/package-lock.json
-5
View File
@@ -1,5 +0,0 @@
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# Check("../.."): every manifest.
modules/*/module.json
-3
View File
@@ -1,3 +0,0 @@
module git.novox.be/novox/mesh-catalog/checks/nobinaries
go 1.22
-84
View File
@@ -1,84 +0,0 @@
// Package nobinaries refuses a compiled program anywhere in the repository (issue 1577; first a checks/ binary slipped past a modules/-only rule): it is build output, made by the module's build step,
// and `go build` in a module's folder writes one beside go.mod where `git add -A` takes it.
package nobinaries
import (
"bytes"
"fmt"
"io"
"os"
"os/exec"
"path/filepath"
"strings"
)
// Kind says what executable format the first bytes of a file are, or "" when they are none.
func Kind(head []byte) string {
switch {
case bytes.HasPrefix(head, []byte("\x7fELF")):
return "ELF"
case bytes.HasPrefix(head, []byte("MZ")):
return "PE"
case bytes.HasPrefix(head, []byte{0xcf, 0xfa, 0xed, 0xfe}), bytes.HasPrefix(head, []byte{0xce, 0xfa, 0xed, 0xfe}),
bytes.HasPrefix(head, []byte{0xfe, 0xed, 0xfa, 0xcf}), bytes.HasPrefix(head, []byte{0xfe, 0xed, 0xfa, 0xce}),
bytes.HasPrefix(head, []byte{0xca, 0xfe, 0xba, 0xbe}) && len(head) >= 8 && head[7] < 0x20 && head[6] == 0 && head[5] == 0 && head[4] == 0:
return "Mach-O"
}
return ""
}
// Files are the files of the catalogue's root that the change holds, anywhere: the tracked ones when the root is a git
// checkout (an ignored binary a local build left is not the change's), else every file there but .git's.
func Files(root string) ([]string, error) {
cmd := exec.Command("git", "-C", root, "ls-files", "-z")
if out, err := cmd.Output(); err == nil && len(out) > 0 {
var files []string
for _, f := range strings.Split(strings.TrimRight(string(out), "\x00"), "\x00") {
if _, err := os.Lstat(filepath.Join(root, f)); err == nil { // a file the change deletes is not there
files = append(files, f)
}
}
return files, nil
}
var files []string
err := filepath.WalkDir(root, func(p string, d os.DirEntry, err error) error {
if err != nil {
return err
}
if d.IsDir() && d.Name() == ".git" {
return filepath.SkipDir
}
if d.IsDir() || !d.Type().IsRegular() {
return nil
}
rel, _ := filepath.Rel(root, p)
files = append(files, rel)
return nil
})
return files, err
}
// Find names each file of files (paths from root) that is a compiled program, with its format.
func Find(root string, files []string) ([]string, error) {
var found []string
for _, f := range files {
p := filepath.Join(root, f)
if st, err := os.Lstat(p); err != nil || !st.Mode().IsRegular() {
continue
}
fh, err := os.Open(p)
if err != nil {
return nil, err
}
head := make([]byte, 8)
n, err := io.ReadFull(fh, head)
fh.Close()
if err != nil && err != io.ErrUnexpectedEOF && err != io.EOF {
return nil, err
}
if k := Kind(head[:n]); k != "" {
found = append(found, fmt.Sprintf("%s (%s)", f, k))
}
}
return found, nil
}
-89
View File
@@ -1,89 +0,0 @@
package nobinaries
import (
"os"
"os/exec"
"path/filepath"
"testing"
)
func TestFindNamesEachCompiledProgramAndNothingElse(t *testing.T) {
root := t.TempDir()
files := map[string][]byte{
"modules/a/a": []byte("\x7fELF\x02\x01\x01\x00rest"),
"modules/b/cmd/b/b.exe": []byte("MZ\x90\x00"),
"modules/c/c": {0xcf, 0xfa, 0xed, 0xfe, 7, 0, 0, 1},
"modules/d/main.go": []byte("package main\n"),
"modules/e/short": []byte("\x7f"),
"modules/f/empty": {},
"modules/g/Foo.class": {0xca, 0xfe, 0xba, 0xbe, 0, 0, 0, 52}, // Java, not Mach-O
"checks/words/words": []byte("\x7fELF\x02\x01\x01\x00rest"), // outside modules/ too
}
var names []string
for f, b := range files {
p := filepath.Join(root, f)
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(p, b, 0o644); err != nil {
t.Fatal(err)
}
names = append(names, f)
}
got, err := Find(root, names)
if err != nil {
t.Fatal(err)
}
if len(got) != 4 {
t.Fatalf("found %v; wanted modules/a/a, modules/b/cmd/b/b.exe and modules/c/c and checks/words/words only", got)
}
}
// The catalogue's own tree holds no compiled program: a compiled program is build output, so remove it and add it to .gitignore.
func TestTheCatalogueHoldsNoCompiledProgram(t *testing.T) {
root := filepath.Join("..", "..")
files, err := Files(root)
if err != nil {
t.Fatal(err)
}
found, err := Find(root, files)
if err != nil {
t.Fatal(err)
}
for _, f := range found {
t.Errorf("%s: a compiled program is build output: remove it and add it to .gitignore (issue 1577)", f)
}
}
// A compiled program outside modules/ is refused too (a binary under checks/ slipped past the modules/-only rule): Files holds
// every tracked file of a git checkout and every file of a plain folder, and Find names the binary in both.
func TestAProgramUnderChecksIsRefused(t *testing.T) {
for _, git := range []bool{true, false} {
root := t.TempDir()
p := filepath.Join(root, "checks", "words", "words")
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(p, []byte("\x7fELF\x02\x01\x01\x00rest"), 0o644); err != nil {
t.Fatal(err)
}
if git {
for _, args := range [][]string{{"init", "-q"}, {"add", "-A"}} {
if out, err := exec.Command("git", append([]string{"-C", root}, args...)...).CombinedOutput(); err != nil {
t.Fatalf("git %v: %v: %s", args, err, out)
}
}
}
files, err := Files(root)
if err != nil {
t.Fatal(err)
}
found, err := Find(root, files)
if err != nil {
t.Fatal(err)
}
if len(found) != 1 {
t.Errorf("git=%v: found %v; wanted checks/words/words (ELF)", git, found)
}
}
}
-1
View File
@@ -1 +0,0 @@
# every tracked file of the repository (git ls-files, no path limit): the whole tree, so this package's verdict is never kept.
-4
View File
@@ -1,4 +0,0 @@
# The modules of this catalogue that still read the controller's assignments (mesh-controller.assignments), each with
# why. The check of this directory (noplacement) refuses any other module that reads them, and an entry here whose module no longer
# does: the list only shrinks, and it is empty before the controller's lease is written to a holder that is no
# controller (novox/hq ADR 0549 §3, §6; issue 774).
-7
View File
@@ -1,7 +0,0 @@
module git.novox.be/novox/mesh-catalog/checks/noplacement
go 1.22
require git.novox.be/novox/mesh-catalog/checks/trunk v0.0.0
replace git.novox.be/novox/mesh-catalog/checks/trunk => ../trunk
-170
View File
@@ -1,170 +0,0 @@
// Command noplacement refuses what the repository path of placement left behind (novox/hq ADR 0567 §10, §12).
//
// Placement, which module runs on which node, is bus state held by the seat mesh-placement; no file of the
// catalogue says it, and no manifest reads it from the delivery indexer, which no longer keeps it. It refuses:
//
// 1. a `placement` directory (or file) at the root of the catalogue;
//
// 2. a `particulars` file at the root (the mesh's node names come from the checker's run, §16);
//
// 3. a manifest naming a state, stream, event or bucket of the indexer's placement (`mesh-delivery-indexer.placement…`
// or `mesh-delivery-indexer_placement…`) in any string value, such as `reads`;
//
// 4. a manifest, other than mesh-controller's, that reads the controller's assignments (`mesh-controller.assignments`):
// every reader takes placement from the seat mesh-placement (novox/hq ADR 0549 §3, ADR 0567), and the lease write is
// refused while one still reads them (ADR 0549 §6; issue 774). The modules that still read them for what the
// placement does not carry are named in checks/noplacement/assignments-readers, each with why; that list only
// shrinks: an entry whose module no longer reads them is refused too.
//
// go run . <catalogue root>
//
// Exits 1 on a finding.
package main
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
)
// Prefixes are the names of the indexer's placement parts; a manifest string starting with one names them.
var Prefixes = []string{"mesh-delivery-indexer.placement", "mesh-delivery-indexer_placement"}
// Findings is every refusal in the catalogue at root, sorted.
func Findings(root string) ([]string, error) {
var out []string
if _, err := os.Lstat(filepath.Join(root, "placement")); err == nil {
out = append(out, "placement: the repository holds no placement; it is bus state of the seat mesh-placement (novox/hq ADR 0567 §10)")
}
if _, err := os.Lstat(filepath.Join(root, "particulars")); err == nil {
out = append(out, "particulars: the repository lists no node names; the checker hands them to the check (novox/hq ADR 0567 §16)")
}
manifests, err := filepath.Glob(filepath.Join(root, "modules", "*", "module.json"))
if err != nil {
return nil, err
}
sort.Strings(manifests)
for _, file := range manifests {
raw, err := os.ReadFile(file)
if err != nil {
return nil, err
}
var doc any
if err := json.Unmarshal(raw, &doc); err != nil {
return nil, fmt.Errorf("%s: %w", file, err)
}
module := filepath.Base(filepath.Dir(file))
for _, s := range named(doc, nil) {
out = append(out, fmt.Sprintf("%s: names %q, which the indexer no longer keeps; placement is the seat mesh-placement's (novox/hq ADR 0567 §12)", module, s))
}
}
readers, err := assignmentsReaders(root)
if err != nil {
return nil, err
}
out = append(out, readers...)
sort.Strings(out)
return out, nil
}
// AssignmentsRead is the controller's assignments as a manifest reads them, and Allowed the file, relative to the
// catalogue's root, that names the modules still reading them: `<module> <why>` per line, `#` a comment.
const (
AssignmentsRead = "mesh-controller.assignments"
Allowed = "checks/noplacement/assignments-readers"
)
// assignmentsReaders is every finding on the reads of the controller's assignments: a module other than the controller
// reading them that Allowed does not name, and a module Allowed names that no longer reads them (issue 774).
func assignmentsReaders(root string) ([]string, error) {
allowed := map[string]bool{}
raw, err := os.ReadFile(filepath.Join(root, Allowed))
if err != nil && !os.IsNotExist(err) {
return nil, err
}
for _, line := range strings.Split(string(raw), "\n") {
if f := strings.Fields(line); len(f) > 0 && !strings.HasPrefix(f[0], "#") {
allowed[f[0]] = true
}
}
manifests, err := filepath.Glob(filepath.Join(root, "modules", "*", "module.json"))
if err != nil {
return nil, err
}
var found []string
reads := map[string]bool{}
for _, path := range manifests {
raw, err := os.ReadFile(path)
if err != nil {
return nil, err
}
var m struct {
Module string `json:"module"`
Reads []string `json:"reads"`
}
if json.Unmarshal(raw, &m) != nil || m.Module == "mesh-controller" {
continue // a manifest that does not read is the manifest checks' to refuse
}
for _, r := range m.Reads {
if strings.TrimSpace(r) != AssignmentsRead {
continue
}
reads[m.Module] = true
if !allowed[m.Module] {
found = append(found, fmt.Sprintf("%s reads %s: read where modules run from mesh-delivery-assigner.placement, "+
"the seat mesh-placement's (novox/hq ADR 0549 §3, ADR 0567); the lease write is refused while a module reads "+
"the assignments (ADR 0549 §6)", m.Module, AssignmentsRead))
}
}
}
for module := range allowed {
if !reads[module] {
found = append(found, fmt.Sprintf("%s names %s, which no longer reads %s: take it off the list, which only shrinks",
Allowed, module, AssignmentsRead))
}
}
return found, nil
}
// named collects the strings of a decoded manifest that start with one of the Prefixes.
func named(v any, into []string) []string {
switch v := v.(type) {
case string:
for _, p := range Prefixes {
if strings.HasPrefix(v, p) {
return append(into, v)
}
}
case []any:
for _, e := range v {
into = named(e, into)
}
case map[string]any:
for _, e := range v {
into = named(e, into)
}
}
return into
}
func main() {
if len(os.Args) != 2 {
fmt.Fprintln(os.Stderr, "usage: noplacement <catalogue root>")
os.Exit(2)
}
found, err := Findings(os.Args[1])
if err != nil {
fmt.Println(err)
os.Exit(1)
}
for _, f := range found {
fmt.Println(f)
}
if len(found) > 0 {
os.Exit(1)
}
fmt.Println("no placement directory, no manifest reads the indexer's placement, and only the modules listed read the controller's assignments")
}
-88
View File
@@ -1,88 +0,0 @@
package main
import (
"os"
"path/filepath"
"strings"
"testing"
"git.novox.be/novox/mesh-catalog/checks/trunk"
)
func TestFindingsRefuseWhatThePlacementPathLeft(t *testing.T) {
for _, row := range []struct {
name string
files map[string]string
want string // empty: clean
}{
{"clean", map[string]string{"modules/a/module.json": `{"module":"a","reads":["mesh-delivery-assigner.placement"]}`}, ""},
{"a placement directory", map[string]string{"placement/ace.json": `{}`}, "placement: the repository holds no placement"},
{"a particulars file", map[string]string{"particulars": "ace\n"}, "particulars: the repository lists no node names"},
{"a read of the indexer's placement", map[string]string{"modules/a/module.json": `{"module":"a","reads":["mesh-delivery-indexer.placement"]}`}, `a: names "mesh-delivery-indexer.placement"`},
{"a read of its candidates", map[string]string{"modules/b/module.json": `{"module":"b","reads":["mesh-delivery-indexer.placement-candidates"]}`}, "placement-candidates"},
{"its event", map[string]string{"modules/c/module.json": `{"module":"c","on":[{"event":"mesh-delivery-indexer.placement-changed"}]}`}, "placement-changed"},
// Issue 774 (ADR 0549 §3, §6): a reader of the controller's assignments is refused unless the list names it, the
// controller's own manifest is not judged, and a name on the list whose module no longer reads them is refused.
{"an unlisted reader of the assignments", map[string]string{"modules/r/module.json": `{"module":"r","reads":["mesh-controller.assignments"]}`}, "r reads mesh-controller.assignments"},
{"a listed reader and the controller", map[string]string{"modules/l/module.json": `{"module":"l","reads":["mesh-controller.assignments"]}`,
"modules/mesh-controller/module.json": `{"module":"mesh-controller","reads":["mesh-controller.assignments"]}`, Allowed: "# why\nl its plan\n"}, ""},
{"a listed module that reads them no more", map[string]string{Allowed: "gone no longer reads\n"}, "names gone, which no longer reads"},
{"the seat's own state is allowed", map[string]string{"modules/d/module.json": `{"module":"d","reads":["mesh-placement.placement"]}`}, ""},
} {
t.Run(row.name, func(t *testing.T) {
root := t.TempDir()
for path, text := range row.files {
p := filepath.Join(root, path)
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(p, []byte(text), 0o644); err != nil {
t.Fatal(err)
}
}
got, err := Findings(root)
if err != nil {
t.Fatal(err)
}
if all := strings.Join(got, "\n"); (row.want == "") != (all == "") || !strings.Contains(all, row.want) {
t.Fatalf("findings = %q, want one containing %q", all, row.want)
}
})
}
}
// The script's run of this check over the real tree (novox/hq ADR 0529 clause 2).
func TestTheCatalogueHoldsNoPlacementOfTheRepositoryPath(t *testing.T) {
got, err := Findings(filepath.Join("..", ".."))
if err != nil {
t.Fatal(err)
}
for _, f := range got {
t.Error(f)
}
// Issue 774 (ADR 0549 §3, §6; the comparison of novox/hq ADR 0529 clause 2): the list of modules still reading the
// controller's assignments gains none the trunk's copy at the merge base does not hold. Its first copy gains nothing.
path, ok := trunk.Copy(t, Allowed)
if !ok {
return
}
names := func(path string) map[string]bool {
raw, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
out := map[string]bool{}
for _, line := range strings.Split(string(raw), "\n") {
if f := strings.Fields(line); len(f) > 0 && !strings.HasPrefix(f[0], "#") {
out[f[0]] = true
}
}
return out
}
base := names(path)
for module := range names(filepath.Join("..", "..", Allowed)) {
if !base[module] {
t.Errorf("%s gains %s, which the trunk's copy does not hold: the list only shrinks", Allowed, module)
}
}
}
-6
View File
@@ -1,6 +0,0 @@
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# the root, for a placement directory, and every manifest, for a read of the indexer's placement states.
?placement
modules/*/module.json
-3
View File
@@ -1,3 +0,0 @@
module checks/noscripts
go 1.22
-19
View File
@@ -1,19 +0,0 @@
package noscripts
import (
"errors"
"os"
"path/filepath"
"testing"
)
// The catalogue's checks are Go tests (novox/hq ADR 0529 clause 5, ADR 0445): the checker runs every nested Go module's tests by
// the language's convention and no script at the root says what else to run. A script back at the root would be a check nobody
// runs (ADR 0503 §3: since R4e the checker has not run them), so it is refused.
func TestNoScriptOfTheOldCheckStandsAtTheRoot(t *testing.T) {
for _, name := range []string{"merge-check.sh", "merge-check-typescript.sh", "check-nested-modules.sh"} {
if _, err := os.Stat(filepath.Join("..", "..", name)); !errors.Is(err, os.ErrNotExist) {
t.Errorf("%s stands at the catalogue's root (%v): what it ran is a Go test in checks/ or a step of the build module", name, err)
}
}
}
-7
View File
@@ -1,7 +0,0 @@
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# that none of the old check scripts stands at the root.
?merge-check.sh
?merge-check-typescript.sh
?check-nested-modules.sh
-4
View File
@@ -1,7 +1,3 @@
module git.novox.be/novox/mesh-catalog/checks/owned
go 1.22
require git.novox.be/novox/mesh-catalog/checks/trunk v0.0.0
replace git.novox.be/novox/mesh-catalog/checks/trunk => ../trunk
-5
View File
@@ -1,5 +0,0 @@
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# Check("../.."): every manifest; its lists are this package's own files.
modules/*/module.json
-41
View File
@@ -1,41 +0,0 @@
package main
import (
"testing"
"git.novox.be/novox/mesh-catalog/checks/trunk"
)
// The script's run of this check over the real tree (novox/hq ADR 0529 clause 2): every rule holds.
func TestTheCatalogueKeepsEveryRuleOfOwnedPaths(t *testing.T) {
found, err := Check("../..", nil)
if err != nil {
t.Fatal(err)
}
for _, f := range found {
t.Errorf("%s: %s: %s (rule %d, novox/hq ADR 0422 §8)", f.Module, f.ID, f.Why, f.Rule)
}
}
// And the list of system directories' known exceptions holds no line the trunk's copy at the merge base does not (rule 5). A trunk
// with none had no list: its first copy. Skipped, and said, where no trunk is beside.
func TestTheSystemListHoldsNoLineTheTrunksDoesNot(t *testing.T) {
path, ok := trunk.Copy(t, "checks/owned/system-known")
var base []string
if ok {
lines, err := readLines(path)
if err != nil {
t.Fatal(err)
}
base = append([]string{}, lines...)
}
found, err := Check("../..", base)
if err != nil {
t.Fatal(err)
}
for _, f := range found {
if f.Rule == 5 {
t.Errorf("%s: %s: %s (rule %d, issue 1841)", f.Module, f.ID, f.Why, f.Rule)
}
}
}
+7 -22
View File
@@ -2,9 +2,8 @@
// issue 1838): a module is for any mesh, so no manifest, file, code string, identifier, comment, test name or
// fixture of a covered module names a node of this one.
//
// The names are the mesh's own, not the repository's: the checker hands them in the environment value
// MESH_NODE_NAMES, comma-separated, from the node records it reads (novox/hq ADR 0567 §16). A run with none fails
// loudly, since it would pass on anything. The modules covered are in `covered` at checks/particulars/covered, one a line. A name is matched
// The names are in the file `particulars` at the catalogue's root, one a line, `#` comments — the one place
// they may be written. The modules covered are in `covered` at checks/particulars/covered, one a line. A name is matched
// whole and in any case; a hyphen, an underscore or a CamelCase boundary parts one, so a longer word holding a name is another word.
// The forge is not a node name: its host (git.novox.be) and the organisation path (novox/<repo>) are blanked
// before a line is read.
@@ -35,28 +34,14 @@ func main() {
if len(os.Args) > 1 {
root = os.Args[1]
}
os.Exit(check(root, namesFromEnv()))
}
// NamesEnv is the environment value that carries the mesh's node names (mesh-sdk delivery.NodeNamesEnv).
const NamesEnv = "MESH_NODE_NAMES"
// namesFromEnv is the names NamesEnv holds, comma-separated, trimmed, empty entries left out.
func namesFromEnv() []string {
var out []string
for _, n := range strings.Split(os.Getenv(NamesEnv), ",") {
if n = strings.TrimSpace(n); n != "" {
out = append(out, n)
}
}
return out
os.Exit(check(root))
}
// check runs the check over a catalogue and returns the exit code: 0 clean, 1 a finding, 2 a list unreadable.
func check(root string, names []string) int {
var err error
if len(names) == 0 {
err = fmt.Errorf("no node names (%s is empty) — the check would pass on anything", NamesEnv)
func check(root string) int {
names, err := readList(filepath.Join(root, "particulars"))
if err == nil && len(names) == 0 {
err = fmt.Errorf("particulars lists no name — the check would pass on anything")
}
var covered []string
if err == nil {
+7 -15
View File
@@ -74,24 +74,26 @@ func TestANodeNamedPathIsRefusedThroughRun(t *testing.T) {
}
}
func TestCheckReturnsOneOnAFindingAndTwoOnNoNames(t *testing.T) {
func TestCheckReturnsOneOnAFindingAndTwoOnAnUnreadableList(t *testing.T) {
root := t.TempDir()
write := func(p, s string) {
p = filepath.Join(root, p)
os.MkdirAll(filepath.Dir(p), 0o755)
os.WriteFile(p, []byte(s), 0o644)
}
write("particulars", "ace\n")
write("checks/particulars/covered", "kernel\n")
write("modules/kernel/a", "clean\n")
if got := check(root, []string{"ace"}); got != 0 {
if got := check(root); got != 0 {
t.Errorf("clean = %d, want 0", got)
}
write("modules/kernel/a", "on ace\n")
if got := check(root, []string{"ace"}); got != 1 {
if got := check(root); got != 1 {
t.Errorf("finding = %d, want 1", got)
}
if got := check(root, nil); got != 2 {
t.Errorf("no names = %d, want 2", got)
os.Remove(filepath.Join(root, "particulars"))
if got := check(root); got != 2 {
t.Errorf("no list = %d, want 2", got)
}
}
@@ -112,13 +114,3 @@ func TestOnlyCoveredModulesAreRead(t *testing.T) {
t.Errorf("got %v", got)
}
}
// The script's run of this check over the real tree (novox/hq ADR 0529 clause 2): no node name of this mesh in a covered module.
func TestTheCoveredModulesNameNoNode(t *testing.T) {
if _, set := os.LookupEnv(NamesEnv); !set {
t.Skip("ADR 0567 §16: this checker hands no node names (before D14 is delivered); the skip goes once it is")
}
if code := check("../..", namesFromEnv()); code != 0 {
t.Fatalf("the check of the real catalogue exits %d (see what it printed above)", code)
}
}
-5
View File
@@ -1,5 +0,0 @@
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# check("../.."): every file of the modules the covered list names. The node names come from the run (MESH_NODE_NAMES).
modules
-4
View File
@@ -1,7 +1,3 @@
module git.novox.be/novox/mesh-catalog/checks/paths
go 1.22
require git.novox.be/novox/mesh-catalog/checks/trunk v0.0.0
replace git.novox.be/novox/mesh-catalog/checks/trunk => ../trunk
+1 -2
View File
@@ -251,8 +251,7 @@ func main() {
}
var grown []string
if baseDir == "" {
fmt.Fprintln(os.Stderr, "NOT COMPARED: no base branch's lists were given (-base <dir>), so additions to them were not judged; that is a failure, not a pass (issue 1295)")
os.Exit(2)
fmt.Println("NOT COMPARED: no base branch's lists were given, so additions to them were not judged")
} else {
baseShared, _, err1 := list(filepath.Join(baseDir, "shared"))
baseKnown, _, err2 := list(filepath.Join(baseDir, "known"))
-5
View File
@@ -1,5 +0,0 @@
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# Check("../.."): every manifest and every file of a module.
modules
-63
View File
@@ -1,63 +0,0 @@
package main
import (
"path/filepath"
"testing"
"git.novox.be/novox/mesh-catalog/checks/trunk"
)
// The script's run of this check over the real tree (novox/hq ADR 0529 clause 2): no module names another's state on its machine,
// and the known list holds only what still occurs.
func TestNoModuleNamesAnothersState(t *testing.T) {
root := filepath.Join("..", "..")
found, err := Check(root)
if err != nil {
t.Fatal(err)
}
shared, _, err := list("shared")
if err != nil {
t.Fatal(err)
}
known, _, err := list("known")
if err != nil {
t.Fatal(err)
}
refused, gone := Judge(found, shared, known)
for _, f := range refused {
t.Error(f)
}
for _, k := range gone {
t.Errorf("%q is in checks/paths/known and no longer occurs: take it out, so it cannot come back", k)
}
}
// And neither list gains a line the trunk's copies at the merge base do not hold, unless the line says why after `# allow:`. Fails, saying
// NOT COMPARED, where no trunk is beside (issue 1295).
func TestTheListsOfPathsHoldNoLineTheTrunksDoNot(t *testing.T) {
sharedPath, okShared := trunk.Copy(t, "checks/paths/shared")
knownPath, okKnown := trunk.Copy(t, "checks/paths/known")
if !okShared || !okKnown {
t.Log("the trunk holds not both lists: their first copies, to which nothing was added")
return
}
shared, sharedWhy, err := list("shared")
if err != nil {
t.Fatal(err)
}
known, knownWhy, err := list("known")
if err != nil {
t.Fatal(err)
}
baseShared, _, err := list(sharedPath)
if err != nil {
t.Fatal(err)
}
baseKnown, _, err := list(knownPath)
if err != nil {
t.Fatal(err)
}
for _, g := range append(Grown("shared", shared, sharedWhy, baseShared), Grown("known", known, knownWhy, baseKnown)...) {
t.Error(g)
}
}
-1
View File
@@ -1,4 +1,3 @@
# Paths every program on a machine reads as the machine's own, not as a message from the module that
# writes them (checks/paths, novox/hq ADR 0260). One per line, with why.
/etc/hosts # every resolver on the machine reads it; the hostname module writes it for all of them
/var/lib/mesh-build-node-cache # allow: the Go build module gives its integration level the node's installed Node toolchains read-only, so the Node build module's tests build offline (issue 1891, ADR 0533)
-4
View File
@@ -1,7 +1,3 @@
module git.novox.be/novox/mesh-catalog/checks/predefined
go 1.22
require git.novox.be/novox/mesh-catalog/checks/trunk v0.0.0
replace git.novox.be/novox/mesh-catalog/checks/trunk => ../trunk
-6
View File
@@ -1,6 +0,0 @@
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# the root's list of predefined keys and every manifest that names one.
predefined.json
modules/*/module.json
-41
View File
@@ -1,41 +0,0 @@
package main
import (
"path/filepath"
"sort"
"testing"
"git.novox.be/novox/mesh-catalog/checks/trunk"
)
// The script's run of this check over the real tree (novox/hq ADR 0529 clause 2): predefined.json removes no key a module still
// names, judged against the trunk's copy at the merge base. A trunk with none had no keys; with no trunk to read, nothing can be
// seen removed, and the test says so.
func TestThePredefinedListRemovesNoKeyAModuleStillNames(t *testing.T) {
tree, err := Keys(filepath.Join("..", "..", File))
if err != nil {
t.Fatal(err)
}
path, ok := trunk.Copy(t, File)
if !ok {
t.Log("no trunk copy of " + File + ": no removal can be seen")
return
}
base, err := Keys(path)
if err != nil {
t.Fatal(err)
}
named, err := Named(filepath.Join("..", ".."))
if err != nil {
t.Fatal(err)
}
removed := Removed(base, tree, named)
keys := make([]string, 0, len(removed))
for k := range removed {
keys = append(keys, k)
}
sort.Strings(keys)
for _, k := range keys {
t.Errorf("%q is removed from %s and still named by %v", k, File, removed[k])
}
}
-4
View File
@@ -1,7 +1,3 @@
module git.novox.be/novox/mesh-catalog/checks/secrets
go 1.22
require git.novox.be/novox/mesh-catalog/checks/trunk v0.0.0
replace git.novox.be/novox/mesh-catalog/checks/trunk => ../trunk
-5
View File
@@ -1,5 +0,0 @@
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# Check("../.."): every manifest and the files it names.
modules
-25
View File
@@ -1,25 +0,0 @@
package main
import (
"testing"
"git.novox.be/novox/mesh-catalog/checks/trunk"
)
// The script's run of this check over the real tree (novox/hq ADR 0529 clause 2): the known list gains no line the trunk's copy
// at the merge base does not hold. Without a trunk copy the lines in seed are the trunk's.
func TestTheKnownListHoldsNoLineTheTrunksDoesNot(t *testing.T) {
known, err := Known("known")
if err != nil {
t.Fatal(err)
}
base := seed
if path, ok := trunk.Copy(t, "checks/secrets/known"); ok {
if base, err = Known(path); err != nil {
t.Fatal(err)
}
}
for _, line := range Added(known, base) {
t.Errorf("checks/secrets/known gains %q, which the trunk's copy does not hold: the list only shrinks — protect the key instead (novox/hq issue 342)", line)
}
}
-5
View File
@@ -1,5 +0,0 @@
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# Check("../.."): every manifest.
modules/*/module.json
-4
View File
@@ -1,7 +1,3 @@
module git.novox.be/novox/mesh-catalog/checks/templates
go 1.22
require git.novox.be/novox/mesh-catalog/checks/trunk v0.0.0
replace git.novox.be/novox/mesh-catalog/checks/trunk => ../trunk
-5
View File
@@ -1,5 +0,0 @@
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# Check("../.."): every manifest and its templates.
modules
-32
View File
@@ -1,32 +0,0 @@
package main
import (
"testing"
"git.novox.be/novox/mesh-catalog/checks/trunk"
)
// The script's comparison of this list with the trunk's (novox/hq ADR 0529 clause 2): the list of modules keeping file text inline
// gains none the trunk's copy at the merge base does not hold. The catalogue itself is judged against the list by
// TestTheCatalogueKeepsFileTextInlineOnlyWhereListed. Skipped, and said, where no trunk is beside.
func TestTheInlineListHoldsNoModuleTheTrunksDoesNot(t *testing.T) {
path, ok := trunk.Copy(t, "checks/templates/inline")
found, err := Check("../..")
if err != nil {
t.Fatal(err)
}
listed, err := Listed(listFile)
if err != nil {
t.Fatal(err)
}
base := seed
if ok {
if base, err = Listed(path); err != nil {
t.Fatal(err)
}
}
_, _, added := Judge(found, listed, base)
for _, m := range added {
t.Errorf("checks/templates/inline gains %s, which the trunk's copy does not hold: the list only shrinks", m)
}
}
-5
View File
@@ -1,5 +0,0 @@
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# Check("../.."): every manifest.
modules/*/module.json
-3
View File
@@ -1,3 +0,0 @@
module git.novox.be/novox/mesh-catalog/checks/trunk
go 1.22
-65
View File
@@ -1,65 +0,0 @@
// Package trunk is where a catalogue check finds the trunk's copy of a list that may only shrink (novox/hq ADR 0529 clause 2):
// the file as the catalogue's base commit holds it, to judge a change's list against.
//
// The checker fetches the change's own repository at its base commit beside the change, as it fetches a sibling, and says where
// in MESH_CHECK_BESIDE; the copy is under the repository's name (the same convention the other repositories' tests use). With
// MESH_CHECK_BESIDE unset, or set without the catalogue's copy in it, the test fails saying NOT COMPARED WITH THE TRUNK and
// why: the checker places the change's own repository since R4g (issue 1295), so a missing copy is a fault, never a pass. A trunk that holds no such file had
// none: the list's first copy, which nothing was added to.
package trunk
import (
"errors"
"fmt"
"os"
"path/filepath"
"testing"
)
// Repository is the catalogue's name beside a check.
const Repository = "mesh-catalog"
// BesideEnv is where a merge check says the clones beside the one checked are.
const BesideEnv = "MESH_CHECK_BESIDE"
// Copy is the trunk's copy of the file at rel (a path from the catalogue's root), written to a file of the test's own: its path,
// and whether the trunk holds one (false: its first copy). It fails the test when no copy of the catalogue is beside.
func Copy(t testing.TB, rel string) (string, bool) {
t.Helper()
raw, found, err := Read(rel)
switch {
case err != nil:
t.Fatal(err)
case !found:
t.Logf("the trunk holds no %s: its first copy", rel)
return "", false
}
path := filepath.Join(t.TempDir(), filepath.Base(rel))
if err := os.WriteFile(path, raw, 0o644); err != nil {
t.Fatal(err)
}
return path, true
}
// Read is the trunk's copy of rel and whether the trunk holds one. err is a trunk not beside the check (MESH_CHECK_BESIDE unset,
// or without the catalogue's directory), or a copy unreadable: a comparison that cannot be made is never a pass.
func Read(rel string) (content []byte, found bool, err error) {
beside := os.Getenv(BesideEnv)
if beside == "" {
return nil, false, fmt.Errorf("NOT COMPARED WITH THE TRUNK: %s is not set, so the trunk's copy of %s is not beside this run; the checker places the change's own repository there (issue 1295)", BesideEnv, rel)
}
dir := filepath.Join(beside, Repository)
if _, err := os.Stat(dir); errors.Is(err, os.ErrNotExist) {
return nil, false, fmt.Errorf("NOT COMPARED WITH THE TRUNK: %s is not beside this check in %s=%s, so %s is not judged against the trunk's copy; the checker places the change's own repository there at its trunk (issue 1295)", Repository, BesideEnv, beside, rel)
} else if err != nil {
return nil, false, err
}
raw, err := os.ReadFile(filepath.Join(dir, rel))
if errors.Is(err, os.ErrNotExist) {
return nil, false, nil
}
if err != nil {
return nil, false, err
}
return raw, true, nil
}
-43
View File
@@ -1,43 +0,0 @@
package trunk
import (
"os"
"path/filepath"
"strings"
"testing"
)
func TestTheCloneBesideTheCheckIsTheTrunk(t *testing.T) {
beside := t.TempDir()
dir := filepath.Join(beside, Repository, "checks", "x")
if err := os.MkdirAll(dir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "known"), []byte("a\nb\n"), 0o644); err != nil {
t.Fatal(err)
}
t.Setenv(BesideEnv, beside)
raw, found, err := Read("checks/x/known")
if err != nil || !found || string(raw) != "a\nb\n" {
t.Fatalf("read %q found %v err %v", raw, found, err)
}
if _, found, err := Read("checks/x/absent"); err != nil || found {
t.Fatalf("a file the trunk does not hold: found %v err %v; wanted its first copy", found, err)
}
}
func TestWithNothingBesideTheComparisonFailsAndSaysNotMade(t *testing.T) {
t.Setenv(BesideEnv, "")
_, found, err := Read("checks/x/known")
if found || err == nil || !strings.Contains(err.Error(), "NOT COMPARED") {
t.Fatalf("found %v, err %v; wanted a failure saying NOT COMPARED", found, err)
}
}
func TestACloneBesideWithoutTheCatalogueFailsAndSaysNotMade(t *testing.T) {
t.Setenv(BesideEnv, t.TempDir()) // the siblings are there, the catalogue's own base is not
_, found, err := Read("checks/x/known")
if found || err == nil || !strings.Contains(err.Error(), "NOT COMPARED") || !strings.Contains(err.Error(), "not beside") {
t.Fatalf("found %v, err %v; wanted a failure naming the missing copy", found, err)
}
}
-3
View File
@@ -1,3 +0,0 @@
# The modules of the catalogue whose texts still name the build agent or the toolchain images, and why (checks/words,
# buildagent_test.go). One module directory a line, with its reason after a #. It only shrinks.
modules/mesh-build-go # build.go:45 says "the build agent's recipe" in a step's description: the asks lane's module (issue 1887), not E6's to edit
-57
View File
@@ -1,57 +0,0 @@
package main
import (
"os"
"strings"
"testing"
)
// The build agent and the toolchain images are gone (novox/hq ADR 0524 §5, issue 1295 and E6 of milestone R4g): no text a module
// can show names them. Where the word stands for a reason that is another lane's or still waits for its change, buildagent-allowed
// names the module's directory with that reason and the date it is dated from; the list only shrinks: a directory on it that no
// longer names the word is refused, so the line is taken out.
var gone = []string{"build-agent", "build agent", "toolchain image"}
func allowedModules(t *testing.T) map[string]string {
t.Helper()
raw, err := os.ReadFile("buildagent-allowed")
if err != nil {
t.Fatal(err)
}
out := map[string]string{}
for _, line := range strings.Split(string(raw), "\n") {
entry, why, _ := strings.Cut(line, "#")
if entry = strings.TrimSpace(entry); entry != "" && !strings.HasPrefix(line, "#") {
if strings.TrimSpace(why) == "" {
t.Fatalf("%q is allowed without a reason", entry)
}
out[entry] = strings.TrimSpace(why)
}
}
return out
}
func TestNoTextOfTheCatalogueNamesTheBuildAgentOutsideTheModulesAllowed(t *testing.T) {
texts, _, err := collect("../..")
if err != nil {
t.Fatal(err)
}
allowed := allowedModules(t)
seen := map[string]bool{}
for _, f := range check(gone, texts) {
module := f
if rest, ok := strings.CutPrefix(f, "modules/"); ok {
module = "modules/" + strings.SplitN(rest, "/", 2)[0]
}
if _, ok := allowed[module]; ok {
seen[module] = true
continue
}
t.Errorf("%s (the build agent and the toolchain images are gone: say what stands in their place)", f)
}
for module, why := range allowed {
if !seen[module] {
t.Errorf("buildagent-allowed names %s (%s), which names the word no more: take the line out, so it cannot come back", module, why)
}
}
}
-3
View File
@@ -58,9 +58,6 @@ func checkLines(lines []retiredLine, texts []text) []string {
for _, l := range lines {
rx := linePattern(l)
for _, t := range texts {
if !strings.Contains(t.s, l.daemon) { // the pattern holds the daemon's name whole: the cheap test first
continue
}
if m := rx.FindStringSubmatchIndex(t.s); m != nil {
line := t.line
if line > 0 {
+36 -71
View File
@@ -43,63 +43,18 @@ func main() {
if len(os.Args) > 1 {
root = os.Args[1]
}
os.Exit(run(root))
}
// run reads every text a module can show under root and answers the command's exit code: 0 clean, 1 a finding, 2 a list or a
// file unreadable. The command prints what it found; a test of the real catalogue reads the code.
func run(root string) int {
words, err := readList(filepath.Join(root, "retired-words"))
if err != nil {
fmt.Fprintln(os.Stderr, "words:", err)
return 2
os.Exit(2)
}
if len(words) == 0 {
fmt.Fprintln(os.Stderr, "words: retired-words lists no word — the check would pass on anything")
return 2
os.Exit(2)
}
texts, files, err := collect(root)
if err != nil {
fmt.Fprintln(os.Stderr, "words:", err)
return 2
}
findings := check(words, texts)
for _, f := range findings {
fmt.Println(f)
}
if len(findings) > 0 {
fmt.Printf("words: %d use(s) of a word the glossary retired (novox/hq ADR 0244) — say it in the glossary's word\n", len(findings))
return 1
}
// And no daemon's line (novox/hq ADR 0272): what a module tells the operator to type is a `nox` line.
lines, err := readLines(filepath.Join(root, "retired-lines"))
if err != nil {
fmt.Fprintln(os.Stderr, "words:", err)
return 2
}
if len(lines) == 0 {
fmt.Fprintln(os.Stderr, "words: retired-lines lists no line — the check would pass on anything")
return 2
}
if findings := checkLines(lines, texts); len(findings) > 0 {
for _, f := range findings {
fmt.Println(f)
}
fmt.Printf("words: %d daemon's line(s) in what a module shows — the operator types one command, nox (novox/hq ADR 0272)\n",
len(findings))
return 1
}
fmt.Printf("words: %d retired words and %d daemons' lines, none in the %d strings of %d files\n", len(words), len(lines),
len(texts), files)
return 0
}
// collect reads every text a module can show under root: the strings of its own code that is not a test, and every string of
// its manifests.
func collect(root string) ([]text, int, error) {
var texts []text
files := 0
err := filepath.WalkDir(filepath.Join(root, "modules"), func(path string, d fs.DirEntry, err error) error {
err = filepath.WalkDir(filepath.Join(root, "modules"), func(path string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
@@ -132,7 +87,38 @@ func collect(root string) ([]text, int, error) {
texts = append(texts, found...)
return nil
})
return texts, files, err
if err != nil {
fmt.Fprintln(os.Stderr, "words:", err)
os.Exit(2)
}
findings := check(words, texts)
for _, f := range findings {
fmt.Println(f)
}
if len(findings) > 0 {
fmt.Printf("words: %d use(s) of a word the glossary retired (novox/hq ADR 0244) — say it in the glossary's word\n", len(findings))
os.Exit(1)
}
// And no daemon's line (novox/hq ADR 0272): what a module tells the operator to type is a `nox` line.
lines, err := readLines(filepath.Join(root, "retired-lines"))
if err != nil {
fmt.Fprintln(os.Stderr, "words:", err)
os.Exit(2)
}
if len(lines) == 0 {
fmt.Fprintln(os.Stderr, "words: retired-lines lists no line — the check would pass on anything")
os.Exit(2)
}
if findings := checkLines(lines, texts); len(findings) > 0 {
for _, f := range findings {
fmt.Println(f)
}
fmt.Printf("words: %d daemon's line(s) in what a module shows — the operator types one command, nox (novox/hq ADR 0272)\n",
len(findings))
os.Exit(1)
}
fmt.Printf("words: %d retired words and %d daemons' lines, none in the %d strings of %d files\n", len(words), len(lines),
len(texts), files)
}
// readList reads retired-words: one word or phrase a line; blank lines and lines starting with # are not words.
@@ -166,22 +152,9 @@ func pattern(word string) *regexp.Regexp {
func check(words []string, texts []text) []string {
var out []string
// A text can hold a word only if it holds the word's first part, in any case: the regular expression is run on those alone,
// which keeps the check of the whole catalogue within the unit level's bound.
lower := make([]string, len(texts))
for i, t := range texts {
lower[i] = strings.ToLower(t.s)
}
for _, w := range words {
rx := pattern(w)
first := ""
if parts := strings.Fields(w); len(parts) > 0 {
first = strings.ToLower(parts[0])
}
for i, t := range texts {
if !strings.Contains(lower[i], first) {
continue
}
for _, t := range texts {
if m := rx.FindStringSubmatchIndex(t.s); m != nil {
line := t.line
if line > 0 {
@@ -206,9 +179,6 @@ func excerpt(s string, from, to int) string {
return strings.Join(strings.Fields(s[a:b]), " ")
}
// modulePath matches a string that is only a module or import path ("git.example.org/org/repo/go/pkg").
var modulePath = regexp.MustCompile(`^[a-z0-9.-]+\.[a-z]+/[A-Za-z0-9_./-]+$`)
// goStrings returns a Go file's string literals, adjacent ones joined across `+` as the program joins them.
func goStrings(path, rel string) ([]text, error) {
src, err := os.ReadFile(path)
@@ -234,11 +204,6 @@ func goStrings(path, rel string) ([]text, error) {
if err != nil {
v = lit
}
if modulePath.MatchString(v) {
// An import path or another module path names code, not words an agent reads.
cur, joining = nil, false
continue
}
if cur != nil && joining {
cur.s += v
} else {
-8
View File
@@ -87,11 +87,3 @@ func TestRetiredLinesAreReadAsDaemonAndCommand(t *testing.T) {
t.Fatal("a line with no command was read")
}
}
// The script's run of this check over the real tree (novox/hq ADR 0529 clause 2): no word the glossary retired, and no
// daemon's line, in any text a module's tools can show an agent.
func TestTheCatalogueHoldsNoRetiredWordAndNoDaemonsLine(t *testing.T) {
if code := run("../.."); code != 0 {
t.Fatalf("the check of the real catalogue exits %d (see what it printed above)", code)
}
}
-7
View File
@@ -1,7 +0,0 @@
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# run("../.."): the root's lists of retired words and lines, and every module's texts.
retired-words
retired-lines
modules
+1 -1
View File
@@ -1,5 +1,5 @@
58
The long-running resources of this catalogue that do not say how they are ready (`health`, novox/hq ADR 0240
rule 8), as `module check` counts them. It may only go down: checks/health fails a change that raises it, and
rule 8), as `module check` counts them. It may only go down: merge-check.sh fails a change that raises it, and
one that lowers it writes the new number on the first line. From 2026-11-18, or once it is 0, a long-running
resource without `health` is refused.
+140
View File
@@ -0,0 +1,140 @@
#!/bin/sh
# The catalogue's TypeScript part of its own check (novox/hq issue 302's `mesh-check-also`), declared in
# merge-check.sh's header and run by the build seat in the TypeScript toolchain, after the Go part. The media
# catalogue's twin is issue 311. Before this, a TypeScript module's change ran nothing of its own before it
# merged: the Go part said "NOT TESTED HERE", and a type error in the forge module would first have shown in
# its walk.
#
# For every touched module that has a package.json, in a copy of it outside the clone:
#
# 1. its dependencies are the toolchain's own (/app/node_modules: the mesh's SDK, Node's types and what the
# runtime carries), copied beside it, never fetched: the check holds no credential for the package
# registry. A module that depends on a package the toolchain does not hold **fails the check**, by name:
# it could not be type-checked or tested, and what could not run is a failure, never a note (novox/hq
# issue 385);
# 2. every source that is not a test is type-checked with the toolchain's compiler and the flags a build
# compiles with — the same check the build makes, before the merge rather than in the walk;
# 3. its tests (test/*.test.ts) run under Node's own runner with types stripped, as its own `npm test` does
# after its build: the sources are compiled to dist/ and beside themselves first, so a test that imports
# the compiled module, or a source that imports its sibling by its .js name, finds it.
#
# Said, never passed silently: a module with no tests is said NOT TESTED on every run that touches it —
# nothing of it could fail. A module whose tests exist and could not run fails (novox/hq issue 385).
#
# By hand: MESH_CHECK_MODULES=gitea sh merge-check-typescript.sh, in the toolchain image (it reads
# /app/node_modules), or with TOOLCHAIN_MODULES pointing at a node_modules that holds the same.
#
# **In an agent's session, outside the toolchain image** (novox/hq issue 742), the same node_modules is
# assembled first, in a throwaway directory under the check's own workspace, the way the image's `deps` stage
# assembles it: the toolchain's own package.json and lock (mesh-tools' node-tools/, read from the forge this
# clone came from, so the list is never copied here) installed with `npm ci`, then the SDK the mesh last
# published (`@novox/mesh-sdk@latest`), the @novox scope from the forge's package registry, read without a
# credential. Node and npm are the dev-tools module's; nothing is installed on the machine, no npm
# configuration or cache of the account's is read or written. Only when neither TOOLCHAIN_MODULES nor
# /app/node_modules is there: the forge's check, in the image, fetches nothing, as before. The forge is
# MESH_FORGE (default: the origin remote's), the owner MESH_FORGE_OWNER (default: the origin's owner).
set -eu
work=$(mktemp -d)
trap 'rm -rf "$work"' EXIT
deps="${TOOLCHAIN_MODULES:-/app/node_modules}"
if [ -z "${TOOLCHAIN_MODULES:-}" ] && [ ! -f "$deps/typescript/bin/tsc" ]; then
origin=$(git remote get-url origin 2>/dev/null || true)
forge="${MESH_FORGE:-$(printf '%s\n' "$origin" | sed -n 's#^\(https\{0,1\}://[^/]*\)/.*#\1#p')}"
owner="${MESH_FORGE_OWNER:-$(printf '%s\n' "$origin" | sed -n 's#^https\{0,1\}://[^/]*/\([^/]*\)/.*#\1#p')}"
if [ -z "$forge" ] || [ -z "$owner" ]; then
echo "NOT CHECKED: no TypeScript toolchain at $deps, and no forge to assemble one from (origin is '$origin'; set MESH_FORGE and MESH_FORGE_OWNER, or TOOLCHAIN_MODULES)"
exit 1
fi
echo "no TypeScript toolchain at $deps: assembling one from $forge/$owner/mesh-tools node-tools/ and $owner's package registry (novox/hq issue 742)"
tc="$work/toolchain"
mkdir -p "$tc"
printf '@%s:registry=%s/api/packages/%s/npm/\n' "$owner" "$forge" "$owner" >"$work/npmrc"
# Read with Node's own fetch (dev-tools declares Node, not curl); a status that is not 2xx fails.
if ! { node -e '
const [base, dir] = process.argv.slice(1);
Promise.all(["package.json", "package-lock.json"].map(async (f) => {
const r = await fetch(base + f);
if (!r.ok) throw new Error(base + f + ": " + r.status);
require("fs").writeFileSync(dir + "/" + f, await r.text());
})).catch((e) => { console.error(e.message); process.exit(1); });' \
"$forge/$owner/mesh-tools/raw/branch/main/node-tools/" "$tc" &&
(cd "$tc" && export npm_config_userconfig="$work/npmrc" npm_config_cache="$work/npm-cache" &&
npm ci --ignore-scripts --no-audit --no-fund --loglevel=error &&
npm install --ignore-scripts --no-audit --no-fund --no-save --loglevel=error "@$owner/mesh-sdk@latest"); }; then
echo "NOT CHECKED: the TypeScript toolchain could not be assembled from $forge (novox/hq issue 742)"
exit 1
fi
deps="$tc/node_modules"
fi
tsc="$deps/typescript/bin/tsc"
if [ ! -f "$tsc" ]; then
echo "NOT CHECKED: no TypeScript compiler at $tsc — this part runs in the TypeScript toolchain"
exit 1
fi
# The modules the change reaches: the planner's answer (hq ADR 0238), and every module a changed file is in —
# a change to a module's tests alone moves no build (hq ADR 0267), and its tests must still run.
touched=$({
printf '%s\n' "${MESH_CHECK_MODULES:-}" | tr ',' '\n' | sed -e 's#^modules/##' -e '/\//d'
printf '%s\n' "${MESH_CHECK_CHANGED:-}" | tr ',' '\n' | sed -n 's#^modules/\([^/]*\)/.*#\1#p'
} | sed '/^$/d' | sort -u)
failed=""
for m in $touched; do
d="modules/$m"
[ -f "$d/package.json" ] || continue
copy="$work/$m"
mkdir -p "$copy"
(cd "$d" && tar --exclude=./node_modules --exclude=./dist -cf - .) | (cd "$copy" && tar -xf -)
# What it depends on, against what the toolchain holds.
missing=$(node -e '
const p = require(process.argv[1]); const have = require("fs").readdirSync(process.argv[2]);
const scoped = (n) => n.startsWith("@") ? n.split("/")[0] : n;
const names = Object.keys({ ...(p.dependencies || {}), ...(p.devDependencies || {}) });
const held = (n) => have.includes(n) || (n.startsWith("@") && have.includes(scoped(n)) &&
require("fs").existsSync(process.argv[2] + "/" + n));
console.log(names.filter((n) => n !== "typescript" && !held(n)).join(" "));' "$copy/package.json" "$deps")
if [ -n "$missing" ]; then
echo "FAILED: $d could not be checked — it depends on $missing, which the toolchain does not hold and the check cannot fetch (novox/hq issue 385)"
failed="$failed $m"
continue
fi
mkdir -p "$copy/node_modules"
for p in "$deps"/*; do
case "$(basename "$p")" in typescript | esbuild | @esbuild) continue ;; esac
cp -r "$p" "$copy/node_modules/"
done
sources=$(cd "$copy" && find . -name '*.ts' ! -name '*.d.ts' ! -path './test/*' ! -path './node_modules/*' ! -path './dist/*' | sort)
if [ -n "$sources" ]; then
echo "type-checking $d (TypeScript)"
# shellcheck disable=SC2086 # the module's sources, none with a space in its name
if ! (cd "$copy" && node "$tsc" --noEmit --module NodeNext --moduleResolution NodeNext --target ES2022 \
--types node $sources); then
failed="$failed $m"
continue
fi
else
echo "$d has no TypeScript sources of its own to type-check: its tests read its files"
fi
if ! ls "$copy"/test/*.test.ts >/dev/null 2>&1; then
echo "NOT TESTED: $d is TypeScript and has no tests of its own"
continue
fi
if [ -n "$sources" ]; then
# shellcheck disable=SC2086
(cd "$copy" && node "$tsc" --module NodeNext --moduleResolution NodeNext --target ES2022 --types node \
--rootDir . --outDir dist $sources && node "$tsc" --module NodeNext --moduleResolution NodeNext \
--target ES2022 --types node --rootDir . --outDir . $sources)
fi
echo "testing $d (TypeScript)"
if ! (cd "$copy" && node --test --test-reporter=spec --experimental-strip-types \
--no-warnings=ExperimentalWarning test/*.test.ts); then
failed="$failed $m"
fi
done
if [ -n "$failed" ]; then
echo "TypeScript modules that failed:$failed"
exit 1
fi
Executable
+233
View File
@@ -0,0 +1,233 @@
#!/bin/sh
# mesh-check-toolchain: go
# mesh-check-also: typescript merge-check-typescript.sh
#
# The catalogue's own check (novox/hq ADR 0237 as amended): the second layer of a pull request's merge
# check, `mesh/repo-check`, run by the build seat in the mesh's Go toolchain.
#
# The gate — the manifests the change touches through the running controller's module check, every machine
# of the facts snapshot composed with them and validated by the node-engine's own validator, the replays
# against this tree — is the build seat's first layer (`mesh/merge-gate`), run because the mesh's module
# graph builds these modules from here. It is not repeated here. This is the repository's own:
#
# 1. every manifest through the controller's module check, so one module's change cannot leave another
# it shares a rule with refused (MESH_GATE is the controller the mesh runs) — and the count of
# long-running resources without `health` held to the number in health-undeclared, which only goes down;
# 2. the Go tests of every module the change touches that has them, under the race detector when the
# toolchain has a C compiler. A requirement of the mesh's own forge the module cache does not hold (a
# pinned SDK commit) is taken from its repository's clone beside the check (checks/deps); a module whose
# dependencies still cannot be had **fails the check**, by name and why — its tests could not run, and a
# test that could not run is a failure, never a note (novox/hq issue 385). The integration tests (`//go:build
# integration`) run here, with -tags integration, and only here. A module written in
# TypeScript is type-checked and tested by
# merge-check-typescript.sh, declared above, in the TypeScript toolchain (novox/hq issues 302, 311);
# 3. the words (novox/hq ADR 0244): no word the glossary retired for the tools, as copied in
# retired-words, in any text a module's tools can show an agent — every string in its own code that is
# not a test, and every description in its manifest. Run over every module, not only the touched ones,
# so a word newly retired is found everywhere it stands. The check's own test runs first.
# 4. the paths (novox/hq ADR 0260): no module names a path under /run, /var/lib, /usr/local/lib or /etc
# that another module's manifest places — modules talk over the bus, not through each other's files.
# Over every module; checks/paths/shared and checks/paths/known narrow it, and known only shrinks.
# 5. the agent's home (novox/hq ADR 0266): no module unpacks an archive, or writes into a file, under
# ${machine:agent-home} — the node-engine does both after checking the path, and the agent account owns
# that home and could swap a link in between.
# 6. the secrets (novox/hq issue 342): no mergeable file fills a key with a vault secret that a setting could
# replace — the key is `protected`, or left out of the file's `takes-settings`. checks/secrets/known names
# the nested ones the controller cannot protect alone yet, and only shrinks.
# 7. the list of predefined keys (novox/hq ADR 0348 clause 7): a change to predefined.json removes no key a
# module still names, judged against the trunk's copy at the merge base. Whether a removed key still holds a
# value at the mesh or a node is mesh-config's to refuse; this check reads no store. The list itself and
# every manifest against it are judged by the controller's module check in step 1.
# 8. the config form (novox/hq ADR 0348 clauses 1 and 11): no module says `settings`, `takes-settings` or
# `${setting:<key>}`, in its manifest or a template it names, or has a mergeable file naming no keys (which
# takes its whole layer), unless checks/config/unmoved lists it as not yet moved; that list only shrinks,
# judged against the trunk's copy at the merge base.
# 9. the file text (novox/hq ADR 0349 §2): no module keeps a file's text inline in its manifest (`content`)
# unless checks/templates/inline lists it as not yet moved; each names its template in files/. That list
# only shrinks, judged against the trunk's copy at the merge base.
# 10. the mail-store seat (novox/hq ADR 0378 §3, issue 800): no manifest but modules/mesh-mail claims it.
# 11. owned paths (novox/hq ADR 0422 §8, issue 1024; it holds ~/.ssh for ssh-client as ADR 0415 §6 asked):
# every directory and archive says `unknown` (move-out, module or program) and a valid `knows`; `move-out`
# only under a home; no module declares a path inside a directory of another module; and none is, lies in or
# holds a directory a system package owns under /var/lib (checks/owned/system, issue 1841).
# checks/sshdir (ADR 0415 §6) stays while main runs it; it goes under issue 1035 once main runs checks/owned.
# 12. the image builder (novox/hq ADR 0487 §3, issue 1704): only modules/buildah provides rootless-image-build.
# 13. the image's place (novox/hq ADR 0487 §1, issue 1707): every image builds from a Dockerfile in its own module's directory.
set -eu
(cd checks/words && go test -count=1 . && go run . ../..)
# Node names (novox/hq ADR 0507 §9, issue 1838): no node name of this mesh in a covered module; the names are in
# `particulars`, the covered modules in checks/particulars/covered.
(cd checks/particulars && go test -count=1 . && go run . ../..)
# The toolchains (novox/hq ADR 0477 clause 4): the package go only in modules/go, nodejs and npm only in modules/nodejs.
(cd checks/toolchains && go test -count=1 . && go run . ../..)
# The image builder (novox/hq ADR 0487 §3): only modules/buildah provides rootless-image-build, docker never.
(cd checks/imagebuild && go test -count=1 . && go run . ../..)
# The image's place (novox/hq ADR 0487 §1): every image builds from a Dockerfile in its own module's directory.
(cd checks/imageplace && go test -count=1 . && go run . ../..)
# The Node lock files (novox/hq ADR 0469 §5, issue 1868): a TypeScript module that names packages beyond the SDK commits its lock.
(cd checks/locks && go test -count=1 . && go run . ../..)
# The agent's home (novox/hq ADR 0266): no archive, and no file written into, under it.
(cd checks/agenthome && go test -count=1 . && go run . ../..)
# The mail-store seat (novox/hq ADR 0378 §3): only mesh-mail may claim it.
(cd checks/mailstore && go test -count=1 . && go run . ../..)
# ~/.ssh (novox/hq ADR 0415 §6): only ssh-client may name a path under it. Main's merge check runs this line, so
# checks/sshdir stays until main runs checks/owned; it goes under issue 1035.
(cd checks/sshdir && go test -count=1 . && go run . ../..)
# Owned paths (novox/hq ADR 0422 §8): unknown files named per directory; no path inside another module's directory;
# none in a system package's directory under /var/lib (issue 1841), the known exceptions only shrinking: judged
# against the trunk's copy of checks/owned/system-known at the merge base; a trunk without one is the list's first copy.
owned_base=""
if merge_base=$(git merge-base HEAD "origin/${MESH_CHECK_BASE:-main}" 2>/dev/null); then
if git cat-file -e "$merge_base:checks/owned/system-known" 2>/dev/null; then
owned_base=$(mktemp)
git show "$merge_base:checks/owned/system-known" >"$owned_base"
fi
else
echo "NOT COMPARED WITH THE TRUNK: no merge base with origin/${MESH_CHECK_BASE:-main}; checks/owned/system-known is held to the list it was first written with"
fi
(cd checks/owned && go test -count=1 . && if [ -n "$owned_base" ]; then go run . -base "$owned_base" ../..; else go run . ../..; fi)
# The secrets (novox/hq issue 342): a password is not a setting, so no setting can replace the vault's. The
# known list only shrinks: judged against the trunk's copy at the merge base, a line it does not hold is refused.
secrets_base=""
secrets_ref="origin/${MESH_CHECK_BASE:-main}"
if merge_base=$(git merge-base HEAD "$secrets_ref" 2>/dev/null); then
if git cat-file -e "$merge_base:checks/secrets/known" 2>/dev/null; then
secrets_base=$(mktemp)
git show "$merge_base:checks/secrets/known" >"$secrets_base"
fi
else
echo "NOT COMPARED WITH THE TRUNK: no merge base with $secrets_ref; checks/secrets/known is held to the list it was first written with"
fi
(cd checks/secrets && go test -count=1 . && if [ -n "$secrets_base" ]; then go run . -base "$secrets_base" ../..; else go run . ../..; fi)
# The list of predefined keys (novox/hq ADR 0348 clause 7): no key removed that a module still names. The trunk's
# list at the merge base is what it is judged against; a trunk without the file had none.
predefined_base=""
if merge_base=$(git merge-base HEAD "origin/${MESH_CHECK_BASE:-main}" 2>/dev/null); then
predefined_base=$(mktemp)
if ! git show "$merge_base:predefined.json" >"$predefined_base" 2>/dev/null; then
echo '{}' >"$predefined_base"
fi
else
echo "NOT COMPARED WITH THE TRUNK: no merge base with origin/${MESH_CHECK_BASE:-main}; no removal from predefined.json can be seen"
fi
(cd checks/predefined && go test -count=1 . && if [ -n "$predefined_base" ]; then go run . -base "$predefined_base" ../..; else go run . ../..; fi)
# The config form (novox/hq ADR 0348 §1, §11): a moved module says no old name, and the list of modules not yet moved
# only shrinks, judged against the trunk's copy at the merge base; a trunk without one is the list's first copy.
unmoved_base=""
if merge_base=$(git merge-base HEAD "origin/${MESH_CHECK_BASE:-main}" 2>/dev/null); then
if git cat-file -e "$merge_base:checks/config/unmoved" 2>/dev/null; then
unmoved_base=$(mktemp)
git show "$merge_base:checks/config/unmoved" >"$unmoved_base"
fi
else
echo "NOT COMPARED WITH THE TRUNK: no merge base with origin/${MESH_CHECK_BASE:-main}; checks/config/unmoved is held to the list it was first written with"
fi
(cd checks/config && go test -count=1 . && if [ -n "$unmoved_base" ]; then go run . -base "$unmoved_base" ../..; else go run . ../..; fi)
# The file text (novox/hq ADR 0349 §2): a file's text is a template in files/, outside the modules not yet moved; the
# list of those only shrinks, judged against the trunk's copy at the merge base, a trunk without one being its first.
inline_base=""
if merge_base=$(git merge-base HEAD "origin/${MESH_CHECK_BASE:-main}" 2>/dev/null); then
if git cat-file -e "$merge_base:checks/templates/inline" 2>/dev/null; then
inline_base=$(mktemp)
git show "$merge_base:checks/templates/inline" >"$inline_base"
fi
else
echo "NOT COMPARED WITH THE TRUNK: no merge base with origin/${MESH_CHECK_BASE:-main}; checks/templates/inline is held to the list it was first written with"
fi
(cd checks/templates && go test -count=1 . && if [ -n "$inline_base" ]; then go run . -base "$inline_base" ../..; else go run . ../..; fi)
# The lists that narrow the paths check may not grow unseen: the base branch's copies are what they are
# judged against (MESH_CHECK_BASE, main when unset), read from the clone the check runs in.
base_lists=""
base_ref="origin/${MESH_CHECK_BASE:-main}"
if git rev-parse --verify --quiet "$base_ref^{commit}" >/dev/null 2>&1 &&
git cat-file -e "$base_ref:checks/paths/known" 2>/dev/null && git cat-file -e "$base_ref:checks/paths/shared" 2>/dev/null; then
base_lists=$(mktemp -d)
git show "$base_ref:checks/paths/known" >"$base_lists/known"
git show "$base_ref:checks/paths/shared" >"$base_lists/shared"
fi
(cd checks/paths && go test -count=1 . && if [ -n "$base_lists" ]; then go run . -base "$base_lists" ../..; else go run . ../..; fi)
if [ -n "${MESH_GATE:-}" ]; then
checked=$("$MESH_GATE" module check modules/*/module.json) || { printf '%s\n' "$checked"; exit 1; }
# **The count only goes down** (novox/hq ADR 0240 rule 8): the long-running resources that do not say how
# they are ready, as the controller counts them, against the number kept in health-undeclared. A change
# that raises it fails; one that lowers it writes the new number there, so it can never rise again.
kept=$(sed -n 's/^\([0-9][0-9]*\).*/\1/p' health-undeclared | head -n 1)
counted=$(printf '%s\n' "$checked" | sed -n 's/^long-running resources without health: \([0-9][0-9]*\)$/\1/p')
if [ -z "$counted" ]; then
echo "NOT COUNTED: the controller the mesh runs is older than the count of resources without health (ADR 0240 Phase B)"
elif [ "$counted" -gt "$kept" ]; then
echo "the long-running resources without health rose from $kept to $counted: declare how each new one is ready (ADR 0240 rule 8)"
exit 1
elif [ "$counted" -lt "$kept" ]; then
echo "the long-running resources without health fell from $kept to $counted: write $counted in health-undeclared, so the count cannot rise again"
exit 1
else
echo "long-running resources without health: $counted, as kept"
fi
else
echo "NOT CHECKED: no controller was built beside this check, so the manifests were not read by one"
fi
race=""
if command -v gcc >/dev/null 2>&1; then race="-race"; else echo "NOT RACE-CHECKED: the toolchain holds no C compiler"; fi
# The modules the change reaches are the controller's planner's answer (MESH_CHECK_MODULES, hq ADR 0238):
# a module by its name, a new one by its directory. By hand, without it, the directories the changed files
# are in.
if [ -n "${MESH_CHECK_MODULES:-}" ]; then
touched=$(printf '%s\n' "$MESH_CHECK_MODULES" | tr ',' '\n' | sed -e 's#^modules/##' -e '/\//d' | sort -u)
else
touched=$(printf '%s\n' "${MESH_CHECK_CHANGED:-}" | tr ',' '\n' | sed -n 's#^modules/\([^/]*\)/.*#\1#p' | sort -u)
fi
# A module's dependencies: fetched, or a forge requirement taken from its clone beside the check
# (MESH_CHECK_BESIDE); its tests run against them (checks/deps, novox/hq issue 385). The helper's own test
# runs first.
(cd checks/deps && go test -count=1 .)
deps=$(mktemp -d)
trap 'rm -rf "$deps"' EXIT
(cd checks/deps && go build -o "$deps/deps" .)
untestable=""
for m in $touched; do
[ -d "modules/$m" ] || continue
if [ ! -f "modules/$m/go.mod" ]; then
[ -f "modules/$m/package.json" ] && echo "modules/$m is TypeScript: merge-check-typescript.sh type-checks and tests it"
continue
fi
if ! modfile=$(GOPRIVATE=git.novox.be "$deps/deps" -beside "${MESH_CHECK_BESIDE:-}" -work "$deps" "modules/$m"); then
echo "FAILED: modules/$m could not be tested — $modfile"
untestable="$untestable modules/$m"
continue
fi
[ -n "$modfile" ] && echo "modules/$m takes its forge dependencies from the clones beside the check"
echo "testing modules/$m"
unformatted=$(cd "modules/$m" && gofmt -l .)
if [ -n "$unformatted" ]; then
echo "modules/$m is not gofmt'd: $unformatted"
exit 1
fi
cgo=0
[ -n "$race" ] && cgo=1
# The integration tests (a file whose first line is `//go:build integration`: a test that starts a bus, a store or
# another process) run here and nowhere else: the unit suite stays under a second (novox/hq playbook 07).
(cd "modules/$m" && CGO_ENABLED=$cgo GOPRIVATE=git.novox.be go vet -tags integration ${modfile:+"-modfile=$modfile"} ./... &&
CGO_ENABLED=$cgo GOPRIVATE=git.novox.be go test -tags integration ${modfile:+"-modfile=$modfile"} $race -count=1 ./...)
# The docker tests of the bus's snapshot program and entrypoint (novox/hq ADR 0480 clause 6) start throwaway nats
# containers, and the Go toolchain container holds no docker, so above they skip with a "NOT RUN" line. Where the
# build seat says docker exists (MESH_CHECK_DOCKER=1) they run here by name, and a missing docker fails the check
# (MESH_REQUIRE_DOCKER=1 makes the tests fail instead of skip). Without it the step says it did not run.
if [ "$m" = nats ]; then
if [ "${MESH_CHECK_DOCKER:-}" = 1 ]; then
(cd modules/nats && MESH_REQUIRE_DOCKER=1 CGO_ENABLED=$cgo GOPRIVATE=git.novox.be go test -tags integration ${modfile:+"-modfile=$modfile"} -count=1 \
-run 'TestTheDeclaredDumpRunsInTheBussImage|TestTheEntrypointReloadsTheServerWithoutDroppingAConnection|TestTheStoreSurvivesTheNewDeclarationTheOldAndTheNewAgain|TestASnapshotOfALiveBusRestoresToTheSameContent|TestTheSnapshotUserCannotChangeTheBus' \
./cmd/mesh-nats-snapshot) || { echo "the bus's docker tests failed"; exit 1; }
else
echo "NOT RUN: the bus's five docker tests (modules/nats/cmd/mesh-nats-snapshot) skipped: this check has no docker (MESH_CHECK_DOCKER is not 1)"
fi
fi
done
if [ -n "$untestable" ]; then
echo "the check fails: the tests of$untestable could not run (novox/hq issue 385)"
exit 1
fi
@@ -1,9 +0,0 @@
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# the copies of this package in every module that carries one.
modules/xorg/internal/desktop/*.go
modules/lemurs/internal/desktop/*.go
modules/i3/internal/desktop/*.go
modules/xterm/internal/desktop/*.go
modules/adwaita/internal/desktop/*.go
@@ -23,7 +23,7 @@ import (
// manifest's.
const NamesFileEnv = "MESH_ARTIFACT_STORE_NAMES_FILE"
// CatalogueModules and CatalogueModule are the catalog's tools these definitions are read through.
// CatalogueModules and CatalogueModule are the catalogue's tools these definitions are read through.
const (
CatalogueModules = "mesh-catalog.catalog_modules"
CatalogueModule = "mesh-catalog.catalog_module"
@@ -143,21 +143,21 @@ func PinsIn(manifest json.RawMessage, names StoreNames) []pinned {
return out
}
// Definitions is what the catalog's definitions pin in the store, and what of them could not be read.
// Definitions is what the catalogue's definitions pin in the store, and what of them could not be read.
type Definitions struct {
// Pins are by Key(repository, digest).
Pins map[string][]Pin
// Read is how many definitions were read; Unread says, per module, why its definition was not.
Read int
Unread map[string]string
// NotAsked says why the catalog could not be asked at all, when it could not.
// NotAsked says why the catalogue could not be asked at all, when it could not.
NotAsked string
// NamesNotRead says why the store's older names could not be read, when they could not.
NamesNotRead string
Names StoreNames
}
// Catalogue reads the definitions through the catalog's tools, and keeps each definition it read by its
// Catalogue reads the definitions through the catalogue's tools, and keeps each definition it read by its
// module and commit, which never change together.
type Catalogue struct {
Ask Asking
@@ -171,21 +171,21 @@ type catalogueEntry struct {
Commit string `json:"commit"`
}
// Read asks the catalog for every module and reads each definition, bounded by DefinitionsTimeout.
// Read asks the catalogue for every module and reads each definition, bounded by DefinitionsTimeout.
func (c *Catalogue) Read(ctx context.Context, names StoreNames) *Definitions {
defs := &Definitions{Pins: map[string][]Pin{}, Unread: map[string]string{}, Names: names}
if c == nil || c.Ask == nil {
defs.NotAsked = "this bundle has no way to ask the catalog"
defs.NotAsked = "this bundle has no way to ask the catalogue"
return defs
}
raw, err := c.Ask(CatalogueModules, map[string]any{})
if err != nil {
defs.NotAsked = "the catalog's modules could not be asked: " + err.Error()
defs.NotAsked = "the catalogue's modules could not be asked: " + err.Error()
return defs
}
answer, output, ok := answerOf(raw)
if !ok {
defs.NotAsked = "the catalog refused its modules: " + lastLine(output)
defs.NotAsked = "the catalogue refused its modules: " + lastLine(output)
return defs
}
var list struct {
@@ -193,7 +193,7 @@ func (c *Catalogue) Read(ctx context.Context, names StoreNames) *Definitions {
Error string `json:"error"`
}
if err := json.Unmarshal(answer, &list); err != nil || list.Error != "" || list.Modules == nil {
defs.NotAsked = "the catalog's modules answer is not readable: " + firstOf(list.Error, errText(err), lastLine(output))
defs.NotAsked = "the catalogue's modules answer is not readable: " + firstOf(list.Error, errText(err), lastLine(output))
return defs
}
c.mu.Lock()
@@ -1,129 +0,0 @@
package main
// The artifact store's door and the bus, as the holder reaches them through this module's runtime.
import (
"context"
"encoding/json"
"fmt"
"os"
"strings"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"artifactstoretools/internal/registry"
)
// door is registry.Store at the store's address as this machine reaches it.
type door struct{ s registry.Store }
func (d *door) Address() string { return d.s.Address }
func (d *door) Hold(ctx context.Context, r string) (bool, error) { return d.s.Hold(ctx, r) }
func (d *door) Holds(ctx context.Context, r string) error { return d.s.Holds(ctx, r) }
func (d *door) IsIndex(ctx context.Context, r string) (bool, error) {
return d.s.IsIndex(ctx, r)
}
func (d *door) LetGo(ctx context.Context, r string) error { return d.s.LetGo(ctx, r) }
func (d *door) SpareKept(ctx context.Context, keptImages, touching []string) error {
spare, err := d.s.SpareKeptIndexes(ctx, keptImages, touching)
if err == nil {
d.s.Spare = spare
}
return err
}
// DoorAt reads the store's address (host:port) from the file the manifest fills from the seat mesh-artifact-store's
// reach, each time, so a store that moved is reached where it is now.
func DoorAt(file string) func() (Door, error) {
return func() (Door, error) {
if file == "" {
return nil, fmt.Errorf("this bundle was given no file naming the artifact store's address")
}
raw, err := os.ReadFile(file)
if err != nil {
return nil, fmt.Errorf("the artifact store's address: %w", err)
}
address := strings.TrimSpace(string(raw))
if address == "" {
return nil, fmt.Errorf("the artifact store's address is empty in %s", file)
}
return &door{s: registry.Store{Address: address}}, nil
}
}
// runtimeBus is the holder's bus through the runtime: its reads, its findings and its seat's event.
type runtimeBus struct{}
func (runtimeBus) Keys(state string) ([]string, error) { return stdio.State(state).Keys() }
func (runtimeBus) Get(state, key string) ([]byte, bool, error) {
e, err := stdio.State(state).Get(key)
if err != nil || e == nil {
return nil, false, err
}
return e.Value, true, nil
}
func (runtimeBus) History(state, key string) ([]stdio.StateRevision, error) {
return stdio.State(state).History(key)
}
func (runtimeBus) LogSince(read string, at time.Time) ([]stdio.LogEntry, error) {
var out []stdio.LogEntry
page, err := stdio.Log(read).Since(at, "", 0)
for err == nil {
out = append(out, page.Entries...)
if page.Next == 0 {
return out, nil
}
page, err = stdio.Log(read).Read(page.Next, "", 0)
}
return nil, err
}
func (runtimeBus) Put(state, key string, value any) error {
_, err := stdio.State(state).Put(key, value)
return err
}
func (runtimeBus) Delete(state, key string) error { return stdio.State(state).Delete(key) }
func (runtimeBus) Publish(subject string, body any) error {
return stdio.SeatPublish(subject, body, "")
}
// follow watches the builder's done keys and the wants: a build's end sweeps (once the lease names no controller) and
// observes again; a want changed observes again. Event-driven: nothing reads them on a timer.
func follow(h *Holder) {
<-stdio.Served()
watch := func(state string, handle func(stdio.StateChange) error) {
for {
if err := stdio.State(state).Watch("", handle); err == nil {
return
} else {
h.logf("watching %s: %v; asked again in 10s", state, err)
}
time.Sleep(10 * time.Second)
}
}
observe := func() {
if err := h.Observe(context.Background()); err != nil {
h.logf("image-missing could not be observed: %v", err)
}
}
watch(BuildsRead, func(c stdio.StateChange) error {
if c.Current || c.Op != stdio.OpPut {
return nil
}
var b struct {
State string `json:"state"`
}
if json.Unmarshal(c.Value, &b) == nil && b.State == "done" {
go func() { h.AfterBuild(c.Key); observe() }()
}
return nil
})
watch(WantsRead, func(c stdio.StateChange) error {
if !c.Current {
go observe()
}
return nil
})
observe()
}
File diff suppressed because it is too large Load Diff
@@ -1,260 +0,0 @@
//go:build integration
package main
// The seat mesh-artifact-sweep's holder whole against a real bus (novox/hq ADR 0571 clause 2, issue 1955): the builder's done
// keys, the wants and the controller's lease in a real nats-server's key-value buckets, a build's end heard by a watch
// of the builds as the runtime hands it over, each deletion recorded by a request to the seat mesh-hand-acts on the bus,
// artifacts-gone said on the seat's subject, and S25's finding written to the holder's own bucket. Only the store's
// door is in memory: its HTTP is the copied registry package's, tested there.
import (
"context"
"encoding/json"
"strings"
"sync"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/handacts"
"git.novox.be/novox/mesh-sdk/go/testbus"
)
// realBus is the holder's Bus over a real nats-server: each read state a bucket named as the runtime names it.
type realBus struct {
ctx context.Context
nc *nats.Conn
js jetstream.JetStream
// log is the builder's log as the runtime's log verbs would read it (mesh-tools adds them; the sdk's own read of a
// log is tested in mesh-sdk's process package).
log []stdio.LogEntry
}
func (b realBus) LogSince(_ string, at time.Time) ([]stdio.LogEntry, error) {
var out []stdio.LogEntry
for _, e := range b.log {
if !e.At.Before(at) {
out = append(out, e)
}
}
return out, nil
}
func bucketOf(state string) string { return strings.Replace(state, ".", "_", 1) }
func (b realBus) kv(state string) (jetstream.KeyValue, error) {
return b.js.KeyValue(b.ctx, bucketOf(state))
}
func (b realBus) Keys(state string) ([]string, error) {
kv, err := b.kv(state)
if err != nil {
return nil, err
}
keys, err := kv.Keys(b.ctx)
if err == jetstream.ErrNoKeysFound {
return nil, nil
}
return keys, err
}
func (b realBus) Get(state, key string) ([]byte, bool, error) {
kv, err := b.kv(state)
if err != nil {
return nil, false, err
}
e, err := kv.Get(b.ctx, key)
if err == jetstream.ErrKeyNotFound {
return nil, false, nil
}
if err != nil {
return nil, false, err
}
return e.Value(), true, nil
}
func (b realBus) History(state, key string) ([]stdio.StateRevision, error) {
kv, err := b.kv(state)
if err != nil {
return nil, err
}
h, err := kv.History(b.ctx, key)
if err != nil {
return nil, err
}
var out []stdio.StateRevision
for _, e := range h {
out = append(out, stdio.StateRevision{Key: key, Value: e.Value(), Revision: e.Revision(), At: e.Created()})
}
return out, nil
}
func (b realBus) Put(state, key string, value any) error {
kv, err := b.kv(state)
if err != nil {
return err
}
raw, _ := json.Marshal(value)
_, err = kv.Put(b.ctx, key, raw)
return err
}
func (b realBus) Delete(state, key string) error {
kv, err := b.kv(state)
if err != nil {
return err
}
return kv.Delete(b.ctx, key)
}
func (b realBus) Publish(subject string, body any) error {
raw, _ := json.Marshal(body)
return b.nc.Publish(subject, raw)
}
func TestTheSeatSweepsAtABuildsEndOnARealBusOnceTheLeaseIsAway(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
nc, err := nats.Connect(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
defer nc.Close()
js, err := jetstream.New(nc)
if err != nil {
t.Fatal(err)
}
for _, s := range []string{BuildsRead, WantsRead, HavesRead, LeaseRead, ManifestsRead, "artifact-store-tools." + FindingsState} {
if _, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: bucketOf(s), History: 5}); err != nil {
t.Fatal(err)
}
}
bus := realBus{ctx: ctx, nc: nc, js: js}
// The mesh as aMesh lays it, copied onto the real buckets: letta's seven builds, a want, the store's listing.
mem, memb, door, _ := aMesh(t, &buscompose.LeaseHolder{Instance: "c1", Host: "novox", Epoch: 3})
bus.log = memb.log
for _, s := range []string{BuildsRead, WantsRead, LeaseRead, ManifestsRead} {
keys, _ := memb.Keys(s)
for _, k := range keys[:min(len(keys), 6)] { // the seventh build ends later, on the bus
raw, _, _ := memb.Get(s, k)
if err := bus.Put(s, k, json.RawMessage(raw)); err != nil {
t.Fatal(err)
}
}
}
time.Sleep(20 * time.Millisecond) // each build's end is its key's time on the bus: the seventh the newest
// The seat mesh-hand-acts, answering every act it records; artifacts-gone heard on the seat's subject.
var mu sync.Mutex
var acts, gone []string
if _, err := nc.Subscribe("mesh.seat."+handacts.Seat+".tool.hand-act", func(m *nats.Msg) {
mu.Lock()
acts = append(acts, string(m.Data))
mu.Unlock()
_ = m.Respond([]byte(`{"ok":true}`))
}); err != nil {
t.Fatal(err)
}
if _, err := nc.Subscribe(GoneSubject, func(m *nats.Msg) {
mu.Lock()
gone = append(gone, string(m.Data))
mu.Unlock()
}); err != nil {
t.Fatal(err)
}
h := &Holder{Bus: findingsAt{realBus: bus}, Door: mem.Door, Listed: mem.Listed, Now: time.Now, Logf: t.Logf}
h.Acts = handacts.Recorder{LeaseAway: h.LeaseAway, Ask: func(key string, body any) (json.RawMessage, error) {
raw, _ := json.Marshal(body)
m, err := nc.Request("mesh.seat."+strings.TrimPrefix(strings.Replace(key, ".", ".tool.", 1), "seat:"), raw, 5*time.Second)
if err != nil {
return nil, err
}
return m.Data, nil
}}
// A build's end heard from the builds bucket, as follow hands it over.
kv, _ := bus.kv(BuildsRead)
w, err := kv.WatchAll(ctx, jetstream.UpdatesOnly())
if err != nil {
t.Fatal(err)
}
ended := func() {
keys, _ := memb.Keys(BuildsRead)
raw, _, _ := memb.Get(BuildsRead, keys[6])
if err := bus.Put(BuildsRead, keys[6], json.RawMessage(raw)); err != nil {
t.Fatal(err)
}
select {
case e := <-w.Updates():
h.AfterBuild(e.Key())
case <-ctx.Done():
t.Fatal("the build's end was not heard")
}
}
// While the lease names the controller, a build's end lets nothing go.
ended()
if len(door.asked) != 0 {
t.Fatalf("before the lease write the store was asked %v", door.asked)
}
// The lease handed away: the next build's end holds the kept archives and lets go of letta's 2nd, which the 7th moved
// past the five newest and no want names (the want keeps the 1st), recording each deletion.
if err := bus.Put(LeaseRead, buscompose.ControllerLeaseKey, buscompose.LeaseHolder{HandedTo: "mesh-grants", Why: "the lease write"}); err != nil {
t.Fatal(err)
}
ended()
deadline := time.Now().Add(5 * time.Second)
for {
mu.Lock()
n := len(gone)
mu.Unlock()
if n > 0 || time.Now().After(deadline) {
break
}
time.Sleep(20 * time.Millisecond)
}
mu.Lock()
defer mu.Unlock()
if len(acts) != 3 || !strings.Contains(acts[0], archiveRef("letta", 2)) || !strings.Contains(acts[1], imageRef("letta", 2)) ||
!strings.Contains(acts[2], mirror(1)) {
t.Fatalf("the hand acts recorded are %v; the store was asked %v", acts, door.asked)
}
if len(gone) != 1 || !strings.Contains(gone[0], `"fingerprint":"`+fp(2)+`"`) {
t.Fatalf("artifacts-gone said %v", gone)
}
// S25 through the real findings bucket: letta's wanted image missing from the store is a finding there.
view, _ := mem.Listed(ctx)
repo, digest, _ := splitReference(imageRef("letta", 1))
delete(view.L.Repos[repo].Revisions, digest)
if err := h.Observe(ctx); err != nil {
t.Fatal(err)
}
if keys, _ := bus.Keys("artifact-store-tools." + FindingsState); len(keys) != 1 || keys[0] != "module.letta.ace.image-missing" {
t.Fatalf("the findings are %v", keys)
}
}
// findingsAt reads and writes the holder's own findings in its bucket, artifact-store-tools_findings.
type findingsAt struct{ realBus }
func (f findingsAt) own(state string) string {
if state == FindingsState {
return Module + "." + state
}
return state
}
func (f findingsAt) Keys(state string) ([]string, error) { return f.realBus.Keys(f.own(state)) }
func (f findingsAt) Get(state, key string) ([]byte, bool, error) {
return f.realBus.Get(f.own(state), key)
}
func (f findingsAt) Put(state, key string, value any) error {
return f.realBus.Put(f.own(state), key, value)
}
func (f findingsAt) Delete(state, key string) error { return f.realBus.Delete(f.own(state), key) }
@@ -1,351 +0,0 @@
package main
import (
"context"
"encoding/json"
"fmt"
"sort"
"strings"
"sync"
"testing"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/handacts"
"git.novox.be/novox/mesh-sdk/go/health"
"artifactstoretools/internal/registry"
)
// memBus is the holder's bus in memory: states by name, each key's history, and what was published.
type memBus struct {
mu sync.Mutex
states map[string]map[string][]byte
history map[string][]stdio.StateRevision
log []stdio.LogEntry
published []string
}
func (b *memBus) LogSince(read string, at time.Time) ([]stdio.LogEntry, error) {
var out []stdio.LogEntry
for _, e := range b.log {
if read == delivery.BuildLogRead && !e.At.Before(at) {
out = append(out, e)
}
}
return out, nil
}
func (b *memBus) Keys(state string) ([]string, error) {
b.mu.Lock()
defer b.mu.Unlock()
var out []string
for k := range b.states[state] {
out = append(out, k)
}
sort.Strings(out)
return out, nil
}
func (b *memBus) Get(state, key string) ([]byte, bool, error) {
b.mu.Lock()
defer b.mu.Unlock()
v, ok := b.states[state][key]
return v, ok, nil
}
func (b *memBus) History(state, key string) ([]stdio.StateRevision, error) {
return b.history[state+" "+key], nil
}
func (b *memBus) Put(state, key string, value any) error {
raw, _ := json.Marshal(value)
b.mu.Lock()
defer b.mu.Unlock()
if b.states[state] == nil {
b.states[state] = map[string][]byte{}
}
b.states[state][key] = raw
return nil
}
func (b *memBus) Delete(state, key string) error {
b.mu.Lock()
defer b.mu.Unlock()
delete(b.states[state], key)
return nil
}
func (b *memBus) Publish(subject string, body any) error {
raw, _ := json.Marshal(body)
b.mu.Lock()
defer b.mu.Unlock()
b.published = append(b.published, subject+" "+string(raw))
return nil
}
// memDoor is the store's door in memory: what it holds, which references are indexes, and what was asked.
type memDoor struct {
holds map[string]bool
indexes map[string]bool
asked []string
}
func (d *memDoor) Address() string { return "10.0.0.5:5000" }
func (d *memDoor) Hold(_ context.Context, r string) (bool, error) {
d.asked = append(d.asked, "hold "+r)
if !d.holds[r] {
return false, registry.Gone
}
return false, nil
}
func (d *memDoor) Holds(_ context.Context, r string) error {
if !d.holds[r] {
return registry.Gone
}
return nil
}
func (d *memDoor) IsIndex(_ context.Context, r string) (bool, error) { return d.indexes[r], nil }
func (d *memDoor) LetGo(_ context.Context, r string) error {
d.asked = append(d.asked, "let go "+r)
delete(d.holds, r)
return nil
}
func (d *memDoor) SpareKept(context.Context, []string, []string) error {
d.asked = append(d.asked, "spare")
return nil
}
// mirror is a copy of an upstream base in the store, the n-th.
func mirror(n int) string {
return "artifact-store://upstream/docker.io/library/alpine@sha256:" + strings.Repeat(fmt.Sprintf("%x", n%16), 64)
}
// pinnedImage is the image letta's manifest names by itself, at the store's address.
const pinnedImage = "10.0.0.5:5000/letta/pinned@sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc"
func digestOf(ref string) string { _, d, _ := splitReference(ref); return d }
func fp(n int) string { return "src2:" + strings.Repeat(fmt.Sprintf("%x", n%16), 64) }
func imageRef(module string, n int) string {
return "artifact-store://" + module + "/image@sha256:" + strings.Repeat(fmt.Sprintf("%x", n%16), 64)
}
func archiveRef(module string, n int) string {
return "artifact-store://" + module + "/files/blobs/sha256:" + strings.Repeat(fmt.Sprintf("%x", (n+8)%16), 64)
}
// A mesh with seven done builds of letta, the n-th ending n hours after a day began, each with an image and an archive;
// a want names the first on ace; the store lists them all and one image no build names; the lease as given.
func aMesh(t *testing.T, lease *buscompose.LeaseHolder) (*Holder, *memBus, *memDoor, *[]string) {
t.Helper()
day := time.Date(2026, 10, 1, 0, 0, 0, 0, time.UTC)
b := &memBus{states: map[string]map[string][]byte{}, history: map[string][]stdio.StateRevision{}}
door := &memDoor{holds: map[string]bool{}, indexes: map[string]bool{}}
listing := &Listing{Blobs: map[string]int64{}, Repos: map[string]*Repo{}}
for n := 1; n <= 7; n++ {
key, _ := delivery.BuilderBuildKey("letta", fp(n))
raw, err := delivery.EncodeBuilderBuild(delivery.BuilderBuild{State: delivery.BuildDone, Manifest: strings.Repeat("d", 64), Artifacts: []delivery.BuiltArtifact{
{Name: "image", Kind: "image", Digest: digestOf(imageRef("letta", n)), Reference: imageRef("letta", n)},
{Name: "files", Kind: "archive", Digest: digestOf(archiveRef("letta", n)), Reference: archiveRef("letta", n)},
{Name: "pkg", Kind: "package", Digest: "sha256:" + strings.Repeat("f", 64), Reference: "letta@1.0." + fmt.Sprint(n)}}})
if err != nil {
t.Fatal(err)
}
_ = b.Put(BuildsRead, key, json.RawMessage(raw))
b.history[BuildsRead+" "+key] = []stdio.StateRevision{{Key: key, Value: raw, At: day.Add(time.Duration(n) * time.Hour)}}
for _, r := range []string{imageRef("letta", n), archiveRef("letta", n)} {
door.holds[r] = true
repo, digest, kind := splitReference(r)
if kind == "archive" {
listing.Blobs[digest] = 10
} else {
listing.repo(repo).Revisions[digest] = true
}
}
}
listing.repo("letta/image").Revisions["sha256:"+strings.Repeat("e", 64)] = true // no build names it
listing.repo("letta/pinned").Revisions["sha256:"+strings.Repeat("c", 64)] = true
// Three copies of a base: letta's 2nd stood on the first, its 7th on the second, and no build on the third.
for n := 1; n <= 3; n++ {
repo, digest, _ := splitReference(mirror(n))
listing.repo(repo).Revisions[digest] = true
door.holds[mirror(n)] = true
}
for n := 1; n <= 7; n++ {
e := delivery.BuildLogEntry{Event: delivery.BuildEndedEvent, Module: "letta", FP: fp(n), Repository: "novox/x", Commit: "c",
Node: "ace", At: day.Add(time.Duration(n) * time.Hour), Outcome: "done"}
switch n {
case 2:
e.Against = []string{mirror(1)}
case 7:
e.Against = []string{mirror(2)}
}
raw, _ := delivery.EncodeBuildLogEntry(e)
b.log = append(b.log, stdio.LogEntry{Sequence: uint64(n), Key: "letta", Value: raw, At: e.At})
}
manifest, _ := delivery.EncodeManifest(delivery.Manifest{Module: "letta",
JSON: `{"module":"letta","resources":[{"id":"app","type":"container","image":"` + pinnedImage + `"},` +
`{"id":"built","type":"container","image":"${artifact:image}"}]}`})
mk, _ := delivery.ManifestsKey(strings.Repeat("d", 64))
_ = b.Put(ManifestsRead, mk, json.RawMessage(manifest))
want, _ := json.Marshal(delivery.Want{V: delivery.Version, FP: fp(1), Self: true})
_ = b.Put(WantsRead, "want.ace.letta", json.RawMessage(want))
if lease != nil {
_ = b.Put(LeaseRead, buscompose.ControllerLeaseKey, lease)
}
var acts []string
h := &Holder{Bus: b, Door: func() (Door, error) { return door, nil },
Listed: func(context.Context) (*View, error) { return &View{L: listing}, nil },
Now: func() time.Time { return day.Add(48 * time.Hour) },
Acts: handacts.Recorder{Ask: func(key string, body any) (json.RawMessage, error) {
raw, _ := json.Marshal(body)
acts = append(acts, key+" "+string(raw))
return json.RawMessage(`{}`), nil
}}}
h.Acts.LeaseAway = h.LeaseAway
return h, b, door, &acts
}
// ADR 0571 clause 2: the keep set from the store's listing, the wants and the five newest builds; nothing let go of while
// the lease names a controller; once it names another holder, a build's end lets go of the eligible within the bound,
// leaves an index for a confirmed collect, records each deletion and says artifacts-gone.
func TestTheSeatKeepsTheFiveNewestAndTheWantedAndSweepsOnlyOnceTheLeaseIsAway(t *testing.T) {
controller := &buscompose.LeaseHolder{Instance: "c1", Host: "novox", Epoch: 3}
handed := &buscompose.LeaseHolder{HandedTo: "mesh-grants", HandedBy: "person.jochen", Why: "the lease write"}
h, _, _, _ := aMesh(t, controller)
r, err := h.Artifacts("")
if err != nil {
t.Fatal(err)
}
states := map[string]string{}
for _, x := range r.References {
states[x.Reference] = x.State
}
// letta's 3rd to 7th are the five newest; its 1st a want names on ace, however old; its 2nd is eligible; the package
// and the manifest no build names are no reference of the seat's.
for n, want := range map[int]string{1: "kept", 2: "eligible", 3: "kept", 7: "kept"} {
if states[imageRef("letta", n)] != want || states[archiveRef("letta", n)] != want {
t.Errorf("letta's build %d is %s and %s, not %s", n, states[imageRef("letta", n)], states[archiveRef("letta", n)], want)
}
}
// And the copies: the one letta's 2nd stood on is eligible with it, the 7th's kept; the third no build names.
if states[mirror(1)] != "eligible" || states[mirror(2)] != "kept" || states[mirror(3)] != "" {
t.Errorf("the copies are %s, %s and %q", states[mirror(1)], states[mirror(2)], states[mirror(3)])
}
if m, err := h.MirrorsNow(); err != nil || len(m.Copies) != 3 || m.Copies[2].State != "unrecorded" || m.Held != "" {
t.Errorf("mirrors answered %+v: %v", m, err)
}
if len(r.References) != 16 || r.Counts["kept"] != 13 || r.Counts["eligible"] != 3 || r.Store != "10.0.0.5:5000" {
t.Errorf("the references are %d, counted %v, at %q", len(r.References), r.Counts, r.Store)
}
for _, c := range []struct {
name string
lease *buscompose.LeaseHolder
index bool
most int
letGo []string
left int
skipped int
unlog int
}{
{name: "the lease names a controller", lease: controller},
{name: "no lease key", lease: nil},
{name: "handed away", lease: handed, most: MostPerSweep, letGo: []string{archiveRef("letta", 2), imageRef("letta", 2), mirror(1)}},
{name: "an index waits", lease: handed, index: true, most: MostPerSweep, letGo: []string{archiveRef("letta", 2), mirror(1)}, skipped: 1},
{name: "bounded", lease: handed, most: 1, letGo: []string{archiveRef("letta", 2)}, left: 2},
// A kept build the log no longer holds: what it stood on is unknown, so no copy goes.
{name: "a kept build's bases unknown", lease: handed, most: MostPerSweep, letGo: []string{archiveRef("letta", 2), imageRef("letta", 2)}, unlog: 3},
} {
t.Run(c.name, func(t *testing.T) {
h, b, door, acts := aMesh(t, c.lease)
door.indexes[imageRef("letta", 2)] = c.index
if c.unlog > 0 {
b.log = append(b.log[:c.unlog-1:c.unlog-1], b.log[c.unlog:]...)
}
if c.most == 0 {
h.AfterBuild("letta.x")
if _, err := h.Collect("tidy", true, 0); err == nil {
t.Error("a confirmed collect was let through while the controller sweeps")
}
if len(door.asked) != 0 || len(*acts) != 0 || len(b.published) != 0 {
t.Fatalf("before the lease write the store was asked %v, acts %v, said %v", door.asked, *acts, b.published)
}
return
}
a, err := h.Sweep(context.Background(), Sweeping{Most: c.most, By: "after the build letta.x"})
if err != nil {
t.Fatal(err)
}
if fmt.Sprint(a.LetGo) != fmt.Sprint(c.letGo) || a.Left != c.left || a.Skipped != c.skipped || a.Held != 6 || a.Missing != 0 {
t.Fatalf("let go of %v (left %d, skipped %d, held %d, missing %d), want %v", a.LetGo, a.Left, a.Skipped, a.Held, a.Missing, c.letGo)
}
// Every kept archive held before the first deletion.
first := -1
for i, x := range door.asked {
if strings.HasPrefix(x, "let go ") && first < 0 {
first = i
}
}
if first != 6 {
t.Errorf("the store was asked %v: the six kept archives are held first", door.asked)
}
if len(*acts) != len(c.letGo) || !strings.HasPrefix((*acts)[0], "seat:mesh-hand-acts.hand-act ") ||
!strings.Contains((*acts)[0], c.letGo[0]) {
t.Errorf("the hand acts recorded are %v", *acts)
}
if len(b.published) != 1 || !strings.HasPrefix(b.published[0], GoneSubject+" ") ||
!strings.Contains(b.published[0], `"fingerprint":"`+fp(2)+`"`) {
t.Errorf("said %v", b.published)
}
})
}
}
// S25 (image-missing): a want whose build names an image the store does not hold is a finding in the seat's own state,
// written once and deleted once the store holds it again; images names what a machine's wants and haves name.
func TestAWantedImageTheStoreLacksIsAFindingAndImagesNamesWantsAndHaves(t *testing.T) {
h, b, _, _ := aMesh(t, nil)
view, _ := h.Listed(context.Background())
repo, digest, _ := splitReference(imageRef("letta", 1))
delete(view.L.Repos[repo].Revisions, digest)
h.Listed = func(context.Context) (*View, error) { return view, nil }
if err := h.Observe(context.Background()); err != nil {
t.Fatal(err)
}
key := health.FindingKey(health.ScopeModule, "letta.ace", ImageMissing)
raw, found, _ := b.Get(FindingsState, key)
f, err := health.DecodeFinding(raw)
if !found || err != nil || f.Subject.Machine != "ace" || f.Source != "S25" || !strings.Contains(f.Evidence[0].Said, imageRef("letta", 1)) {
t.Fatalf("the finding is %+v (%v, %v)", f, found, err)
}
view.L.Repos[repo].Revisions[digest] = true
if err := h.Observe(context.Background()); err != nil {
t.Fatal(err)
}
if _, found, _ := b.Get(FindingsState, key); found {
t.Fatal("the finding stayed once the store held the image again")
}
// An image the manifest pins by itself, under the store's address, is judged as the controller judged it.
delete(view.L.Repos["letta/pinned"].Revisions, "sha256:"+strings.Repeat("c", 64))
if err := h.Observe(context.Background()); err != nil {
t.Fatal(err)
}
if raw, found, _ := b.Get(FindingsState, key); !found || !strings.Contains(string(raw), "letta/pinned@sha256:") {
t.Fatalf("the pinnedImage image missing is %s", raw)
}
have, _ := json.Marshal(delivery.Have{V: delivery.Version, FP: fp(7), Phase: "running"})
_ = b.Put(HavesRead, "ace.letta", json.RawMessage(have))
images, err := h.ImagesOf("ace", "10.0.0.5:5000")
if err != nil || len(images.Images) != 3 || !images.SentKnown || images.Images[0].Image != "10.0.0.5:5000/letta/image@sha256:"+strings.Repeat("1", 64) ||
fmt.Sprint(images.Images[0].In) != "[declared]" || fmt.Sprint(images.Images[1].In) != "[sent]" ||
images.Images[2].Image != pinnedImage || fmt.Sprint(images.Images[2].In) != "[declared sent]" {
t.Fatalf("the images of ace are %+v: %v", images, err)
}
}
@@ -1,38 +1,30 @@
// The artifact-store-tools module's Go bundle (novox/hq ADR 0251 §1–3, ADR 0571 clause 2, to-be 51): a process the
// node's runtime launches and speaks MCP over stdio to. It says what the artifact store holds — its repositories, its
// size, and what the mesh keeps of each manifest — read from the store's own files through the store's own container,
// and it holds the seat mesh-artifact-sweep: the keep set, the sweep at each build's end once the controller's lease names
// no controller, and S25's image-missing.
// The artifact-store-tools module's Go bundle (novox/hq ADR 0251 §1–3, to-be 51): a process the node's
// runtime launches and speaks MCP over stdio to. It says what the artifact store holds — its
// repositories, its size, and what the controller's records say of each manifest — read from the
// store's own files through the store's own container, and writing nothing. A collection is asked of
// the controller, which decides and records what it lets go of (ADR 0189).
//
// **A module beside the store, not the store's own.** A module that provides the artifact store cannot also build an
// artifact: building publishes to the store, so the store would be needed to create itself. So these tools are a
// second module, assigned to the machine that holds the store. stdout is the protocol; what this bundle says, it says
// on stderr.
// **A module beside the store, not the store's own.** A module that provides the artifact store cannot
// also build an artifact: building publishes to the store, so the store would be needed to create
// itself, and the controller refuses that manifest. So these tools are a second module, assigned to the
// machine that holds the store. stdout is the protocol; what this bundle says, it says on stderr.
package main
import (
"fmt"
"os"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/handacts"
)
func logf(format string, a ...any) { fmt.Fprintf(os.Stderr, format+"\n", a...) }
func main() {
s := &Store{
Container: os.Getenv("MESH_ARTIFACT_STORE_CONTAINER"),
Run: ExecRunner,
UID: os.Getuid(),
}
h := &Holder{Bus: runtimeBus{}, Door: DoorAt(os.Getenv("MESH_ARTIFACT_STORE_ADDRESS_FILE")), Listed: s.View,
Now: time.Now, Logf: logf}
h.Acts = handacts.Recorder{Ask: stdio.Ask, LeaseAway: h.LeaseAway}
go follow(h)
// An empty name serves as the module the runtime names (MESH_SERVED_MODULE): artifact-store-tools.
if err := stdio.Serve("", append(Tools(s, h, &Catalogue{Ask: stdio.Ask}), SeatTools(h, s)...)); err != nil {
if err := stdio.Serve("", Tools(s, Controller{Ask: stdio.Ask}, &Catalogue{Ask: stdio.Ask})); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
@@ -1,122 +0,0 @@
package main
// The seat mesh-artifact-sweep's verbs (novox/hq ADR 0571 clause 2; catalogue.ArtifactSweepSeat), each a tool named
// `artifact-store.<verb>` as the runtime serves a seat's verb, answering as the controller's verb answered: {output,
// ok, answer}.
import (
"fmt"
"strings"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
// verbAnswer is the controller's verbs' answer shape, which the builder and docker read (answerOf).
type verbAnswer struct {
Output string `json:"output"`
OK bool `json:"ok"`
Answer any `json:"answer"`
}
// SeatTools are the seat's verbs over h; s reads the store's listing for mirrors.
func SeatTools(h *Holder, s *Store) []stdio.Tool {
seat := catalogue.ArtifactSweepSeat + "."
return []stdio.Tool{
{Name: seat + "artifacts", Description: "Every artifact the mesh made that the store holds: kept and why, eligible, " +
"or with collected those it no longer holds.",
Run: func(args map[string]any) (any, error) {
r, err := h.Artifacts(text(args, "repository"))
if err != nil {
return nil, err
}
if text(args, "collected") != "true" {
refs := r.References[:0:0]
for _, x := range r.References {
if x.State != "collected" {
refs = append(refs, x)
}
}
r.References = refs
}
return verbAnswer{Output: fmt.Sprintf("%d kept, %d eligible, %d collected\n", r.Counts["kept"],
r.Counts["eligible"], r.Counts["collected"]), OK: true, Answer: r}, nil
}},
{Name: seat + "held", Description: "Whether the store holds each reference given: held, gone or unknown with why.",
Run: func(args map[string]any) (any, error) {
a, err := h.Held(listOf(args["references"]))
if err != nil {
return nil, err
}
return verbAnswer{Output: fmt.Sprintf("the artifact store at %s holds %d of the %d reference(s) asked\n",
a.Store, len(a.Held), len(a.Held)+len(a.Gone)+len(a.Unknown)), OK: true, Answer: a}, nil
}},
{Name: seat + "collect", Description: "The sweep now: a dry run without confirm; with confirm and why, every kept " +
"archive held, then each eligible artifact let go of, each recorded in the hand-act log.",
Run: func(args map[string]any) (any, error) {
most, err := count(args, "most", MostPerSweep, 1, MostAtAll)
if err != nil {
return nil, err
}
a, err := h.Collect(text(args, "why"), text(args, "confirm") == "true", most)
if err != nil {
return nil, err
}
return verbAnswer{Output: fmt.Sprintf("let go of %d, would let go of %d, %d left\n", len(a.LetGo),
len(a.WouldLetGo), a.Left), OK: true, Answer: a}, nil
}},
{Name: seat + "mirrors", Description: "The bases the mesh copied into its store, each kept (a kept build stood on " +
"it), eligible (only builds no longer kept stood on it) or unrecorded (no build the builder's log holds stood " +
"on it: never removed).",
Run: func(args map[string]any) (any, error) {
if text(args, "record") != "" {
return nil, fmt.Errorf("mirrors records nothing by hand on this seat: a copy is recorded by the builds the " +
"builder's log says stood on it (novox/hq ADR 0571 clause 1)")
}
m, err := h.MirrorsNow()
if err != nil {
return nil, err
}
return verbAnswer{Output: fmt.Sprintf("%d copies\n", len(m.Copies)), OK: true, Answer: m}, nil
}},
{Name: seat + "images", Description: "Every container image one machine's wants and haves name.",
Run: func(args map[string]any) (any, error) {
node := text(args, "node")
if node == "" {
return nil, fmt.Errorf("images names the machine")
}
address := ""
if d, err := h.Door(); err == nil {
address = d.Address()
}
a, err := h.ImagesOf(node, address)
if err != nil {
return nil, err
}
return verbAnswer{Output: fmt.Sprintf("%s: %d images\n", node, len(a.Images)), OK: true, Answer: a}, nil
}},
}
}
// listOf is a list argument: a list, or one text separated by commas.
func listOf(v any) []string {
var out []string
add := func(s string) {
for _, p := range strings.Split(s, ",") {
if p = strings.TrimSpace(p); p != "" && !contains(out, p) {
out = append(out, p)
}
}
}
switch x := v.(type) {
case string:
add(x)
case []any:
for _, e := range x {
if s, ok := e.(string); ok {
add(s)
}
}
}
return out
}
@@ -1,8 +1,6 @@
package main
import (
"git.novox.be/novox/mesh-sdk/go/catalogue"
"context"
"encoding/json"
"fmt"
@@ -413,11 +411,6 @@ func TestTheToolsServedAreTheToolsTheManifestNames(t *testing.T) {
var m struct {
Tools []string `json:"tools"`
Invokes []string `json:"invokes"`
Reads []string `json:"reads"`
Claims []struct {
Name string `json:"name"`
Serves []string `json:"serves"`
} `json:"claims"`
}
if err := json.Unmarshal(raw, &m); err != nil {
t.Fatal(err)
@@ -435,30 +428,10 @@ func TestTheToolsServedAreTheToolsTheManifestNames(t *testing.T) {
if !reflect.DeepEqual(served, listed) {
t.Fatalf("served %v, manifest %v", served, listed)
}
// ADR 0571 clause 2: it holds the seat mesh-artifact-sweep and serves its verbs, and calls no verb of the controller's;
// the lease is the one controller state it reads (clause 10's exemption).
want := []string{"seat:mesh-hand-acts.hand-act", CatalogueModules, CatalogueModule}
want := []string{"seat:mesh-controller.artifacts", "seat:mesh-controller.collect", CatalogueModules, CatalogueModule}
if !reflect.DeepEqual(m.Invokes, want) {
t.Errorf("invokes %v", m.Invokes)
}
var verbs []string
for _, tl := range SeatTools(&Holder{}, &Store{}) {
verbs = append(verbs, strings.TrimPrefix(tl.Name, catalogue.ArtifactSweepSeat+"."))
}
seat, _ := catalogue.SeatNamed(catalogue.ArtifactSweepSeat)
var contract []string
for _, v := range seat.Serves {
contract = append(contract, v.Name)
}
if len(m.Claims) != 1 || m.Claims[0].Name != catalogue.ArtifactSweepSeat || !reflect.DeepEqual(m.Claims[0].Serves, verbs) ||
!reflect.DeepEqual(verbs, contract) {
t.Errorf("claims %+v, serves %v, the seat's verbs %v", m.Claims, verbs, contract)
}
for _, r := range m.Reads {
if strings.HasPrefix(r, "mesh-controller.") && r != LeaseRead {
t.Errorf("reads the controller's %s", r)
}
}
}
func TestASocketRefusalIsAskedAgainThroughSudo(t *testing.T) {
@@ -11,16 +11,9 @@ import (
stdio "git.novox.be/novox/mesh-sdk/go"
)
// Recorded is what says which references the mesh keeps and lets go of them: the seat's holder (Holder, novox/hq ADR
// 0571 clause 2), or before it the controller (Controller), each in the controller's answer's shape.
type Recorded interface {
Artifacts(repository string) (*Records, error)
Collect(why string, confirm bool, most int) (*CollectAnswer, error)
}
// Tools are the artifact store's tools (novox/hq ADR 0251 §1). The first three only read; artifact_store_collect
// only reads too, unless it is a real run, and then it asks the seat's sweep, which decides and deletes.
func Tools(s *Store, c Recorded, cat *Catalogue) []stdio.Tool {
// only reads too, unless it is a real run, and then it asks the controller, which decides and deletes.
func Tools(s *Store, c Controller, cat *Catalogue) []stdio.Tool {
ctx := context.Background()
repoArg := map[string]any{"type": "string", "description": "one repository, as the store names it (<module>/<artifact>)"}
return []stdio.Tool{
@@ -61,7 +54,7 @@ func Tools(s *Store, c Recorded, cat *Catalogue) []stdio.Tool {
Name: "artifact_store_references",
Description: "What the controller's records say of each manifest the artifact store holds: kept (a definition " +
"names it, or one of the five most recent builds of a module the mesh holds), named by a definition (no " +
"record names it, and a definition in the catalog pins it by digest, under the store's current name or " +
"record names it, and a definition in the catalogue pins it by digest, under the store's current name or " +
"an older one: the module is said), the holder of a kept archive, " +
"eligible (the mesh made it and keeps it for no reason), the holder of an eligible archive, let go yet present, " +
"a named document the controller keeps under a tag, or unrecorded — no record names it. A platform manifest " +
@@ -1 +0,0 @@
${seat:mesh-artifact-store:reach}
+2 -19
View File
@@ -1,22 +1,5 @@
module artifactstoretools
go 1.26.0
go 1.22
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030001116-2da0de75c3f0
github.com/nats-io/nats.go v1.54.0
)
require (
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op // indirect
github.com/google/go-tpm v0.9.8 // indirect
github.com/klauspost/compress v1.20.0 // indirect
github.com/minio/highwayhash v1.0.4 // indirect
github.com/nats-io/jwt/v2 v2.8.1 // indirect
github.com/nats-io/nats-server/v2 v2.11.17 // indirect
github.com/nats-io/nkeys v0.4.16 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
golang.org/x/crypto v0.57.0 // indirect
golang.org/x/sys v0.48.0 // indirect
golang.org/x/time v0.15.0 // indirect
)
require git.novox.be/novox/mesh-sdk/go v0.1.7
+2 -27
View File
@@ -1,27 +1,2 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030001116-2da0de75c3f0 h1:lKU3HjNXGiVbLW2hUgFjU/TMTUYMlNnnmZxG5T7biQA=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030001116-2da0de75c3f0/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
github.com/google/go-tpm v0.9.8/go.mod h1:h9jEsEECg7gtLis0upRBQU+GhYVH6jMjrFxI8u6bVUY=
github.com/klauspost/compress v1.20.0 h1:a3C1ke2ohxFymNlb2HWAHjDeKCI90scRskErZkR0ezA=
github.com/klauspost/compress v1.20.0/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/minio/highwayhash v1.0.4 h1:asJizugGgchQod2ja9NJlGOWq4s7KsAWr5XUc9Clgl4=
github.com/minio/highwayhash v1.0.4/go.mod h1:GGYsuwP/fPD6Y9hMiXuapVvlIUEhFhMTh0rxU3ik1LQ=
github.com/nats-io/jwt/v2 v2.8.1 h1:V0xpGuD/N8Mi+fQNDynXohVvp7ZztevW5io8CUWlPmU=
github.com/nats-io/jwt/v2 v2.8.1/go.mod h1:nWnOEEiVMiKHQpnAy4eXlizVEtSfzacZ1Q43LIRavZg=
github.com/nats-io/nats-server/v2 v2.11.17 h1:GKEghcFK6A+aFx11Yf1LjgLC3txAwvyhnYzhBIQZA8I=
github.com/nats-io/nats-server/v2 v2.11.17/go.mod h1:B1sFVz4StNosQ903ak4N1G01Fl/9f8e06mXpFIE2K24=
github.com/nats-io/nats.go v1.54.0 h1:vsXoOxjHp/GmPUN+EcI7uOf/uB+iAP+kEsAFNQN0yzA=
github.com/nats-io/nats.go v1.54.0/go.mod h1:y+DZoD1oBOYfZTU681eTUiUjI0vbqYGixNVFHcjHJ0k=
github.com/nats-io/nkeys v0.4.16 h1:rd5oAuLOb8mnAycB0xleuEBNS1pVVnN0fv/FF34Eypg=
github.com/nats-io/nkeys v0.4.16/go.mod h1:llLgWoI0o4z/Q57q2R1kHfmocyhGV6VG/U18Glg1Afs=
github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw=
github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
git.novox.be/novox/mesh-sdk/go v0.1.7 h1:C0sTQmtTiyYH7bnqZb7PusXnqA37gKuT7Nqjn9gG47w=
git.novox.be/novox/mesh-sdk/go v0.1.7/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
@@ -1,57 +0,0 @@
package registry
import (
"context"
"crypto/sha256"
"encoding/hex"
"fmt"
"io"
"net/http"
)
// blobBound is the most GetBlob reads: a templates archive is a few kilobytes; anything near this is no archive a
// take-in reads text from.
const blobBound = 64 << 20
// GetBlob is the archive a recorded reference names (`artifact-store://<repository>/blobs/sha256:<hex>`), read whole
// and checked against the digest it names: bytes that hash to another digest are refused, never read (R4e: the
// controller reads a templates archive by the reference of a builder's done key). Gone when the store does not hold
// it; ErrNotOurs for a reference outside the store or one that is not an archive.
func (s Store) GetBlob(ctx context.Context, reference string) ([]byte, error) {
repository, digest, archive, err := s.archive(reference)
if err != nil {
return nil, err
}
if !archive {
return nil, fmt.Errorf("%w: %s is not an archive", ErrNotOurs, reference)
}
request, err := http.NewRequestWithContext(ctx, http.MethodGet, s.url(repository, "blobs", digest), nil)
if err != nil {
return nil, err
}
response, err := s.client().Do(request)
if err != nil {
return nil, fmt.Errorf("cannot reach the artifact store at %s: %w", s.Address, err)
}
defer response.Body.Close()
switch response.StatusCode {
case http.StatusOK:
case http.StatusNotFound:
return nil, Gone
default:
return nil, fmt.Errorf("the artifact store answered %s for %s/blobs/%s", response.Status, repository, digest)
}
body, err := io.ReadAll(io.LimitReader(response.Body, blobBound+1))
if err != nil {
return nil, fmt.Errorf("reading %s/blobs/%s: %w", repository, digest, err)
}
if len(body) > blobBound {
return nil, fmt.Errorf("%s/blobs/%s is larger than %d bytes, which no archive read here is", repository, digest, blobBound)
}
sum := sha256.Sum256(body)
if got := "sha256:" + hex.EncodeToString(sum[:]); got != digest {
return nil, fmt.Errorf("the artifact store answered %s/blobs/%s with bytes of the digest %s: refused", repository,
digest, got)
}
return body, nil
}
@@ -1,44 +0,0 @@
package registry
import (
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"net/http"
"net/http/httptest"
"strings"
"testing"
)
// A take-in reads a templates archive by its reference, and only bytes of the digest the reference names.
func TestGetBlobReadsOnlyTheBytesOfItsDigest(t *testing.T) {
good := []byte("the archive")
sum := sha256.Sum256(good)
digest := "sha256:" + hex.EncodeToString(sum[:])
liar := "sha256:" + strings.Repeat("0", 64)
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch {
case strings.HasSuffix(r.URL.Path, "/x/files/blobs/"+digest), strings.HasSuffix(r.URL.Path, "/x/files/blobs/"+liar):
_, _ = w.Write(good)
default:
w.WriteHeader(http.StatusNotFound)
}
}))
defer server.Close()
s := Store{Address: strings.TrimPrefix(server.URL, "http://")}
ctx := context.Background()
body, err := s.GetBlob(ctx, "artifact-store://x/files/blobs/"+digest)
if err != nil || string(body) != string(good) {
t.Fatalf("the archive: %q %v", body, err)
}
if _, err := s.GetBlob(ctx, "artifact-store://x/files/blobs/"+liar); err == nil || !strings.Contains(err.Error(), "refused") {
t.Fatalf("bytes of another digest are read: %v", err)
}
if _, err := s.GetBlob(ctx, "artifact-store://x/other/blobs/"+digest); !errors.Is(err, Gone) {
t.Fatalf("a blob the store does not hold: %v", err)
}
if _, err := s.GetBlob(ctx, "registry.example/x@"+digest); !errors.Is(err, ErrNotOurs) {
t.Fatalf("a reference outside the store: %v", err)
}
}
@@ -1,345 +0,0 @@
package registry
import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"io"
"net/http"
"strconv"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
// Every archive the store keeps is held by a manifest (novox/hq issue 253, ADR 0189).
//
// **The store's collector marks only from manifests.** The mesh's store is a stock registry, and
// its nightly `registry garbage-collect` walks every manifest in every repository, marks the blobs
// those manifests name, and deletes every blob it did not mark. An image is a manifest, so what
// the mesh keeps of an image survives. An archive was not: the builder put it in the store as a
// bare blob — upload, then `PUT ?digest=` — and nothing in the store names it. To the collector a
// bare blob is unreferenced, so the first real collection would have deleted every archive the
// mesh holds, kept or not, and every machine pinning a bundle would have found it gone. The
// collector runs `--dry-run` until this is true.
//
// **So each archive gets a holder**: the smallest OCI image manifest that names it — the empty
// config, one layer, nothing else — put in the archive's own repository, by digest, untagged. The
// collector marks it and so keeps the archive; the sweep lets go of an archive by deleting its
// holder first, which is what lets the bytes go at the next collection.
//
// **Nothing a machine reads changes.** The recorded reference stays
// `artifact-store://<module>/<artifact>/blobs/sha256:…`, and machines fetch the blob exactly as
// before. The holder is the store's bookkeeping, not a second way to reach anything.
//
// **Deterministic, so it never needs recording.** The holder is composed from the archive's digest
// and size alone, in a fixed field order with no timestamps or annotations, so the sweep can
// compute which manifest holds any archive from the reference it already has plus one HEAD for the
// size. No schema change, no second record that could disagree with the store.
const (
// mediaManifest is the type a holder is put and asked for as.
mediaManifest = "application/vnd.oci.image.manifest.v1+json"
// mediaEmpty is the OCI empty descriptor's type: a config that says nothing, for a manifest
// whose only purpose is to name its layer.
mediaEmpty = "application/vnd.oci.empty.v1+json"
// emptyDigest is the digest of `{}`, the empty config's content, fixed by the OCI spec.
emptyDigest = "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a"
// mediaArchive is the layer type an archive is held as. Every archive the builder publishes is
// `pack`'s gzipped tar, so this is the true type and not a placeholder — and it is a constant,
// not read from anywhere, because the holder must be recomputable from the reference alone.
mediaArchive = "application/vnd.oci.image.layer.v1.tar+gzip"
)
// emptyConfig is the content emptyDigest names.
var emptyConfig = []byte("{}")
// manifestAccept is what a manifest is asked for as. A registry answers a manifest HEAD only in a
// type the caller named, and answers 404 to a bare one for a manifest it holds perfectly well
// (measured 2026-09-28; internal/builder/registry.go says how that was found).
var manifestAccept = []string{
mediaManifest,
"application/vnd.docker.distribution.manifest.v2+json",
}
type descriptor struct {
MediaType string `json:"mediaType"`
Digest string `json:"digest"`
Size int64 `json:"size"`
}
type holderManifest struct {
SchemaVersion int `json:"schemaVersion"`
MediaType string `json:"mediaType"`
Config descriptor `json:"config"`
Layers []descriptor `json:"layers"`
}
// Holder is the manifest that holds an archive in the store, and its digest.
//
// A pure function of the archive's digest and size: the same two in give the same bytes out,
// always, because `encoding/json` writes a struct's fields in their declared order and there is
// nothing here that varies by when or where it was composed.
func Holder(digest string, size int64) (body []byte, holder string) {
body, err := json.Marshal(holderManifest{
SchemaVersion: 2,
MediaType: mediaManifest,
Config: descriptor{MediaType: mediaEmpty, Digest: emptyDigest, Size: int64(len(emptyConfig))},
Layers: []descriptor{{MediaType: mediaArchive, Digest: digest, Size: size}},
})
if err != nil {
// Marshalling a struct of strings and integers cannot fail.
panic(err)
}
sum := sha256.Sum256(body)
return body, "sha256:" + hex.EncodeToString(sum[:])
}
// Hold makes sure the store holds this archive by a manifest, and says whether it had to write one.
//
// Takes a reference as the mesh records it. An image is its own manifest and needs no holder, so
// it answers false and nothing is asked. Idempotent: a holder already there is left alone, which
// is what lets the sweep run it over every kept archive on every build and so backfill the bare
// blobs published before holders existed (novox/hq issue 253).
//
// Gone when the store does not hold the archive at all: there is nothing to hold, and that is a
// fact the caller reports rather than one this invents a remedy for.
func (s Store) Hold(ctx context.Context, reference string) (bool, error) {
repository, digest, archive, err := s.archive(reference)
if err != nil || !archive {
return false, err
}
size, err := s.blobSize(ctx, repository, digest)
if err != nil {
return false, err
}
return s.HoldBlob(ctx, repository, digest, size)
}
// Held is whether the store holds this archive by its manifest. Asks and changes nothing — the
// question an operator needs answered with "none unheld" before the collector is let loose.
//
// An image answers true: it is its own manifest. An archive the store does not have answers Gone.
func (s Store) Held(ctx context.Context, reference string) (bool, error) {
repository, digest, archive, err := s.archive(reference)
if err != nil {
return false, err
}
if !archive {
return true, nil
}
size, err := s.blobSize(ctx, repository, digest)
if err != nil {
return false, err
}
_, holder := Holder(digest, size)
return s.has(ctx, s.url(repository, "manifests", holder), manifestAccept...)
}
// HoldBlob puts the holder for a blob of this digest and size into its repository, unless it is
// there already. Answers whether it wrote one.
//
// The builder calls this with the size it has just uploaded; the sweep, through Hold, with the size
// the store reports. Both arrive at the same holder, which is the point of composing it.
func (s Store) HoldBlob(ctx context.Context, repository, digest string, size int64) (bool, error) {
if s.Address == "" {
return false, fmt.Errorf("this mesh has no artifact store on its network to hold %s/%s in", repository, digest)
}
body, holder := Holder(digest, size)
there, err := s.has(ctx, s.url(repository, "manifests", holder), manifestAccept...)
if err != nil {
return false, err
}
if there {
return false, nil
}
// The config must be in the repository before a manifest naming it is accepted: a registry
// refuses a manifest whose blobs it cannot find there, which is the property that makes a
// holder mean something.
if err := s.putBlob(ctx, repository, emptyDigest, emptyConfig); err != nil {
return false, err
}
// **By digest, never by tag.** A tag would be one more name to move and one more thing the
// collector's `--delete-untagged` would read as meaningful; the mesh names nothing by tag that
// it pins by digest, and an untagged manifest is kept by plain collection.
request, err := http.NewRequestWithContext(ctx, http.MethodPut,
s.url(repository, "manifests", holder), bytes.NewReader(body))
if err != nil {
return false, err
}
request.Header.Set("Content-Type", mediaManifest)
response, err := s.client().Do(request)
if err != nil {
return false, err
}
defer response.Body.Close()
if response.StatusCode != http.StatusCreated {
said, _ := io.ReadAll(io.LimitReader(response.Body, 4096))
return false, fmt.Errorf("the artifact store refused to hold %s/%s: %s %s",
repository, digest, response.Status, strings.TrimSpace(string(said)))
}
return true, nil
}
// letGoOfHolder deletes the manifest holding an archive, before the archive's own link goes.
//
// **Holder first.** Deleting the blob link alone leaves a manifest still naming the blob, and the
// collector would keep its bytes for ever on the strength of it — the sweep would record the
// archive collected while the disk said otherwise. Deleting the holder first and failing before
// the link goes leaves an unheld archive that the next sweep still offers, which is safe.
//
// A store that no longer has the blob answers Gone: without its size the holder cannot be named,
// and without the blob there is nothing left for a holder to keep. A store that never had a holder
// for it — an archive published before holders, never backfilled — answers 404 to the delete, and
// that is the outcome wanted.
func (s Store) letGoOfHolder(ctx context.Context, repository, digest string) error {
size, err := s.blobSize(ctx, repository, digest)
if err != nil {
return err
}
_, holder := Holder(digest, size)
err = s.remove(ctx, s.url(repository, "manifests", holder), repository+"/manifests/"+holder)
if err == Gone {
return nil
}
return err
}
// archive reads a recorded reference into its repository and digest, and whether it is an archive
// at all. Refuses as ErrNotOurs anything the mesh did not put in its own store.
func (s Store) archive(reference string) (repository, digest string, archive bool, err error) {
path, kept := catalogue.InArtifactStore(reference)
if !kept {
return "", "", false, fmt.Errorf("%w: %s", ErrNotOurs, reference)
}
if s.Address == "" {
return "", "", false, fmt.Errorf("this mesh has no artifact store on its network to ask about %s", reference)
}
repository, kind, digest, err := split(path)
if err != nil {
return "", "", false, err
}
return repository, digest, kind == "blobs", nil
}
// blobSize is how large the store says a blob is; Gone when it does not have it.
func (s Store) blobSize(ctx context.Context, repository, digest string) (int64, error) {
request, err := http.NewRequestWithContext(ctx, http.MethodHead, s.url(repository, "blobs", digest), nil)
if err != nil {
return 0, err
}
response, err := s.client().Do(request)
if err != nil {
return 0, fmt.Errorf("cannot reach the artifact store at %s: %w", s.Address, err)
}
defer response.Body.Close()
switch response.StatusCode {
case http.StatusOK:
case http.StatusNotFound:
return 0, Gone
default:
return 0, fmt.Errorf("the artifact store answered %s for %s/blobs/%s", response.Status, repository, digest)
}
// Read from the header rather than ContentLength: a HEAD's ContentLength is what the response
// says it would have sent, which Go reports faithfully, but a proxy in between is free to drop
// it, and the header is what the registry itself wrote.
if length := response.Header.Get("Content-Length"); length != "" {
if n, err := strconv.ParseInt(length, 10, 64); err == nil && n >= 0 {
return n, nil
}
}
if response.ContentLength >= 0 {
return response.ContentLength, nil
}
return 0, fmt.Errorf("the artifact store holds %s/blobs/%s and will not say how large it is", repository, digest)
}
// putBlob uploads a small blob unless the repository already has it: ask where, then put it there
// naming the digest — the registry's own two steps, the same the builder takes for an archive.
func (s Store) putBlob(ctx context.Context, repository, digest string, body []byte) error {
if there, err := s.has(ctx, s.url(repository, "blobs", digest)); err != nil {
return err
} else if there {
return nil
}
start, err := http.NewRequestWithContext(ctx, http.MethodPost,
"http://"+s.Address+"/v2/"+repository+"/blobs/uploads/", nil)
if err != nil {
return err
}
begun, err := s.client().Do(start)
if err != nil {
return fmt.Errorf("cannot start an upload to %s: %w", repository, err)
}
begun.Body.Close()
if begun.StatusCode != http.StatusAccepted {
return fmt.Errorf("the artifact store answered %s when asked where to put a blob in %s", begun.Status, repository)
}
where := begun.Header.Get("Location")
if where == "" {
return fmt.Errorf("the artifact store accepted an upload to %s and said nowhere to put it", repository)
}
if strings.HasPrefix(where, "/") {
where = "http://" + s.Address + where
}
separator := "?"
if strings.Contains(where, "?") {
separator = "&"
}
put, err := http.NewRequestWithContext(ctx, http.MethodPut, where+separator+"digest="+digest, bytes.NewReader(body))
if err != nil {
return err
}
put.Header.Set("Content-Type", "application/octet-stream")
done, err := s.client().Do(put)
if err != nil {
return err
}
defer done.Body.Close()
if done.StatusCode != http.StatusCreated {
said, _ := io.ReadAll(io.LimitReader(done.Body, 4096))
return fmt.Errorf("the artifact store refused a blob in %s: %s %s", repository, done.Status, strings.TrimSpace(string(said)))
}
return nil
}
// has is whether the store answers 200 for a HEAD at that URL.
func (s Store) has(ctx context.Context, url string, accept ...string) (bool, error) {
request, err := http.NewRequestWithContext(ctx, http.MethodHead, url, nil)
if err != nil {
return false, err
}
for _, media := range accept {
request.Header.Add("Accept", media)
}
response, err := s.client().Do(request)
if err != nil {
return false, fmt.Errorf("cannot reach the artifact store at %s: %w", s.Address, err)
}
defer response.Body.Close()
switch response.StatusCode {
case http.StatusOK:
return true, nil
case http.StatusNotFound:
return false, nil
default:
return false, fmt.Errorf("the artifact store answered %s for %s", response.Status, url)
}
}
func (s Store) url(repository, kind, digest string) string {
return "http://" + s.Address + "/v2/" + repository + "/" + kind + "/" + digest
}
func (s Store) client() *http.Client {
if s.HTTP != nil {
return s.HTTP
}
return &http.Client{Timeout: 30 * time.Second}
}
@@ -1,268 +0,0 @@
package registry
import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"net/http/httptest"
"strconv"
"strings"
"sync"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
// Every kept archive is held by a manifest (novox/hq issue 253, ADR 0189).
//
// Against an in-memory registry that keeps blobs and manifests per repository and refuses what a
// registry refuses — a blob whose digest does not match, a manifest whose digest does not match
// or whose blobs the repository does not have, a manifest asked for without an Accept naming its
// type. What is asserted is this side's decisions; the live test below asserts the registry's.
type memRegistry struct {
mu sync.Mutex
blobs map[string][]byte // repository + "@" + digest
manifests map[string][]byte // repository + "@" + digest
writes []string // every PUT and DELETE, as "METHOD path"
}
func digestOf(body []byte) string {
sum := sha256.Sum256(body)
return "sha256:" + hex.EncodeToString(sum[:])
}
func (m *memRegistry) serve(t *testing.T) Store {
t.Helper()
m.blobs = map[string][]byte{}
m.manifests = map[string][]byte{}
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
m.mu.Lock()
defer m.mu.Unlock()
path := strings.TrimPrefix(r.URL.Path, "/v2/")
if r.Method == http.MethodPut || r.Method == http.MethodDelete {
m.writes = append(m.writes, r.Method+" "+r.URL.Path)
}
switch {
case r.Method == http.MethodPost && strings.HasSuffix(path, "/blobs/uploads/"):
repository := strings.TrimSuffix(path, "/blobs/uploads/")
w.Header().Set("Location", "/upload/"+repository+"?state=x")
w.WriteHeader(http.StatusAccepted)
case r.Method == http.MethodPut && strings.HasPrefix(r.URL.Path, "/upload/"):
repository := strings.TrimPrefix(r.URL.Path, "/upload/")
body, _ := io.ReadAll(r.Body)
digest := r.URL.Query().Get("digest")
if digest != digestOf(body) {
w.WriteHeader(http.StatusBadRequest)
return
}
m.blobs[repository+"@"+digest] = body
w.WriteHeader(http.StatusCreated)
case strings.Contains(path, "/blobs/"):
repository, digest, _ := strings.Cut(path, "/blobs/")
key := repository + "@" + digest
body, ok := m.blobs[key]
if !ok {
w.WriteHeader(http.StatusNotFound)
return
}
switch r.Method {
case http.MethodHead:
w.Header().Set("Content-Length", strconv.Itoa(len(body)))
w.WriteHeader(http.StatusOK)
case http.MethodDelete:
delete(m.blobs, key)
w.WriteHeader(http.StatusAccepted)
default:
w.WriteHeader(http.StatusMethodNotAllowed)
}
case strings.Contains(path, "/manifests/"):
repository, digest, _ := strings.Cut(path, "/manifests/")
key := repository + "@" + digest
switch r.Method {
case http.MethodHead:
if _, ok := m.manifests[key]; !ok || !strings.Contains(r.Header.Get("Accept"), mediaManifest) {
w.WriteHeader(http.StatusNotFound)
return
}
w.WriteHeader(http.StatusOK)
case http.MethodPut:
body, _ := io.ReadAll(r.Body)
if digest != digestOf(body) {
w.WriteHeader(http.StatusBadRequest)
return
}
var named holderManifest
if err := json.Unmarshal(body, &named); err != nil {
w.WriteHeader(http.StatusBadRequest)
return
}
for _, d := range append([]descriptor{named.Config}, named.Layers...) {
if _, ok := m.blobs[repository+"@"+d.Digest]; !ok {
w.WriteHeader(http.StatusBadRequest)
fmt.Fprintf(w, "MANIFEST_BLOB_UNKNOWN %s", d.Digest)
return
}
}
m.manifests[key] = body
w.WriteHeader(http.StatusCreated)
case http.MethodDelete:
if _, ok := m.manifests[key]; !ok {
w.WriteHeader(http.StatusNotFound)
return
}
delete(m.manifests, key)
w.WriteHeader(http.StatusAccepted)
}
default:
w.WriteHeader(http.StatusNotFound)
}
}))
t.Cleanup(server.Close)
return Store{Address: strings.TrimPrefix(server.URL, "http://")}
}
// bare puts an archive in the store the way the builder did before holders: a blob, nothing more.
func (m *memRegistry) bare(repository string, body []byte) string {
m.mu.Lock()
defer m.mu.Unlock()
digest := digestOf(body)
m.blobs[repository+"@"+digest] = body
return catalogue.ArtifactStoreScheme + repository + "/blobs/" + digest
}
func TestTheHolderIsComposedFromTheDigestAndSizeAlone(t *testing.T) {
// The sweep must arrive at the very manifest the builder wrote, with nothing recorded between
// them. Same inputs, same bytes — and a different size is a different holder, so a holder can
// never be mistaken for one of a different blob.
digest := "sha256:" + strings.Repeat("a", 64)
one, first := Holder(digest, 42)
two, second := Holder(digest, 42)
if !bytes.Equal(one, two) || first != second {
t.Fatalf("the same archive composed two holders:\n%s\n%s", one, two)
}
if _, other := Holder(digest, 43); other == first {
t.Fatal("a different size composed the same holder")
}
want := `{"schemaVersion":2,"mediaType":"application/vnd.oci.image.manifest.v1+json",` +
`"config":{"mediaType":"application/vnd.oci.empty.v1+json",` +
`"digest":"sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a","size":2},` +
`"layers":[{"mediaType":"application/vnd.oci.image.layer.v1.tar+gzip","digest":"` + digest + `","size":42}]}`
if string(one) != want {
t.Fatalf("the holder is\n%s\nwant\n%s", one, want)
}
if digestOf(emptyConfig) != emptyDigest {
t.Fatalf("the empty config's digest is %s, not %s", digestOf(emptyConfig), emptyDigest)
}
}
func TestHoldBackfillsABareArchiveAndIsIdempotent(t *testing.T) {
// The archives published before this have no holder. Hold, run over every kept archive on
// every sweep, writes one the first time and nothing after.
m := &memRegistry{}
store := m.serve(t)
ctx := context.Background()
body := []byte("a theme")
reference := m.bare("shell/config", body)
if held, err := store.Held(ctx, reference); err != nil || held {
t.Fatalf("a bare blob reads as held=%v (%v)", held, err)
}
wrote, err := store.Hold(ctx, reference)
if err != nil {
t.Fatal(err)
}
if !wrote {
t.Fatal("holding a bare archive wrote nothing")
}
_, holder := Holder(digestOf(body), int64(len(body)))
if _, ok := m.manifests["shell/config@"+holder]; !ok {
t.Fatalf("the store holds manifests %v; want %s", m.manifests, holder)
}
if held, err := store.Held(ctx, reference); err != nil || !held {
t.Fatalf("after holding, held=%v (%v)", held, err)
}
writes := len(m.writes)
wrote, err = store.Hold(ctx, reference)
if err != nil {
t.Fatal(err)
}
if wrote || len(m.writes) != writes {
t.Fatalf("holding again wrote %v", m.writes[writes:])
}
}
func TestAnImageNeedsNoHolderAndAMissingArchiveIsGone(t *testing.T) {
m := &memRegistry{}
store := m.serve(t)
ctx := context.Background()
// An image is its own manifest: nothing is asked.
wrote, err := store.Hold(ctx, catalogue.ArtifactStoreScheme+"web/app@sha256:"+strings.Repeat("b", 64))
if err != nil || wrote || len(m.writes) != 0 {
t.Fatalf("holding an image wrote=%v err=%v writes=%v", wrote, err, m.writes)
}
// An archive the store does not have is a fact to report, not something to invent a holder for.
_, err = store.Hold(ctx, catalogue.ArtifactStoreScheme+"web/config/blobs/sha256:"+strings.Repeat("c", 64))
if !errors.Is(err, Gone) {
t.Fatalf("holding a missing archive answered %v, want Gone", err)
}
// And a reference that is not the mesh's is refused as such.
if _, err := store.Hold(ctx, "docker.io/library/registry@sha256:abc"); !errors.Is(err, ErrNotOurs) {
t.Fatalf("holding a vendor's image answered %v, want ErrNotOurs", err)
}
}
func TestLettingGoOfAnArchiveDeletesItsHolderFirst(t *testing.T) {
// A holder left behind would keep the bytes through every collection while the record said
// collected; the link deleted first and the holder failing after would be that exactly.
m := &memRegistry{}
store := m.serve(t)
ctx := context.Background()
body := []byte("an old theme")
reference := m.bare("shell/config", body)
if _, err := store.Hold(ctx, reference); err != nil {
t.Fatal(err)
}
m.writes = nil
if err := store.LetGo(ctx, reference); err != nil {
t.Fatal(err)
}
_, holder := Holder(digestOf(body), int64(len(body)))
want := []string{
"DELETE /v2/shell/config/manifests/" + holder,
"DELETE /v2/shell/config/blobs/" + digestOf(body),
}
if strings.Join(m.writes, "\n") != strings.Join(want, "\n") {
t.Fatalf("the store was asked\n%s\nwant\n%s", strings.Join(m.writes, "\n"), strings.Join(want, "\n"))
}
if len(m.manifests) != 0 {
t.Fatalf("a holder survived: %v", m.manifests)
}
// Asked again, the archive is already gone, which is the outcome wanted.
if err := store.LetGo(ctx, reference); !errors.Is(err, Gone) {
t.Fatalf("letting go twice answered %v, want Gone", err)
}
}
func TestLettingGoOfAnUnheldArchiveStillDeletesIt(t *testing.T) {
// An archive published before holders and let go of before any sweep held it: the holder's
// delete answers 404, which is the outcome wanted, and the blob still goes.
m := &memRegistry{}
store := m.serve(t)
reference := m.bare("shell/config", []byte("never held"))
if err := store.LetGo(context.Background(), reference); err != nil {
t.Fatal(err)
}
if len(m.blobs) != 0 {
t.Fatalf("the blob survived: %v", m.blobs)
}
}
@@ -1,176 +0,0 @@
package registry
import (
"context"
"errors"
"fmt"
"net/http"
"net/http/httptest"
"slices"
"strings"
"sync"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
// An index is let go of with the platform manifests it names, except those a kept index of the same
// repository names too (novox/hq ADR 0257). Each platform manifest is a manifest of the repository in
// its own right, and the store's collector keeps every manifest a repository holds — so an index let go
// of alone frees nothing of the images it names.
func digestN(n int) string { return fmt.Sprintf("sha256:%064x", n) }
// indexStore holds indexes and images in one repository, answers GETs with their documents, and records
// every delete.
type indexStore struct {
mu sync.Mutex
indexes map[string][]string // digest → the platform manifests it names
images map[string]bool
deleted []string
}
func (s *indexStore) serve(t *testing.T) Store {
t.Helper()
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
s.mu.Lock()
defer s.mu.Unlock()
digest := r.URL.Path[strings.LastIndex(r.URL.Path, "/")+1:]
switch r.Method {
case http.MethodGet:
if children, ok := s.indexes[digest]; ok {
var named []string
for _, c := range children {
named = append(named, `{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"`+c+`","size":1}`)
}
w.Header().Set("Content-Type", "application/vnd.oci.image.index.v1+json")
_, _ = w.Write([]byte(`{"schemaVersion":2,"mediaType":"application/vnd.oci.image.index.v1+json","manifests":[` +
strings.Join(named, ",") + `]}`))
return
}
if s.images[digest] {
w.Header().Set("Content-Type", "application/vnd.oci.image.manifest.v1+json")
_, _ = w.Write([]byte(`{"schemaVersion":2,"mediaType":"application/vnd.oci.image.manifest.v1+json","layers":[]}`))
return
}
w.WriteHeader(http.StatusNotFound)
case http.MethodHead:
if _, ok := s.indexes[digest]; ok || s.images[digest] {
w.WriteHeader(http.StatusOK)
return
}
w.WriteHeader(http.StatusNotFound)
case http.MethodDelete:
if _, ok := s.indexes[digest]; !ok && !s.images[digest] {
w.WriteHeader(http.StatusNotFound)
return
}
s.deleted = append(s.deleted, digest)
delete(s.indexes, digest)
delete(s.images, digest)
w.WriteHeader(http.StatusAccepted)
default:
w.WriteHeader(http.StatusBadRequest)
}
}))
t.Cleanup(server.Close)
return Store{Address: strings.TrimPrefix(server.URL, "http://")}
}
func TestAnIndexGoesWithItsPlatformsAndAKeptIndexKeepsItsOwn(t *testing.T) {
// Two indexes of one image's repository: the old one names platforms 11 and 12, the kept one 12 and 13.
s := &indexStore{
indexes: map[string][]string{digestN(1): {digestN(11), digestN(12)}, digestN(2): {digestN(12), digestN(13)}},
images: map[string]bool{digestN(11): true, digestN(12): true, digestN(13): true},
}
store := s.serve(t)
repository := "upstream/docker.io/library/golang"
old := catalogue.ArtifactStoreScheme + repository + "@" + digestN(1)
kept := catalogue.ArtifactStoreScheme + repository + "@" + digestN(2)
spare, err := store.SpareKeptIndexes(context.Background(), []string{kept}, []string{old})
if err != nil {
t.Fatal(err)
}
store.Spare = spare
if err := store.LetGo(context.Background(), old); err != nil {
t.Fatal(err)
}
// Its own platform first, the index last; the platform the kept index names is left in place.
if !slices.Equal(s.deleted, []string{digestN(11), digestN(1)}) {
t.Fatalf("deleted %v; want the old index's own platform, then the index", s.deleted)
}
if !s.images[digestN(12)] || !s.images[digestN(13)] {
t.Fatal("a platform a kept index names was let go of")
}
}
func TestWithoutSparingAnIndexGoesAloneAndAnImageNamesNoPlatforms(t *testing.T) {
s := &indexStore{indexes: map[string][]string{digestN(1): {digestN(11)}}, images: map[string]bool{digestN(11): true, digestN(5): true}}
store := s.serve(t)
// No Spare: as before ADR 0257, the index alone.
if err := store.LetGo(context.Background(), catalogue.ArtifactStoreScheme+"web/base@"+digestN(1)); err != nil {
t.Fatal(err)
}
if !slices.Equal(s.deleted, []string{digestN(1)}) {
t.Fatalf("without a Spare, deleted %v", s.deleted)
}
// An image is one delete, with a Spare or without.
s.deleted = nil
spare, err := store.SpareKeptIndexes(context.Background(), nil, []string{catalogue.ArtifactStoreScheme + "web/app@" + digestN(5)})
if err != nil {
t.Fatal(err)
}
store.Spare = spare
if err := store.LetGo(context.Background(), catalogue.ArtifactStoreScheme+"web/app@"+digestN(5)); err != nil {
t.Fatal(err)
}
if !slices.Equal(s.deleted, []string{digestN(5)}) {
t.Fatalf("an image was let go of as %v", s.deleted)
}
// An index the store no longer holds is Gone, and nothing is deleted on its account.
s.deleted = nil
err = store.LetGo(context.Background(), catalogue.ArtifactStoreScheme+"web/base@"+digestN(9))
if !errors.Is(err, Gone) || len(s.deleted) != 0 {
t.Fatalf("an index the store does not hold: %v, deleted %v", err, s.deleted)
}
}
func TestASpareThatCannotBeReadKeepsTheIndex(t *testing.T) {
s := &indexStore{indexes: map[string][]string{digestN(1): {digestN(11)}}, images: map[string]bool{digestN(11): true}}
store := s.serve(t)
store.Spare = func(context.Context, string) (map[string]bool, error) { return nil, fmt.Errorf("the store went away") }
err := store.LetGo(context.Background(), catalogue.ArtifactStoreScheme+"web/base@"+digestN(1))
if err == nil || !strings.Contains(err.Error(), "was kept") {
t.Fatalf("a spare that could not be read: %v", err)
}
if len(s.deleted) != 0 {
t.Fatalf("deleted %v without knowing what a kept index names", s.deleted)
}
}
// The spare list is read whole before the sweep: a kept index that cannot be read is an error before
// anything is deleted, and a repository not read before keeps its index.
func TestASpareListIsReadWholeBeforeAnythingGoes(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusInternalServerError)
}))
t.Cleanup(server.Close)
store := Store{Address: strings.TrimPrefix(server.URL, "http://")}
kept := catalogue.ArtifactStoreScheme + "web/base@" + digestN(2)
old := catalogue.ArtifactStoreScheme + "web/base@" + digestN(1)
if _, err := store.SpareKeptIndexes(context.Background(), []string{kept}, []string{old}); err == nil {
t.Fatal("a kept index the store would not answer for was taken as naming nothing")
}
s := &indexStore{indexes: map[string][]string{digestN(1): {digestN(11)}}, images: map[string]bool{digestN(11): true}}
good := s.serve(t)
spare, err := good.SpareKeptIndexes(context.Background(), nil, []string{catalogue.ArtifactStoreScheme + "other/repo@" + digestN(3)})
if err != nil {
t.Fatal(err)
}
good.Spare = spare
if err := good.LetGo(context.Background(), old); err == nil || len(s.deleted) != 0 {
t.Fatalf("an index in a repository not read before the sweep was let go of: %v, deleted %v", err, s.deleted)
}
}
@@ -1,325 +0,0 @@
// Package registry speaks to the mesh's artifact store over its own door.
//
// Only what the mesh needs that nothing else does: letting go of something it put there
// (novox/hq ADR 0189, issue 108), and holding every archive it keeps by a manifest so the store's
// own collector does not take it (novox/hq issue 253). Pushing is the builder's, through the container runtime; reading
// is every machine's, through its runtime. This is the one operation that belongs to the thing
// holding the records, because it is the only one that is a decision rather than a transfer.
//
// Copied from mesh-controller internal/artifacts at e04846e9 (novox/hq ADR 0571 clause 2): the seat mesh-artifact-sweep lets
// go of what it no longer keeps exactly as the controller's sweep did; the controller's copy goes with its lease.
package registry
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"strings"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
// Store is the artifact store at an address, as this machine reaches it.
type Store struct {
// Address is `host:port` — the store as the caller reaches it now, composed and never
// recorded (novox/hq 04-ISSUES/102).
Address string
// HTTP is the client used; nil is a client with a modest timeout.
HTTP *http.Client
// Spare says, for one repository, which manifests a kept index there names: the platform
// manifests an index being let go of must leave in place (novox/hq ADR 0257). Nil spares every
// platform manifest — an index is then let go of alone, as it was before.
Spare func(ctx context.Context, repository string) (map[string]bool, error)
}
// Gone is the answer when the store does not hold it: the outcome wanted, already true.
var Gone = errors.New("the store does not hold it")
// ErrNotOurs is a reference this sweep will not address: not the mesh's own, or naming nothing
// the store holds by digest.
//
// **A fact about the record, not about the store** (novox/hq issue 226). The two deserve opposite
// responses — skip one and go on, abandon the sweep for the other — and collapsing them into "an
// error" is how a cautious loop became one that did nothing while reporting the right number.
var ErrNotOurs = errors.New("not a reference into the mesh's artifact store")
// LetGo asks the store to drop one artifact the mesh recorded making.
//
// Takes a reference as the mesh records it — `artifact-store://<module>/<artifact>@sha256:…` for
// an image, `…/blobs/sha256:…` for an archive — because that is the identity every record uses,
// and composes the address here at the moment of use.
//
// An archive is let go of in two deletes, its holder manifest and then the blob's link (novox/hq
// issue 253); an image in one.
//
// Returns Gone when the store answers that it does not have it. That is not a failure: the sweep
// wants the artifact absent, and it is. It is distinguished from success only so a caller can say
// which of the two happened.
func (s Store) LetGo(ctx context.Context, reference string) error {
// **Strict, and deliberately** (novox/hq issue 226). Only a reference the mesh keeps in its
// own vocabulary is addressed here. `Recorded` would read `docker.io/library/registry@sha256:…`
// as the mesh's too — it cannot tell one registry host from another — so normalising belongs
// where the provenance is known, which is the sweep reading its own build records, not here
// where the only job is to refuse anything that is not plainly ours.
path, kept := catalogue.InArtifactStore(reference)
if !kept {
// Nothing the mesh put in its own store. Refused rather than attempted: composing a
// delete for a reference of unknown shape is how a sweep reaches something that is not
// the mesh's. Distinguished from a store that refuses, so a sweep skips this and goes on.
return fmt.Errorf("%w: %s", ErrNotOurs, reference)
}
if s.Address == "" {
return fmt.Errorf("this mesh has no artifact store on its network to ask about %s", reference)
}
repository, kind, digest, err := split(path)
if err != nil {
return err
}
if kind == "blobs" {
// **An archive's holder goes before the archive** (novox/hq issue 253): a manifest left
// naming the blob would keep its bytes through every collection while the record said
// collected. Gone here means the store has no such blob, so there is nothing to let go.
if err := s.letGoOfHolder(ctx, repository, digest); err != nil {
return err
}
}
if kind == "manifests" && s.Spare != nil {
// **An index's platform manifests go before the index** (novox/hq ADR 0257). Each was put
// under its own digest, so each is a manifest of the repository in its own right, and the
// store's collector keeps every manifest a repository holds: an index let go of alone frees
// no byte of the images it names. Before, not after, so a sweep cut short leaves the index
// to be asked about again, and with it the list of what is still to go.
if err := s.letGoOfPlatforms(ctx, repository, digest, reference); err != nil {
return err
}
}
return s.remove(ctx, s.url(repository, kind, digest), reference)
}
// letGoOfPlatforms deletes the manifests an index names, except those a kept index of the same
// repository also names. A manifest that is not an index names none.
func (s Store) letGoOfPlatforms(ctx context.Context, repository, digest, reference string) error {
children, err := s.Platforms(ctx, repository, digest)
if err != nil || len(children) == 0 {
// Gone: there is no index to read, and the delete that follows says so.
if errors.Is(err, Gone) {
return nil
}
return err
}
spared, err := s.Spare(ctx, repository)
if err != nil {
return fmt.Errorf("cannot say which platform manifests of %s a kept index names, so %s was kept: %w",
repository, reference, err)
}
for _, child := range children {
if spared[child] {
continue
}
if err := s.remove(ctx, s.url(repository, "manifests", child), reference+" (its "+child+")"); err != nil && !errors.Is(err, Gone) {
return err
}
}
return nil
}
// anyManifest is every manifest the store may hold, indexes included, so it answers with the
// document as it is.
var anyManifest = []string{
"application/vnd.oci.image.index.v1+json",
"application/vnd.docker.distribution.manifest.list.v2+json",
"application/vnd.oci.image.manifest.v1+json",
"application/vnd.docker.distribution.manifest.v2+json",
}
// Platforms is the manifests an index names, by digest; none for a manifest that is not an index.
// Gone when the store does not hold it.
func (s Store) Platforms(ctx context.Context, repository, digest string) ([]string, error) {
request, err := http.NewRequestWithContext(ctx, http.MethodGet, s.url(repository, "manifests", digest), nil)
if err != nil {
return nil, err
}
request.Header.Set("Accept", strings.Join(anyManifest, ", "))
response, err := s.client().Do(request)
if err != nil {
return nil, err
}
defer response.Body.Close()
switch response.StatusCode {
case http.StatusOK:
case http.StatusNotFound:
return nil, Gone
default:
return nil, fmt.Errorf("the artifact store answered %s when asked for %s@%s", response.Status, repository, digest)
}
var document struct {
Manifests []struct {
Digest string `json:"digest"`
} `json:"manifests"`
}
if err := json.NewDecoder(io.LimitReader(response.Body, 4<<20)).Decode(&document); err != nil {
return nil, fmt.Errorf("%s@%s is not a manifest: %w", repository, digest, err)
}
var out []string
for _, m := range document.Manifests {
if strings.HasPrefix(m.Digest, "sha256:") {
out = append(out, m.Digest)
}
}
return out, nil
}
// SpareKeptIndexes is a Spare for a sweep, read whole before it starts: for every repository an
// image reference of `touching` is in, each kept manifest there and every platform manifest a kept
// index there names. Any read that fails is an error, and the sweep lets nothing go. A repository not
// read here is answered with an error, so an index in it is kept. A kept index the store no longer
// holds names nothing.
func (s Store) SpareKeptIndexes(ctx context.Context, kept, touching []string) (func(context.Context, string) (map[string]bool, error), error) {
imageIn := func(reference string) (repository, digest string, ok bool) {
path, ours := catalogue.InArtifactStore(reference)
if !ours {
return "", "", false
}
repository, kind, digest, err := split(path)
if err != nil || kind != "manifests" {
return "", "", false
}
return repository, digest, true
}
read := map[string]map[string]bool{}
for _, reference := range touching {
if repository, _, ok := imageIn(reference); ok {
read[repository] = map[string]bool{}
}
}
for _, reference := range kept {
repository, digest, ok := imageIn(reference)
spared, touched := read[repository]
if !ok || !touched {
continue
}
spared[digest] = true
children, err := s.Platforms(ctx, repository, digest)
if errors.Is(err, Gone) {
continue
}
if err != nil {
return nil, fmt.Errorf("reading the kept %s@%s: %w", repository, digest, err)
}
for _, c := range children {
spared[c] = true
}
}
return func(_ context.Context, repository string) (map[string]bool, error) {
spared, done := read[repository]
if !done {
return nil, fmt.Errorf("%s was not read before the sweep began", repository)
}
return spared, nil
}, nil
}
// remove asks the store to delete what is at url. Gone when it has no such thing.
func (s Store) remove(ctx context.Context, url, what string) error {
request, err := http.NewRequestWithContext(ctx, http.MethodDelete, url, nil)
if err != nil {
return err
}
response, err := s.client().Do(request)
if err != nil {
return err
}
defer response.Body.Close()
switch response.StatusCode {
case http.StatusAccepted, http.StatusOK, http.StatusNoContent:
return nil
case http.StatusNotFound:
return Gone
case http.StatusMethodNotAllowed:
// The registry was started without deletion enabled. Said plainly, because the remedy is
// a setting on the store's module and not anything about this artifact.
return fmt.Errorf(
"the artifact store refuses deletion: its server was started without it enabled "+
"(REGISTRY_STORAGE_DELETE_ENABLED), so nothing can be collected until the store "+
"module is applied again (novox/hq ADR 0189). Asking about %s", what)
default:
return fmt.Errorf("the artifact store answered %s for %s", response.Status, what)
}
}
// split reads a recorded path into the repository, which endpoint names the thing, and the digest.
//
// Two shapes, which are the two the mesh records: `<repository>@sha256:<hex>` is a manifest, and
// `<repository>/blobs/sha256:<hex>` is a blob.
func split(path string) (repository, kind, digest string, err error) {
if before, after, ok := strings.Cut(path, "@sha256:"); ok {
return before, "manifests", "sha256:" + after, nil
}
if before, after, ok := strings.Cut(path, "/blobs/sha256:"); ok {
return before, "blobs", "sha256:" + after, nil
}
return "", "", "", fmt.Errorf("%w: %q names nothing the store holds by digest", ErrNotOurs, path)
}
// IsIndex is whether a recorded reference is an index the store holds: one that names manifests. A
// blob, or a manifest that names none, is not. Gone when the store does not hold it.
func (s Store) IsIndex(ctx context.Context, reference string) (bool, error) {
path, ours := catalogue.InArtifactStore(reference)
if !ours {
return false, fmt.Errorf("%w: %s", ErrNotOurs, reference)
}
repository, kind, digest, err := split(path)
if err != nil || kind != "manifests" {
return false, err
}
children, err := s.Platforms(ctx, repository, digest)
return len(children) > 0, err
}
// HoldsManifest is whether the store holds the manifest a recorded image reference names.
func (s Store) HoldsManifest(ctx context.Context, reference string) (bool, error) {
path, ours := catalogue.InArtifactStore(reference)
if !ours {
return false, fmt.Errorf("%w: %s", ErrNotOurs, reference)
}
repository, kind, digest, err := split(path)
if err != nil {
return false, err
}
if kind != "manifests" {
return false, fmt.Errorf("%w: %s names a blob, not a manifest", ErrNotOurs, reference)
}
return s.has(ctx, s.url(repository, kind, digest), anyManifest...)
}
// Holds is whether the store holds what a recorded reference names: the blob of an archive, the manifest of an image.
// Gone when it does not; asks and changes nothing (ADR 0526 clause 3: a registration by hand is refused at bytes that
// are not there).
func (s Store) Holds(ctx context.Context, reference string) error {
path, ours := catalogue.InArtifactStore(reference)
if !ours {
return fmt.Errorf("%w: %s", ErrNotOurs, reference)
}
repository, kind, digest, err := split(path)
if err != nil {
return err
}
if s.Address == "" {
return fmt.Errorf("this mesh has no artifact store on its network to ask about %s", reference)
}
if kind == "blobs" {
_, err := s.blobSize(ctx, repository, digest)
return err
}
held, err := s.has(ctx, s.url(repository, kind, digest), anyManifest...)
if err != nil {
return err
}
if !held {
return Gone
}
return nil
}
@@ -1,134 +0,0 @@
package registry
import (
"context"
"errors"
"net/http"
"net/http/httptest"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
// Asking the store to let go of what the mesh no longer keeps (novox/hq ADR 0189, issue 108).
//
// A fake store records what it was asked to delete, so what is asserted is the mesh's decision
// and the shape of the request — not the registry's behaviour, which is the registry's to test.
func fakeStore(t *testing.T, answer int) (Store, *[]string) {
t.Helper()
var asked []string
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method == http.MethodHead && strings.Contains(r.URL.Path, "/blobs/") {
// An archive's size, asked so its holder can be named (novox/hq issue 253). A store
// that does not have the thing does not have its blob either.
if answer == http.StatusNotFound {
w.WriteHeader(http.StatusNotFound)
return
}
w.Header().Set("Content-Length", "7")
w.WriteHeader(http.StatusOK)
return
}
if r.Method != http.MethodDelete {
t.Errorf("the store was asked %s %s; collecting is a delete", r.Method, r.URL.Path)
}
asked = append(asked, r.URL.Path)
w.WriteHeader(answer)
}))
t.Cleanup(server.Close)
return Store{Address: strings.TrimPrefix(server.URL, "http://")}, &asked
}
func TestAnImageAndAnArchiveAreAskedForAtTheirOwnEndpoints(t *testing.T) {
// The two shapes the mesh records: a manifest by digest, and a blob by digest. They are
// different endpoints, and asking at the wrong one answers 404 — which this would then
// record as collected, leaving the bytes on disk for ever while the record says otherwise.
store, asked := fakeStore(t, http.StatusAccepted)
ctx := context.Background()
image := catalogue.ArtifactStoreScheme + "web/app@sha256:abc123"
archive := catalogue.ArtifactStoreScheme + "web/config/blobs/sha256:def456"
if err := store.LetGo(ctx, image); err != nil {
t.Fatal(err)
}
if err := store.LetGo(ctx, archive); err != nil {
t.Fatal(err)
}
// The archive's holder goes first, then the archive (novox/hq issue 253).
_, holder := Holder("sha256:def456", 7)
want := []string{
"/v2/web/app/manifests/sha256:abc123",
"/v2/web/config/manifests/" + holder,
"/v2/web/config/blobs/sha256:def456",
}
if strings.Join(*asked, " ") != strings.Join(want, " ") {
t.Fatalf("the store was asked %v; want %v", *asked, want)
}
}
func TestAStoreThatDoesNotHaveItAnswersGone(t *testing.T) {
// The outcome wanted, already true. Told apart from success only so the sweep can say which
// happened; both are recorded, because retrying for ever is the thing to avoid.
store, _ := fakeStore(t, http.StatusNotFound)
err := store.LetGo(context.Background(), catalogue.ArtifactStoreScheme+"web/app@sha256:abc123")
if !errors.Is(err, Gone) {
t.Fatalf("a store that does not hold it answered %v, want Gone", err)
}
}
func TestAStoreWithDeletionOffSaysSoAndNamesTheRemedy(t *testing.T) {
// The registry answers 405 when it was started without deletion enabled. The remedy is a
// setting on the store's module, and saying "405" would send somebody to the wrong place.
store, _ := fakeStore(t, http.StatusMethodNotAllowed)
err := store.LetGo(context.Background(), catalogue.ArtifactStoreScheme+"web/app@sha256:abc123")
if err == nil {
t.Fatal("a store that refuses deletion was read as success")
}
if !strings.Contains(err.Error(), "REGISTRY_STORAGE_DELETE_ENABLED") {
t.Fatalf("the refusal does not name the remedy: %v", err)
}
}
func TestAReferenceThatIsNotTheMeshsOwnIsNeverAsked(t *testing.T) {
// The whole safety of the sweep is that it names only what the mesh recorded putting there.
// A reference of another shape — a vendor's image, a package version — is refused rather
// than composed into a delete somewhere that is not the mesh's store.
store, asked := fakeStore(t, http.StatusAccepted)
for _, reference := range []string{
"docker.io/library/registry@sha256:abc123",
"registry@sha256:abc123",
"1.4.2",
} {
if err := store.LetGo(context.Background(), reference); err == nil {
t.Errorf("%s was asked about; it is not a reference into the mesh's store", reference)
}
}
if len(*asked) != 0 {
t.Fatalf("the store was asked about %v", *asked)
}
}
// A reference this sweep will not address says so as ErrNotOurs, which is a fact about the
// record and not about the store (novox/hq issue 226).
//
// The sweep skips one and abandons itself for the other, so they cannot be the same error. The
// first live run met a reference recorded with the store's old address, read the refusal as "the
// store refuses everything", and collected none of the 1681 it had found.
func TestAReferenceThisSweepWillNotAddressIsToldApartFromAStoreRefusing(t *testing.T) {
store, asked := fakeStore(t, http.StatusAccepted)
for _, reference := range []string{
"docker.io/library/registry@sha256:abc123",
"127.0.0.1:5100/mesh-tools/build@sha256:abc123",
"1.4.2",
} {
err := store.LetGo(context.Background(), reference)
if !errors.Is(err, ErrNotOurs) {
t.Errorf("%s answered %v; a sweep must be able to skip it and go on", reference, err)
}
}
if len(*asked) != 0 {
t.Fatalf("the store was asked about %v", *asked)
}
}
+11 -46
View File
@@ -1,52 +1,26 @@
{
"module": "artifact-store-tools",
"version": "1",
"claims": [
{
"name": "mesh-artifact-sweep",
"scope": "mesh",
"serves": [
"artifacts",
"held",
"collect",
"mirrors",
"images"
]
}
],
"requires": [
"artifact-store"
],
"invokes": [
"seat:mesh-hand-acts.hand-act",
"seat:mesh-controller.artifacts",
"seat:mesh-controller.collect",
"mesh-catalog.catalog_modules",
"mesh-catalog.catalog_module"
],
"reads": [
"mesh-delivery-assigner.wants",
"mesh-delivery-builder.builds",
"mesh-delivery-builder.build-log",
"mesh-delivery-indexer.manifests",
"node-engine.have",
"mesh-controller.lease"
],
"state": [
"findings"
],
"tools": [
"artifact_store_repositories",
"artifact_store_usage",
"artifact_store_references",
"artifact_store_collect"
],
"config": {
"older-store-names": {
"type": "string",
"description": "The names the artifact store had before, as host names or host:port separated by spaces, or none when it had no other. artifact_store_references reads every definition under these as well as under the store's current name, so an image a definition pins under one is said to be named by a definition (novox/hq issue 323).",
"pattern": "none|[A-Za-z0-9.:-]+( [A-Za-z0-9.:-]+)*",
"default": "none"
}
},
"config": {
"older-store-names": {
"type": "string",
"description": "The names the artifact store had before, as host names or host:port separated by spaces, or none when it had no other. artifact_store_references reads every definition under these as well as under the store's current name, so an image a definition pins under one is said to be named by a definition (novox/hq issue 323).",
"pattern": "none|[A-Za-z0-9.:-]+( [A-Za-z0-9.:-]+)*",
"default": "none"
}
},
"resources": [
{
"id": "state",
@@ -62,14 +36,6 @@
"mode": "0600",
"trusted": false,
"template": "files/store-names"
},
{
"id": "store-address",
"type": "file",
"path": "${dir:state}/store-address",
"mode": "0600",
"trusted": false,
"template": "files/store-address"
}
],
"build": {
@@ -86,8 +52,7 @@
],
"env": {
"MESH_ARTIFACT_STORE_CONTAINER": "mesh-registry",
"MESH_ARTIFACT_STORE_NAMES_FILE": "${dir:state}/store-names",
"MESH_ARTIFACT_STORE_ADDRESS_FILE": "${dir:state}/store-address"
"MESH_ARTIFACT_STORE_NAMES_FILE": "${dir:state}/store-names"
}
},
{
+360
View File
@@ -0,0 +1,360 @@
# betterbird
Betterbird, the mail and calendar client of g14 and shanks (a Thunderbird fork, with the operator's
Exchange calendar through the Owl add-on), as a module (novox/hq issue 345, issue 586, ADR 0208, ADR 0316,
ADR 0317, ADR 0329, to-be 56). It requires `x11-display`, so it is assigned only where a display server is held
on the same machine. It installs Betterbird from a pinned upstream release, ships Owl with it through
Betterbird's enterprise policy, and owns Betterbird's whole configuration as settings: the preferences, the
mail accounts without their credentials, the identities with their signatures, the CalDAV calendars, the
CardDAV address books, the filters and the tags.
It exists because a symptom had no tools: on a workstation, a calendar reminder appeared as a
transparent window with nothing in it (2026-10-09). Nothing in the mesh could say which window that
was, how the compositor treats it, or which preferences Betterbird ran with. The diagnosis is in hq
issue 345.
## Owns
| what | where | class |
|---|---|---|
| Betterbird, the upstream release (`archive`, pinned by URL and sha256) | `/opt/betterbird` | owned, whole |
| the enterprise policy (`install/distribution/policies.json`) | `/opt/betterbird/distribution` | owned, whole |
| Owl's xpi (`owl`, pinned by the vendor's sha256) | `/opt/betterbird/owl` | owned, whole |
| the module's defaults (`install/defaults-pref/mesh-defaults.js`: the preferences kept at Betterbird's own default, no value of the operator's) and the spelling dictionaries' default (`vendor-prefs.js`) | `/opt/betterbird/defaults/pref` | owned: its two files |
| the tools' program, for the start's preparation | `/usr/local/lib/mesh-betterbird` | owned, whole |
| the start | `/usr/local/bin/betterbird` | owned, whole |
| the launcher entry | `/usr/local/share/applications/eu.betterbird.Betterbird.desktop` | owned, whole |
| the module's directory, with the settings as the node-engine writes them (`prefs.json`, `accounts.json`, `configuration.json`, `tags.json`), the records of what was adopted or created, and the filter files it replaced (`kept/`) | `~/.config/betterbird/` | owned, whole |
| the profile's `user.js`, placed by the tools | the profile | the module's, or never touched |
| the lists in the profile's `prefs.js` (accounts, outgoing servers, an adopted account's identities, the calendars' order), and the preferences of what the module created and the settings no longer hold | the profile | written into: the module's keys only, only while Betterbird is closed |
| each declared account's `msgFilterRules.dat` (setting `filters`) | the account's directory in the profile | owned, whole, only while Betterbird is closed |
| the profile's own values of the preferences the module manages, in `prefs.js` | the profile | cleared only by `betterbird_config_apply` with apply, Betterbird quit |
| the profile: mail, address books, calendars, logins | `~/.thunderbird/` | found (ADR 0182) |
**The community package `betterbird-bin` is declared absent** (ADR 0180), before anything is placed in
`/opt/betterbird`: the node-engine removes it with `pacman -R`, then unpacks the release where the package
had it. Betterbird chooses its profile by its installation's path, so the same profile opens after the
switch. A package left in place (its removal failed) holds the path, and the release is refused there by
the node-engine rather than mixed in. The module creates no link: the start, a launcher entry naming the
icon in the installation, and the dictionaries as a default preference replace the package's links.
**Every new build waits for the operator** (`upgrade: record`): the first push to a node removes the
package and replaces what a running Betterbird has open, so the operator quits Betterbird, then pushes.
An upgrade is a change to the pins, reviewed and delivered like any other (ADR 0316 §5).
**The pins.** The release is `betterbird-153.4.0esr-bb10.en-US.linux-x86_64.tar.xz` (81,620,684 bytes) and
Owl is 1.5.6.1 (430,201 bytes), each pinned by sha256 in `module.json`. The build seat fetches each once,
refuses it when the digest differs, keeps it under its sha256, and packs it into the artifact store; a node
receives the mesh's archive by digest and never asks the vendor. **Owl is never in this repository**: its
licence forbids copying it to third parties, and its vendor serves only its newest version, at one
address. Keeping the pinned copy in the mesh's private artifact store for the operator's own machines was
the operator's call (2026-10-12, ADR 0316 §6).
**Owl's range.** Owl declares which Betterbird versions it supports (`strict_min_version` to
`strict_max_version`: 115 to 156 for 1.5.6.1). `owlSupports` in `manifest_test.go` holds the range read
from each pinned xpi, and the module's tests refuse a Betterbird pin outside it: moving Owl's pin means
reading the new range there (ADR 0316 §8).
**The policy** turns Betterbird's own updater off (`DisableAppUpdate`, which Betterbird's update check
honours) and installs Owl into every profile from `file:///opt/betterbird/owl/owl.xpi` as
`force_installed`, with `updates_disabled`. A profile that already runs another Owl keeps it: the policy
installs the pinned one where Owl is missing. A newer Owl in the profile (g14 runs 1.5.7) is said as a
fact by `betterbird_status`; Owl installed by policy cannot be removed in Betterbird, so the remedy is
moving the pin. An older one is a finding of `betterbird_check`, with the same remedy.
**Betterbird never asks to be the default client.** The policy locks `mail.shell.checkDefaultClient` off
(`Preferences`, `Status: locked`): Betterbird's own default (`all-thunderbird.js`) is to ask at every start,
with a window to make it the default for mail, news, feeds and calendar files, while that preference is true
(`messenger.js`). Betterbird 153 has no policy of its own for it (its `policies-schema.json`), and its
`Preferences` policy accepts the `mail.` prefix. A lock, not a default: the window's own "always check" box
writes the preference into the profile, which wins over a default; a lock wins over the profile and greys the
box out in the settings. `betterbird_check` says when Betterbird would ask (novox/hq issue 345). **Owl's licence stays the person's**: the mesh holds no licence, key or payment (ADR 0316 §9).
**No start of its own.** Betterbird is started by the operator from the launcher. The start
(`/usr/local/bin/betterbird`) first runs `betterbird-tools prepare`, which, with no Betterbird of the
account running, places the user.js and writes what Betterbird rewrites itself (the lists, the removals, the
filter files); then it
starts Betterbird either way. `betterbird_restart` starts it as the transient user unit
`mesh-betterbird`, so its output then goes to the journal.
## The preferences: locks, the setting prefs, and Betterbird's own defaults
Betterbird's preferences come in three layers (novox/hq issue 345, issue 586). **The operator's own choices
live in the module's configuration, never in the catalogue**: the module ships only its policy's locks and the
names it keeps at Betterbird's own default.
| layer | where | who changes it | holds |
|---|---|---|---|
| **locked** | `/opt/betterbird/distribution/policies.json` (`install/distribution/`) | only a reviewed catalogue change | what must never differ on a node; Betterbird's settings show it greyed out |
| **declared** | the setting `prefs`, mesh-wide with a node's layer over it, written to `~/.config/betterbird/prefs.json` and placed in the profile's `user.js` | the operator, through the settings layers (trusted: the terminal or the operator's warrant) | the operator's setup, and a node's exceptions, such as a font size for a larger screen |
| **Betterbird's default** | Betterbird's own defaults; `mesh-defaults.js` names those the module manages there | a catalogue change | the preferences kept at Betterbird's default |
A locked value wins over everything; a declared one (in `user.js`) wins over the profile's `prefs.js` at every
start; `prefs.js` wins over a default.
**The setting `prefs` is one entry per preference** (issue 586): `{"prefs": {"mail.uifontsize": 14, …}}`, the
value true, false, a whole number or a string. The node-engine merges the layers key by key, so a node's layer
overrides one preference, or takes one out with `null`. The operator's setup approved on 2026-10-12 (compact
density, 14-point text, the vertical view with multi-line cards and threads, no new-mail pop-up, sound or tray
icon, the calendar's day from 09:00 to 20:00, reminders missed while asleep shown, chat off, remote content
blocked, Do Not Track; 21 preferences) moved from the catalogue's defaults file into the mesh-wide layer of
`prefs`, set on the operator's Approve.
**The order that leaves no node on Betterbird's own defaults** (issue 586): merge this build (the `record`
upgrade policy holds it back from the nodes); propose the mesh-wide layer through the controller's `settings`
verb with `propose`, carrying the current `accounts` verbatim and `prefs` with the 21 values; the operator
approves; only then push the desktop nodes. Caveat: between that approval and a node's push, any other send to a
desktop node writes the map's text into the old build's user.js, and `betterbird_check` there says the user.js is
refused until the push.
**Locked** (the policy): Betterbird's updater off (`DisableAppUpdate`); telemetry off (`DisableTelemetry`);
add-ons change only through the mesh's pins (`ExtensionUpdate` false); and three preferences locked off:
`mail.shell.checkDefaultClient` (the ask to be the default client), `mail.inappnotifications.enabled` (the
vendor's in-app notices), and `mailnews.start_page.enabled` (the start page, which otherwise loads from the
vendor at every start).
**Kept at Betterbird's default** (`// keeps Betterbird's default: <name>` in `mesh-defaults.js`): the theme,
system colours, always-underlined links, manual spam marking, and the composer's attachment pane. A value the
profile saved for one of them is drift.
**Drift.** `betterbird_config` shows each managed preference (locked, declared, kept) through the layers, and
flags **drift**: a declared value the profile's `user.js` does not hold yet (remedy: `betterbird_user_js` with
apply, then `betterbird_restart`), or a value the profile saved where the module keeps Betterbird's default.
`betterbird_check` reports it.
- `betterbird_config_apply` with `apply` takes a kept preference's saved value out of `prefs.js`, so Betterbird's
default holds at the next start. It acts only while Betterbird is quit (no main process, and no `lock` in the
profile), and says what it cleared.
- **The theme is the exception.** Betterbird keeps the active theme in its add-on database (`extensions.json`),
and `extensions.activeThemeID` only mirrors it. Another theme is drift whose remedy is to enable the default
theme in the add-on manager. `betterbird_config_apply` never clears it, and `keep` refuses it.
- `betterbird_config` with `keep` proposes this node's layer of `prefs` holding the profile's values of the named
preferences beside what the layer holds, through the controller's settings verb, for the operator's Approve.
**One owner per value.** `prefs` refuses a preference whose name can hold a credential, an account's
(`mail.account.`, `mail.identity.`, `mail.server.`, `mail.smtpserver.` and the account lists, ADR 0316 §11), and
those the settings below own: `calendar.registry.`, `calendar.list.`, `ldap_2.servers.` and `mailnews.tags.`
(ADR 0329 §6).
## The user.js
Betterbird reads `user.js` in its profile when it starts, and a preference set there wins over the one it saved
in `prefs.js`. The profile's directory has a random name per machine (`<random>.default-default`), which a
manifest cannot name. So the tools compose the user.js from the settings — `prefs`, then the declared accounts,
then the declared identities, calendars, address books and tags under the keys each resolved to — and place it
in the profile: at the start from the launcher, `betterbird_user_js` with `apply`, or `betterbird_restart`.
**The profile is found** as the one the running Betterbird holds open; with none running, as the
installation's default in `profiles.ini` (`[Install…] Default`), else the profile marked `Default=1`.
**A user.js the module did not write is never overwritten.** The module's begins with the line
`// Betterbird's user.js (module betterbird`; any other is the operator's, and `apply` refuses with what to do.
## The accounts: declared without credentials (ADR 0316 §10–§16, ADR 0317)
The setting `accounts` reaches the machine as `~/.config/betterbird/accounts.json`, a trusted mergeable
file: a map of accounts by name, each with `address`, `from-name` and `label` (optional), `incoming` (`kind`
imap or pop3, `host`, `port`, `security` tls, starttls or none, `sign-in` password, encrypted-password,
oauth, kerberos, ntlm or tls-certificate, `user`) and `outgoing` (the same, without a kind; its sign-in may
be none), and, only in a node's own layer, `key`. **Never a password or a token**: the whole file is
refused when any key in it is shaped like one, and an account signs in at its first use, in Betterbird's
own window, where Betterbird keeps the result in its login store. The work account through Owl is never
declared: the person creates it in Owl's setup (ADR 0316 §13).
At every apply (the start from the launcher, `betterbird_user_js` with apply, `betterbird_restart`), each
declared account, in its place in the setting, is (ADR 0317 §1):
1. **kept by its recorded keys**, when the module recorded it on this node for this profile;
2. **its creation**, when it is under the module's own keys (`account_mesh_<name>`, `id_mesh_<name>`,
`server_mesh_<name>`, `smtp_mesh_<name>`);
3. **adopted by the key a node's layer names** (`key`), when that account is in the list, its default
identity has the address and it is not Owl's;
4. **adopted by its address**, when exactly one account's default identity has it: its keys (account,
default identity, incoming server, and the outgoing server that identity names) become the module's, and
the account lists are left as they are;
5. **created** under the module's own keys when no account has it: its keys are added at the end of
`mail.accountmanager.accounts` and `mail.smtpservers` in `prefs.js`, every other key kept in its place,
only while Betterbird is not running.
**Refused, for that account alone and with the remedy** (§7): two accounts with the address (remove the
duplicate in Betterbird, or name the `key`), Owl's account at the address, a named key that does not
qualify, an account of another kind than declared (never converted), an identity that names no outgoing
server, and an outgoing server that another identity, neither adopted nor the module's, also uses
(Betterbird lets identities share one, and the declared values would change that other account's sending;
ADR 0317 §2 leaves the identity's server and the lists as they are, so the module refuses rather than give
it a server of its own). A recorded adoption follows the identity's current outgoing server at every apply,
under the same rule. The other accounts go ahead; the start and `betterbird_check` say it.
`betterbird_user_js` answers an account's preferences by name with their values withheld.
The declared fields are written in the user.js under the account's keys, so they win at every start; only
the fields the setting names, never the account's directory, folders, filters, caches, mail or kept
sign-in, so an adopted account keeps its sign-in (§5). The apply names, by preference name only, each one
whose value it changed (§4). What was adopted or created is recorded in
`~/.config/betterbird/accounts-record.json`. **Leaving the setting** releases an adopted account (its
lines and its record go, the account stays the person's) and takes a created account's keys out of the
lists (its mail stays in the profile).
**The value is derived, not retyped** (§9): `betterbird_accounts` with `propose` builds `accounts` from the
profile (every account but Owl's, no credential) and proposes it in the mesh-wide layer as it stands, through
the controller's settings verb; the operator approves on a channel that proves who answers. Its answer shows
the value it proposed (ADR 0329 §10).
**A node with no profile yet.** The preparation needs the profile Betterbird makes at its first start: on a
node set up from nothing, the declared accounts are written at the second start from the launcher (or at
`betterbird_restart`).
## Identities, calendars, address books, filters and tags (ADR 0329)
Five more settings, mesh-wide with a node's layer over each (an entry set to `null` in a node's layer is not on
that node). `identities`, `calendars`, `address-books` and `filters` reach the machine as
`~/.config/betterbird/configuration.json`, trusted as `accounts` is; `tags` as `tags.json`, trusted too since issue 751 (the operator's decision), so a change to it is the operator's to approve.
**No value holds a password, a token or a cookie**: a key shaped like one is refused, and so is an address whose
user part carries a password (§2). A setting holds no line break, so a signature is a list of its lines.
| setting | an entry | matched in the profile by |
|---|---|---|
| `identities` | by a name: `account` (its name in `accounts`), and for another identity than the account's default `address` and `name`; `reply-to`, `organization`, `signature` (lines), `signature-html`, `signature-above-quote`; `key` in a node's layer | the account's default (no `address`), or an identity of that account with the address |
| `calendars` | by a name: `name`, `url` (CalDAV), `color`, `read-only`, `reminders`, `user`, `invitations` (an identities entry); `key` in a node's layer | its address |
| `address-books` | by a name: `name`, `url` (CardDAV), `user`; `key` in a node's layer | its address |
| `tags` | by its key: `name`, `color`, `order` | its key |
| `filters` | by an account's name: the ordered list, each with `name`, `enabled`, `when` (new-mail, manual, after-junk, sending, archiving, periodic), `match` (all, any, every-message), `conditions` (`field`, `op`, `value`, in Betterbird's words) and `actions` (`action` in Betterbird's words; `folder` as a path within the account, or `value`) | owned whole per account |
**Adopted as accounts are** (§7): an entry recorded on this node is kept by its key; one under the module's own
key (`ident_mesh_<name>`, `mesh_<name>` for a calendar or an address book) is its creation; a `key` in the
node's layer is adopted when the profile holds it and it matches; else one match is adopted, none is created
under the module's key, and several are refused with the remedy. The declared fields go into the user.js under
that key. An entry leaving the setting releases what was adopted (its lines go) and removes the preferences of
what was created, never a calendar's events, an address book's contacts or a cache. An entry no setting declares
stays the person's. Owl's accounts and calendars, an ICS calendar (its address is often the secret), local
calendars and address books are never declared (§13).
**Filters are owned whole per declared account** (§8): the module writes that account's `msgFilterRules.dat`
from the list, each folder's path turned into its address on this node, so a filter added in Betterbird is gone
at the next apply (drift shows it first). Before its first write it keeps the replaced file in
`~/.config/betterbird/kept/<profile>/`; when the account leaves `filters`, the module stops writing and the file
stays the person's; `betterbird_filters` says where the kept copy is. A filter a setting cannot hold as it is (a
folder of another account, an add-on's action, a news filter) is named and not derived.
**Written only while Betterbird is closed** (§9): the lists in `prefs.js` and the filter files, because
Betterbird rewrites them as it runs and quits; the user.js is placed as before. `betterbird_check` counts per
kind the changes that wait.
**Derived, then proposed one layer at a time** (§10): `betterbird_derive` asks every desktop node holding
Betterbird for what its profile holds. An entry on every such node goes into the mesh-wide layer with the value of
the node the derive was called on, and the answer lists each field another node holds differently (which then
shows as drift there); one on some nodes only goes into those nodes' own layers. An account whose filter file
holds no filter is not put in `filters`. With `propose`, the next layer (the mesh-wide one, then each node's) is
proposed with the layer as it stands beside the derived kinds; never while a proposal of the module's waits, and
never while a desktop node does not answer. `prefs` on the call carries values of the setting `prefs` into the
mesh-wide layer. A `key` is never proposed for the mesh-wide layer.
## Tools
They are served by the node's runtime as the operator account (ADR 0175). **They show the operator's own
configuration with its values, and never a credential** (ADR 0329 §1, §2): every answer passes the module's
credential check, which withholds a field whose name says password, token, secret or cookie and the password
part of any address. No answer carries mail, a calendar entry or a contact.
| tool | does |
|---|---|
| `betterbird_status` (r) | whether it runs (pid, since, scope, children); what is installed and from where, its version beside the newest published build; the policy; Owl's version against the pin and whether it is active; where its output goes; the profile and how it was found; the user.js (declared, placed, in effect, with every declared entry as it resolved); the declared accounts by count; the add-ons; the preferences that decide how its windows and notifications look |
| `betterbird_prefs` (r) | named preferences or prefixes: value and source, and the default with the file that gives it. Withheld without its value: a name that can hold a credential, one that names an account, server, calendar or person, and every `extensions.*` but the theme's id. Values masked (the home as `~`, addresses and hosts) |
| `betterbird_config` (r; a with `keep`) | each managed preference: its lock, the declared value, Betterbird's default where the module keeps that, the profile's saved value, the value in effect, and drift; with `keep`, this node's layer of `prefs` proposed with the profile's values |
| `betterbird_config_apply` (r; a with `apply`) | a kept preference's drifting values listed; with `apply`, while Betterbird is quit, taken out of `prefs.js` |
| `betterbird_user_js` (r; a with `apply`) | the user.js composed from the settings against the profile's; `apply` places it, writes what Betterbird rewrites while it is closed, and counts what waits |
| `betterbird_log` (r) | the last lines of the journal of `mesh-betterbird` (or of the reminder test), addresses and credentials masked |
| `betterbird_restart` (a) | ends Betterbird (asked to quit, forced after 8 s), places the user.js and writes what waited, starts it as `mesh-betterbird`. **Refused while a composer window is open**, unless `discard_unsent` is true, which only the operator's yes justifies |
| `betterbird_check` (r) | installed from the mesh's release; the policy and the module's defaults placed; no managed preference drifting; no ask to be the default mail client; Owl at its pin and active; one running; profile found; the user.js valid, placed, in effect; the declared accounts in the list, none refused, each signed in once; per kind, the declared entries drifting or refused and the changes waiting for Betterbird to close; its output read |
| `betterbird_accounts` (r; a with `propose`) | the profile's accounts with their values and the declared account each is, Owl's by name and address and marked; with `propose`, `accounts` built from them and proposed in the mesh-wide layer as it stands |
| `betterbird_identities`, `betterbird_calendars`, `betterbird_address_books`, `betterbird_filters`, `betterbird_tags` (r) | the profile's entries with their keys and values beside the declared ones and how each resolved (adopted, created or refused with why); an ICS calendar by its name only, Owl's marked; `betterbird_filters` also where the kept filter files are |
| `betterbird_derive` (r; a with `propose`) | the five kinds derived from the desktop nodes' profiles, per layer, with the fields another node holds differently and what was left out; with `propose`, the next layer proposed; `here` answers this node's profile, as another node asks it |
| `betterbird_drift` (r) | per kind and declared entry: holds, differs (each field with both values), lacks, or refused; and the profile's entries nothing declares |
| `betterbird_reminder_test` (a) | a reminder on demand without the operator's calendar (below) |
### A reminder on demand
**It opens windows on the operator's screen**, and its description says so first, so a caller tells the
operator before calling it. On 2026-10-12 at 12:05 a test Betterbird with an empty profile carrying the
operator's theme looked like the operator's configuration lost; so the test is **visibly a test**: its
profile carries a `userChrome.css` that frames every window of the test Betterbird in red and runs a red
band reading MESH TEST across its top, and its event is called "MESH TEST reminder (a throw-away
Betterbird of the mesh, not your calendar)". The reminder window itself is left unstyled, because how
Betterbird draws it is what the test looks at; its title is the event's. A window title cannot be set by a
preference, so the marking is the band, the frame and the event's name.
`betterbird_reminder_test` starts a **second** Betterbird with `--no-remote` in a throw-away profile
under `/run/user/<uid>/mesh-betterbird-reminder-test/`, as the user unit
`mesh-betterbird-reminder-test` (ended after 10 minutes at most). The profile holds the module's
user.js (the setting `prefs`; never a declared account) and one calendar: a local `.ics` file with one
event, "MESH TEST reminder", whose reminder is
due `after` seconds from the call (default 40). `carry` copies named preferences of the operator's
profile into it (a theme, a colour switch), under the same limits as `betterbird_prefs`. The test's own
preferences come last in its user.js, so nothing declared or carried overrides them; one of them turns
Betterbird's update check off (`betterbird.update.checkIntervalDays` 0), because a fresh profile checks
at its start and, when a newer build is published, opens the About window on the operator's screen
(seen 2026-10-12). Then:
- `xorg_windows` with class `Calendar` answers the reminder window: role `AlarmWindow`, type `Dialog`,
its depth (32: a visual with alpha, which the compositor blends), its map state;
- `xorg_screenshot` with that window shows it as the screen shows it.
The unit removes its profile when it ends, however it ends (`ExecStopPost`): at its 10 minutes, on
`stop`, or when the test window is closed. The test Betterbird is never counted as the operator's: its
command line names its own profile, and `status`, `check`, `restart` and the user.js leave it out.
The running Betterbird, its profile and its calendars are not touched: `--no-remote` keeps the two
apart. **`profiles.ini` and `installs.ini` are not rewritten either**, measured on the workstation
(2026-10-09): their content and modification time (June 2023) were the same before a `--profile` start,
while it ran, and after it ended.
## The reminder window, as measured (2026-10-09, workstation, i3 and picom with glx)
| | |
|---|---|
| X class / instance | `eu.betterbird.Betterbird` / `Calendar` |
| role, type | `AlarmWindow`, `_NET_WM_WINDOW_TYPE_DIALOG` (i3 floats it) |
| depth | 32 (ARGB visual), like every Betterbird window there |
| size | 600×300, kept in the profile's `xulstore.json` |
A window of depth 32 that its program has not drawn is transparent under a compositor, not black:
that is why the symptom looks the way it does. Why Betterbird did not draw it is hq issue 345's
question.
## What changes when it is pushed
The module is assigned to shanks today (look-only, from issue 345's first part); g14 runs Betterbird
outside the mesh. With this version, a push to a node (each waits for the operator: `upgrade: record`):
| | on a node with the community package (g14, shanks) |
|---|---|
| the package | `betterbird-bin` removed (quit Betterbird first) |
| `/opt/betterbird` | the mesh's 153.4.0esr-bb10, in the same place, so the same profile opens |
| the policy, Owl | placed; Owl installed into the profile by Betterbird at its next start where it is missing |
| `/usr/local` | the start, the launcher entry, the tools' program |
| `~/.config/betterbird/` | `prefs.json`, `accounts.json`, `configuration.json`, `tags.json` (each empty until its setting is approved) |
| the profile | untouched until the next start from the launcher, `betterbird_user_js` with apply, or `betterbird_restart` |
| the configuration | the policy's locks at Betterbird's next start; the settings in the user.js from the next start. Pushed in the order above, the operator's 21 preferences are in `prefs` before the build arrives; pushed before that layer is approved, Betterbird runs with its own defaults for them until it is |
## Relies on
- A display server on the same machine (`x11-display`, ADR 0208 §3).
- **`xorg`** for `xorg_windows` and `xorg_screenshot`, which look at Betterbird's windows. This module
carries no window code of its own.
- `systemd-run` and `journalctl` of the account's user manager.
- The build seat's archive pinned by URL and sha256 (mesh-controller, ADR 0316 §2): a controller that does
not know it refuses this manifest.
- The controller's `settings` verb, for every proposal and to read a layer as it stands; its `nodes` and `node`
verbs, for which machines hold the module; and the module's own `betterbird_derive` on those machines
(`invokes`).
## Files it does not know
Each directory or archive this module declares says how it treats a file it did not write (ADR 0422, ADR 0416).
- `~/.config/betterbird`: holds only what this module declares and its own tools' records: `accounts-record.json` (`cmd/betterbird-tools/adoption.go:36`), `configuration-record.json` (`owned.go:33`) and the `kept/` folder of replaced filter files (`filters.go:261`); anything else is moved out by the node-engine to `~/.local/state/mesh-moved/betterbird/` with a list, never deleted (ADR 0422 §4).
- `/opt/betterbird`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
- `/opt/betterbird/owl`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
- `/opt/betterbird/distribution`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
- `/opt/betterbird/defaults/pref`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
- `/usr/local/lib/mesh-betterbird`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
@@ -0,0 +1,842 @@
package main
// The operator's mail accounts, declared without credentials (novox/hq ADR 0316 §10–§14, to-be 56 §4).
//
// The setting `accounts` reaches the machine as ~/.config/betterbird/accounts.json, a file the node-engine
// writes from the module's settings layers. Per account it names the address, the incoming server and the
// outgoing server; never a password or a token, and the whole file is refused when any key in it is shaped
// like one. Betterbird's accounts are preferences, so a declared account becomes:
//
// - its declared fields as preferences in the user.js the module places, where they win at every start
// (§11): under the keys of the account the profile already holds at that address, which the module adopts
// (novox/hq ADR 0317), or, where the profile holds none, under keys of the module's own that end in
// mesh_<name> (account_mesh_<name>, id_mesh_<name>, server_mesh_<name>, smtp_mesh_<name>);
// - for an account the module created, its key in the account list (mail.accountmanager.accounts) and in the
// list of outgoing servers (mail.smtpservers) of the profile's prefs.js: written into, never owned, and
// only while Betterbird is not running, because it rewrites prefs.js when it quits (§12). Every other key
// keeps its place, and an adopted account's keys are in the lists already.
//
// Which account is adopted, created or refused is adoption.go's.
//
// Nobody's password passes through here: an account signs in at its first use, in Betterbird's own window,
// and Betterbird keeps the result in its login store (§14). Of that store the module reads only the server
// names each entry is kept for, which Betterbird writes in the clear, to count the declared accounts that
// have not signed in yet (§16); never a user name or a password, which are encrypted there anyway.
import (
"bufio"
"bytes"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"regexp"
"sort"
"strconv"
"strings"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// accountsFile is the setting accounts on the machine, written by the node-engine.
const accountsFile = ".config/betterbird/accounts.json"
// Account is one declared mail account, in the shape of the setting accounts: keyed by its name.
type Account struct {
Address string `json:"address"`
// FromName is the name the account's mail is sent under. Absent: Betterbird's, none.
FromName string `json:"from-name,omitempty"`
// Label is what Betterbird shows the account as. Absent: its address.
Label string `json:"label,omitempty"`
Incoming Server `json:"incoming"`
Outgoing Server `json:"outgoing"`
// Key is the profile's account key to adopt, given only in a node's own layer to settle an address the
// profile holds more than once (novox/hq ADR 0317 §8).
Key string `json:"key,omitempty"`
}
// Server is an account's incoming or outgoing server: never a password or a token.
type Server struct {
// Kind is imap or pop3, for the incoming server; the outgoing one is smtp and says nothing.
Kind string `json:"kind,omitempty"`
Host string `json:"host"`
Port int `json:"port,omitempty"`
Security string `json:"security"`
SignIn string `json:"sign-in"`
User string `json:"user,omitempty"`
}
// The words a server's connection security and sign-in method are said in, and Betterbird's numbers for
// them (socketType / try_ssl, authMethod).
var (
securities = map[string]int{"none": 0, "starttls": 2, "tls": 3}
signIns = map[string]int{"none": 1, "password": 3, "encrypted-password": 4, "kerberos": 5, "ntlm": 6,
"tls-certificate": 7, "oauth": 10}
)
func wordFor(words map[string]int, n int) string {
for w, v := range words {
if v == n {
return w
}
}
return ""
}
var (
accountName = regexp.MustCompile(`^[a-z][a-z0-9]{0,30}$`)
addressShape = regexp.MustCompile(`^[^\s@"\\]+@[A-Za-z0-9.\-]+\.[A-Za-z]{2,}$`)
hostShape = regexp.MustCompile(`^[A-Za-z0-9](?:[A-Za-z0-9.\-]*[A-Za-z0-9])?$`)
profileKey = regexp.MustCompile(`^[A-Za-z0-9_]{1,64}$`)
// plainText is a value written into a preference: no quote, backslash or line break.
plainText = regexp.MustCompile(`^[^"\\\n\r]*$`)
// secretKey is a key the accounts file may never carry, at any depth: the setting holds no credential.
secretKey = regexp.MustCompile(`(?i)(pass|secret|token|oauth.?refresh|refresh|credential|api.?key|private.?key|cookie|bearer)`)
)
// keysOf are the module's own keys for a declared account: each ends in mesh_<name>.
func keysOf(name string) (account, identity, server, smtp string) {
return "account_mesh_" + name, "id_mesh_" + name, "server_mesh_" + name, "smtp_mesh_" + name
}
// ownKey is a key of an account list the module wrote, and the name it was declared under.
var ownKey = regexp.MustCompile(`^(?:account|smtp)_mesh_([a-z][a-z0-9]{0,30})$`)
// readAccounts reads the declared accounts. A missing file is no account (the module is older here); a file
// that carries a key shaped like a credential, an unknown field or a value that cannot be written is refused
// whole, naming the key and never a value.
func (m *Machine) readAccounts() (map[string]Account, bool, error) {
raw, err := readBounded(m.home(accountsFile))
if errors.Is(err, os.ErrNotExist) {
return map[string]Account{}, false, nil
}
if err != nil {
return nil, false, err
}
accounts, err := parseAccounts(raw)
return accounts, true, err
}
func parseAccounts(raw []byte) (map[string]Account, error) {
var tree any
if err := json.Unmarshal(raw, &tree); err != nil {
return nil, fmt.Errorf("~/%s is not JSON: %v", accountsFile, err)
}
if where := secretKeyIn(tree, ""); where != "" {
// An account's name is said by nothing: its segment of the path is withheld.
if parts := strings.SplitN(where, ".", 3); len(parts) == 3 && parts[0] == "accounts" {
where = "accounts.<an account>." + parts[2]
}
return nil, fmt.Errorf("the setting accounts carries %s, a key shaped like a credential: an account is declared "+
"without its password or token, which it signs in with at its first use (novox/hq ADR 0316 §10, §14). "+
"Take it out at the controller's terminal", where)
}
var top map[string]json.RawMessage
if err := json.Unmarshal(raw, &top); err != nil {
return nil, fmt.Errorf("~/%s is not an object: %v", accountsFile, err)
}
accounts := map[string]Account{}
if body, ok := top["accounts"]; ok && string(body) != "null" {
dec := json.NewDecoder(bytes.NewReader(body))
dec.DisallowUnknownFields()
if err := dec.Decode(&accounts); err != nil {
return nil, fmt.Errorf("the setting accounts is not a map of accounts by name: %v", err)
}
}
var problems []string
names := sortedNames(accounts)
for i, name := range names {
problems = append(problems, accountProblems(fmt.Sprintf("declared account %d of %d", i+1, len(names)),
name, accounts[name])...)
}
if len(problems) > 0 {
return nil, fmt.Errorf("the setting accounts is refused: %s", strings.Join(problems, "; "))
}
return accounts, nil
}
// secretKeyIn is the path of the first key shaped like a credential in a JSON tree, or "".
func secretKeyIn(node any, at string) string {
switch v := node.(type) {
case map[string]any:
keys := make([]string, 0, len(v))
for k := range v {
keys = append(keys, k)
}
sort.Strings(keys)
for _, k := range keys {
here := k
if at != "" {
here = at + "." + k
}
if secretKey.MatchString(k) {
return here
}
if w := secretKeyIn(v[k], here); w != "" {
return w
}
}
case []any:
for i, x := range v {
if w := secretKeyIn(x, fmt.Sprintf("%s[%d]", at, i)); w != "" {
return w
}
}
}
return ""
}
func sortedNames(accounts map[string]Account) []string {
names := make([]string, 0, len(accounts))
for n := range accounts {
names = append(names, n)
}
sort.Strings(names)
return names
}
// accountProblems is what is wrong with one declared account, said of it by its place (label) and never by a
// value of it, its name included (novox/hq ADR 0316 §16).
func accountProblems(label, name string, a Account) []string {
var out []string
say := func(format string, args ...any) { out = append(out, label+": "+fmt.Sprintf(format, args...)) }
if !accountName.MatchString(name) {
say("an account's name is lower-case letters and digits, starting with a letter (it names the module's keys, mesh_<name>)")
}
if !addressShape.MatchString(a.Address) {
say("the address is not a mail address")
}
if a.Key != "" && !profileKey.MatchString(a.Key) {
say("key is an account key of the profile, such as account2")
}
for field, v := range map[string]string{"from-name": a.FromName, "label": a.Label} {
if !plainText.MatchString(v) {
say("%s holds a quote, a backslash or a line break", field)
}
}
for side, s := range map[string]Server{"incoming": a.Incoming, "outgoing": a.Outgoing} {
switch {
case side == "incoming" && s.Kind != "imap" && s.Kind != "pop3":
say("the incoming server's kind is imap or pop3")
case side == "outgoing" && s.Kind != "" && s.Kind != "smtp":
say("the outgoing server is smtp")
}
if !hostShape.MatchString(s.Host) {
say("the %s server's host is not a host name", side)
}
if s.Port < 0 || s.Port > 65535 {
say("the %s server's port is not a port", side)
}
if _, ok := securities[s.Security]; !ok {
say("the %s server's security is tls, starttls or none", side)
}
if _, ok := signIns[s.SignIn]; !ok || (side == "incoming" && s.SignIn == "none") {
say("the %s server's sign-in is password, encrypted-password, oauth, kerberos, ntlm or tls-certificate%s",
side, map[bool]string{true: ", or none", false: ""}[side == "outgoing"])
}
if !plainText.MatchString(s.User) {
say("the %s server's user holds a quote, a backslash or a line break", side)
}
}
return out
}
// quoted is a preference's string value as user.js writes it.
func quoted(s string) string { return strconv.Quote(s) }
// declaredPrefs are a declared account's fields as preferences under the keys it resolved to, name to value
// as user.js writes it, in a fixed order. Only the fields the setting names (novox/hq ADR 0317 §5): never
// the account's directory, folders, filters or caches. An adopted account keeps its kind, the outgoing server
// its identity names, and its own name and sender's name where the declaration says none; a created one
// gets every field.
func declaredPrefs(k keySet, a Account, adopted bool) [][2]string {
label := a.Label
if label == "" && !adopted {
label = a.Address
}
inPort := a.Incoming.Port
if inPort == 0 {
inPort = defaultPort(a.Incoming.Kind, a.Incoming.Security)
}
outPort := a.Outgoing.Port
if outPort == 0 {
outPort = defaultPort("smtp", a.Outgoing.Security)
}
var out [][2]string
add := func(key, value string) { out = append(out, [2]string{key, value}) }
if !adopted {
add("mail.account."+k.Account+".identities", quoted(k.Identity))
add("mail.account."+k.Account+".server", quoted(k.Server))
}
if a.FromName != "" || !adopted {
add("mail.identity."+k.Identity+".fullName", quoted(a.FromName))
}
add("mail.identity."+k.Identity+".useremail", quoted(a.Address))
if !adopted {
add("mail.identity."+k.Identity+".smtpServer", quoted(k.SMTP))
add("mail.identity."+k.Identity+".valid", "true")
add("mail.server."+k.Server+".type", quoted(a.Incoming.Kind))
}
add("mail.server."+k.Server+".hostname", quoted(a.Incoming.Host))
add("mail.server."+k.Server+".port", strconv.Itoa(inPort))
add("mail.server."+k.Server+".socketType", strconv.Itoa(securities[a.Incoming.Security]))
add("mail.server."+k.Server+".authMethod", strconv.Itoa(signIns[a.Incoming.SignIn]))
add("mail.server."+k.Server+".userName", quoted(a.Incoming.User))
if label != "" {
add("mail.server."+k.Server+".name", quoted(label))
}
add("mail.smtpserver."+k.SMTP+".hostname", quoted(a.Outgoing.Host))
add("mail.smtpserver."+k.SMTP+".port", strconv.Itoa(outPort))
add("mail.smtpserver."+k.SMTP+".try_ssl", strconv.Itoa(securities[a.Outgoing.Security]))
add("mail.smtpserver."+k.SMTP+".authMethod", strconv.Itoa(signIns[a.Outgoing.SignIn]))
add("mail.smtpserver."+k.SMTP+".username", quoted(a.Outgoing.User))
if label != "" {
add("mail.smtpserver."+k.SMTP+".description", quoted(label))
}
return out
}
// defaultPort is the port a server listens on when the declaration says none.
func defaultPort(kind, security string) int {
switch {
case kind == "imap" && security == "tls":
return 993
case kind == "imap":
return 143
case kind == "pop3" && security == "tls":
return 995
case kind == "pop3":
return 110
case security == "tls":
return 465
case security == "starttls":
return 587
}
return 25
}
// composeUserJS is the user.js the module places in the profile: the declared file (the setting prefs), then
// each adopted or created account's declared fields under its keys. A refused account has none.
func composeUserJS(declaredText string, accounts map[string]Account, resolved []resolvedAccount) string {
var lines []string
for _, r := range resolved {
if r.How == refused {
continue
}
lines = append(lines, fmt.Sprintf("// declared account %d: %s under %s", r.Place, r.How, r.Keys.Account))
for _, p := range declaredPrefs(r.Keys, accounts[r.name], r.How == adopted) {
lines = append(lines, fmt.Sprintf("user_pref(%s, %s);", quoted(p[0]), p[1]))
}
}
if len(lines) == 0 {
return declaredText
}
return strings.TrimRight(declaredText, "\n") + "\n\n" +
"// The accounts the setting accounts declares (novox/hq ADR 0316 §11, ADR 0317): the fields it names, under\n" +
"// the keys of the account adopted or created. No password or token: an account signs in at its first use.\n" +
strings.Join(lines, "\n") + "\n"
}
// profileAccount is one account the profile holds, as its preferences say it.
type profileAccount struct {
Key, Server, Kind string
// Identities are its identities, the default first, and Addresses their addresses in that order.
Identities []string
Addresses []string
Owl, Ours bool
}
// profileAccounts reads the accounts the profile's list names, from prefs.js and user.js (user.js wins).
func profileAccounts(prefs map[string]string) []profileAccount {
var out []profileAccount
for _, key := range listOf(prefs["mail.accountmanager.accounts"]) {
a := profileAccount{Key: key, Server: unquote(prefs["mail.account."+key+".server"])}
a.Kind = unquote(prefs["mail.server."+a.Server+".type"])
a.Owl = strings.HasPrefix(a.Kind, "owl")
a.Ours = ownKey.MatchString(key)
for _, id := range listOf(prefs["mail.account."+key+".identities"]) {
a.Identities = append(a.Identities, id)
a.Addresses = append(a.Addresses, unquote(prefs["mail.identity."+id+".useremail"]))
}
out = append(out, a)
}
return out
}
// mergedPrefs are a profile's preferences as Betterbird starts with them: prefs.js, then user.js over it.
func mergedPrefs(profile string) map[string]string {
prefsJS, _ := readBounded(filepath.Join(profile, "prefs.js"))
userJS, _ := readBounded(filepath.Join(profile, "user.js"))
out := readPrefs(prefsJS)
for k, v := range readPrefs(userJS) {
out[k] = v
}
return out
}
// listOf reads a comma-separated list preference's value.
func listOf(v string) []string {
var out []string
for _, k := range strings.Split(unquote(v), ",") {
if k = strings.TrimSpace(k); k != "" {
out = append(out, k)
}
}
return out
}
// unquote is a preference's string value without its quotes; any other value as written.
func unquote(v string) string {
if s, err := strconv.Unquote(v); err == nil {
return s
}
return v
}
// writeAccountLists writes the module's keys into the profile's account list and list of outgoing servers
// (§12): the declared accounts' keys added at the end, the module's keys of accounts no longer declared taken
// out, every other key and line kept as it was. Only while Betterbird is not running. Says what changed.
func (m *Machine) writeAccountLists(profile string, resolved []resolvedAccount) (string, error) {
path := filepath.Join(profile, "prefs.js")
raw, err := readBounded(path)
if err != nil && !errors.Is(err, os.ErrNotExist) {
return "", err
}
var wantAccounts, wantSMTP []string
for _, r := range resolved {
if r.How == created {
wantAccounts, wantSMTP = append(wantAccounts, r.Keys.Account), append(wantSMTP, r.Keys.SMTP)
}
}
out, changed := withListed(string(raw), map[string][]string{
"mail.accountmanager.accounts": wantAccounts, "mail.smtpservers": wantSMTP})
if !changed {
return "unchanged", nil
}
// Only while Betterbird is closed: no main process and no lock on the profile (novox/hq ADR 0316 §12, ADR 0329 §9).
if why := m.closedFor(profile); why != "" {
return "", errors.New(why + ": the account list is written at its next start from the launcher, or by betterbird_restart")
}
mode := os.FileMode(0o600)
if st, err := os.Stat(path); err == nil {
mode = st.Mode().Perm()
}
if err := writeAtomic(path, []byte(out), mode); err != nil {
return "", err
}
return fmt.Sprintf("written: %d account(s) the module created in the account list", len(wantAccounts)), nil
}
// withListed is prefs.js with each list preference holding the module's wanted keys: its own keys that are
// not wanted taken out, the wanted ones not there added at its end, everything else as it was. A list the
// file does not hold yet is added as a line, when anything is wanted in it.
func withListed(text string, want map[string][]string) (string, bool) {
changed := false
done := map[string]bool{}
var lines []string
s := bufio.NewScanner(strings.NewReader(text))
s.Buffer(make([]byte, 64<<10), 4<<20)
for s.Scan() {
line := s.Text()
if mm := prefLine.FindStringSubmatch(line); mm != nil {
if keys, ok := want[mm[1]]; ok && !done[mm[1]] {
done[mm[1]] = true
next := listedWith(listOf(mm[2]), keys)
if next != strings.Join(listOf(mm[2]), ",") {
line = fmt.Sprintf("user_pref(%s, %s);", quoted(mm[1]), quoted(next))
changed = true
}
}
}
lines = append(lines, line)
}
names := make([]string, 0, len(want))
for n := range want {
names = append(names, n)
}
sort.Strings(names)
for _, n := range names {
if !done[n] && len(want[n]) > 0 {
lines = append(lines, fmt.Sprintf("user_pref(%s, %s);", quoted(n), quoted(strings.Join(want[n], ","))))
changed = true
}
}
out := strings.Join(lines, "\n")
if text == "" || strings.HasSuffix(text, "\n") {
out += "\n"
}
return out, changed
}
// listedWith is a list with the module's own keys made the wanted ones, every other key in its place.
func listedWith(have, wanted []string) string {
want := map[string]bool{}
for _, k := range wanted {
want[k] = true
}
var out []string
there := map[string]bool{}
for _, k := range have {
if ownKey.MatchString(k) && !want[k] {
continue
}
out = append(out, k)
there[k] = true
}
for _, k := range wanted {
if !there[k] {
out = append(out, k)
}
}
return strings.Join(out, ",")
}
// AccountCount is what status and check say of the accounts: numbers, places and keys, never an address, a
// host or a name (novox/hq ADR 0316 §16, ADR 0317 §3).
type AccountCount struct {
Declared int `json:"declared"`
// InList are the declared accounts the profile's account list names, adopted or created.
InList int `json:"declared_in_the_list"`
// Others are the accounts the profile holds that no declaration names (the work account in Owl among them).
Others int `json:"other_accounts"`
// WithoutSignIn are the declared accounts with no sign-in kept in Betterbird's login store yet.
WithoutSignIn int `json:"declared_without_a_sign_in,omitempty"`
// Each is every declared account by its place in the setting: adopted, created or refused, and its keys.
Each []resolvedAccount `json:"each,omitempty"`
Problem string `json:"problem,omitempty"`
}
// countAccounts counts the declared accounts against the profile's.
func (m *Machine) countAccounts(profile, profileDir string) AccountCount {
var c AccountCount
accounts, _, err := m.readAccounts()
if err != nil {
c.Problem = err.Error()
return c
}
c.Declared = len(accounts)
prefs := mergedPrefs(profile)
have := profileAccounts(prefs)
c.Each = resolveAccounts(accounts, prefs, m.readRecord(profileDir))
mine := map[string]bool{}
refusedN := 0
for _, r := range c.Each {
if r.How == refused {
refusedN++
continue
}
mine[r.Keys.Account] = true
}
for _, a := range have {
switch {
case mine[a.Key]:
c.InList++
case a.Kind != "none":
c.Others++
}
}
if refusedN > 0 {
c.Problem = fmt.Sprintf("%d declared account(s) are refused: each says why and what to do", refusedN)
}
c.WithoutSignIn = len(withoutSignIn(accounts, keptSignIns(profile)))
return c
}
// keptSignIns are the server names Betterbird's login store keeps a sign-in for: the `hostname` of each
// entry of logins.json, which Betterbird writes in the clear. Nothing else of the file is read.
func keptSignIns(profile string) map[string]bool {
raw, err := readBounded(filepath.Join(profile, "logins.json"))
out := map[string]bool{}
if err != nil {
return out
}
var store struct {
Logins []struct {
Hostname string `json:"hostname"`
} `json:"logins"`
}
if json.Unmarshal(raw, &store) != nil {
return out
}
for _, l := range store.Logins {
out[strings.ToLower(l.Hostname)] = true
}
return out
}
// oauthIssuers are the sign-in services of the providers whose servers sign in through OAuth, by a suffix of
// the server's host: their kept sign-in is under the service's name, not the server's.
var oauthIssuers = map[string]string{"gmail.com": "oauth://accounts.google.com",
"googlemail.com": "oauth://accounts.google.com", "office365.com": "oauth://login.microsoftonline.com",
"outlook.com": "oauth://login.microsoftonline.com"}
// withoutSignIn are the declared accounts whose incoming server has no sign-in kept yet. A server signed in
// to without a password (none, kerberos, ntlm, a certificate) keeps nothing and is never counted.
func withoutSignIn(accounts map[string]Account, kept map[string]bool) []string {
var out []string
for _, name := range sortedNames(accounts) {
in := accounts[name].Incoming
switch in.SignIn {
case "password", "encrypted-password":
if !kept[strings.ToLower(in.Kind+"://"+in.Host)] {
out = append(out, name)
}
case "oauth":
issuer := ""
for suffix, i := range oauthIssuers {
if strings.HasSuffix(strings.ToLower(in.Host), suffix) {
issuer = i
}
}
found := false
for k := range kept {
if (issuer != "" && k == issuer) || (issuer == "" && strings.HasPrefix(k, "oauth://")) {
found = true
}
}
if !found {
out = append(out, name)
}
}
}
return out
}
// AccountsAnswer is what betterbird_accounts answers its caller: the profile's accounts with the operator's own
// values (novox/hq ADR 0329 §1, §12), Owl's marked and by name and address only (§13), and counts and kinds.
// Never a password, a token or anything of the login store (§2).
type AccountsAnswer struct {
Profile string `json:"profile"`
// Accounts are the profile's accounts, in its list's order.
Accounts []listedAccount `json:"accounts"`
// Derived is the value of the setting accounts the propose built, as it was proposed (ADR 0329 §10).
Derived map[string]Account `json:"derived,omitempty"`
// Declarable are the profile's mail accounts a declaration can hold (IMAP and POP3), by kind.
Declarable map[string]int `json:"declarable"`
// KeptByOwl are the accounts Owl keeps (the work account through Exchange): never declared (ADR 0316 §13).
KeptByOwl int `json:"kept_by_owl"`
// NotDeclarable are accounts of another kind (local folders, feeds, news, chat), by kind.
NotDeclarable map[string]int `json:"not_declarable,omitempty"`
// AlreadyDeclared are the declarable accounts under the module's own keys.
AlreadyDeclared int `json:"under_the_modules_keys"`
// Proposed is the controller's answer to the proposal, with every address and host masked.
Proposed string `json:"proposed,omitempty"`
Note string `json:"note"`
}
// listedAccount is one account of the profile as betterbird_accounts lists it.
type listedAccount struct {
Key string `json:"key"`
Kind string `json:"kind"`
// DeclaredAs is the name the setting accounts declares it under, or empty: the person's own.
DeclaredAs string `json:"declared_as,omitempty"`
Owl bool `json:"owl,omitempty"`
// Name is what Betterbird shows it as, and Address its default identity's.
Name string `json:"name,omitempty"`
Address string `json:"address,omitempty"`
Value *Account `json:"value,omitempty"`
Note string `json:"note,omitempty"`
}
// profileDeclarable reads the profile's accounts: the value a declaration of them would be, by name, and
// what is answered of them.
func (m *Machine) profileDeclarable() (map[string]Account, AccountsAnswer, error) {
p, err := m.profile()
if err != nil {
return nil, AccountsAnswer{}, err
}
prefs := mergedPrefs(p.path)
a := AccountsAnswer{Profile: p.Dir, Accounts: []listedAccount{}, Declarable: map[string]int{}, NotDeclarable: map[string]int{}}
value := map[string]Account{}
used := map[string]bool{}
declaredAs := map[string]string{}
if accounts, _, err := m.readAccounts(); err == nil {
for _, r := range resolveAccounts(accounts, prefs, m.readRecord(p.Dir)) {
if r.How != refused {
declaredAs[r.Keys.Account] = r.name
}
}
}
for _, pa := range profileAccounts(prefs) {
listed := listedAccount{Key: pa.Key, Kind: pa.Kind, DeclaredAs: declaredAs[pa.Key], Owl: pa.Owl,
Name: unquote(prefs["mail.server."+pa.Server+".name"])}
if len(pa.Addresses) > 0 {
listed.Address = pa.Addresses[0]
}
switch {
case pa.Owl:
a.KeptByOwl++
listed.Note = "Owl's: never declared (novox/hq ADR 0316 §13)"
a.Accounts = append(a.Accounts, listed)
continue
case pa.Kind != "imap" && pa.Kind != "pop3":
kind := pa.Kind
if kind == "none" {
kind = "local folders"
}
a.NotDeclarable[kind]++
listed.Note = "not an account a declaration holds"
a.Accounts = append(a.Accounts, listed)
continue
}
a.Declarable[pa.Kind]++
if pa.Ours {
a.AlreadyDeclared++
}
address := ""
if len(pa.Addresses) > 0 {
address = pa.Addresses[0]
}
name := nameFor(pa, address, used)
if declaredAs[pa.Key] != "" {
name = declaredAs[pa.Key]
}
used[name] = true
acc := readAccount(prefs, pa)
value[name] = acc
listed.Value = &acc
a.Accounts = append(a.Accounts, listed)
}
if len(a.NotDeclarable) == 0 {
a.NotDeclarable = nil
}
return value, a, nil
}
// asker calls a verb of the mesh as this module, through the runtime: a variable so a test answers.
var asker = stdio.Ask
// ProfileAccounts says, by count and kind, which accounts the profile in use holds; with propose it builds
// the setting accounts from them (every account but Owl's, without any credential) and proposes it through
// the controller's settings verb for the operator's Approve (novox/hq ADR 0317 §9, ADR 0277). The value
// never comes back to the caller: the operator reads it whole at their desk (`settings proposals <id>`) and
// checks the fingerprint the ask shows.
func (m *Machine) ProfileAccounts(propose bool) (AccountsAnswer, error) {
value, a, err := m.profileDeclarable()
if err != nil {
return a, err
}
a.Note = "the profile's accounts with their values, never a password or token (novox/hq ADR 0329 §1, §2). With " +
"propose, the setting accounts is built from them (never Owl's) and proposed to the operator in the mesh-wide " +
"layer as it stands, who approves it on a channel that proves who answers (novox/hq ADR 0317 §9)"
if !propose {
return a, nil
}
if len(value) == 0 {
return a, errors.New("the profile holds no IMAP or POP3 account to declare: nothing was proposed")
}
var problems []string
for i, name := range sortedNames(value) {
if p := accountProblems("", name, value[name]); len(p) > 0 {
problems = append(problems, fmt.Sprintf("account %d of %d: %d field(s) cannot be declared as the profile "+
"holds them", i+1, len(value), len(p)))
}
}
if len(problems) > 0 {
return a, fmt.Errorf("nothing was proposed: %s; betterbird_prefs shows the account's preferences by name",
strings.Join(problems, "; "))
}
// A layer is replaced whole: the proposal carries the mesh-wide layer as it stands, with this value of accounts.
layer, err := readLayer("")
if err != nil {
return a, fmt.Errorf("nothing was proposed: %v", err)
}
layer["accounts"] = value
a.Derived = value
said, err := proposeLayer("", layer, false)
a.Proposed = m.maskValue(said)
return a, err
}
// verbSaid is a verb's words and whether it went ahead, whatever wraps them: the controller's {ok, output},
// or the protocol's content list.
func verbSaid(raw json.RawMessage) (string, bool) {
var s string
if json.Unmarshal(raw, &s) == nil {
return verbSaid(json.RawMessage(s))
}
var m map[string]json.RawMessage
if json.Unmarshal(raw, &m) != nil {
return string(raw), true
}
if content, has := m["content"]; has {
var items []struct {
Text string `json:"text"`
}
var e struct {
IsError bool `json:"isError"`
}
_ = json.Unmarshal(raw, &e)
if json.Unmarshal(content, &items) == nil && len(items) > 0 {
out, ok := verbSaid(json.RawMessage(items[0].Text))
return out, ok && !e.IsError
}
}
if okRaw, has := m["ok"]; has {
var ok bool
_ = json.Unmarshal(okRaw, &ok)
var output string
_ = json.Unmarshal(m["output"], &output)
return output, ok
}
return string(raw), true
}
// readAccount reads one profile account's preferences into the shape of the setting.
func readAccount(prefs map[string]string, pa profileAccount) Account {
num := func(key string, def int) int {
if n, err := strconv.Atoi(prefs[key]); err == nil {
return n
}
return def
}
text := func(key string) string { return unquote(prefs[key]) }
srv := "mail.server." + pa.Server + "."
acc := Account{Incoming: Server{Kind: pa.Kind, Host: text(srv + "hostname"), User: text(srv + "userName"),
Security: wordFor(securities, num(srv+"socketType", 0)), SignIn: wordFor(signIns, num(srv+"authMethod", 3))}}
acc.Incoming.Port = num(srv+"port", defaultPort(pa.Kind, acc.Incoming.Security))
ids := listOf(prefs["mail.account."+pa.Key+".identities"])
if len(ids) > 0 {
id := "mail.identity." + ids[0] + "."
acc.Address, acc.FromName = text(id+"useremail"), text(id+"fullName")
smtp := text(id + "smtpServer")
if smtp == "" {
smtp = text("mail.smtp.defaultserver")
}
if smtp != "" {
s := "mail.smtpserver." + smtp + "."
acc.Outgoing = Server{Host: text(s + "hostname"), User: text(s + "username"),
Security: wordFor(securities, num(s+"try_ssl", 0)), SignIn: wordFor(signIns, num(s+"authMethod", 3))}
acc.Outgoing.Port = num(s+"port", defaultPort("smtp", acc.Outgoing.Security))
}
}
if label := text(srv + "name"); label != "" && label != acc.Address {
acc.Label = label
}
return acc
}
// nameFor is the name a profile account is answered under: its own name when the module declared it, else
// the first label of its address's domain, made unique.
func nameFor(pa profileAccount, address string, used map[string]bool) string {
if mm := ownKey.FindStringSubmatch(pa.Key); mm != nil {
return mm[1]
}
base := "account"
if _, domain, ok := strings.Cut(address, "@"); ok {
label, _, _ := strings.Cut(strings.ToLower(domain), ".")
label = regexp.MustCompile(`[^a-z0-9]`).ReplaceAllString(label, "")
if label != "" && label[0] >= 'a' && label[0] <= 'z' {
base = label
}
}
if len(base) > 28 {
base = base[:28]
}
name := base
for i := 2; used[name]; i++ {
name = base + strconv.Itoa(i)
}
return name
}
@@ -0,0 +1,567 @@
package main
import (
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
)
// The setting accounts (novox/hq ADR 0316 §10–§16, ADR 0317 §9 for betterbird_accounts): declared without
// credentials, written under the module's own keys, the account list written into and never owned, refused
// rather than duplicated, and said by count.
const personal = `{"accounts": {
"personal": {"address": "me@mail.example.org", "from-name": "Me",
"incoming": {"kind": "imap", "host": "imap.example.org", "security": "tls", "sign-in": "password", "user": "me"},
"outgoing": {"host": "smtp.example.org", "port": 587, "security": "starttls", "sign-in": "password", "user": "me"}}
}}`
// profileWithAccounts is a profile holding an account of the person's at another address, Owl's work account,
// local folders, and an account list in that order.
func (f *fake) profileWithAccounts() {
f.betterbird()
f.write(profileDir+"/prefs.js", "// Mozilla User Preferences\n"+
`user_pref("mail.account.account1.server", "server1");`+"\n"+
`user_pref("mail.account.account1.identities", "id1");`+"\n"+
`user_pref("mail.account.account2.server", "server2");`+"\n"+
`user_pref("mail.account.account3.server", "server3");`+"\n"+
`user_pref("mail.account.account3.identities", "id3");`+"\n"+
`user_pref("mail.accountmanager.accounts", "account1,account3,account2");`+"\n"+
`user_pref("mail.identity.id1.useremail", "other@mail.example.org");`+"\n"+
`user_pref("mail.identity.id1.fullName", "Other");`+"\n"+
`user_pref("mail.identity.id1.smtpServer", "smtp1");`+"\n"+
`user_pref("mail.identity.id3.useremail", "work@corp.example.com");`+"\n"+
`user_pref("mail.server.server1.type", "imap");`+"\n"+
`user_pref("mail.server.server1.hostname", "imap.example.org");`+"\n"+
`user_pref("mail.server.server1.socketType", 3);`+"\n"+
`user_pref("mail.server.server1.authMethod", 10);`+"\n"+
`user_pref("mail.server.server1.userName", "other");`+"\n"+
`user_pref("mail.server.server1.directory", "/home/operator/.thunderbird/x/ImapMail/imap.example.org");`+"\n"+
`user_pref("mail.server.server2.type", "none");`+"\n"+
`user_pref("mail.server.server3.type", "owl-ews");`+"\n"+
`user_pref("mail.smtpserver.smtp1.hostname", "smtp.example.org");`+"\n"+
`user_pref("mail.smtpserver.smtp1.try_ssl", 3);`+"\n"+
`user_pref("mail.smtpserver.smtp1.authMethod", 10);`+"\n"+
`user_pref("mail.smtpserver.smtp1.username", "other");`+"\n"+
`user_pref("mail.smtpservers", "smtp1");`+"\n"+
`user_pref("mailnews.tags.version", 2);`+"\n")
}
func (f *fake) declareAccounts(body string) {
f.write(testHome+"/"+accountsFile, body)
}
func (f *fake) read(path string) string {
f.t.Helper()
raw, err := os.ReadFile(filepath.Join(f.Root, path))
if err != nil {
f.t.Fatal(err)
}
return string(raw)
}
func TestAnAccountsValueCarryingAPasswordLikeFieldIsRefusedWithoutItsValue(t *testing.T) {
for _, body := range []string{
`{"accounts": {"personal": {"address": "me@mail.example.org", "password": "hunter2"}}}`,
`{"accounts": {"personal": {"incoming": {"oauth_refresh": "1//hunter2"}}}}`,
`{"token": "hunter2", "accounts": {}}`,
} {
_, err := parseAccounts([]byte(body))
if err == nil || !strings.Contains(err.Error(), "shaped like a credential") || strings.Contains(err.Error(), "hunter2") {
t.Errorf("%s: %v", body, err)
}
}
// An unknown field is refused too, and a value that cannot be declared, said by the account's place.
if _, err := parseAccounts([]byte(`{"accounts": {"personal": {"address": "me@mail.example.org", "keychain": 1}}}`)); err == nil {
t.Error("an unknown field was taken")
}
_, err := parseAccounts([]byte(strings.Replace(personal, `"security": "tls"`, `"security": "maybe"`, 1)))
if err == nil || !strings.Contains(err.Error(), "declared account 1 of 1") || strings.Contains(err.Error(), "personal") ||
strings.Contains(err.Error(), "example.org") {
t.Errorf("a bad value is said by the account's place, never its name or a value: %v", err)
}
}
func TestTheSettingPrefsRefusesAnAccountsPreferences(t *testing.T) {
for _, name := range []string{"mail.account.account1.server", "mail.identity.id1.useremail", "mail.server.server1.hostname",
"mail.smtpserver.smtp1.hostname", "mail.accountmanager.accounts", "mail.smtpservers"} {
err := validUserJS(mark + ")\nuser_pref(\"" + name + "\", \"x\");\n")
if err == nil || !strings.Contains(err.Error(), "setting accounts") {
t.Errorf("%s was taken in prefs: %v", name, err)
}
}
if err := validUserJS(mark + ")\nuser_pref(\"calendar.alarms.show\", true);\n"); err != nil {
t.Fatal(err)
}
}
// noPackage answers pacman as a machine without the community package.
func (f *fake) noPackage() {
f.answer = func(name string, args []string) Output {
if name == "pacman" {
return Output{Code: 1}
}
return Output{}
}
}
// userPrefLines are the user_pref lines of a user.js, by preference name.
func userPrefLines(t *testing.T, text string) map[string]string {
t.Helper()
out := map[string]string{}
for _, line := range strings.Split(text, "\n") {
if !strings.HasPrefix(line, "user_pref(") {
continue
}
if !statement.MatchString(line) {
t.Errorf("not a statement Betterbird reads as one: %s", line)
}
out[line[len(`user_pref("`):strings.Index(line, `",`)]] = line
}
return out
}
func TestADeclaredAccountTheProfileDoesNotHoldIsCreatedUnderTheModulesKeysAndListed(t *testing.T) {
f := newFake(t)
f.profileWithAccounts()
f.noPackage()
links(t, nil)
f.declare("// none")
f.declareAccounts(personal)
// Betterbird is running: the user.js is placed, the list waits for its next start.
u, err := f.userJS(true)
if err != nil {
t.Fatal(err)
}
if !strings.HasPrefix(u.Lists, "not written") || !strings.Contains(u.Lists, "running") {
t.Fatalf("the list is never written while Betterbird runs: %+v", u)
}
if len(u.Each) != 1 || u.Each[0].How != created || u.Each[0].Keys != (keySet{"account_mesh_personal", "id_mesh_personal",
"server_mesh_personal", "smtp_mesh_personal"}) {
t.Fatalf("%+v", u.Each)
}
placed := f.read(profileDir + "/user.js")
for name := range userPrefLines(t, placed) {
if key := strings.Split(name, ".")[2]; !strings.HasSuffix(key, "mesh_personal") {
t.Errorf("a created account's preference outside the module's keys: %s", name)
}
if strings.Contains(strings.ToLower(name), "password") || strings.Contains(name, "directory") {
t.Errorf("the module writes no credential and none of Betterbird's own preferences: %s", name)
}
}
for _, want := range []string{`user_pref("mail.server.server_mesh_personal.port", 993);`,
`user_pref("mail.server.server_mesh_personal.socketType", 3);`,
`user_pref("mail.smtpserver.smtp_mesh_personal.try_ssl", 2);`,
`user_pref("mail.identity.id_mesh_personal.useremail", "me@mail.example.org");`} {
if !strings.Contains(placed, want) {
t.Errorf("the user.js lacks %s:\n%s", want, placed)
}
}
// Quit: the list gains the module's keys at its end, every other line as it was.
before := f.read(profileDir + "/prefs.js")
os.RemoveAll(filepath.Join(f.Root, "proc", "4000"))
if err := f.Prepare(); err != nil {
t.Fatal(err)
}
after := f.read(profileDir + "/prefs.js")
if !strings.Contains(after, `user_pref("mail.accountmanager.accounts", "account1,account3,account2,account_mesh_personal");`) ||
!strings.Contains(after, `user_pref("mail.smtpservers", "smtp1,smtp_mesh_personal");`) {
t.Fatalf("the lists:\n%s", after)
}
bl, al := strings.Split(before, "\n"), strings.Split(after, "\n")
if len(bl) != len(al) {
t.Fatalf("lines were added or lost: %d then %d", len(bl), len(al))
}
for i := range bl {
if bl[i] != al[i] && !strings.Contains(bl[i], "mail.accountmanager.accounts") && !strings.Contains(bl[i], "mail.smtpservers") {
t.Errorf("line %d changed: %q to %q", i+1, bl[i], al[i])
}
}
// The record lost: an account under the module's own keys is still its creation, never adopted.
os.Remove(filepath.Join(f.Root, testHome, recordFile))
u, err = f.userJS(true)
if err != nil || u.Each[0].How != created {
t.Fatalf("without the record, the module's keys still say created: %+v %v", u.Each, err)
}
// The account leaves the setting: only the module's keys go, the mail stays in the profile.
f.declareAccounts(`{"accounts": {}}`)
if err := f.Prepare(); err != nil {
t.Fatal(err)
}
if got := f.read(profileDir + "/prefs.js"); got != before {
t.Fatalf("removing the module's keys left the file otherwise as it was:\n%s", got)
}
}
// adoptable is a declaration of the profile's account1 (other@mail.example.org): equal but for its outgoing
// port, said explicitly, and its incoming user.
const adoptable = `{"accounts": {
"mail": {"address": "Other@mail.example.org",
"incoming": {"kind": "imap", "host": "imap.example.org", "security": "tls", "sign-in": "oauth", "user": "other2"},
"outgoing": {"host": "smtp.example.org", "port": 465, "security": "tls", "sign-in": "oauth", "user": "other"}}}}`
func TestADeclaredAccountTheProfileHoldsIsAdoptedUnderItsOwnKeysAndTheListIsLeftAlone(t *testing.T) {
f := newFake(t)
f.profileWithAccounts()
f.noPackage()
links(t, nil)
f.declare("// none")
f.declareAccounts(adoptable)
os.RemoveAll(filepath.Join(f.Root, "proc", "4000"))
before := f.read(profileDir + "/prefs.js")
u, err := f.userJS(true)
if err != nil {
t.Fatal(err)
}
if len(u.Each) != 1 || u.Each[0].How != adopted || u.Each[0].Keys != (keySet{"account1", "id1", "server1", "smtp1"}) {
t.Fatalf("adopted under the profile's keys: %+v", u.Each)
}
// The address's case and the user name differ; the default port declared explicitly is no change. Names
// only, no value.
if strings.Join(u.Each[0].Changed, ",") != "mail.identity.id1.useremail,mail.server.server1.userName" {
t.Fatalf("the changed preferences, by name: %v", u.Each[0].Changed)
}
if f.read(profileDir+"/prefs.js") != before || u.Lists != "unchanged" {
t.Fatalf("an adopted account's keys are in the lists already: nothing written to prefs.js (%s)", u.Lists)
}
lines := userPrefLines(t, f.read(profileDir+"/user.js"))
for name := range lines {
key := strings.Split(name, ".")[2]
if key != "account1" && key != "id1" && key != "server1" && key != "smtp1" {
t.Errorf("an adopted account's preference outside its keys: %s", name)
}
for _, never := range []string{"directory", ".type", "smtpServer", "identities", ".server"} {
if strings.HasSuffix(name, never) {
t.Errorf("the module wrote %s, which is Betterbird's or the account's own", name)
}
}
}
if !strings.Contains(lines["mail.server.server1.userName"], `"other2"`) {
t.Fatalf("the declared value wins at every start: %v", lines)
}
// The status says it by place and keys, never by address, host or name.
s, err := f.Status()
if err != nil {
t.Fatal(err)
}
raw, _ := json.Marshal(s.Accounts)
if s.Accounts.InList != 1 || s.Accounts.Others != 1 || strings.Contains(string(raw), "example") ||
strings.Contains(string(raw), `"mail"`) || !strings.Contains(string(raw), `"account1"`) {
t.Fatalf("%s", raw)
}
// Leaving the setting releases it: its lines and its record go, the account and the lists stay.
f.declareAccounts(`{"accounts": {}}`)
if err := f.Prepare(); err != nil {
t.Fatal(err)
}
if f.read(profileDir+"/prefs.js") != before || strings.Contains(f.read(profileDir+"/user.js"), "server1") ||
strings.Contains(f.read(testHome+"/"+recordFile), "account1") {
t.Fatal("a released account stays the person's, with no line or record of the module's")
}
}
func TestAnAmbiguousOwlsOrOtherKindOfMatchIsRefusedForThatAccountAlone(t *testing.T) {
f := newFake(t)
f.profileWithAccounts()
f.noPackage()
links(t, nil)
f.declare("// none")
os.RemoveAll(filepath.Join(f.Root, "proc", "4000"))
// A second account with account1's address.
prefs := f.read(profileDir + "/prefs.js")
prefs = strings.Replace(prefs, `"account1,account3,account2"`, `"account1,account3,account2,account4"`, 1) +
`user_pref("mail.account.account4.server", "server4");` + "\n" +
`user_pref("mail.account.account4.identities", "id4");` + "\n" +
`user_pref("mail.identity.id4.useremail", "other@mail.example.org");` + "\n" +
`user_pref("mail.identity.id4.smtpServer", "smtp4");` + "\n" +
`user_pref("mail.smtpserver.smtp4.hostname", "smtp.example.org");` + "\n" +
`user_pref("mail.server.server4.type", "imap");` + "\n"
f.write(profileDir+"/prefs.js", prefs)
both := strings.Replace(adoptable, `{"accounts": {`, `{"accounts": {
"zz": {"address": "new@mail.example.org",
"incoming": {"kind": "imap", "host": "imap.example.org", "security": "tls", "sign-in": "password"},
"outgoing": {"host": "smtp.example.org", "security": "tls", "sign-in": "password"}},`, 1)
f.declareAccounts(both)
err := f.Prepare()
if err == nil || !strings.Contains(err.Error(), "declared account 1 of 2: 2 accounts in the profile have its address (account1, account4)") ||
!strings.Contains(err.Error(), "key") || strings.Contains(err.Error(), "example.org") {
t.Fatalf("refused by place, with the matches' keys and the remedy, never the address: %v", err)
}
u, _ := f.userJS(false)
if u.Each[0].How != refused || u.Each[1].How != created {
t.Fatalf("the other declared account goes ahead: %+v", u.Each)
}
if strings.Contains(f.read(profileDir+"/user.js"), "server1") || strings.Contains(f.read(profileDir+"/user.js"), "server4") {
t.Fatal("a refused account gets no line")
}
// The node's layer names the key: that one is adopted.
f.declareAccounts(strings.Replace(both, `"address": "Other@mail.example.org",`, `"address": "Other@mail.example.org", "key": "account4",`, 1))
if err := f.Prepare(); err != nil {
t.Fatalf("%v", err)
}
if u, _ := f.userJS(false); u.Each[0].How != adopted || u.Each[0].Keys.Account != "account4" {
t.Fatalf("%+v", u.Each)
}
os.Remove(filepath.Join(f.Root, testHome, recordFile))
for key, want := range map[string]string{"account9": "not in the profile's account list", "account3": "Owl's",
"account2": "another address"} {
f.declareAccounts(strings.Replace(adoptable, `"address": "Other@mail.example.org",`, `"address": "Other@mail.example.org", "key": "`+key+`",`, 1))
if err := f.Prepare(); err == nil || !strings.Contains(err.Error(), want) {
t.Errorf("key %s: %v", key, err)
}
}
// Owl's account at the address, and an account of another kind than declared.
f.declareAccounts(`{"accounts": {"work": {"address": "work@corp.example.com",
"incoming": {"kind": "imap", "host": "outlook.example.com", "security": "tls", "sign-in": "oauth"},
"outgoing": {"host": "smtp.example.com", "security": "tls", "sign-in": "oauth"}}}}`)
if err := f.Prepare(); err == nil || !strings.Contains(err.Error(), "Owl's") {
t.Errorf("Owl's account is never adopted: %v", err)
}
f.write(profileDir+"/prefs.js", strings.Replace(prefs, `"mail.server.server4.type", "imap"`, `"mail.server.server4.type", "pop3"`, 1))
f.declareAccounts(strings.Replace(adoptable, `"address": "Other@mail.example.org",`, `"address": "Other@mail.example.org", "key": "account4",`, 1))
if err := f.Prepare(); err == nil || !strings.Contains(err.Error(), "never converts") {
t.Errorf("an account of another kind is refused, not converted: %v", err)
}
}
func TestDeclaredAccountsWithoutASignInAreCountedFromTheServerNamesAlone(t *testing.T) {
f := newFake(t)
f.profileWithAccounts()
links(t, nil)
f.declareAccounts(`{"accounts": {
"personal": {"address": "me@mail.example.org", "incoming": {"kind": "imap", "host": "imap.example.org", "security": "tls", "sign-in": "password"},
"outgoing": {"host": "smtp.example.org", "security": "tls", "sign-in": "password"}},
"google": {"address": "me@gmail.com", "incoming": {"kind": "imap", "host": "imap.gmail.com", "security": "tls", "sign-in": "oauth"},
"outgoing": {"host": "smtp.gmail.com", "security": "tls", "sign-in": "oauth"}},
"lan": {"address": "me@lan.example.org", "incoming": {"kind": "pop3", "host": "pop.example.org", "security": "tls", "sign-in": "kerberos"},
"outgoing": {"host": "smtp.example.org", "security": "tls", "sign-in": "none"}}}}`)
// The username and password fields are encrypted and never read; a fixture holding junk there proves it.
f.write(profileDir+"/logins.json", `{"logins": [
{"hostname": "imap://imap.example.org", "httpRealm": "imap://imap.example.org", "encryptedUsername": "MEIE...", "encryptedPassword": "MDoE..."},
{"hostname": "smtp://smtp.example.org", "encryptedUsername": "x", "encryptedPassword": "y"}]}`)
c := f.countAccounts(filepath.Join(f.Root, profileDir), "~/.thunderbird/60016jsp.default-default")
if c.Declared != 3 || c.WithoutSignIn != 1 || c.Others != 2 || c.InList != 0 {
t.Fatalf("three declared, the OAuth one not signed in, the person's and Owl's other (local folders not counted): %+v", c)
}
raw, _ := json.Marshal(c)
if strings.Contains(string(raw), "example") || strings.Contains(string(raw), "gmail") {
t.Fatalf("a count names nothing: %s", raw)
}
}
// ADR 0329 §1, §10, §12, §13: betterbird_accounts lists the accounts with their values, Owl's by name and address
// and marked; its propose carries the mesh-wide layer as it stands with accounts in it, and answers the value.
func TestBetterbirdAccountsListsTheValuesAndProposesThemInTheLayerAsItStands(t *testing.T) {
f := newFake(t)
f.profileWithAccounts()
f.write(profileDir+"/logins.json", `{"logins":[{"hostname":"imap://imap.example.org","encryptedUsername":"MDoEEPgAAAAA","encryptedPassword":"hunter2secret"}]}`)
links(t, nil)
a, err := f.ProfileAccounts(false)
if err != nil {
t.Fatal(err)
}
if a.Declarable["imap"] != 1 || a.KeptByOwl != 1 || a.NotDeclarable["local folders"] != 1 || a.Proposed != "" {
t.Fatalf("%+v", a)
}
raw, _ := json.Marshal(credentialChecked(a))
for _, shown := range []string{"other@mail.example.org", "imap.example.org", "smtp.example.org", `"port":993`, `"work@corp.example.com"`} {
if !strings.Contains(string(raw), shown) {
t.Errorf("the answer shows the operator's own %s (ADR 0329 §1): %s", shown, raw)
}
}
for _, never := range []string{"hunter2", "MDoEEP"} {
if strings.Contains(string(raw), never) {
t.Errorf("the answer carries the login store's %q", never)
}
}
for _, l := range a.Accounts {
if l.Key == "account3" && (!l.Owl || l.Value != nil || l.Address != "work@corp.example.com") {
t.Errorf("Owl's account is listed by name and address, marked, never as a value to declare: %+v", l)
}
}
var asked []map[string]any
asker = func(key string, body any) (json.RawMessage, error) {
if key != "seat:mesh-controller.settings" {
t.Fatalf("asked %s", key)
}
args := body.(map[string]any)
asked = append(asked, args)
if args["propose"] == nil {
return json.RawMessage(`{"ok": true, "output": "{\n \"tags\": {\"$label1\": {\"name\": \"Important\"}}\n}\n approved by the operator\nbetterbird on the whole mesh, every value"}`), nil
}
return json.RawMessage(`{"ok": true, "output": "proposed p-17 for betterbird on the whole mesh: accounts (other@mail.example.org); fingerprint ab12"}`), nil
}
t.Cleanup(func() { asker = nil })
a, err = f.ProfileAccounts(true)
if err != nil {
t.Fatal(err)
}
if len(asked) != 2 || asked[1]["module"] != "betterbird" || asked[1]["propose"] != "true" || asked[1]["node"] != nil ||
asked[1]["replace"] != nil {
t.Fatalf("the layer read, then proposed for the whole mesh, never set, never removing: %v", asked)
}
var layer map[string]json.RawMessage
if err := json.Unmarshal([]byte(asked[1]["values"].(string)), &layer); err != nil || string(layer["tags"]) == "" {
t.Fatalf("the proposal carries the layer as it stands: %v %s", err, asked[1]["values"])
}
value, err := parseAccounts([]byte(asked[1]["values"].(string)))
if err != nil {
t.Fatalf("the proposed value is one the module takes: %v", err)
}
other, ok := value["mail"]
if len(value) != 1 || !ok || other.Address != "other@mail.example.org" || other.Incoming.SignIn != "oauth" ||
other.Incoming.Port != 993 || other.Outgoing.Security != "tls" || other.Outgoing.Port != 465 {
t.Fatalf("the value holds the person's account and not Owl's: %+v", value)
}
if a.Derived["mail"].Address != "other@mail.example.org" {
t.Fatalf("the answer shows the value it proposed (ADR 0329 §10): %+v", a.Derived)
}
if strings.Contains(a.Proposed, "other@") || !strings.Contains(a.Proposed, "p-17") {
t.Fatalf("the controller's words come back with every address masked: %s", a.Proposed)
}
}
func TestAnAdoptedAccountNeverWritesIntoAnOutgoingServerAnotherIdentityUses(t *testing.T) {
f := newFake(t)
f.profileWithAccounts()
f.noPackage()
links(t, nil)
f.declare("// none")
os.RemoveAll(filepath.Join(f.Root, "proc", "4000"))
// A second account of the person's, not declared, sends through smtp1 too.
prefs := strings.Replace(f.read(profileDir+"/prefs.js"), `"account1,account3,account2"`, `"account1,account3,account2,account5"`, 1) +
`user_pref("mail.account.account5.server", "server5");` + "\n" +
`user_pref("mail.account.account5.identities", "id5");` + "\n" +
`user_pref("mail.identity.id5.useremail", "fifth@mail.example.org");` + "\n" +
`user_pref("mail.identity.id5.smtpServer", "smtp1");` + "\n" +
`user_pref("mail.server.server5.type", "imap");` + "\n"
f.write(profileDir+"/prefs.js", prefs)
f.declareAccounts(adoptable)
err := f.Prepare()
if err == nil || !strings.Contains(err.Error(), "also used by identity id5") || strings.Contains(err.Error(), "example.org") {
t.Fatalf("refused with the remedy, by keys: %v", err)
}
if user, _ := os.ReadFile(filepath.Join(f.Root, profileDir, "user.js")); strings.Contains(string(user), "smtpserver.smtp1") {
t.Fatal("the shared outgoing server was written")
}
// Recorded adoption, then the person moves the identity to a server of its own: the module follows it.
f.write(profileDir+"/prefs.js", strings.Replace(prefs, `"mail.identity.id5.smtpServer", "smtp1"`, `"mail.identity.id5.smtpServer", "smtp5"`, 1))
if err := f.Prepare(); err != nil {
t.Fatal(err)
}
moved := strings.Replace(f.read(profileDir+"/prefs.js"), `"mail.identity.id1.smtpServer", "smtp1"`, `"mail.identity.id1.smtpServer", "smtp9"`, 1) +
`user_pref("mail.smtpserver.smtp9.hostname", "smtp.example.org");` + "\n"
f.write(profileDir+"/prefs.js", moved)
u, err := f.userJS(true)
if err != nil || u.Each[0].How != adopted || u.Each[0].Keys.SMTP != "smtp9" {
t.Fatalf("the identity's current outgoing server is followed: %+v %v", u.Each, err)
}
user := f.read(profileDir + "/user.js")
if strings.Contains(user, "smtpserver.smtp1.") || !strings.Contains(user, "smtpserver.smtp9.hostname") {
t.Fatalf("never the old one:\n%s", user)
}
// And the move back onto a shared server is refused again.
f.write(profileDir+"/prefs.js", strings.Replace(moved, `"mail.identity.id5.smtpServer", "smtp5"`, `"mail.identity.id5.smtpServer", "smtp9"`, 1))
if err := f.Prepare(); err == nil || !strings.Contains(err.Error(), "also used by identity id5") {
t.Fatalf("%v", err)
}
}
func TestTheUserJSAnswerWithholdsTheAccountsValues(t *testing.T) {
f := newFake(t)
f.profileWithAccounts()
f.noPackage()
links(t, nil)
f.declare(`user_pref("mail.compose.default_to", "x");`)
f.declareAccounts(personal)
u, err := f.userJS(false)
if err != nil {
t.Fatal(err)
}
for _, never := range []string{"example.org", `"me"`, `"Me"`} {
if strings.Contains(u.Content, never) {
t.Errorf("the answer carries %s:\n%s", never, u.Content)
}
}
if !strings.Contains(u.Content, `user_pref("mail.server.server_mesh_personal.hostname", "<withheld>");`) {
t.Fatalf("an account's preference is named and its value withheld:\n%s", u.Content)
}
// What is placed in the profile keeps the values: only the answer withholds them.
if f.read(testHome+"/"+accountsFile) == "" {
t.Fatal("no declaration")
}
}
func TestACredentialRefusalNamesNoAccount(t *testing.T) {
_, err := parseAccounts([]byte(`{"accounts": {"personal": {"incoming": {"password": "hunter2"}}}}`))
if err == nil || strings.Contains(err.Error(), "personal") || !strings.Contains(err.Error(), "accounts.<an account>.incoming.password") {
t.Fatalf("%v", err)
}
}
func TestAnAccountsOwnAliasesAndNonMailIdentitiesDoNotShareItsOutgoingServer(t *testing.T) {
f := newFake(t)
f.profileWithAccounts()
f.noPackage()
links(t, nil)
f.declare("// none")
os.RemoveAll(filepath.Join(f.Root, "proc", "4000"))
// account1 has an alias identity on smtp1; smtp1 is the default server; Owl's account and local folders
// have identities that name no outgoing server.
prefs := strings.Replace(f.read(profileDir+"/prefs.js"), `"mail.account.account1.identities", "id1"`, `"mail.account.account1.identities", "id1,id1b"`, 1)
prefs = strings.Replace(prefs, `"mail.account.account2.server", "server2");`, `"mail.account.account2.server", "server2");`+"\n"+
`user_pref("mail.account.account2.identities", "id2");`, 1) +
`user_pref("mail.identity.id1b.useremail", "alias@mail.example.org");` + "\n" +
`user_pref("mail.identity.id1b.smtpServer", "smtp1");` + "\n" +
`user_pref("mail.identity.id2.useremail", "local@mail.example.org");` + "\n" +
`user_pref("mail.smtp.defaultserver", "smtp1");` + "\n"
f.write(profileDir+"/prefs.js", prefs)
f.declareAccounts(adoptable)
if err := f.Prepare(); err != nil {
t.Fatalf("an account's own alias, and identities that do not send by SMTP, are no sharing: %v", err)
}
// An identity of another mail account naming no server does use the default one.
f.write(profileDir+"/prefs.js", strings.Replace(prefs, `"account1,account3,account2"`, `"account1,account3,account2,account6"`, 1)+
`user_pref("mail.account.account6.server", "server6");`+"\n"+
`user_pref("mail.account.account6.identities", "id6");`+"\n"+
`user_pref("mail.identity.id6.useremail", "six@mail.example.org");`+"\n"+
`user_pref("mail.server.server6.type", "pop3");`+"\n")
os.Remove(filepath.Join(f.Root, testHome, recordFile))
if err := f.Prepare(); err == nil || !strings.Contains(err.Error(), "also used by identity id6") {
t.Fatalf("%v", err)
}
}
// Review 121: the account lists wait for Betterbird to close by its profile's lock too, not only by its process.
func TestTheAccountListsWaitWhileTheProfileIsLocked(t *testing.T) {
f := newFake(t)
f.profileWithAccounts()
f.noPackage()
links(t, nil)
f.declare("// none")
f.declareAccounts(personal)
stopBetterbird(t, f)
f.write(profileDir+"/lock", "")
before := f.read(profileDir + "/prefs.js")
u, err := f.userJS(true)
if err != nil {
t.Fatal(err)
}
if !strings.HasPrefix(u.Lists, "not written") || !strings.Contains(u.Lists, "locked") || f.read(profileDir+"/prefs.js") != before {
t.Fatalf("with the profile locked and no Betterbird running, prefs.js is unchanged: %q", u.Lists)
}
os.Remove(filepath.Join(f.Root, profileDir, "lock"))
if u, err = f.userJS(true); err != nil || !strings.HasPrefix(u.Lists, "written") {
t.Fatalf("once unlocked the list is written: %q %v", u.Lists, err)
}
}

Some files were not shown because too many files have changed in this diff Show More