Compare commits

...
Author SHA1 Message Date
jochen b1f2776693 WIP: smb-server module and nfs-server's local-network grant (hq issue 613, ADR 0341)
Paused: the module's manifest test, README and a full merge-check run are
still to do; tests pass locally for both modules.
2026-10-13 11:24:03 +02:00
mesh-admin f35b067504 Merge pull request 'fail2ban: never ban the operator's own addresses, and say a failure from one (hq issue 238, ADR 0339)' (#266) from fix/238-the-operators-own-addresses into main 2026-10-13 09:05:49 +00:00
jochen 93d58f3740 fail2ban: never ban the operator's own addresses, and say a failure from one (hq issue 238, ADR 0339)
mesh/merge-gate head 93d58f3740a0 merged into trunk 5cd608ba8898 — pass: builds fail2ban → ace, g14, novox, shanks; no bus step; every machine composes wit…
mesh/repo-check head 93d58f3740a0 merged into trunk 5cd608ba8898 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-…
mesh/delivery failed: its walk failed, carrying its own commit: a gate on a first machine (what it carried put back), a build, a machine
An agent polling the forge from the operator's home got that home banned for a day and then four
weeks, and nothing in the house reached the control-node. Every jail spared only private ranges.

The mesh-wide setting own-addresses reaches the machine as a mergeable file whose own content is
the empty list, so the module composes while it is unset. A new watch process (root) writes its
entries into a list every jail's base ignoreip names as file:, which fail2ban reloads without a
restart; it checks every jail names that list and keeps each failure a jail ignored because of it
for a day. The new fail2ban_health tool judges the watch, so the controller raises a warning
naming node, jail, address and count. The ban verb refuses an address on the setting or on the
jail's ignore list before the daemon is asked.
2026-10-13 10:54:49 +02:00
mesh-admin 5cd608ba88 Merge pull request 'Deliver a merge whose walk said its modules matched what runs, instead of stopping it as stranded (hq issue 611)' (#265) from fix/611-an-unchanged-build-is-delivered into main 2026-10-13 08:25:47 +00:00
jochen 2f5e97030a Deliver a merge whose walk said its modules matched what runs, instead of stopping it as stranded (hq issue 611)
mesh/merge-gate head 2f5e97030a5c merged into trunk bc3b59227f46 — pass: builds mesh-delivery → novox; no bus step; every machine composes with the change …
mesh/repo-check head 2f5e97030a5c merged into trunk bc3b59227f46 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-…
mesh/delivery delivered
mesh/delivery-group group fix/611-an-unchanged-build-is-delivered delivered: every member is delivered
A walk that moved nothing because every module's recorded build source held none of the changed files now
says so per module (delivery.unchanged). Such a merge moves nothing (ADR 0270 §1): it is delivered, naming
the modules, not stopped 20 minutes later with the operator asked about a change already on every machine.
A module the walk neither built nor said unchanged still stops it (issue 474).
2026-10-13 08:18:25 +02:00
mesh-admin bc3b59227f Merge pull request 'A password, token or API key is the vault's, never a setting (hq issue 342)' (#264) from fix/342-a-secret-is-never-a-setting into main 2026-10-13 05:52:05 +00:00
jochen eb42806379 Make grafana prefer the vault's password, and let the secrets list only shrink (hq issue 342)
mesh/merge-gate head eb4280637906 merged into trunk 53e51864969f — pass: builds baserow, gitea, grafana, home-assistant, icecast, influxdb, letta, mailu, n…
mesh/repo-check head eb4280637906 merged into trunk 53e51864969f — pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): main's version judged it; the ch…
mesh/delivery delivered
mesh/delivery-group group fix/342-a-secret-is-never-a-setting delivered: every member is delivered
grafana's client logged in with a token setting ahead of the vault's admin password,
and its runtime config let a layer set token. The token setting is now ignored and
said, token is protected, and the vault's password comes first.

checks/secrets/known could gain a line hiding a newly unprotected key. merge-check.sh
now judges it against the trunk's copy at the merge base, and refuses an added line.
2026-10-13 07:43:08 +02:00
jochen fe2040c65e Keep every password, token and API key the vault's, never a setting (hq issue 342)
mesh/merge-gate head fe2040c65ea2 merged into trunk 53e51864969f — pass: builds baserow, gitea, grafana, home-assistant, icecast, influxdb, letta, mailu, n…
mesh/repo-check head fe2040c65ea2 merged into trunk 53e51864969f — pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): main's version judged it; the ch…
mesh/delivery-group group fix/342-a-secret-is-never-a-setting ready: every member ready, and composed together they pass
mesh/delivery superseded: a newer head of the same pull request
A setting outranked the vault: baserow, grafana, letta and unifi merged a layer's
password over the ${secret:...} in their runtime config, and the gitea,
home-assistant, icecast, influxdb, mailu and nextcloud clients read the settings
file before the vault's file. A secret in a setting cannot be rotated and is read
back by the settings verb.

The four files now keep their password protected, so the controller refuses a
setting of it. The six clients ignore a secret in the settings file and say so
once, by key and never with the value; icecast reads its admin password from the
vault's file. checks/secrets refuses a mergeable file whose secret a layer could
replace; searxng's nested server.secret_key is listed as known until the
controller protects a nested key.
2026-10-13 07:32:19 +02:00
mesh-admin 53e5186496 Merge pull request 'build-agent: the package registry is optional, and genesis hands it the forge it raised (hq issue 121, ADR 0336)' (#263) from fix/121-genesis-package-registry into main 2026-10-13 03:37:08 +00:00
jochen e0bc919d49 Let genesis assign the build agent before the package registry exists (hq issue 121, ADR 0336)
mesh/merge-gate head e0bc919d4945 merged into trunk f88e3452f8b8 — pass: builds build-agent → ace, g14, novox, shanks; no bus step; every machine composes …
mesh/repo-check head e0bc919d4945 merged into trunk f88e3452f8b8 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-…
mesh/delivery covered: a later merge that contains it was delivered: novox/mesh-catalog@2f5e97030a5c (merged as 5cd608ba into main, walk plan-17918799551…
mesh/delivery-group group fix/121-genesis-package-registry delivered: every member is delivered
build-agent's npm-package-registry becomes optional, so genesis's step 13 is no
longer refused with 'nothing provides npm-package-registry'. Until gitea is
assigned, the agent on genesis's node reads the genesis-package-registry
own-secret the installer seals at the packages step; it is issued outside the
mesh, so no other machine is given one. The binding and its credential are
composed only while bound (ADR 0330 §6), and no file names them otherwise.
2026-10-13 02:50:39 +02:00
mesh-admin f88e3452f8 Merge pull request 'betterbird: own Betterbird's whole configuration as settings, shown without a credential (hq issue 586, ADR 0329)' (#262) from fix/586-betterbird-owns-its-configuration into main 2026-10-13 00:25:41 +00:00
mesh-admin 827b89dbf1 Merge pull request 'Mark grafana's and home-assistant's soft requirements optional (hq issue 166, ADR 0330)' (#259) from fix/166-optional-requirements into main 2026-10-13 00:22:35 +00:00
jochen c9a711c259 betterbird: write the account lists only while the profile is closed, and never propose over an unreadable layer (hq issue 586, review 121)
mesh/merge-gate head c9a711c259ed merged into trunk 827b89dbf109 — pass: builds betterbird, sent nowhere; no bus step; 2 wait(s) for a person; every machin…
mesh/repo-check head c9a711c259ed merged into trunk 827b89dbf109 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-…
mesh/delivery delivered
writeAccountLists checked only for a running Betterbird and wrote without a sync;
it now waits for the profile's lock too and writes atomically. readLayer read any
answer that was not JSON as an empty layer, and a proposal replaces the layer
whole, so a changed wording would have dropped accounts; it now refuses. The
README says the rollout order that leaves no node on Betterbird's defaults.
2026-10-13 02:12:06 +02:00
jochen 1835f0efad betterbird: own Betterbird's whole configuration as settings, shown without a credential (hq issue 586, ADR 0329)
mesh/merge-gate head 1835f0efad0f merged into trunk 39e7a06e6293 — pass: builds betterbird, sent nowhere; no bus step; 2 wait(s) for a person; every machin…
mesh/repo-check head 1835f0efad0f merged into trunk 39e7a06e6293 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-…
mesh/delivery superseded: a newer head of the same pull request
The module owned only the accounts' server fields, so identities, signatures,
calendars, address books, filters and tags could differ between desktop nodes
unseen, and the operator's own preferences were shipped in the catalogue.

prefs becomes one entry per preference (a node's layer overrides one key) and the
defaults file keeps no value of the operator's; identities, calendars,
address-books, filters and tags become settings, adopted like accounts, filters
owned whole per account, and what Betterbird rewrites written only while it is
closed. New tools list each kind, derive the layers from every desktop node and
propose one at a time, and show drift; every answer passes a credential check.
2026-10-13 02:03:04 +02:00
mesh-admin 39e7a06e62 Merge pull request 'lab: runs name the one genesis bundle (issue 146)' (#261) from fix/146-lab-names-the-one-bundle into main 2026-10-12 23:22:05 +00:00
jochen 84911b94f3 Give the lab's runs the one genesis bundle (hq issue 146, ADR 0331)
mesh/merge-gate head 84911b94f3be merged into trunk 8f80cd471634 — pass: builds lab → shanks; no bus step; every machine composes with the change as it did…
mesh/repo-check head 84911b94f3be merged into trunk 8f80cd471634 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-…
mesh/delivery delivered
mesh-host now carries one genesis bundle, foundation-first-node.lock; the -nats one the
lab's runs named is deleted, so every run would have raised from a file that is gone.
2026-10-13 01:18:08 +02:00
jochen 4193eacec4 Mark grafana's and home-assistant's soft requirements optional (hq issue 166, ADR 0330)
mesh/merge-gate head 4193eacec459 merged into trunk 39e7a06e6293 — pass: builds grafana, home-assistant → ace; no bus step; every machine composes with the…
mesh/repo-check head 4193eacec459 merged into trunk 39e7a06e6293 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-…
mesh/delivery delivered
grafana runs without influxdb and home-assistant without the Servarr apps or a broker, yet a mesh
without them could not assign either. Their requirements are now optional: bound when a provider
runs, absent otherwise, with nothing installed for them. grafana's data source and its credential are
delivered only while influxdb-api is bound, into directories it mounts, so an absent file never
becomes a directory docker made in its place.
2026-10-13 00:41:30 +02:00
84 changed files with 9310 additions and 478 deletions
+3
View File
@@ -0,0 +1,3 @@
module git.novox.be/novox/mesh-catalog/checks/secrets
go 1.22
+4
View File
@@ -0,0 +1,4 @@
# Secrets nested below a key a mergeable file takes for its other values, which the controller cannot
# protect alone yet (it protects top-level keys): "<module> <resource> <dotted key>". This list only shrinks:
# a line that no longer names a finding is refused, and so is one the trunk's copy does not hold (hq issue 342).
searxng settings server.secret_key
+244
View File
@@ -0,0 +1,244 @@
// Command secrets refuses, in every module's manifest, a mergeable file that a setting could fill with a
// password, a token or an API key in place of the vault's (novox/hq issue 342).
//
// A password is not a setting: it comes from the key vault, and it can be rotated there. A file that says
// `"merge": "json"` takes the keys of the mesh-wide and the node's layers over its own content, so a key the
// content fills with `${secret:<name>}` is one a layer could replace, and the vault's value would lose to it
// until somebody removed it by hand. The controller refuses a setting of a key the file keeps as `protected`,
// naming the layer and the key, never the value. This check holds every such file to that: each top-level key
// whose value holds `${secret:…}`, anywhere below it, is either listed in the file's `protected`, or left out
// of the file's `takes-settings` (novox/hq ADR 0326), so that no layer reaches it.
//
// A secret nested below a key the file takes for its other values (`server.secret_key` below `server`) cannot
// be protected alone today: the controller protects top-level keys. Such a file is named in the file `known`
// beside this check, one line per key, until the controller refuses a nested key. The list only shrinks: a
// line there that no longer names a finding is refused, and so is a line the trunk's copy (the merge base's,
// given with -base) does not hold. While the trunk holds no copy, the lines in seed are the trunk's.
//
// go run . [-base <the trunk's known>] <catalogue root>
//
// Exits 1 on a finding.
package main
import (
"bufio"
"encoding/json"
"flag"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
)
// Finding is one key of one mergeable file a layer could fill in place of the vault.
type Finding struct {
Module, ID, Key string
}
func (f Finding) String() string { return f.Module + " " + f.ID + " " + f.Key }
// Check reads every manifest under root/modules and answers each secret-bearing key a layer could replace.
func Check(root string) ([]Finding, error) {
manifests, err := filepath.Glob(filepath.Join(root, "modules", "*", "module.json"))
if err != nil {
return nil, err
}
sort.Strings(manifests)
var out []Finding
for _, path := range manifests {
raw, err := os.ReadFile(path)
if err != nil {
return nil, err
}
var m struct {
Module string `json:"module"`
Resources []map[string]any `json:"resources"`
}
if err := json.Unmarshal(raw, &m); err != nil {
return nil, fmt.Errorf("%s: %w", path, err)
}
for _, r := range m.Resources {
if how, _ := r["merge"].(string); how == "" {
continue
}
id, _ := r["id"].(string)
content, _ := r["content"].(string)
var doc map[string]any
if strings.TrimSpace(content) == "" {
continue
}
if err := json.Unmarshal([]byte(content), &doc); err != nil {
return nil, fmt.Errorf("%s: %s merges as JSON and its content is not JSON: %w", m.Module, id, err)
}
protected := set(r["protected"])
takes, names := r["takes-settings"]
taken := set(takes)
for _, key := range sortedKeys(doc) {
for _, at := range secretsBelow(key, doc[key]) {
switch {
case protected[key]:
case names && !taken[key]:
default:
out = append(out, Finding{m.Module, id, at})
}
}
}
}
}
return out, nil
}
// secretsBelow answers the dotted path of every string at or below value that names a vault secret.
func secretsBelow(at string, value any) []string {
switch v := value.(type) {
case string:
if strings.Contains(v, "${secret:") {
return []string{at}
}
case map[string]any:
var out []string
for _, k := range sortedKeys(v) {
out = append(out, secretsBelow(at+"."+k, v[k])...)
}
return out
case []any:
var out []string
for i, e := range v {
out = append(out, secretsBelow(fmt.Sprintf("%s[%d]", at, i), e)...)
}
return out
}
return nil
}
func set(v any) map[string]bool {
out := map[string]bool{}
list, _ := v.([]any)
for _, e := range list {
if s, ok := e.(string); ok {
out[s] = true
}
}
return out
}
func sortedKeys(m map[string]any) []string {
keys := make([]string, 0, len(m))
for k := range m {
keys = append(keys, k)
}
sort.Strings(keys)
return keys
}
// Known reads the lines of the known list: "<module> <resource> <key>", # comments and blank lines skipped.
func Known(path string) (map[string]bool, error) {
f, err := os.Open(path)
if os.IsNotExist(err) {
return map[string]bool{}, nil
}
if err != nil {
return nil, err
}
defer f.Close()
out := map[string]bool{}
s := bufio.NewScanner(f)
for s.Scan() {
line := strings.TrimSpace(s.Text())
if line == "" || strings.HasPrefix(line, "#") {
continue
}
out[strings.Join(strings.Fields(line), " ")] = true
}
return out, s.Err()
}
// seed is the known list as it was first written, standing for the trunk's copy until the trunk holds one.
var seed = map[string]bool{
"searxng settings server.secret_key": true,
}
// Added answers the lines of known that base does not hold: a line may leave the list, never join it.
func Added(known, base map[string]bool) []string {
var out []string
for line := range known {
if !base[line] {
out = append(out, line)
}
}
sort.Strings(out)
return out
}
// Judge splits the findings into those refused and the known lines that no longer name one.
func Judge(found []Finding, known map[string]bool) (refused []Finding, stale []string) {
seen := map[string]bool{}
for _, f := range found {
seen[f.String()] = true
if !known[f.String()] {
refused = append(refused, f)
}
}
for line := range known {
if !seen[line] {
stale = append(stale, line)
}
}
sort.Strings(stale)
return refused, stale
}
// remedy says how a finding is put right.
func remedy(key string) string {
top, _, nested := strings.Cut(key, ".")
if !nested {
return fmt.Sprintf("list %q in the file's \"protected\", so the controller refuses a setting of it", top)
}
return fmt.Sprintf("the controller protects a top-level key only; list %q in the file's \"protected\" if "+
"nothing else below it is the operator's, or leave it out of the file's \"takes-settings\"", top)
}
func main() {
basePath := flag.String("base", "", "the trunk's copy of known (the merge base's); seed when absent")
flag.Parse()
root := "."
if flag.NArg() > 0 {
root = flag.Arg(0)
}
found, err := Check(root)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
known, err := Known("known")
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
base := seed
if *basePath != "" {
if base, err = Known(*basePath); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
added := Added(known, base)
for _, line := range added {
fmt.Printf("checks/secrets/known gains %q, which the trunk's copy does not hold: the list only shrinks — "+
"protect the key instead (novox/hq issue 342)\n", line)
}
refused, stale := Judge(found, known)
for _, f := range refused {
fmt.Printf("%s: %s fills %s with a vault secret, and a setting could replace it (novox/hq issue 342): %s\n",
f.Module, f.ID, f.Key, remedy(f.Key))
}
for _, line := range stale {
fmt.Printf("checks/secrets/known names %q, which no longer holds a secret a setting could replace: "+
"remove the line\n", line)
}
if len(refused) > 0 || len(stale) > 0 || len(added) > 0 {
os.Exit(1)
}
fmt.Printf("secrets: no setting can replace a vault secret in a mergeable file (%d known, nested)\n", len(known))
}
+125
View File
@@ -0,0 +1,125 @@
package main
import (
"os"
"path/filepath"
"testing"
)
func manifest(t *testing.T, root, module, resources string) {
t.Helper()
dir := filepath.Join(root, "modules", module)
if err := os.MkdirAll(dir, 0o755); err != nil {
t.Fatal(err)
}
body := `{"module":"` + module + `","resources":[` + resources + `]}`
if err := os.WriteFile(filepath.Join(dir, "module.json"), []byte(body), 0o644); err != nil {
t.Fatal(err)
}
}
func TestASecretASettingCouldReplaceIsRefused(t *testing.T) {
root := t.TempDir()
// Refused: the key holds the vault's secret and nothing keeps a layer off it.
manifest(t, root, "a", `{"id":"cfg","type":"file","merge":"json","content":"{\"password\": \"${secret:admin}\"}"}`)
// Refused: nested below a key the file takes.
manifest(t, root, "b", `{"id":"cfg","type":"file","merge":"json","takes-settings":["server"],`+
`"content":"{\"server\": {\"secret_key\": \"${secret:s}\", \"port\": 1}}"}`)
// Kept: protected.
manifest(t, root, "c", `{"id":"cfg","type":"file","merge":"json","protected":["password"],`+
`"content":"{\"password\": \"${secret:admin}\", \"user\": \"admin\"}"}`)
// Kept: the file names its keys, and not this one.
manifest(t, root, "d", `{"id":"cfg","type":"file","merge":"json","takes-settings":["ui"],`+
`"content":"{\"token\": \"${secret:t}\", \"ui\": {}}"}`)
// Kept: a file that does not merge takes no setting.
manifest(t, root, "e", `{"id":"cfg","type":"file","content":"{\"token\": \"${secret:t}\"}"}`)
// Refused: protecting another key does not protect this one; a list element counts too.
manifest(t, root, "f", `{"id":"cfg","type":"file","merge":"json","protected":["user"],`+
`"content":"{\"users\": [{\"password\": \"${secret:p}\"}]}"}`)
found, err := Check(root)
if err != nil {
t.Fatal(err)
}
want := []string{"a cfg password", "b cfg server.secret_key", "f cfg users[0].password"}
if len(found) != len(want) {
t.Fatalf("found %v, want %v", found, want)
}
for i, f := range found {
if f.String() != want[i] {
t.Fatalf("found %v, want %v", found, want)
}
}
}
func TestTheKnownListOnlyShrinks(t *testing.T) {
found := []Finding{{"b", "cfg", "server.secret_key"}, {"a", "cfg", "password"}}
refused, stale := Judge(found, map[string]bool{"b cfg server.secret_key": true, "gone cfg token": true})
if len(refused) != 1 || refused[0].String() != "a cfg password" {
t.Fatalf("refused %v", refused)
}
if len(stale) != 1 || stale[0] != "gone cfg token" {
t.Fatalf("stale %v", stale)
}
}
func TestTheCatalogueHoldsNoSecretASettingCouldReplace(t *testing.T) {
found, err := Check("../..")
if err != nil {
t.Fatal(err)
}
known, err := Known("known")
if err != nil {
t.Fatal(err)
}
refused, stale := Judge(found, known)
if len(refused) > 0 || len(stale) > 0 {
t.Fatalf("refused %v, stale %v", refused, stale)
}
}
func TestALineTheTrunkDoesNotHoldIsRefused(t *testing.T) {
trunk := map[string]bool{"b cfg server.secret_key": true}
// A key unprotected on the branch, and its line added to the list: refused, though it matches a finding.
branch := map[string]bool{"b cfg server.secret_key": true, "a cfg password": true}
if added := Added(branch, trunk); len(added) != 1 || added[0] != "a cfg password" {
t.Fatalf("added %v", added)
}
// A line removed is no addition.
if added := Added(map[string]bool{}, trunk); len(added) != 0 {
t.Fatalf("added %v", added)
}
// Until the trunk holds a copy, the seed stands for it, and the list as committed is within it.
known, err := Known("known")
if err != nil {
t.Fatal(err)
}
if added := Added(known, seed); len(added) != 0 {
t.Fatalf("known holds lines the seed does not: %v", added)
}
}
func TestTheCheckRefusesALineAddedAgainstTheTrunk(t *testing.T) {
root := t.TempDir()
manifest(t, root, "a", `{"id":"cfg","type":"file","merge":"json","content":"{\"password\": \"${secret:admin}\"}"}`)
dir := t.TempDir()
base := filepath.Join(dir, "base")
if err := os.WriteFile(base, []byte("# the trunk's\n"), 0o644); err != nil {
t.Fatal(err)
}
branch := map[string]bool{"a cfg password": true}
found, err := Check(root)
if err != nil {
t.Fatal(err)
}
trunk, err := Known(base)
if err != nil {
t.Fatal(err)
}
refused, stale := Judge(found, branch)
if len(refused) != 0 || len(stale) != 0 {
t.Fatalf("the line hides the finding from Judge alone: %v %v", refused, stale)
}
if added := Added(branch, trunk); len(added) != 1 {
t.Fatalf("an added line passed: %v", added)
}
}
+1 -1
View File
@@ -1,4 +1,4 @@
67
66
The long-running resources of this catalogue that do not say how they are ready (`health`, novox/hq ADR 0240
rule 8), as `module check` counts them. It may only go down: merge-check.sh fails a change that raises it, and
one that lowers it writes the new number on the first line. From 2026-11-18, or once it is 0, a long-running
+16
View File
@@ -30,11 +30,27 @@
# 5. the agent's home (novox/hq ADR 0266): no module unpacks an archive, or writes into a file, under
# ${machine:agent-home} — the node-engine does both after checking the path, and the agent account owns
# that home and could swap a link in between.
# 6. the secrets (novox/hq issue 342): no mergeable file fills a key with a vault secret that a setting could
# replace — the key is `protected`, or left out of the file's `takes-settings`. checks/secrets/known names
# the nested ones the controller cannot protect alone yet, and only shrinks.
set -eu
(cd checks/words && go test -count=1 . && go run . ../..)
# The agent's home (novox/hq ADR 0266): no archive, and no file written into, under it.
(cd checks/agenthome && go test -count=1 . && go run . ../..)
# The secrets (novox/hq issue 342): a password is not a setting, so no setting can replace the vault's. The
# known list only shrinks: judged against the trunk's copy at the merge base, a line it does not hold is refused.
secrets_base=""
secrets_ref="origin/${MESH_CHECK_BASE:-main}"
if merge_base=$(git merge-base HEAD "$secrets_ref" 2>/dev/null); then
if git cat-file -e "$merge_base:checks/secrets/known" 2>/dev/null; then
secrets_base=$(mktemp)
git show "$merge_base:checks/secrets/known" >"$secrets_base"
fi
else
echo "NOT COMPARED WITH THE TRUNK: no merge base with $secrets_ref; checks/secrets/known is held to the list it was first written with"
fi
(cd checks/secrets && go test -count=1 . && if [ -n "$secrets_base" ]; then go run . -base "$secrets_base" ../..; else go run . ../..; fi)
# The lists that narrow the paths check may not grow unseen: the base branch's copies are what they are
# judged against (MESH_CHECK_BASE, main when unset), read from the clone the check runs in.
base_lists=""
+2 -1
View File
@@ -103,7 +103,8 @@
"path": "${dir:mesh-state}/config.json",
"mode": "0600",
"content": "{\n \"password\": \"${secret:admin}\",\n \"host\": \"${bound:route:name}\"\n}\n",
"merge": "json"
"merge": "json",
"protected": ["password"]
}
],
"build": {
+129 -88
View File
@@ -1,10 +1,12 @@
# betterbird
Betterbird, the mail and calendar client of g14 and shanks (a Thunderbird fork, with the operator's
Exchange calendar through the Owl add-on), as a module (novox/hq issue 345, ADR 0208, ADR 0316, ADR 0317,
to-be 56). It requires `x11-display`, so it is assigned only where a display server is held on the same
machine. It installs Betterbird from a pinned upstream release, ships Owl with it through Betterbird's
enterprise policy, and declares the operator's mail accounts without their credentials.
Exchange calendar through the Owl add-on), as a module (novox/hq issue 345, issue 586, ADR 0208, ADR 0316,
ADR 0317, ADR 0329, to-be 56). It requires `x11-display`, so it is assigned only where a display server is held
on the same machine. It installs Betterbird from a pinned upstream release, ships Owl with it through
Betterbird's enterprise policy, and owns Betterbird's whole configuration as settings: the preferences, the
mail accounts without their credentials, the identities with their signatures, the CalDAV calendars, the
CardDAV address books, the filters and the tags.
It exists because a symptom had no tools: on a workstation, a calendar reminder appeared as a
transparent window with nothing in it (2026-10-09). Nothing in the mesh could say which window that
@@ -18,13 +20,14 @@ issue 345.
| Betterbird, the upstream release (`archive`, pinned by URL and sha256) | `/opt/betterbird` | owned, whole |
| the enterprise policy (`install/distribution/policies.json`) | `/opt/betterbird/distribution` | owned, whole |
| Owl's xpi (`owl`, pinned by the vendor's sha256) | `/opt/betterbird/owl` | owned, whole |
| the module's defaults (`install/defaults-pref/mesh-defaults.js`) and the spelling dictionaries' default (`vendor-prefs.js`) | `/opt/betterbird/defaults/pref` | owned: its two files |
| the module's defaults (`install/defaults-pref/mesh-defaults.js`: the preferences kept at Betterbird's own default, no value of the operator's) and the spelling dictionaries' default (`vendor-prefs.js`) | `/opt/betterbird/defaults/pref` | owned: its two files |
| the tools' program, for the start's preparation | `/usr/local/lib/mesh-betterbird` | owned, whole |
| the start | `/usr/local/bin/betterbird` | owned, whole |
| the launcher entry | `/usr/local/share/applications/eu.betterbird.Betterbird.desktop` | owned, whole |
| the module's directory, with its `user.js` (setting `prefs`), `accounts.json` (setting `accounts`) and the record of adopted and created accounts | `~/.config/betterbird/` | owned, whole |
| the module's directory, with the settings as the node-engine writes them (`prefs.json`, `accounts.json`, `configuration.json`, `tags.json`), the records of what was adopted or created, and the filter files it replaced (`kept/`) | `~/.config/betterbird/` | owned, whole |
| the profile's `user.js`, placed by the tools | the profile | the module's, or never touched |
| the account lists in the profile's `prefs.js` | the profile | written into: the module's keys only |
| the lists in the profile's `prefs.js` (accounts, outgoing servers, an adopted account's identities, the calendars' order), and the preferences of what the module created and the settings no longer hold | the profile | written into: the module's keys only, only while Betterbird is closed |
| each declared account's `msgFilterRules.dat` (setting `filters`) | the account's directory in the profile | owned, whole, only while Betterbird is closed |
| the profile's own values of the preferences the module manages, in `prefs.js` | the profile | cleared only by `betterbird_config_apply` with apply, Betterbird quit |
| the profile: mail, address books, calendars, logins | `~/.thunderbird/` | found (ADR 0182) |
@@ -69,97 +72,85 @@ box out in the settings. `betterbird_check` says when Betterbird would ask (novo
**No start of its own.** Betterbird is started by the operator from the launcher. The start
(`/usr/local/bin/betterbird`) first runs `betterbird-tools prepare`, which, with no Betterbird of the
account running, places the user.js and writes the created accounts into the account lists; then it
account running, places the user.js and writes what Betterbird rewrites itself (the lists, the removals, the
filter files); then it
starts Betterbird either way. `betterbird_restart` starts it as the transient user unit
`mesh-betterbird`, so its output then goes to the journal.
## The configuration this module ships
## The preferences: locks, the setting prefs, and Betterbird's own defaults
Betterbird's configuration comes in three layers (novox/hq issue 345). The operator approved the set on
2026-10-12. It is the laptop's setup, with the start page locked off, reminders missed while asleep shown at the
next start, and chat off.
Betterbird's preferences come in three layers (novox/hq issue 345, issue 586). **The operator's own choices
live in the module's configuration, never in the catalogue**: the module ships only its policy's locks and the
names it keeps at Betterbird's own default.
| layer | file | who changes it | holds |
| layer | where | who changes it | holds |
|---|---|---|---|
| **locked** | `/opt/betterbird/distribution/policies.json` (`install/distribution/`) | only a reviewed catalogue change | what must never differ on a node; Betterbird's settings show it greyed out |
| **default** | `/opt/betterbird/defaults/pref/mesh-defaults.js` (`install/defaults-pref/`) | a catalogue change; or the operator in Betterbird's settings, where the change lands in the profile's `prefs.js` | the operator's setup: what a fresh node gets without anyone touching it |
| **node** | the setting `prefs`, written to `~/.config/betterbird/user.js` and placed in the profile's `user.js` | the operator, per node, through the settings layers | a node's exceptions, such as a font size for a larger screen |
| **declared** | the setting `prefs`, mesh-wide with a node's layer over it, written to `~/.config/betterbird/prefs.json` and placed in the profile's `user.js` | the operator, through the settings layers (trusted: the terminal or the operator's warrant) | the operator's setup, and a node's exceptions, such as a font size for a larger screen |
| **Betterbird's default** | Betterbird's own defaults; `mesh-defaults.js` names those the module manages there | a catalogue change | the preferences kept at Betterbird's default |
A locked value wins over everything. A node layer (`user.js`) wins over the profile's `prefs.js`, and
`prefs.js` wins over a default.
A locked value wins over everything; a declared one (in `user.js`) wins over the profile's `prefs.js` at every
start; `prefs.js` wins over a default.
**The setting `prefs` is one entry per preference** (issue 586): `{"prefs": {"mail.uifontsize": 14, …}}`, the
value true, false, a whole number or a string. The node-engine merges the layers key by key, so a node's layer
overrides one preference, or takes one out with `null`. The operator's setup approved on 2026-10-12 (compact
density, 14-point text, the vertical view with multi-line cards and threads, no new-mail pop-up, sound or tray
icon, the calendar's day from 09:00 to 20:00, reminders missed while asleep shown, chat off, remote content
blocked, Do Not Track; 21 preferences) moved from the catalogue's defaults file into the mesh-wide layer of
`prefs`, set on the operator's Approve.
**The order that leaves no node on Betterbird's own defaults** (issue 586): merge this build (the `record`
upgrade policy holds it back from the nodes); propose the mesh-wide layer through the controller's `settings`
verb with `propose`, carrying the current `accounts` verbatim and `prefs` with the 21 values; the operator
approves; only then push the desktop nodes. Caveat: between that approval and a node's push, any other send to a
desktop node writes the map's text into the old build's user.js, and `betterbird_check` there says the user.js is
refused until the push.
**Locked** (the policy): Betterbird's updater off (`DisableAppUpdate`); telemetry off (`DisableTelemetry`);
add-ons change only through the mesh's pins (`ExtensionUpdate` false); and three preferences locked off:
`mail.shell.checkDefaultClient` (the ask to be the default client), `mail.inappnotifications.enabled` (the
vendor's in-app notices), and `mailnews.start_page.enabled` (the start page, which otherwise loads from the
vendor at every start). The `Preferences` policy accepts only some prefixes. `privacy.` and `toolkit.` are not
among them, so such a preference can only be a default.
vendor at every start).
**Default** (`mesh-defaults.js`, each line commented with why):
**Kept at Betterbird's default** (`// keeps Betterbird's default: <name>` in `mesh-defaults.js`): the theme,
system colours, always-underlined links, manual spam marking, and the composer's attachment pane. A value the
profile saved for one of them is drift.
- display: compact density (`mail.uidensity` 0), 14-point interface text, and message bodies not darkened;
- layout: the vertical view with multi-line rows, the list as cards, threads grouped;
- no confirmation on Shift+Delete;
- new mail: no pop-up, no sound, no tray icon, and alerts through the desktop's notifier if a node turns them on;
- calendar: the day from 09:00 to 20:00, deleting without asking, the node's time zone, reminders shown, and
reminders missed while asleep shown at the next start;
- chat off;
- remote content blocked, and Do Not Track sent.
**Drift.** `betterbird_config` shows each managed preference (locked, declared, kept) through the layers, and
flags **drift**: a declared value the profile's `user.js` does not hold yet (remedy: `betterbird_user_js` with
apply, then `betterbird_restart`), or a value the profile saved where the module keeps Betterbird's default.
`betterbird_check` reports it.
A line `// keeps Betterbird's default: <name>` names a preference the module manages without a value of its own.
For these the operator's setup is Betterbird's own default: the theme, system colours, always-underlined
links, manual spam marking, and the composer's attachment pane.
**Drift.** A default does not beat a value the profile saved in `prefs.js`. That value may be a choice made in
the settings window, or one the profile already held. `betterbird_config` shows each managed preference through
the layers, and flags **drift**: a saved value that beats the module's default without a node layer saying so.
`betterbird_check` reports it. There are two remedies:
- `betterbird_config_apply` with `apply` takes the drifting values out of `prefs.js`, so the module's value
holds at the next start. It acts only while Betterbird is quit (no main process, and no `lock` in the profile),
because Betterbird rewrites `prefs.js` when it quits, and it says what it cleared. Without `apply` it lists
what it would clear.
- `betterbird_config_apply` with `apply` takes a kept preference's saved value out of `prefs.js`, so Betterbird's
default holds at the next start. It acts only while Betterbird is quit (no main process, and no `lock` in the
profile), and says what it cleared.
- **The theme is the exception.** Betterbird keeps the active theme in its add-on database (`extensions.json`),
and `extensions.activeThemeID` only mirrors it. So the tools read the theme from the add-on database, and the
module's value is `default-theme@mozilla.org`. Another theme is drift whose remedy is to enable the default
and `extensions.activeThemeID` only mirrors it. Another theme is drift whose remedy is to enable the default
theme in the add-on manager. `betterbird_config_apply` never clears it, and `keep` refuses it.
- `betterbird_config` with `keep` proposes this node's layer of `prefs` holding the profile's values of the named
preferences beside what the layer holds, through the controller's settings verb, for the operator's Approve.
**Never an account.** A name the tools withhold (a credential's, an account's, a server's), or an account's own
preference, is never managed, even when the defaults file lists it. `betterbird_config` leaves it out and names it
in `refused`, `betterbird_config_apply` never clears it, and `betterbird_check` says to take it out of the file.
- `betterbird_config` with `keep` proposes, through the controller's settings verb, a node layer of `prefs` that
holds the profile's value for this machine. The operator approves it on a channel that proves who answers.
The tool never writes `user.js` itself. The setting is one string, so the layer carries what `prefs` says on
this machine today, with the kept values added.
**One owner per value.** `prefs` refuses a preference whose name can hold a credential, an account's
(`mail.account.`, `mail.identity.`, `mail.server.`, `mail.smtpserver.` and the account lists, ADR 0316 §11), and
those the settings below own: `calendar.registry.`, `calendar.list.`, `ldap_2.servers.` and `mailnews.tags.`
(ADR 0329 §6).
## The user.js: how a preference is declared
## The user.js
Betterbird reads `user.js` in its profile when it starts, and a preference set there wins over the
one it saved in `prefs.js`. The profile's directory has a random name per machine
(`<random>.default-default`), which a manifest cannot name. So:
1. the node-engine writes `~/.config/betterbird/user.js` from the module's setting `prefs`, one line of
`user_pref("name", value);` statements (default: a comment, no preference);
2. the module's tools copy that file whole into the profile: `betterbird_user_js` with `apply`, or
`betterbird_restart`, which places it before it starts Betterbird again;
3. Betterbird reads it at that start.
Betterbird reads `user.js` in its profile when it starts, and a preference set there wins over the one it saved
in `prefs.js`. The profile's directory has a random name per machine (`<random>.default-default`), which a
manifest cannot name. So the tools compose the user.js from the settings — `prefs`, then the declared accounts,
then the declared identities, calendars, address books and tags under the keys each resolved to — and place it
in the profile: at the start from the launcher, `betterbird_user_js` with `apply`, or `betterbird_restart`.
**The profile is found** as the one the running Betterbird holds open; with none running, as the
installation's default in `profiles.ini` (`[Install…] Default`), else the profile marked `Default=1`.
On the workstation measured these differ: `Profile1` is marked `Default=1`, and the installation's
default, the one in use, is the other.
**A user.js the module did not write is never overwritten.** The module's begins with the line
`// Betterbird's user.js (module betterbird`; any other is the operator's, and `apply` refuses with
what to do. The tools refuse a declared file that holds anything but comments and `user_pref`
statements, or that sets a preference whose name can hold a credential.
`// Betterbird's user.js (module betterbird`; any other is the operator's, and `apply` refuses with what to do.
The file is **trusted** (`"trusted": true`): a preference can weaken how the mail client checks a server,
so the setting is changed only at the controller's terminal or proposed and set on the operator's warrant
(novox/hq issue 339, ADR 0277). An account's preferences (`mail.account.`, `mail.identity.`, `mail.server.`,
`mail.smtpserver.` and the account lists) are refused in `prefs`: an account is declared in `accounts`, so
it has one owner (ADR 0316 §11).
## The accounts: declared without credentials (ADR 0316 §10–§16, ADR 0317)
The setting `accounts` reaches the machine as `~/.config/betterbird/accounts.json`, a trusted mergeable
@@ -205,32 +196,80 @@ lines and its record go, the account stays the person's) and takes a created acc
lists (its mail stays in the profile).
**The value is derived, not retyped** (§9): `betterbird_accounts` with `propose` builds `accounts` from the
profile (every account but Owl's, no credential) and proposes it through the controller's settings verb;
the operator reads it whole at their desk (`settings proposals <id>`) and approves on a channel that proves
who answers. Its answer to its caller holds counts and kinds only.
profile (every account but Owl's, no credential) and proposes it in the mesh-wide layer as it stands, through
the controller's settings verb; the operator approves on a channel that proves who answers. Its answer shows
the value it proposed (ADR 0329 §10).
**A node with no profile yet.** The preparation needs the profile Betterbird makes at its first start: on a
node set up from nothing, the declared accounts are written at the second start from the launcher (or at
`betterbird_restart`).
## Identities, calendars, address books, filters and tags (ADR 0329)
Five more settings, mesh-wide with a node's layer over each (an entry set to `null` in a node's layer is not on
that node). `identities`, `calendars`, `address-books` and `filters` reach the machine as
`~/.config/betterbird/configuration.json`, trusted as `accounts` is; `tags` as `tags.json`, not trusted (§5).
**No value holds a password, a token or a cookie**: a key shaped like one is refused, and so is an address whose
user part carries a password (§2). A setting holds no line break, so a signature is a list of its lines.
| setting | an entry | matched in the profile by |
|---|---|---|
| `identities` | by a name: `account` (its name in `accounts`), and for another identity than the account's default `address` and `name`; `reply-to`, `organization`, `signature` (lines), `signature-html`, `signature-above-quote`; `key` in a node's layer | the account's default (no `address`), or an identity of that account with the address |
| `calendars` | by a name: `name`, `url` (CalDAV), `color`, `read-only`, `reminders`, `user`, `invitations` (an identities entry); `key` in a node's layer | its address |
| `address-books` | by a name: `name`, `url` (CardDAV), `user`; `key` in a node's layer | its address |
| `tags` | by its key: `name`, `color`, `order` | its key |
| `filters` | by an account's name: the ordered list, each with `name`, `enabled`, `when` (new-mail, manual, after-junk, sending, archiving, periodic), `match` (all, any, every-message), `conditions` (`field`, `op`, `value`, in Betterbird's words) and `actions` (`action` in Betterbird's words; `folder` as a path within the account, or `value`) | owned whole per account |
**Adopted as accounts are** (§7): an entry recorded on this node is kept by its key; one under the module's own
key (`ident_mesh_<name>`, `mesh_<name>` for a calendar or an address book) is its creation; a `key` in the
node's layer is adopted when the profile holds it and it matches; else one match is adopted, none is created
under the module's key, and several are refused with the remedy. The declared fields go into the user.js under
that key. An entry leaving the setting releases what was adopted (its lines go) and removes the preferences of
what was created, never a calendar's events, an address book's contacts or a cache. An entry no setting declares
stays the person's. Owl's accounts and calendars, an ICS calendar (its address is often the secret), local
calendars and address books are never declared (§13).
**Filters are owned whole per declared account** (§8): the module writes that account's `msgFilterRules.dat`
from the list, each folder's path turned into its address on this node, so a filter added in Betterbird is gone
at the next apply (drift shows it first). Before its first write it keeps the replaced file in
`~/.config/betterbird/kept/<profile>/`; when the account leaves `filters`, the module stops writing and the file
stays the person's; `betterbird_filters` says where the kept copy is. A filter a setting cannot hold as it is (a
folder of another account, an add-on's action, a news filter) is named and not derived.
**Written only while Betterbird is closed** (§9): the lists in `prefs.js` and the filter files, because
Betterbird rewrites them as it runs and quits; the user.js is placed as before. `betterbird_check` counts per
kind the changes that wait.
**Derived, then proposed one layer at a time** (§10): `betterbird_derive` asks every desktop node holding
Betterbird for what its profile holds. An entry on every such node goes into the mesh-wide layer with the value of
the node the derive was called on, and the answer lists each field another node holds differently (which then
shows as drift there); one on some nodes only goes into those nodes' own layers. An account whose filter file
holds no filter is not put in `filters`. With `propose`, the next layer (the mesh-wide one, then each node's) is
proposed with the layer as it stands beside the derived kinds; never while a proposal of the module's waits, and
never while a desktop node does not answer. `prefs` on the call carries values of the setting `prefs` into the
mesh-wide layer. A `key` is never proposed for the mesh-wide layer.
## Tools
They are served by the node's runtime as the operator account (ADR 0175). **No answer carries mail, a
calendar entry, an address, a host, an account's name or a credential**: accounts are said by count, place
and key.
They are served by the node's runtime as the operator account (ADR 0175). **They show the operator's own
configuration with its values, and never a credential** (ADR 0329 §1, §2): every answer passes the module's
credential check, which withholds a field whose name says password, token, secret or cookie and the password
part of any address. No answer carries mail, a calendar entry or a contact.
| tool | does |
|---|---|
| `betterbird_status` (r) | whether it runs (pid, since, scope, children); what is installed and from where (the mesh's release, or the community package before the switch), its version and build beside the newest published build (Betterbird's own `versions.txt`); the policy; Owl's version against the pin and whether it is active; where its output goes; the profile and how it was found; the user.js (declared, placed, in effect); the declared accounts by count and, per place, adopted, created or refused with their keys; the add-ons in the profile (ids, versions); the preferences that decide how its windows and notifications look |
| `betterbird_prefs` (r) | named preferences or prefixes (ending in `.` or `*`, at most 200; an empty name, `*` or `.` is refused): value and source (the policy's lock, user.js, prefs.js or the default), and the default itself with the file that gives it. Betterbird's own defaults are read from `omni.ja` (`greprefs.js`, `defaults/pref/*.js`) and the installation's `defaults/pref`, the module's from its files there. Of several files, the one first by name holds, as Gecko reads them. A preference compiled into Betterbird is said as built in. Withheld without its value: a name that can hold a credential (`password`, `token`, `oauth`, `signon`, …); one that names an account, server, calendar, recent folder or person (`mail.server.`, `mail.identity.`, `mail.last_msg…`, `messenger.account.`, `mail.cloud_files.`, `calendar.registry.`, and of `mailnews.*` only a name that holds a server, account, identity or tag); and every `extensions.*` but the theme's id (`extensions.activeThemeID`). In every value answered, the home is `~`, and a mail address, a `user%40domain@host` URI and a host name are masked |
| `betterbird_config` (r; a with `keep`) | each managed preference: its lock, the module's default (or Betterbird's, where the module keeps that), the node layer, the profile's saved value, the value in effect, and drift; with `keep`, a node layer holding the profile's values proposed for this machine |
| `betterbird_config_apply` (r; a with `apply`) | the drifting values listed; with `apply`, while Betterbird is quit, taken out of `prefs.js`, each said with the value it had and the one that now holds |
| `betterbird_user_js` (r; a with `apply`) | the declared user.js against the profile's; `apply` places it |
| `betterbird_log` (r) | the last lines of the journal of `mesh-betterbird` (or of the reminder test), addresses and credentials masked (the value after `password=`, `token:` and the like, and the token after `Bearer`/`Basic`). A Betterbird the module did not start prints elsewhere, and the answer says where |
| `betterbird_restart` (a) | ends Betterbird (asked to quit, forced after 8 s), places the user.js where allowed, starts it as `mesh-betterbird`. **Refused while a composer window is open** (X instance `Msgcompose`, whatever its class), because the unsent message would be lost, unless `discard_unsent` is true, which only the operator's yes justifies. It answers within the runtime's 30 s: a start not yet seen after 4 s is said in `note`, and `betterbird_status` shows it |
| `betterbird_check` (r) | installed from the mesh's release and not behind the newest published build; the policy placed, and the module's defaults with it; no managed preference drifting from them; no ask at the start to be the default mail client; Owl at its pin and active; one running; profile found; the user.js on the machine, valid, placed, in effect; the declared accounts in the list, none refused, each signed in once; its output read |
| `betterbird_accounts` (r; a with `propose`) | the profile's accounts by count and kind (declarable, Owl's, others); with `propose`, the setting `accounts` built from them and proposed to the operator (ADR 0317 §9) |
| `betterbird_status` (r) | whether it runs (pid, since, scope, children); what is installed and from where, its version beside the newest published build; the policy; Owl's version against the pin and whether it is active; where its output goes; the profile and how it was found; the user.js (declared, placed, in effect, with every declared entry as it resolved); the declared accounts by count; the add-ons; the preferences that decide how its windows and notifications look |
| `betterbird_prefs` (r) | named preferences or prefixes: value and source, and the default with the file that gives it. Withheld without its value: a name that can hold a credential, one that names an account, server, calendar or person, and every `extensions.*` but the theme's id. Values masked (the home as `~`, addresses and hosts) |
| `betterbird_config` (r; a with `keep`) | each managed preference: its lock, the declared value, Betterbird's default where the module keeps that, the profile's saved value, the value in effect, and drift; with `keep`, this node's layer of `prefs` proposed with the profile's values |
| `betterbird_config_apply` (r; a with `apply`) | a kept preference's drifting values listed; with `apply`, while Betterbird is quit, taken out of `prefs.js` |
| `betterbird_user_js` (r; a with `apply`) | the user.js composed from the settings against the profile's; `apply` places it, writes what Betterbird rewrites while it is closed, and counts what waits |
| `betterbird_log` (r) | the last lines of the journal of `mesh-betterbird` (or of the reminder test), addresses and credentials masked |
| `betterbird_restart` (a) | ends Betterbird (asked to quit, forced after 8 s), places the user.js and writes what waited, starts it as `mesh-betterbird`. **Refused while a composer window is open**, unless `discard_unsent` is true, which only the operator's yes justifies |
| `betterbird_check` (r) | installed from the mesh's release; the policy and the module's defaults placed; no managed preference drifting; no ask to be the default mail client; Owl at its pin and active; one running; profile found; the user.js valid, placed, in effect; the declared accounts in the list, none refused, each signed in once; per kind, the declared entries drifting or refused and the changes waiting for Betterbird to close; its output read |
| `betterbird_accounts` (r; a with `propose`) | the profile's accounts with their values and the declared account each is, Owl's by name and address and marked; with `propose`, `accounts` built from them and proposed in the mesh-wide layer as it stands |
| `betterbird_identities`, `betterbird_calendars`, `betterbird_address_books`, `betterbird_filters`, `betterbird_tags` (r) | the profile's entries with their keys and values beside the declared ones and how each resolved (adopted, created or refused with why); an ICS calendar by its name only, Owl's marked; `betterbird_filters` also where the kept filter files are |
| `betterbird_derive` (r; a with `propose`) | the five kinds derived from the desktop nodes' profiles, per layer, with the fields another node holds differently and what was left out; with `propose`, the next layer proposed; `here` answers this node's profile, as another node asks it |
| `betterbird_drift` (r) | per kind and declared entry: holds, differs (each field with both values), lacks, or refused; and the profile's entries nothing declares |
| `betterbird_reminder_test` (a) | a reminder on demand without the operator's calendar (below) |
### A reminder on demand
@@ -293,9 +332,9 @@ outside the mesh. With this version, a push to a node (each waits for the operat
| `/opt/betterbird` | the mesh's 153.4.0esr-bb10, in the same place, so the same profile opens |
| the policy, Owl | placed; Owl installed into the profile by Betterbird at its next start where it is missing |
| `/usr/local` | the start, the launcher entry, the tools' program |
| `~/.config/betterbird/` | `user.js`, `accounts.json` (no account until the setting `accounts` is approved) |
| `~/.config/betterbird/` | `prefs.json`, `accounts.json`, `configuration.json`, `tags.json` (each empty until its setting is approved) |
| the profile | untouched until the next start from the launcher, `betterbird_user_js` with apply, or `betterbird_restart` |
| the configuration | the policy's locks at Betterbird's next start; the module's defaults wherever the profile saved no value of its own; a saved value is drift, said by `betterbird_check`, until `betterbird_config_apply` or a node layer |
| the configuration | the policy's locks at Betterbird's next start; the settings in the user.js from the next start. Pushed in the order above, the operator's 21 preferences are in `prefs` before the build arrives; pushed before that layer is approved, Betterbird runs with its own defaults for them until it is |
## Relies on
@@ -305,4 +344,6 @@ outside the mesh. With this version, a push to a node (each waits for the operat
- `systemd-run` and `journalctl` of the account's user manager.
- The build seat's archive pinned by URL and sha256 (mesh-controller, ADR 0316 §2): a controller that does
not know it refuses this manifest.
- The controller's `settings` verb, for `betterbird_accounts` with `propose` (`invokes`).
- The controller's `settings` verb, for every proposal and to read a layer as it stands; its `nodes` and `node`
verbs, for which machines hold the module; and the module's own `betterbird_derive` on those machines
(`invokes`).
@@ -398,10 +398,6 @@ func unquote(v string) string {
// (§12): the declared accounts' keys added at the end, the module's keys of accounts no longer declared taken
// out, every other key and line kept as it was. Only while Betterbird is not running. Says what changed.
func (m *Machine) writeAccountLists(profile string, resolved []resolvedAccount) (string, error) {
if ps := m.mainProcs(); len(ps) > 0 {
return "", fmt.Errorf("Betterbird is running (pid %d), and it rewrites prefs.js when it quits: the account "+
"list is written at its next start from the launcher, or by betterbird_restart", ps[0].PID)
}
path := filepath.Join(profile, "prefs.js")
raw, err := readBounded(path)
if err != nil && !errors.Is(err, os.ErrNotExist) {
@@ -418,16 +414,15 @@ func (m *Machine) writeAccountLists(profile string, resolved []resolvedAccount)
if !changed {
return "unchanged", nil
}
// Only while Betterbird is closed: no main process and no lock on the profile (novox/hq ADR 0316 §12, ADR 0329 §9).
if why := m.closedFor(profile); why != "" {
return "", errors.New(why + ": the account list is written at its next start from the launcher, or by betterbird_restart")
}
mode := os.FileMode(0o600)
if st, err := os.Stat(path); err == nil {
mode = st.Mode().Perm()
}
tmp := path + ".mesh-new"
if err := os.WriteFile(tmp, []byte(out), mode); err != nil {
return "", err
}
if err := os.Rename(tmp, path); err != nil {
os.Remove(tmp)
if err := writeAtomic(path, []byte(out), mode); err != nil {
return "", err
}
return fmt.Sprintf("written: %d account(s) the module created in the account list", len(wantAccounts)), nil
@@ -608,11 +603,15 @@ func withoutSignIn(accounts map[string]Account, kept map[string]bool) []string {
return out
}
// AccountsAnswer is what betterbird_accounts answers its caller: counts and kinds, never an address, a host or
// a name (novox/hq ADR 0317 §9). The value itself goes only to the operator, in a proposal they read at
// their own desk.
// AccountsAnswer is what betterbird_accounts answers its caller: the profile's accounts with the operator's own
// values (novox/hq ADR 0329 §1, §12), Owl's marked and by name and address only (§13), and counts and kinds.
// Never a password, a token or anything of the login store (§2).
type AccountsAnswer struct {
Profile string `json:"profile"`
// Accounts are the profile's accounts, in its list's order.
Accounts []listedAccount `json:"accounts"`
// Derived is the value of the setting accounts the propose built, as it was proposed (ADR 0329 §10).
Derived map[string]Account `json:"derived,omitempty"`
// Declarable are the profile's mail accounts a declaration can hold (IMAP and POP3), by kind.
Declarable map[string]int `json:"declarable"`
// KeptByOwl are the accounts Owl keeps (the work account through Exchange): never declared (ADR 0316 §13).
@@ -626,6 +625,20 @@ type AccountsAnswer struct {
Note string `json:"note"`
}
// listedAccount is one account of the profile as betterbird_accounts lists it.
type listedAccount struct {
Key string `json:"key"`
Kind string `json:"kind"`
// DeclaredAs is the name the setting accounts declares it under, or empty: the person's own.
DeclaredAs string `json:"declared_as,omitempty"`
Owl bool `json:"owl,omitempty"`
// Name is what Betterbird shows it as, and Address its default identity's.
Name string `json:"name,omitempty"`
Address string `json:"address,omitempty"`
Value *Account `json:"value,omitempty"`
Note string `json:"note,omitempty"`
}
// profileDeclarable reads the profile's accounts: the value a declaration of them would be, by name, and
// what is answered of them.
func (m *Machine) profileDeclarable() (map[string]Account, AccountsAnswer, error) {
@@ -634,13 +647,28 @@ func (m *Machine) profileDeclarable() (map[string]Account, AccountsAnswer, error
return nil, AccountsAnswer{}, err
}
prefs := mergedPrefs(p.path)
a := AccountsAnswer{Profile: p.Dir, Declarable: map[string]int{}, NotDeclarable: map[string]int{}}
a := AccountsAnswer{Profile: p.Dir, Accounts: []listedAccount{}, Declarable: map[string]int{}, NotDeclarable: map[string]int{}}
value := map[string]Account{}
used := map[string]bool{}
declaredAs := map[string]string{}
if accounts, _, err := m.readAccounts(); err == nil {
for _, r := range resolveAccounts(accounts, prefs, m.readRecord(p.Dir)) {
if r.How != refused {
declaredAs[r.Keys.Account] = r.name
}
}
}
for _, pa := range profileAccounts(prefs) {
listed := listedAccount{Key: pa.Key, Kind: pa.Kind, DeclaredAs: declaredAs[pa.Key], Owl: pa.Owl,
Name: unquote(prefs["mail.server."+pa.Server+".name"])}
if len(pa.Addresses) > 0 {
listed.Address = pa.Addresses[0]
}
switch {
case pa.Owl:
a.KeptByOwl++
listed.Note = "Owl's: never declared (novox/hq ADR 0316 §13)"
a.Accounts = append(a.Accounts, listed)
continue
case pa.Kind != "imap" && pa.Kind != "pop3":
kind := pa.Kind
@@ -648,6 +676,8 @@ func (m *Machine) profileDeclarable() (map[string]Account, AccountsAnswer, error
kind = "local folders"
}
a.NotDeclarable[kind]++
listed.Note = "not an account a declaration holds"
a.Accounts = append(a.Accounts, listed)
continue
}
a.Declarable[pa.Kind]++
@@ -659,8 +689,14 @@ func (m *Machine) profileDeclarable() (map[string]Account, AccountsAnswer, error
address = pa.Addresses[0]
}
name := nameFor(pa, address, used)
if declaredAs[pa.Key] != "" {
name = declaredAs[pa.Key]
}
used[name] = true
value[name] = readAccount(prefs, pa)
acc := readAccount(prefs, pa)
value[name] = acc
listed.Value = &acc
a.Accounts = append(a.Accounts, listed)
}
if len(a.NotDeclarable) == 0 {
a.NotDeclarable = nil
@@ -681,9 +717,9 @@ func (m *Machine) ProfileAccounts(propose bool) (AccountsAnswer, error) {
if err != nil {
return a, err
}
a.Note = "counts and kinds only: no address, host or name leaves through this answer. With propose, the " +
"setting accounts is built from these accounts (never Owl's, never a password or token) and proposed to " +
"the operator, who reads it whole at their desk before approving (novox/hq ADR 0317 §9)"
a.Note = "the profile's accounts with their values, never a password or token (novox/hq ADR 0329 §1, §2). With " +
"propose, the setting accounts is built from them (never Owl's) and proposed to the operator in the mesh-wide " +
"layer as it stands, who approves it on a channel that proves who answers (novox/hq ADR 0317 §9)"
if !propose {
return a, nil
}
@@ -701,21 +737,16 @@ func (m *Machine) ProfileAccounts(propose bool) (AccountsAnswer, error) {
return a, fmt.Errorf("nothing was proposed: %s; betterbird_prefs shows the account's preferences by name",
strings.Join(problems, "; "))
}
body, err := json.Marshal(map[string]any{"accounts": value})
// A layer is replaced whole: the proposal carries the mesh-wide layer as it stands, with this value of accounts.
layer, err := readLayer("")
if err != nil {
return a, err
return a, fmt.Errorf("nothing was proposed: %v", err)
}
raw, err := asker("seat:mesh-controller.settings", map[string]any{"module": "betterbird", "values": string(body),
"propose": "true"})
if err != nil {
return a, fmt.Errorf("the controller's settings verb did not answer: %v", err)
}
said, ok := verbSaid(raw)
layer["accounts"] = value
a.Derived = value
said, err := proposeLayer("", layer, false)
a.Proposed = m.maskValue(said)
if !ok {
return a, fmt.Errorf("the controller did not take the proposal: %s", a.Proposed)
}
return a, nil
return a, err
}
// verbSaid is a verb's words and whether it went ahead, whatever wraps them: the controller's {ok, output},
@@ -357,9 +357,12 @@ func TestDeclaredAccountsWithoutASignInAreCountedFromTheServerNamesAlone(t *test
}
}
func TestBetterbirdAccountsAnswersCountsAndProposesTheValueWithoutOwlOrACredential(t *testing.T) {
// ADR 0329 §1, §10, §12, §13: betterbird_accounts lists the accounts with their values, Owl's by name and address
// and marked; its propose carries the mesh-wide layer as it stands with accounts in it, and answers the value.
func TestBetterbirdAccountsListsTheValuesAndProposesThemInTheLayerAsItStands(t *testing.T) {
f := newFake(t)
f.profileWithAccounts()
f.write(profileDir+"/logins.json", `{"logins":[{"hostname":"imap://imap.example.org","encryptedUsername":"MDoEEPgAAAAA","encryptedPassword":"hunter2secret"}]}`)
links(t, nil)
a, err := f.ProfileAccounts(false)
if err != nil {
@@ -368,19 +371,33 @@ func TestBetterbirdAccountsAnswersCountsAndProposesTheValueWithoutOwlOrACredenti
if a.Declarable["imap"] != 1 || a.KeptByOwl != 1 || a.NotDeclarable["local folders"] != 1 || a.Proposed != "" {
t.Fatalf("%+v", a)
}
raw, _ := json.Marshal(a)
for _, never := range []string{"example", "other", "Other", "work@", "smtp1"} {
raw, _ := json.Marshal(credentialChecked(a))
for _, shown := range []string{"other@mail.example.org", "imap.example.org", "smtp.example.org", `"port":993`, `"work@corp.example.com"`} {
if !strings.Contains(string(raw), shown) {
t.Errorf("the answer shows the operator's own %s (ADR 0329 §1): %s", shown, raw)
}
}
for _, never := range []string{"hunter2", "MDoEEP"} {
if strings.Contains(string(raw), never) {
t.Errorf("the answer to the caller names %q: %s", never, raw)
t.Errorf("the answer carries the login store's %q", never)
}
}
for _, l := range a.Accounts {
if l.Key == "account3" && (!l.Owl || l.Value != nil || l.Address != "work@corp.example.com") {
t.Errorf("Owl's account is listed by name and address, marked, never as a value to declare: %+v", l)
}
}
var asked map[string]any
var asked []map[string]any
asker = func(key string, body any) (json.RawMessage, error) {
if key != "seat:mesh-controller.settings" {
t.Fatalf("asked %s", key)
}
asked = body.(map[string]any)
args := body.(map[string]any)
asked = append(asked, args)
if args["propose"] == nil {
return json.RawMessage(`{"ok": true, "output": "{\n \"tags\": {\"$label1\": {\"name\": \"Important\"}}\n}\n approved by the operator\nbetterbird on the whole mesh, every value"}`), nil
}
return json.RawMessage(`{"ok": true, "output": "proposed p-17 for betterbird on the whole mesh: accounts (other@mail.example.org); fingerprint ab12"}`), nil
}
t.Cleanup(func() { asker = nil })
@@ -388,10 +405,15 @@ func TestBetterbirdAccountsAnswersCountsAndProposesTheValueWithoutOwlOrACredenti
if err != nil {
t.Fatal(err)
}
if asked["module"] != "betterbird" || asked["propose"] != "true" || asked["node"] != nil {
t.Fatalf("proposed for the whole mesh, never set: %v", asked)
if len(asked) != 2 || asked[1]["module"] != "betterbird" || asked[1]["propose"] != "true" || asked[1]["node"] != nil ||
asked[1]["replace"] != nil {
t.Fatalf("the layer read, then proposed for the whole mesh, never set, never removing: %v", asked)
}
value, err := parseAccounts([]byte(asked["values"].(string)))
var layer map[string]json.RawMessage
if err := json.Unmarshal([]byte(asked[1]["values"].(string)), &layer); err != nil || string(layer["tags"]) == "" {
t.Fatalf("the proposal carries the layer as it stands: %v %s", err, asked[1]["values"])
}
value, err := parseAccounts([]byte(asked[1]["values"].(string)))
if err != nil {
t.Fatalf("the proposed value is one the module takes: %v", err)
}
@@ -400,6 +422,9 @@ func TestBetterbirdAccountsAnswersCountsAndProposesTheValueWithoutOwlOrACredenti
other.Incoming.Port != 993 || other.Outgoing.Security != "tls" || other.Outgoing.Port != 465 {
t.Fatalf("the value holds the person's account and not Owl's: %+v", value)
}
if a.Derived["mail"].Address != "other@mail.example.org" {
t.Fatalf("the answer shows the value it proposed (ADR 0329 §10): %+v", a.Derived)
}
if strings.Contains(a.Proposed, "other@") || !strings.Contains(a.Proposed, "p-17") {
t.Fatalf("the controller's words come back with every address masked: %s", a.Proposed)
}
@@ -516,3 +541,27 @@ func TestAnAccountsOwnAliasesAndNonMailIdentitiesDoNotShareItsOutgoingServer(t *
t.Fatalf("%v", err)
}
}
// Review 121: the account lists wait for Betterbird to close by its profile's lock too, not only by its process.
func TestTheAccountListsWaitWhileTheProfileIsLocked(t *testing.T) {
f := newFake(t)
f.profileWithAccounts()
f.noPackage()
links(t, nil)
f.declare("// none")
f.declareAccounts(personal)
stopBetterbird(t, f)
f.write(profileDir+"/lock", "")
before := f.read(profileDir + "/prefs.js")
u, err := f.userJS(true)
if err != nil {
t.Fatal(err)
}
if !strings.HasPrefix(u.Lists, "not written") || !strings.Contains(u.Lists, "locked") || f.read(profileDir+"/prefs.js") != before {
t.Fatalf("with the profile locked and no Betterbird running, prefs.js is unchanged: %q", u.Lists)
}
os.Remove(filepath.Join(f.Root, profileDir, "lock"))
if u, err = f.userJS(true); err != nil || !strings.HasPrefix(u.Lists, "written") {
t.Fatalf("once unlocked the list is written: %q %v", u.Lists, err)
}
}
@@ -171,7 +171,8 @@ func refuseSharedOutgoing(resolved []resolvedAccount, prefs map[string]string, h
users := map[string][]string{}
seen := map[string]bool{}
use := func(id string) {
if seen[id] || ours[id] {
// An identity the module created for a declared account sends as that account does (ADR 0329 §7).
if seen[id] || ours[id] || ownIdentity.MatchString(id) {
return
}
seen[id] = true
@@ -9,20 +9,20 @@ package main
// preferences asked for are answered and never one that can hold a credential or names an account
// (passwordSecret, accountPrefix);
// - the profile's extensions.json, for the add-ons' ids and whether each is on;
// - the module's declared user.js (~/.config/betterbird/user.js), which the node-engine writes from
// the module's setting `prefs`, and its declared accounts (~/.config/betterbird/accounts.json, the
// setting `accounts`, accounts.go);
// - the module's settings as the node-engine writes them: prefs (~/.config/betterbird/prefs.json, prefs.go),
// accounts (~/.config/betterbird/accounts.json, accounts.go), and identities, calendars, address-books,
// filters and tags (configuration.json and tags.json, configuration.go);
// - of logins.json, only the server name each kept sign-in is for, which Betterbird keeps in the clear,
// to count the declared accounts that have not signed in yet (novox/hq ADR 0316 §16).
//
// Never read: the mail store, the address books, a user name or password in logins.json, key4.db,
// cert9.db, cookies, the calendar's caches. Nothing of the operator's mail or calendar is answered, and of
// their accounts only counts and kinds.
// Never read: the mail store, the address books' contacts, a user name or password in logins.json, key4.db,
// cert9.db, cookies, the calendar's events and caches. The operator's own configuration is answered with its
// values (novox/hq ADR 0329 §1), never a credential (§2).
//
// The files a tool writes in the profile are user.js, and only the module's own (the declared file and
// the declared accounts' preferences, betterbird_user_js with apply, betterbird_restart, and the start
// from the launcher), and the module's own keys in the account lists of prefs.js, only while Betterbird is
// not running (accounts.go). A user.js that is not the module's is never overwritten.
// The files a tool writes in the profile are user.js, and only the module's own (betterbird_user_js with apply,
// betterbird_restart, and the start from the launcher); and, only while Betterbird is not running, the module's
// own keys in the lists of prefs.js, the removals of what it created, and each declared account's
// msgFilterRules.dat (accounts.go, owned.go). A user.js that is not the module's is never overwritten.
import (
"bufio"
@@ -51,8 +51,6 @@ const (
testUnit = "mesh-betterbird-reminder-test"
// profilesDir is where Betterbird keeps its profiles: Thunderbird's place, which Betterbird shares.
profilesDir = ".thunderbird"
// declared is the module's user.js, written by the node-engine from the setting prefs.
declared = ".config/betterbird/user.js"
// mark is the first line of the module's user.js: how a user.js is known to be the module's.
mark = "// Betterbird's user.js (module betterbird"
)
@@ -440,6 +438,12 @@ func validUserJS(text string) error {
"accounts, never in prefs (novox/hq ADR 0316 §11)", i+1, name)
}
}
for _, p := range configurationOwned {
if strings.HasPrefix(name, p) {
return fmt.Errorf("line %d sets %s, which the setting calendars, address-books or tags owns, never "+
"prefs (novox/hq ADR 0329 §6)", i+1, name)
}
}
}
}
return nil
@@ -489,6 +493,16 @@ type UserJS struct {
Each []resolvedAccount `json:"accounts,omitempty"`
// Lists says whether the keys of the accounts the module created were written into the account lists.
Lists string `json:"account_lists,omitempty"`
// Configuration is every declared identity, calendar, address book and tag by its name: adopted, created or
// refused (with why), and its key (novox/hq ADR 0329 §7).
Configuration []resolvedEntry `json:"configuration,omitempty"`
ConfigurationProblem string `json:"configuration_problem,omitempty"`
// Filters are the declared accounts' filter files.
Filters []filterPlan `json:"filters,omitempty"`
// Closed says what was written of what Betterbird rewrites itself, or why it waits (§9).
Closed string `json:"written_while_closed,omitempty"`
// Waiting counts, per kind, the changes that wait for Betterbird to close.
Waiting map[string]int `json:"waiting_for_betterbird_to_close,omitempty"`
}
// refusedAccounts are the reasons of the declared accounts that were refused.
@@ -508,9 +522,12 @@ func (u UserJS) refusedAccounts() []string {
// it created into the profile's account lists when Betterbird is not running. A refused account gets nothing;
// the others go ahead (novox/hq ADR 0317 §7).
func (m *Machine) userJS(apply bool) (UserJS, error) {
a := UserJS{Declared: "~/" + declared}
declaredText, err := readBounded(m.home(declared))
a.Present = err == nil
a := UserJS{Declared: "~/" + prefsFile}
declaredText, present, derr := m.declaredText()
a.Present = present
if derr != nil {
a.Present, a.Valid = true, derr.Error()
}
p, perr := m.profile()
if perr != nil {
return a, perr
@@ -518,19 +535,27 @@ func (m *Machine) userJS(apply bool) (UserJS, error) {
target := filepath.Join(p.path, "user.js")
a.Profile = m.tilde(strings.TrimPrefix(target, m.Root))
accounts, _, aerr := m.readAccounts()
merged := mergedPrefs(p.path)
if aerr != nil {
a.AccountsProblem = aerr.Error()
accounts = nil
} else {
a.Accounts = len(accounts)
a.Each = resolveAccounts(accounts, mergedPrefs(p.path), m.readRecord(p.Dir))
a.Each = resolveAccounts(accounts, merged, m.readRecord(p.Dir))
}
conf, cerr := m.readConfiguration()
if cerr != nil {
a.ConfigurationProblem = cerr.Error()
}
plan := m.planConfiguration(p, conf, accounts, a.Each, merged)
a.Configuration, a.Filters = plan.Entries, plan.filters
a.Waiting = m.waiting(p.path, plan, a.Each)
var want []byte
if a.Present {
if verr := validUserJS(string(declaredText)); verr != nil {
if a.Present && a.Valid == "" {
if verr := validUserJS(declaredText); verr != nil {
a.Valid = verr.Error()
}
want = []byte(composeUserJS(string(declaredText), accounts, a.Each))
want = []byte(composeUserJS(declaredText, accounts, a.Each) + plan.configBlock())
a.Content = m.answeredUserJS(string(want))
for _, line := range strings.Split(string(want), "\n") {
for _, st := range splitStatements(strings.TrimSpace(line)) {
@@ -554,11 +579,13 @@ func (m *Machine) userJS(apply bool) (UserJS, error) {
if apply {
switch {
case !a.Present:
return a, fmt.Errorf("the module's user.js is not on this machine (~/%s): the module is not pushed here yet", declared)
return a, fmt.Errorf("the setting prefs is not on this machine (~/%s): the module is not pushed here yet", prefsFile)
case a.Valid != "":
return a, fmt.Errorf("the module's user.js is refused: %s; change the setting prefs", a.Valid)
case a.AccountsProblem != "":
return a, fmt.Errorf("nothing was placed: %s", a.AccountsProblem)
case a.ConfigurationProblem != "":
return a, fmt.Errorf("nothing was placed: %s", a.ConfigurationProblem)
}
saved, _ := readBounded(filepath.Join(p.path, "prefs.js"))
savedPrefs := readPrefs(saved)
@@ -590,6 +617,16 @@ func (m *Machine) userJS(apply bool) (UserJS, error) {
} else {
a.Lists = lists
}
if closed, err := m.writeClosed(p, &plan); err != nil {
a.Closed = "not written: " + err.Error()
} else {
a.Closed = closed
}
a.Filters = plan.filters
if err := m.writeConfigRecord(p.Dir, plan.record); err != nil {
return a, fmt.Errorf("the user.js is placed, and what was adopted or created could not be recorded: %v", err)
}
a.Waiting = m.waiting(p.path, plan, a.Each)
}
if ps := m.mainProcs(); len(ps) > 0 {
if st, err := os.Stat(target); err == nil {
@@ -640,6 +677,20 @@ func (m *Machine) Prepare() error {
problems = append(problems, "the account lists were "+u.Lists)
}
problems = append(problems, u.refusedAccounts()...)
if strings.HasPrefix(u.Closed, "not written") {
problems = append(problems, "what Betterbird rewrites itself was "+u.Closed)
}
// Said by kind, name and key only: the start's words reach the journal, which holds no address (ADR 0329 §1).
for _, e := range u.Configuration {
if e.How == refused {
problems = append(problems, fmt.Sprintf("%s %s is refused: %s", e.Kind, e.Name, e.Why))
}
}
for _, fp := range u.Filters {
if fp.Refused != "" {
problems = append(problems, fmt.Sprintf("the filters of %s are refused: %s", fp.Account, fp.Refused))
}
}
if len(problems) > 0 {
return fmt.Errorf("%s", strings.Join(problems, "; "))
}
@@ -843,7 +894,7 @@ func (m *Machine) Restart(discard bool) (RestartAnswer, error) {
a.UserJS = "not placed: " + err.Error()
} else {
a.UserJS = fmt.Sprintf("placed: %d preference(s) from ~/%s, %d declared account(s); account lists %s",
u.Statements, declared, u.Accounts, u.Lists)
u.Statements, prefsFile, u.Accounts, u.Lists)
}
// As detach does, with shorter bounds: the worst case of the whole call (3 s for the window tree,
// 8 s to quit, 3 + 5 s to start, 4 s to see it) stays under the runtime's 30 s.
@@ -962,6 +1013,23 @@ func (m *Machine) Check() (CheckAnswer, error) {
add(fmt.Sprintf("%d declared account(s) have not signed in yet", c.WithoutSignIn),
"a person signs in once, in Betterbird's own window, when it asks (novox/hq ADR 0316 §14)")
}
// The five kinds of ADR 0329: drift and what waits for Betterbird to close, per kind (§9, §11, §12).
if _, err := m.readConfiguration(); err != nil {
add(err.Error(), "change the setting at the controller's terminal, or derive and propose it (betterbird_derive)")
} else if v, err := m.view(); err == nil {
drift := m.driftCounts(v)
waiting := m.waiting(v.p.path, v.plan, v.ra)
for _, k := range allKinds {
if n := drift[k]; n > 0 {
add(fmt.Sprintf("%d declared %s differ from the profile, are not in it yet or are refused", n, k),
"betterbird_drift with kind "+k+" says which and how; the module's user.js holds them from Betterbird's next start (betterbird_restart)")
}
if n := waiting[k]; n > 0 {
add(fmt.Sprintf("%d change(s) to %s are not written yet: they are written only while Betterbird is closed", n, k),
"quit Betterbird and start it from the launcher, or betterbird_restart (novox/hq ADR 0329 §9)")
}
}
}
}
ps := m.mainProcs()
switch {
@@ -975,7 +1043,7 @@ func (m *Machine) Check() (CheckAnswer, error) {
case err != nil:
add("no profile: "+err.Error(), "start Betterbird once")
case !u.Present:
add("the module's user.js is not on this machine (~/"+declared+")", "the module is not pushed here: the controller's push")
add("the setting prefs is not on this machine (~/"+prefsFile+")", "the module is not pushed here: the controller's push")
case u.Valid != "":
add("the module's user.js is refused: "+u.Valid, "change the module's setting prefs")
case u.AccountsProblem != "":
@@ -1,6 +1,7 @@
package main
import (
"encoding/json"
"os"
"path/filepath"
"strings"
@@ -44,8 +45,20 @@ func (f *fake) betterbird() {
f.write(profileDir+"/extensions.json", `{"addons":[{"id":"owl@beonex.com","active":true,"location":"app-profile"},{"id":"x@mozilla.org","active":true,"location":"app-builtin"}]}`)
}
// declare writes the setting prefs as the node-engine does (prefs.json, novox/hq issue 586), from user_pref
// statements: each one entry of the map.
func (f *fake) declare(prefs string) {
f.write(testHome+"/.config/betterbird/user.js", mark+", novox/hq issue 345). Owned by the mesh.\n//\n"+prefs+"\n")
f.t.Helper()
values := map[string]any{}
for name, lit := range statementsOf([]byte(prefs)) {
v, ok := prefValue(lit)
if !ok {
f.t.Fatalf("%s: %s is not a value the setting holds", name, lit)
}
values[name] = v
}
body, _ := json.Marshal(map[string]any{"prefs": values})
f.write(testHome+"/"+prefsFile, string(body))
}
func TestTheProfileIsTheOneTheRunningBetterbirdHoldsElseTheInstallationsDefault(t *testing.T) {
@@ -106,9 +119,9 @@ func TestTheModulesUserJSIsPlacedOnlyWhereNoOperatorsFileIs(t *testing.T) {
t.Fatalf("%+v %v", u, err)
}
placed, _ := os.ReadFile(filepath.Join(f.Root, profileDir, "user.js"))
declared, _ := os.ReadFile(filepath.Join(f.Root, testHome, ".config/betterbird/user.js"))
if string(placed) != string(declared) {
t.Fatal("placed whole, as declared")
declared, _, _ := f.declaredText()
if string(placed) != declared || !strings.Contains(declared, `user_pref("ui.key.x", "a;b");`) {
t.Fatalf("placed whole, as declared:\n%s", placed)
}
if !strings.HasPrefix(u.Effective, "yes") {
t.Fatalf("the fake Betterbird started after the file's time: %q", u.Effective)
@@ -1,18 +1,20 @@
package main
// The configuration the module ships (novox/hq issue 345), in three layers:
// The configuration of Betterbird's preferences (novox/hq issue 345, issue 586), in three layers:
//
// - LOCKED: the enterprise policy (/opt/betterbird/distribution/policies.json). Its Preferences entries with
// Status locked win over everything, and Betterbird's settings show them greyed out.
// - DEFAULT: the module's defaults file (/opt/betterbird/defaults/pref/mesh-defaults.js), the operator's
// setup. Betterbird reads it as a default, so a value the profile saved in prefs.js wins over it: that is
// drift, unless a node layer says so.
// - NODE: the setting prefs, which the node-engine writes to ~/.config/betterbird/user.js and the tools place
// in the profile's user.js, where it wins over prefs.js at every start.
// - DECLARED: the setting prefs, the operator's setup: mesh-wide, with a node's layer over it key by key. The
// node-engine writes it to ~/.config/betterbird/prefs.json and the tools place it in the profile's user.js,
// where it wins over prefs.js at every start.
// - BETTERBIRD'S DEFAULT: what Betterbird starts with. The module's defaults file
// (/opt/betterbird/defaults/pref/mesh-defaults.js) holds no value of the operator's any more (issue 586): it
// names the preferences the module keeps at Betterbird's own default, where a value the profile saved is drift.
//
// betterbird_config shows each managed preference through the layers and flags the drift; betterbird_config_apply
// clears a drifting value from prefs.js while Betterbird is not running; keep proposes a node layer that holds the
// profile's value for this node instead, through the controller's settings verb, never by writing user.js.
// clears a drifting value from prefs.js while Betterbird is not running; keep proposes a node layer of the setting
// prefs that holds the profile's value for this node instead, through the controller's settings verb, never by
// writing user.js.
import (
"archive/zip"
@@ -320,14 +322,14 @@ func statementsOf(raw []byte) map[string]string {
// ConfigPref is one managed preference through the layers.
type ConfigPref struct {
Name string `json:"name"`
// Layer is locked (the policy), default (the module's defaults file) or "Betterbird's default" (a managed
// preference whose module value is Betterbird's own).
// Layer is locked (the policy), declared (the setting prefs), default (a value of the module's defaults file) or
// "Betterbird's default" (a managed preference whose module value is Betterbird's own).
Layer string `json:"layer"`
Locked string `json:"locked,omitempty"`
ModuleDefault string `json:"module_default,omitempty"`
BetterbirdDefault string `json:"betterbird_default,omitempty"`
// NodeLayer is what the setting prefs on this machine says (the module's user.js).
NodeLayer string `json:"node_layer,omitempty"`
// Declared is what the setting prefs says on this machine: mesh-wide, with the node's layer over it.
Declared string `json:"declared,omitempty"`
// Profile is what the profile saved in its prefs.js.
Profile string `json:"profile,omitempty"`
Effective string `json:"effective"`
@@ -349,6 +351,8 @@ type ConfigAnswer struct {
Drifting []string `json:"drifting"`
// Refused are names the module's defaults list that the tools never manage: an account's or a credential's.
Refused []string `json:"refused,omitempty"`
// PrefsProblem says why the setting prefs on this machine is refused.
PrefsProblem string `json:"prefs_problem,omitempty"`
// Proposed is the controller's answer to a node layer proposed with keep.
Proposed string `json:"proposed,omitempty"`
Note string `json:"note,omitempty"`
@@ -372,8 +376,10 @@ func (m *Machine) config() (ConfigAnswer, Profile, error) {
}
locked := m.locks()
own, _ := m.defaults(false)
declaredRaw, _ := readBounded(m.home(declared))
node := statementsOf(declaredRaw)
node, _, nerr := m.readDeclaredPrefs()
if nerr != nil {
a.PrefsProblem = nerr.Error()
}
prefsRaw, _ := readBounded(filepath.Join(p.path, "prefs.js"))
saved := readPrefs(prefsRaw)
userRaw, _ := readBounded(filepath.Join(p.path, "user.js"))
@@ -384,8 +390,13 @@ func (m *Machine) config() (ConfigAnswer, Profile, error) {
names = append(names, n)
}
sort.Strings(names)
for _, n := range sh.order {
if _, isLocked := locked[n]; !isLocked {
listed := map[string]bool{}
for _, n := range names {
listed[n] = true
}
for _, n := range append(append([]string{}, sh.order...), sortedKeys(node)...) {
if !listed[n] {
listed[n] = true
names = append(names, n)
}
}
@@ -395,7 +406,7 @@ func (m *Machine) config() (ConfigAnswer, Profile, error) {
a.Refused = append(a.Refused, n+": "+why)
continue
}
c := ConfigPref{Name: n, NodeLayer: node[n], Profile: saved[n]}
c := ConfigPref{Name: n, Declared: node[n], Profile: saved[n]}
if n == themePref && theme != "" {
// The add-on database says which theme is active; the preference only mirrors it.
c.Profile = quoted(theme)
@@ -410,6 +421,8 @@ func (m *Machine) config() (ConfigAnswer, Profile, error) {
switch v, isLocked := locked[n]; {
case isLocked:
c.Layer, c.Locked, c.module = "locked", v, v
case c.Declared != "":
c.Layer, c.module = "declared", c.Declared
case sh.keeps[n]:
c.Layer = "Betterbird's default"
c.module = c.BetterbirdDefault
@@ -430,23 +443,37 @@ func (m *Machine) config() (ConfigAnswer, Profile, error) {
default:
c.From = "Betterbird's built-in default (in no defaults file)"
}
if why := drift(c); why != "" {
if why := drift(c, placed[n]); why != "" {
c.Drift = why
if n == themePref {
switch {
case n == themePref:
c.Remedy = themeRemedy
case c.Layer == "declared":
c.Remedy = "betterbird_user_js with apply places the setting in the profile's user.js; Betterbird reads it " +
"at its next start (betterbird_restart)"
}
a.Drifting = append(a.Drifting, n)
}
c.NodeLayer, c.Profile, c.Effective = m.answer(n, c.NodeLayer), m.answer(n, c.Profile), m.answer(n, c.Effective)
c.Declared, c.Profile, c.Effective = m.answer(n, c.Declared), m.answer(n, c.Profile), m.answer(n, c.Effective)
a.Prefs = append(a.Prefs, c)
}
return a, p, nil
}
// drift says how the profile's saved value beats the module's, or "": a value in prefs.js that is not the
// module's, where no lock holds and no node layer names the preference.
func drift(c ConfigPref) string {
if c.Locked != "" || c.NodeLayer != "" || c.Profile == "" || c.Profile == c.module {
// drift says how the profile differs from the module's configuration, or "": for a declared preference, the
// profile's user.js not holding the declared value (so Betterbird does not start with it); for a preference kept at
// Betterbird's default, a value in prefs.js that is not it. A lock holds whatever the profile says.
func drift(c ConfigPref, placed string) string {
if c.Locked != "" {
return ""
}
if c.Declared != "" {
if placed == c.Declared {
return ""
}
return fmt.Sprintf("the setting prefs says %s, and the profile's user.js does not hold it yet", c.Declared)
}
if c.Profile == "" || c.Profile == c.module {
return ""
}
want := c.module
@@ -457,7 +484,7 @@ func drift(c ConfigPref) string {
if c.Name == themePref {
where = "the profile's active theme is"
}
return fmt.Sprintf("%s %s, which beats the module's %s, and no node layer says so", where, c.Profile, want)
return fmt.Sprintf("%s %s, which beats the module's %s, and the setting prefs does not say so", where, c.Profile, want)
}
// Config is what betterbird_config answers; with keep it proposes a node layer of the setting prefs that holds
@@ -468,8 +495,9 @@ func (m *Machine) Config(keep []string) (ConfigAnswer, error) {
if err != nil {
return a, err
}
a.Note = "drift is a value the profile saved that beats the module's default without a node layer saying so: " +
"betterbird_config_apply clears it (Betterbird quit), or keep proposes a node layer that holds it for this machine"
a.Note = "drift is a declared value the profile's user.js does not hold yet (betterbird_user_js with apply), or a " +
"value the profile saved where the module keeps Betterbird's default: betterbird_config_apply clears that one " +
"(Betterbird quit), or keep proposes a node layer of the setting prefs that holds it for this machine"
if len(keep) == 0 {
return a, nil
}
@@ -494,50 +522,36 @@ func (m *Machine) Config(keep []string) (ConfigAnswer, error) {
}
kept[n] = saved[n]
}
declaredRaw, err := readBounded(m.home(declared))
if err != nil {
return a, fmt.Errorf("nothing was proposed: the module's user.js is not on this machine (~/%s)", declared)
}
node := nodeName()
if node == "" {
return a, errors.New("nothing was proposed: this runtime does not say which machine it serves (MESH_NODE)")
}
// The setting is one string, so a node layer replaces it whole: it carries what the setting says on this
// machine today, with the kept values in place of any the same names had.
var parts []string
for _, line := range strings.Split(string(declaredRaw), "\n") {
for _, st := range splitStatements(strings.TrimSpace(line)) {
if mm := prefLine.FindStringSubmatch(st); mm != nil {
if _, replaced := kept[mm[1]]; !replaced {
parts = append(parts, fmt.Sprintf("user_pref(%s, %s);", quoted(mm[1]), mm[2]))
}
}
}
// A layer is replaced whole, so the proposal carries this node's layer as it stands, with the kept values in
// the setting prefs beside what it held (novox/hq issue 586: one entry per preference).
layer, err := readLayer(node)
if err != nil {
return a, fmt.Errorf("nothing was proposed: %v", err)
}
prefs, _ := layer["prefs"].(map[string]any)
if prefs == nil {
prefs = map[string]any{}
}
for _, n := range keep {
if v, ok := kept[n]; ok {
parts = append(parts, fmt.Sprintf("user_pref(%s, %s);", quoted(n), v))
delete(kept, n)
v, ok := prefValue(kept[n])
if !ok {
return a, fmt.Errorf("nothing was proposed: the profile's value of %s is not one the setting can hold", n)
}
prefs[n] = v
}
value := strings.Join(parts, " ")
if err := validUserJS(mark + ")\n" + value + "\n"); err != nil {
return a, fmt.Errorf("nothing was proposed: the layer would be refused: %v", err)
}
body, _ := json.Marshal(map[string]string{"prefs": value})
raw, err := asker("seat:mesh-controller.settings", map[string]any{"module": "betterbird", "node": node,
"values": string(body), "propose": "true"})
if err != nil {
return a, fmt.Errorf("the controller's settings verb did not answer: %v", err)
}
said, ok := verbSaid(raw)
layer["prefs"] = prefs
said, err := proposeLayer(node, layer, false)
a.Proposed = m.maskValue(said)
if !ok {
return a, fmt.Errorf("the controller did not take the proposal: %s", a.Proposed)
if err != nil {
return a, err
}
a.Note = "proposed for " + node + ": the operator approves it on a channel that proves who answers; the next " +
"push writes it, and it is placed at Betterbird's next start from the launcher (or betterbird_restart). " +
"The layer carries the whole setting prefs as it stands on this machine, since a layer replaces it whole"
"The layer carries this node's layer as it stands, with the kept preferences in the setting prefs"
return a, nil
}
@@ -13,8 +13,9 @@ import (
// The configuration the module ships (novox/hq issue 345): the policy's locks, the module's defaults file, and the
// tools that show the layers, flag drift, clear it and propose a node layer.
// shippedDefaultsWant is the DEFAULT layer the operator approved on 2026-10-12: the laptop's setup, with
// reminders missed while asleep shown at the next start and chat off.
// shippedDefaultsWant is the setup the operator approved on 2026-10-12 (the laptop's, with reminders missed while
// asleep shown at the next start and chat off), which the module shipped as its defaults file until issue 586 moved
// it into the mesh-wide layer of the setting prefs: the catalogue holds none of it now.
var shippedDefaultsWant = map[string]string{
"mail.uidensity": "0",
"mail.uifontsize": "14",
@@ -147,8 +148,13 @@ func TestTheModuleShipsTheApprovedLocksAndDefaults(t *testing.T) {
text := string(repoFile(t, "install/defaults-pref/mesh-defaults.js"))
sh := parseShipped([]byte(text))
if !reflect.DeepEqual(sh.values, shippedDefaultsWant) {
t.Errorf("the defaults are the approved ones:\n got %v\nwant %v", sh.values, shippedDefaultsWant)
if len(sh.values) != 0 {
t.Errorf("the module's defaults hold no value of the operator's: they are the setting prefs (issue 586): %v", sh.values)
}
for n := range shippedDefaultsWant {
if strings.Contains(text, n) {
t.Errorf("%s is the operator's choice and lives in the setting prefs, never in the catalogue", n)
}
}
var keeps []string
for _, n := range sh.order {
@@ -198,8 +204,8 @@ func TestPrefsSayTheDefaultAndTheFileThatGivesIt(t *testing.T) {
if p := by["offline.autoDetect"]; p.Value != "true" || p.Default != "true" || !strings.Contains(p.DefaultFrom, "all-thunderbird.js") {
t.Errorf("of two files, the one first by name holds, as Gecko reads them: %+v", p)
}
if p := by["mail.threadpane.listview"]; p.Value != "0" || !strings.Contains(p.From, "the module's default") {
t.Errorf("the module's default is over Betterbird's: %+v", p)
if p := by["mail.threadpane.listview"]; p.Value != "1" || strings.Contains(p.From, "module") || strings.Contains(p.DefaultFrom, "module") {
t.Errorf("the module ships no value of the operator's, so Betterbird's own default holds: %+v", p)
}
if p := by["gfx.webrender.software"]; p.Value != "false" || !strings.Contains(p.From, "greprefs.js") {
t.Errorf("%+v", p)
@@ -234,8 +240,8 @@ func TestMailnewsIsWithheldOnlyWhereItNamesAnAccount(t *testing.T) {
}
}
// T1: every managed preference through the layers, and drift where the profile's own value beats the module's
// default without a node layer saying so.
// T1: every managed preference through the layers, and drift: a declared value the profile's user.js does not hold
// yet, or a value the profile saved where the module keeps Betterbird's default.
func TestTheConfigShowsTheLayersAndTheDrift(t *testing.T) {
f := newFake(t)
f.betterbird()
@@ -243,13 +249,14 @@ func TestTheConfigShowsTheLayersAndTheDrift(t *testing.T) {
f.withDefaults()
f.betterbirdsOwn()
links(t, nil)
f.declare(`user_pref("mail.uifontsize", 16);`)
f.declare(`user_pref("mail.uifontsize", 16); user_pref("mail.biff.show_alert", false); user_pref("calendar.alarms.showmissed", true);`)
f.write(profileDir+"/user.js", mark+")\n"+`user_pref("mail.uifontsize", 16);`+"\n"+`user_pref("calendar.alarms.showmissed", true);`+"\n")
f.write(profileDir+"/prefs.js", "// Mozilla User Preferences\n"+
`user_pref("extensions.activeThemeID", "thunderbird-compact-dark@mozilla.org");`+"\n"+
`user_pref("mail.uifontsize", 16);`+"\n"+
`user_pref("mail.uidensity", 0);`+"\n"+
`user_pref("mail.uifontsize", 12);`+"\n"+
`user_pref("mail.biff.show_alert", true);`+"\n"+
`user_pref("mail.spam.manualMark", false);`+"\n"+
`user_pref("mail.spam.manualMark", true);`+"\n"+
`user_pref("browser.display.use_system_colors", false);`+"\n"+
`user_pref("mailnews.start_page.enabled", true);`+"\n")
c, err := f.Config(nil)
if err != nil {
@@ -259,30 +266,34 @@ func TestTheConfigShowsTheLayersAndTheDrift(t *testing.T) {
for _, p := range c.Prefs {
by[p.Name] = p
}
if len(c.Prefs) != 3+len(shippedDefaultsWant)+len(shippedKeepsWant) || !strings.HasPrefix(c.Defaults, "placed") {
t.Fatalf("every lock, default and kept default: %d, %s", len(c.Prefs), c.Defaults)
if len(c.Prefs) != 3+3+len(shippedKeepsWant) || !strings.HasPrefix(c.Defaults, "placed") {
t.Fatalf("every lock, declared preference and kept default: %d, %s", len(c.Prefs), c.Defaults)
}
if !reflect.DeepEqual(c.Drifting, []string{"extensions.activeThemeID", "mail.biff.show_alert"}) {
if !reflect.DeepEqual(c.Drifting, []string{"extensions.activeThemeID", "mail.spam.manualMark", "mail.biff.show_alert"}) {
t.Fatalf("drift: %v", c.Drifting)
}
if p := by["mail.biff.show_alert"]; p.Layer != "default" || p.ModuleDefault != "false" || p.BetterbirdDefault != "true" ||
p.Profile != "true" || p.Effective != "true" || p.From != "the profile's prefs.js" || !strings.Contains(p.Drift, "beats the module's false") {
t.Errorf("%+v", p)
if p := by["mail.biff.show_alert"]; p.Layer != "declared" || p.Declared != "false" || p.Profile != "true" ||
p.Effective != "true" || !strings.Contains(p.Drift, "the setting prefs says false") || !strings.Contains(p.Remedy, "betterbird_user_js") {
t.Errorf("a declared value the profile's user.js does not hold yet: %+v", p)
}
if p := by["mail.spam.manualMark"]; p.Layer != "Betterbird's default" || p.BetterbirdDefault != "false" ||
!strings.Contains(p.Drift, "beats the module's false") || p.Remedy != "" {
t.Errorf("a saved value where the module keeps Betterbird's default: %+v", p)
}
if p := by["extensions.activeThemeID"]; p.Layer != "Betterbird's default" || !strings.Contains(p.Drift, `"default-theme@mozilla.org"`) ||
!strings.Contains(p.Remedy, "add-on manager") || p.Profile != `"thunderbird-compact-dark@mozilla.org"` {
t.Errorf("another theme is drift whose remedy is the add-on manager, its id answered whole: %+v", p)
}
if p := by["mail.spam.manualMark"]; p.Drift != "" || p.BetterbirdDefault != "false" {
if p := by["browser.display.use_system_colors"]; p.Drift != "" {
t.Errorf("a saved value equal to Betterbird's default is not drift: %+v", p)
}
if p := by["mail.uifontsize"]; p.Drift != "" || p.NodeLayer != "16" {
t.Errorf("a node layer that says so is not drift: %+v", p)
if p := by["mail.uifontsize"]; p.Drift != "" || p.Declared != "16" || p.Effective != "16" || !strings.Contains(p.From, "user.js") {
t.Errorf("a declared value the user.js holds wins at the start, whatever prefs.js saved: %+v", p)
}
if p := by["mailnews.start_page.enabled"]; p.Layer != "locked" || p.Effective != "false" || p.Drift != "" {
t.Errorf("a lock wins over the profile: %+v", p)
}
if p := by["calendar.alarms.showmissed"]; p.Effective != "true" || p.From != "the module's default" {
if p := by["calendar.alarms.showmissed"]; p.Effective != "true" || p.Layer != "declared" {
t.Errorf("%+v", p)
}
// T5: the check says the drift, with both remedies.
@@ -292,7 +303,7 @@ func TestTheConfigShowsTheLayersAndTheDrift(t *testing.T) {
}
found := false
for _, x := range ch.Findings {
found = found || (strings.Contains(x.What, "1 managed preference(s) drift") && strings.Contains(x.What, "mail.biff.show_alert") &&
found = found || (strings.Contains(x.What, "1 managed preference(s) drift") && strings.Contains(x.What, "mail.spam.manualMark") &&
strings.Contains(x.Do, "betterbird_config_apply") && strings.Contains(x.Do, "keep"))
}
theme := false
@@ -300,7 +311,11 @@ func TestTheConfigShowsTheLayersAndTheDrift(t *testing.T) {
theme = theme || (strings.HasPrefix(x.What, "extensions.activeThemeID drifts") && strings.Contains(x.Do, "add-on manager") &&
!strings.Contains(x.Do, "betterbird_config_apply"))
}
if !found || !theme {
declared := false
for _, x := range ch.Findings {
declared = declared || (strings.HasPrefix(x.What, "mail.biff.show_alert drifts") && strings.Contains(x.Do, "betterbird_user_js"))
}
if !found || !theme || !declared {
t.Fatalf("the check reports the drift: %+v", ch.Findings)
}
os.Remove(filepath.Join(f.Root, meshDefaults))
@@ -326,7 +341,7 @@ func TestConfigApplyClearsTheDriftOnlyWhileBetterbirdIsQuit(t *testing.T) {
prefs := "// Mozilla User Preferences\n" +
`user_pref("extensions.activeThemeID", "thunderbird-compact-dark@mozilla.org");` + "\n" +
`user_pref("mail.identity.id1.useremail", "someone@example.org");` + "\n" +
`user_pref("mail.biff.show_alert", true);` + "\n" +
`user_pref("mail.spam.manualMark", true);` + "\n" +
`user_pref("mail.uidensity", 0);` + "\n"
f.write(profileDir+"/prefs.js", prefs)
dry, err := f.ConfigApply(false)
@@ -365,7 +380,7 @@ func TestConfigApplyClearsTheDriftOnlyWhileBetterbirdIsQuit(t *testing.T) {
for _, c := range a.Clears {
said[c.Name] = c
}
if c := said["mail.biff.show_alert"]; c.Was != "true" || c.Now != "false" {
if c := said["mail.spam.manualMark"]; c.Was != "true" || c.Now != "false" {
t.Errorf("it says what it cleared and what holds now: %+v", c)
}
if _, cleared := said["extensions.activeThemeID"]; cleared || len(a.NotCleared) != 1 || !strings.Contains(a.NotCleared[0], "add-on manager") {
@@ -384,46 +399,52 @@ func TestConfigApplyClearsTheDriftOnlyWhileBetterbirdIsQuit(t *testing.T) {
}
}
// T6: keeping a value on one node is a node layer proposed through the controller, never a write to user.js.
// T6: keeping a value on one node is that node's layer of the setting prefs proposed through the controller, one
// entry per preference beside what the layer holds (issue 586), never a write to user.js.
func TestKeepProposesANodeLayerThroughTheController(t *testing.T) {
f := newFake(t)
f.betterbird()
f.placedByTheMesh()
f.withDefaults()
f.betterbirdsOwn()
links(t, nil)
f.declare(`user_pref("calendar.alarms.playsound", false); user_pref("mail.uifontsize", 12);`)
f.write(profileDir+"/prefs.js", `user_pref("mail.uifontsize", 16);`+"\n"+`user_pref("mail.biff.show_alert", true);`+"\n")
f.write(profileDir+"/prefs.js", `user_pref("mail.uifontsize", 16);`+"\n"+`user_pref("mail.spam.manualMark", true);`+"\n")
oldNode, oldAsk := nodeName, asker
t.Cleanup(func() { nodeName, asker = oldNode, oldAsk })
nodeName = func() string { return "shanks" }
var verb string
var args map[string]any
var asked []map[string]any
asker = func(v string, a any) (json.RawMessage, error) {
verb, args = v, a.(map[string]any)
args := a.(map[string]any)
asked = append(asked, args)
if args["propose"] == nil {
return json.RawMessage(`{"ok": true, "output": "{\"prefs\": {\"calendar.alarms.playsound\": false}, \"identities\": {\"mail\": {\"account\": \"mail\"}}}\n set at the terminal\nbetterbird on shanks, every value"}`), nil
}
return json.RawMessage(`{"ok": true, "output": "proposed: p-1"}`), nil
}
before := f.read(testHome + "/.config/betterbird/user.js")
c, err := f.Config([]string{"mail.uifontsize", "mail.biff.show_alert"})
before := f.read(testHome + "/" + prefsFile)
c, err := f.Config([]string{"mail.uifontsize", "mail.spam.manualMark"})
if err != nil {
t.Fatal(err)
}
if verb != "seat:mesh-controller.settings" || args["node"] != "shanks" || args["propose"] != "true" || args["module"] != "betterbird" {
t.Fatalf("%s %v", verb, args)
if len(asked) != 2 || asked[0]["node"] != "shanks" || asked[1]["node"] != "shanks" || asked[1]["propose"] != "true" ||
asked[1]["module"] != "betterbird" {
t.Fatalf("the node's layer read, then proposed: %v", asked)
}
var values map[string]string
if err := json.Unmarshal([]byte(args["values"].(string)), &values); err != nil {
var values map[string]any
if err := json.Unmarshal([]byte(asked[1]["values"].(string)), &values); err != nil {
t.Fatal(err)
}
want := `user_pref("calendar.alarms.playsound", false); user_pref("mail.uifontsize", 16); user_pref("mail.biff.show_alert", true);`
if values["prefs"] != want {
t.Fatalf("the layer keeps what the setting says and the kept values:\n got %s\nwant %s", values["prefs"], want)
want := map[string]any{"calendar.alarms.playsound": false, "mail.uifontsize": float64(16), "mail.spam.manualMark": true}
if !reflect.DeepEqual(values["prefs"], want) || values["identities"] == nil {
t.Fatalf("the layer as it stands with the kept values, one entry each:\n got %v\nwant %v", values, want)
}
if c.Proposed != "proposed: p-1" || f.read(testHome+"/.config/betterbird/user.js") != before {
t.Fatalf("proposed, and user.js untouched: %+v", c)
if c.Proposed != "proposed: p-1" || f.read(testHome+"/"+prefsFile) != before {
t.Fatalf("proposed, and nothing written: %+v", c)
}
for _, bad := range [][]string{{"mailnews.start_page.enabled"}, {"some.other.pref"}, {"mail.uidensity"}} {
verb = ""
if _, err := f.Config(bad); err == nil || verb != "" {
for _, bad := range [][]string{{"mailnews.start_page.enabled"}, {"some.other.pref"}, {"calendar.alarms.playsound"}} {
asked = nil
if _, err := f.Config(bad); err == nil || len(asked) != 0 {
t.Errorf("%v: a lock, an unmanaged name or a value the profile did not save is refused: %v", bad, err)
}
}
@@ -0,0 +1,471 @@
package main
// Betterbird's whole configuration, beside the accounts (novox/hq ADR 0329 §4–§6, issue 586).
//
// Five settings, each mesh-wide with a node's layer over it, reach the machine as two files the node-engine
// merges from the module's layers key by key:
//
// - ~/.config/betterbird/configuration.json: identities, calendars, address-books and filters, trusted as
// accounts is (§5), because each names where a password, a reply or forwarded mail goes;
// - ~/.config/betterbird/tags.json: tags, which names no server and sends no mail, and is not trusted.
//
// An entry is keyed by a name of the operator's (lower-case letters and digits, as an account's), a tag by its
// own key. A node's layer may set an entry to null to take it out on that node, and may give an identity, a
// calendar or an address book its `key`: the profile's key to adopt there (§4, §7). No value holds a password,
// a token or a cookie: a key shaped like one is refused whole, and so is an address whose user part carries a
// password (§2). A setting holds no line break (novox/hq issue 339), so a signature is a list of its lines.
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"net/url"
"os"
"regexp"
"sort"
"strings"
)
const (
// configurationFile holds identities, calendars, address-books and filters; tagsFile holds tags.
configurationFile = ".config/betterbird/configuration.json"
tagsFile = ".config/betterbird/tags.json"
)
// The five kinds, in the order the tools answer them.
const (
kindIdentities = "identities"
kindCalendars = "calendars"
kindAddressBooks = "address-books"
kindFilters = "filters"
kindTags = "tags"
)
var allKinds = []string{kindIdentities, kindCalendars, kindAddressBooks, kindFilters, kindTags}
// Identity is one identity of a declared account. The account's default identity is the entry that gives no
// address: its name and address are the ones the setting accounts gives (§4). Any other entry is another
// identity of that account, matched by its address.
type Identity struct {
Account string `json:"account"`
Address string `json:"address,omitempty"`
Name string `json:"name,omitempty"`
ReplyTo string `json:"reply-to,omitempty"`
Organization string `json:"organization,omitempty"`
// Signature is the signature's text, a line to an element.
Signature []string `json:"signature,omitempty"`
SignatureHTML bool `json:"signature-html,omitempty"`
// SignatureAboveQuote puts the reply and its signature above the quoted message (Betterbird's "start my
// reply above the quote", signature below the reply); false is below the quote.
SignatureAboveQuote bool `json:"signature-above-quote,omitempty"`
Key string `json:"key,omitempty"`
}
// Calendar is one CalDAV calendar.
type Calendar struct {
Name string `json:"name"`
URL string `json:"url"`
Color string `json:"color,omitempty"`
ReadOnly bool `json:"read-only,omitempty"`
// Reminders absent is on.
Reminders *bool `json:"reminders,omitempty"`
User string `json:"user,omitempty"`
// Invitations is the identities entry whose address answers invitations; absent, none.
Invitations string `json:"invitations,omitempty"`
Key string `json:"key,omitempty"`
}
// AddressBook is one CardDAV address book.
type AddressBook struct {
Name string `json:"name"`
URL string `json:"url"`
User string `json:"user,omitempty"`
Key string `json:"key,omitempty"`
}
// Tag is one tag, keyed in the setting by its own key.
type Tag struct {
Name string `json:"name"`
Color string `json:"color,omitempty"`
// Order is its place among the tags, from 1.
Order int `json:"order,omitempty"`
}
// Filter is one message filter of an account, in its place in the account's list.
type Filter struct {
Name string `json:"name"`
Enabled bool `json:"enabled"`
// When it runs: new-mail, manual, after-junk, sending, archiving, periodic.
When []string `json:"when"`
// Match is all (every condition holds), any (one does) or every-message (no condition).
Match string `json:"match"`
Conditions []Condition `json:"conditions,omitempty"`
Actions []Action `json:"actions"`
}
// Condition is one of a filter's conditions, in Betterbird's own words: a field, an operator, a value.
type Condition struct {
Field string `json:"field"`
Op string `json:"op"`
Value string `json:"value,omitempty"`
}
// Action is one of a filter's actions, in Betterbird's own words. A folder is named by its path within the
// account (INBOX/Receipts), and becomes that folder's address on each node (§8).
type Action struct {
Action string `json:"action"`
Folder string `json:"folder,omitempty"`
Value string `json:"value,omitempty"`
}
// Configuration is the five settings as the machine has them.
type Configuration struct {
Identities map[string]*Identity `json:"identities,omitempty"`
Calendars map[string]*Calendar `json:"calendars,omitempty"`
AddressBooks map[string]*AddressBook `json:"address-books,omitempty"`
Filters map[string][]Filter `json:"filters,omitempty"`
Tags map[string]*Tag `json:"tags,omitempty"`
}
var (
entryName = regexp.MustCompile(`^[a-z][a-z0-9]{0,30}$`)
tagKey = regexp.MustCompile(`^[$A-Za-z0-9_.\-]{1,64}$`)
colorWord = regexp.MustCompile(`^#[0-9A-Fa-f]{6}$`)
// secretName is a field's name that says it holds a credential: such a field is withheld from every answer (§2).
secretName = regexp.MustCompile(`(?i)(password|passwd|token|secret|cookie)`)
// userPassword is an address whose user part carries a password: scheme://user:password@host.
userPassword = regexp.MustCompile(`(?i)\b([a-z][a-z0-9+.\-]*://[^/:@\s]*):[^@/\s]+@`)
)
// The words a filter is written in.
var (
filterWhen = map[string]int{"new-mail": 0x1, "manual": 0x10, "after-junk": 0x20, "sending": 0x40,
"archiving": 0x80, "periodic": 0x100}
filterFields = map[string]bool{"subject": true, "from": true, "to": true, "cc": true, "to or cc": true,
"all addresses": true, "body": true, "date": true, "priority": true, "status": true, "age in days": true,
"size": true, "tag": true, "junk status": true, "junk percent": true, "junk score origin": true,
"has attachment status": true, "date received": true}
filterOps = map[string]bool{"contains": true, "doesn't contain": true, "is": true, "isn't": true,
"begins with": true, "ends with": true, "is empty": true, "isn't empty": true, "is before": true,
"is after": true, "is greater than": true, "is less than": true, "is higher than": true,
"is lower than": true, "is in ab": true, "isn't in ab": true, "matches": true, "doesn't match": true}
// filterActions are the actions a declared filter may take, and whether each names a folder or a value.
filterActions = map[string]string{"Move to folder": "folder", "Copy to folder": "folder",
"Change priority": "value", "Delete": "", "Mark read": "", "Mark unread": "", "Mark flagged": "",
"Ignore thread": "", "Ignore subthread": "", "Watch thread": "", "AddTag": "value", "JunkScore": "value",
"Forward": "value", "Stop execution": "", "Delete from Pop3 server": "", "Leave on Pop3 server": "",
"Fetch body from Pop3Server": ""}
)
// readConfiguration reads the five settings. A missing file holds none of its kinds (the module is older
// here); a file that carries a key shaped like a credential, an address with a password, an unknown field or a
// value Betterbird cannot be given is refused whole, naming the entry and never a value.
func (m *Machine) readConfiguration() (Configuration, error) {
var c Configuration
var problems []string
for _, file := range []string{configurationFile, tagsFile} {
raw, err := readBounded(m.home(file))
if errors.Is(err, os.ErrNotExist) {
continue
}
if err != nil {
return Configuration{}, err
}
if err := decodeInto(raw, file, &c); err != nil {
problems = append(problems, err.Error())
}
}
if len(problems) > 0 {
return Configuration{}, errors.New(strings.Join(problems, "; "))
}
if p := configurationProblems(c); len(p) > 0 {
return Configuration{}, fmt.Errorf("the configuration is refused: %s", strings.Join(p, "; "))
}
c.dropNulls()
for _, cal := range c.Calendars {
cal.Color = strings.ToLower(cal.Color)
}
for _, t := range c.Tags {
t.Color = strings.ToUpper(t.Color)
}
return c, nil
}
// decodeInto reads one settings file into c, refusing a credential-shaped key or an unknown field.
func decodeInto(raw []byte, file string, c *Configuration) error {
var tree any
if err := json.Unmarshal(raw, &tree); err != nil {
return fmt.Errorf("~/%s is not JSON: %v", file, err)
}
if where := secretKeyIn(tree, ""); where != "" {
return fmt.Errorf("~/%s carries %s, a key shaped like a credential: the configuration is declared without "+
"any password, token or cookie (novox/hq ADR 0329 §2); take it out at the controller's terminal", file, where)
}
dec := json.NewDecoder(bytes.NewReader(raw))
dec.DisallowUnknownFields()
var into Configuration
if err := dec.Decode(&into); err != nil {
return fmt.Errorf("~/%s does not hold the shape of the settings it takes: %v", file, err)
}
if file == tagsFile {
if into.Identities != nil || into.Calendars != nil || into.AddressBooks != nil || into.Filters != nil {
return fmt.Errorf("~/%s holds only tags", file)
}
c.Tags = into.Tags
return nil
}
if into.Tags != nil {
return fmt.Errorf("~/%s holds no tags: tags are their own setting", file)
}
c.Identities, c.Calendars, c.AddressBooks, c.Filters = into.Identities, into.Calendars, into.AddressBooks, into.Filters
return nil
}
// dropNulls takes out the entries a node's layer set to null: not on this node.
func (c *Configuration) dropNulls() {
for k, v := range c.Identities {
if v == nil {
delete(c.Identities, k)
}
}
for k, v := range c.Calendars {
if v == nil {
delete(c.Calendars, k)
}
}
for k, v := range c.AddressBooks {
if v == nil {
delete(c.AddressBooks, k)
}
}
for k, v := range c.Filters {
if v == nil {
delete(c.Filters, k)
}
}
for k, v := range c.Tags {
if v == nil {
delete(c.Tags, k)
}
}
}
// addressPassword says whether an address carries a password in its user part.
func addressPassword(s string) bool {
if userPassword.MatchString(s) {
return true
}
if u, err := url.Parse(s); err == nil && u.User != nil {
_, has := u.User.Password()
return has
}
return false
}
// withoutPassword is an address with the password of its user part as ‹withheld› (§2).
func withoutPassword(s string) string {
return userPassword.ReplaceAllString(s, "${1}:‹withheld›@")
}
// davURL says what is wrong with a calendar's or an address book's address, or "".
func davURL(s string) string {
if addressPassword(s) {
return "its address carries a password in its user part, which no setting holds (novox/hq ADR 0329 §2): " +
"Betterbird asks for the password at the first sign-in"
}
u, err := url.Parse(s)
if err != nil || (u.Scheme != "https" && u.Scheme != "http") || u.Host == "" {
return "its address is an https:// (or http://) address of the server"
}
if strings.ContainsAny(s, lineEnds+`"\`) {
return "its address holds a quote, a backslash or a line break"
}
return ""
}
// configurationProblems is everything wrong with the five settings, each said by its kind and entry name.
func configurationProblems(c Configuration) []string {
var out []string
say := func(kind, name, format string, args ...any) {
out = append(out, fmt.Sprintf("%s %s: ", kind, name)+fmt.Sprintf(format, args...))
}
text := func(kind, name, field, v string) {
if !plainText.MatchString(v) {
say(kind, name, "%s holds a quote, a backslash or a line break", field)
}
}
for _, name := range sortedKeys(c.Identities) {
id := c.Identities[name]
if id == nil {
continue
}
if !entryName.MatchString(name) {
say(kindIdentities, name, "a name is lower-case letters and digits, starting with a letter")
}
if !accountName.MatchString(id.Account) {
say(kindIdentities, name, "account names an entry of the setting accounts")
}
if id.Address == "" && id.Name != "" {
say(kindIdentities, name, "the account's default identity (no address) takes its name from the setting accounts")
}
if id.Address != "" && !addressShape.MatchString(id.Address) {
say(kindIdentities, name, "the address is not a mail address")
}
if id.ReplyTo != "" && !addressShape.MatchString(id.ReplyTo) {
say(kindIdentities, name, "reply-to is not a mail address")
}
for field, v := range map[string]string{"name": id.Name, "organization": id.Organization} {
text(kindIdentities, name, field, v)
}
for _, line := range id.Signature {
if strings.ContainsAny(line, lineEnds) {
say(kindIdentities, name, "a signature's line holds a line break: each line is its own element")
break
}
}
if id.Key != "" && !profileKey.MatchString(id.Key) {
say(kindIdentities, name, "key is an identity key of the profile, such as id3")
}
}
for _, name := range sortedKeys(c.Calendars) {
cal := c.Calendars[name]
if cal == nil {
continue
}
if !entryName.MatchString(name) {
say(kindCalendars, name, "a name is lower-case letters and digits, starting with a letter")
}
if why := davURL(cal.URL); why != "" {
say(kindCalendars, name, "%s", why)
}
if cal.Color != "" && !colorWord.MatchString(cal.Color) {
say(kindCalendars, name, "a colour is #rrggbb")
}
if cal.Invitations != "" && c.Identities[cal.Invitations] == nil {
say(kindCalendars, name, "invitations names %s, which the setting identities does not hold", cal.Invitations)
}
text(kindCalendars, name, "name", cal.Name)
text(kindCalendars, name, "user", cal.User)
if cal.Key != "" && !tagKey.MatchString(cal.Key) {
say(kindCalendars, name, "key is a calendar key of the profile")
}
}
for _, name := range sortedKeys(c.AddressBooks) {
ab := c.AddressBooks[name]
if ab == nil {
continue
}
if !entryName.MatchString(name) {
say(kindAddressBooks, name, "a name is lower-case letters and digits, starting with a letter")
}
if why := davURL(ab.URL); why != "" {
say(kindAddressBooks, name, "%s", why)
}
text(kindAddressBooks, name, "name", ab.Name)
text(kindAddressBooks, name, "user", ab.User)
if ab.Key != "" && !profileKey.MatchString(ab.Key) {
say(kindAddressBooks, name, "key is an address book key of the profile")
}
}
for _, key := range sortedKeys(c.Tags) {
t := c.Tags[key]
if t == nil {
continue
}
if !tagKey.MatchString(key) {
say(kindTags, key, "a tag's key is letters, digits, $, _, . or -")
}
if t.Color != "" && !colorWord.MatchString(t.Color) {
say(kindTags, key, "a colour is #rrggbb")
}
text(kindTags, key, "name", t.Name)
}
for _, account := range sortedKeys(c.Filters) {
if !accountName.MatchString(account) {
say(kindFilters, account, "filters are keyed by an account's name in the setting accounts")
}
for i, f := range c.Filters[account] {
for _, p := range filterProblems(f) {
say(kindFilters, account, "filter %d: %s", i+1, p)
}
}
}
return out
}
// filterValue is what a filter's name, value or folder may hold: what its file can carry without quoting.
var filterValue = regexp.MustCompile(`^[^"\\()\n\r]*$`)
func filterProblems(f Filter) []string {
var out []string
if f.Name == "" || !filterValue.MatchString(f.Name) {
out = append(out, "a name, without a quote, a backslash or a parenthesis")
}
if len(f.When) == 0 {
out = append(out, "when says when it runs: new-mail, manual, after-junk, sending, archiving or periodic")
}
for _, w := range f.When {
if _, ok := filterWhen[w]; !ok {
out = append(out, fmt.Sprintf("when %q is new-mail, manual, after-junk, sending, archiving or periodic", w))
}
}
switch f.Match {
case "all", "any":
if len(f.Conditions) == 0 {
out = append(out, "match all or any needs a condition")
}
case "every-message":
if len(f.Conditions) > 0 {
out = append(out, "match every-message takes no condition")
}
default:
out = append(out, "match is all, any or every-message")
}
for i, c := range f.Conditions {
if !filterFields[c.Field] && !(strings.HasPrefix(c.Field, `"`) && strings.HasSuffix(c.Field, `"`) && len(c.Field) > 2) {
out = append(out, fmt.Sprintf("condition %d: the field %q is not one Betterbird filters on", i+1, c.Field))
}
if !filterOps[c.Op] {
out = append(out, fmt.Sprintf("condition %d: the operator %q is not one of Betterbird's", i+1, c.Op))
}
if !filterValue.MatchString(c.Value) {
out = append(out, fmt.Sprintf("condition %d: the value holds a quote, a backslash, a parenthesis or a line break", i+1))
}
}
if len(f.Actions) == 0 {
out = append(out, "a filter takes an action")
}
for i, a := range f.Actions {
takes, ok := filterActions[a.Action]
switch {
case !ok:
out = append(out, fmt.Sprintf("action %d: %q is not an action a declared filter takes", i+1, a.Action))
case takes == "folder" && (a.Folder == "" || a.Value != ""):
out = append(out, fmt.Sprintf("action %d: %s names a folder, by its path within the account", i+1, a.Action))
case takes == "value" && (a.Value == "" || a.Folder != ""):
out = append(out, fmt.Sprintf("action %d: %s takes a value", i+1, a.Action))
case takes == "" && (a.Value != "" || a.Folder != ""):
out = append(out, fmt.Sprintf("action %d: %s takes no value", i+1, a.Action))
}
if !filterValue.MatchString(a.Value) || !filterValue.MatchString(a.Folder) || strings.HasPrefix(a.Folder, "/") ||
strings.Contains(a.Folder, "://") {
out = append(out, fmt.Sprintf("action %d: a folder is a path within the account, and a value holds no quote, "+
"backslash, parenthesis or line break", i+1))
}
if a.Action == "Forward" && !addressShape.MatchString(a.Value) {
out = append(out, fmt.Sprintf("action %d: Forward names a mail address", i+1))
}
}
return out
}
// sortedFilterAccounts are the accounts filters are declared for.
func (c Configuration) sortedFilterAccounts() []string {
out := make([]string, 0, len(c.Filters))
for a := range c.Filters {
out = append(out, a)
}
sort.Strings(out)
return out
}
@@ -0,0 +1,518 @@
package main
import (
"encoding/json"
"os"
"path/filepath"
"reflect"
"strings"
"testing"
)
// Betterbird's whole configuration (novox/hq ADR 0329, issue 586): identities, calendars, address books, filters
// and tags declared mesh-wide, adopted or created as accounts are, filters owned whole per declared account, what
// Betterbird rewrites written only while it is closed, derived from the nodes' profiles one layer at a time, and
// shown with the operator's own values but never a credential.
// profileWithConfiguration adds to profileWithAccounts' profile: account1's directory and signature, CalDAV
// calendars (one address twice), an ICS and an Owl calendar, a local and a CardDAV address book, two tags, and
// account1's filter file holding one filter of the person's.
func (f *fake) profileWithConfiguration() {
f.t.Helper()
prefs := f.read(profileDir+"/prefs.js") +
`user_pref("mail.server.server1.directory-rel", "[ProfD]ImapMail/imap.example.org");` + "\n" +
`user_pref("mail.server.server2.directory-rel", "[ProfD]Mail/Local Folders");` + "\n" +
`user_pref("mail.identity.id1.htmlSigText", "Regards");` + "\n" +
`user_pref("calendar.registry.c1.type", "caldav");` + "\n" +
`user_pref("calendar.registry.c1.uri", "https://dav.example.org/cal/home/");` + "\n" +
`user_pref("calendar.registry.c1.name", "Home");` + "\n" +
`user_pref("calendar.registry.c1.color", "#3366ff");` + "\n" +
`user_pref("calendar.registry.c2.type", "caldav");` + "\n" +
`user_pref("calendar.registry.c2.uri", "https://dav.example.org/cal/shared/");` + "\n" +
`user_pref("calendar.registry.c2.name", "Shared A");` + "\n" +
`user_pref("calendar.registry.c3.type", "caldav");` + "\n" +
`user_pref("calendar.registry.c3.uri", "https://dav.example.org/cal/shared/");` + "\n" +
`user_pref("calendar.registry.c3.name", "Shared B");` + "\n" +
`user_pref("calendar.registry.c4.type", "ics");` + "\n" +
`user_pref("calendar.registry.c4.uri", "https://feeds.example.org/secret-token-abc.ics");` + "\n" +
`user_pref("calendar.registry.c4.name", "Holidays");` + "\n" +
`user_pref("calendar.registry.c5.type", "owl");` + "\n" +
`user_pref("calendar.registry.c5.uri", "owl://work@corp.example.com/calendar");` + "\n" +
`user_pref("calendar.registry.c5.name", "Work");` + "\n" +
`user_pref("calendar.list.sortOrder", "c1 c2 c3 c4 c5");` + "\n" +
`user_pref("ldap_2.servers.pab.description", "Personal Address Book");` + "\n" +
`user_pref("ldap_2.servers.pab.dirType", 101);` + "\n" +
`user_pref("ldap_2.servers.ab1.description", "Contacts");` + "\n" +
`user_pref("ldap_2.servers.ab1.dirType", 102);` + "\n" +
`user_pref("ldap_2.servers.ab1.carddav.url", "https://dav.example.org/card/me/");` + "\n" +
`user_pref("ldap_2.servers.ab1.carddav.username", "me");` + "\n" +
`user_pref("mailnews.tags.$label1.tag", "Important");` + "\n" +
`user_pref("mailnews.tags.$label1.color", "#FF0000");` + "\n" +
`user_pref("mailnews.tags.personal.tag", "Personal");` + "\n" +
`user_pref("mailnews.tags.personal.color", "#00AA00");` + "\n"
f.write(profileDir+"/prefs.js", prefs)
f.write(profileDir+"/ImapMail/imap.example.org/"+filterFileName, "version=\"9\"\nlogging=\"no\"\nname=\"Old\"\n"+
"enabled=\"yes\"\ntype=\"17\"\naction=\"Mark read\"\ncondition=\"AND (from,contains,news)\"\n")
f.write(profileDir+"/Mail/Local Folders/"+filterFileName, "version=\"9\"\nlogging=\"no\"\n")
f.write(profileDir+"/calendar-data/cache.sqlite", "events")
}
const declaredConfiguration = `{
"identities": {
"mail": {"account": "mail", "signature": ["-- ", "Other"], "signature-above-quote": true},
"mailalias": {"account": "mail", "address": "alias@mail.example.org", "name": "Alias", "reply-to": "other@mail.example.org"}
},
"calendars": {
"home": {"name": "Home", "url": "https://dav.example.org/cal/home", "color": "#3366CC", "user": "other", "invitations": "mail"},
"work": {"name": "Team", "url": "https://dav.example.org/cal/team/"},
"shared": {"name": "Shared", "url": "https://dav.example.org/cal/shared/"}
},
"address-books": {
"contacts": {"name": "Contacts", "url": "https://dav.example.org/card/me/", "user": "other"},
"team": {"name": "Team", "url": "https://dav.example.org/card/team/"}
},
"filters": {}
}`
const declaredTags = `{"tags": {"$label1": {"name": "Important", "color": "#ff0000", "order": 1}, "todo": {"name": "To do", "color": "#0000ff", "order": 2}}}`
func (f *fake) declareConfiguration(body, tags string) {
f.write(testHome+"/"+configurationFile, body)
f.write(testHome+"/"+tagsFile, tags)
}
func entriesBy(u UserJS) map[string]resolvedEntry {
out := map[string]resolvedEntry{}
for _, e := range u.Configuration {
out[e.Kind+"/"+e.Name] = e
}
return out
}
// §2, §4, §6: the settings' shapes; a credential, an address with a password, a key on a tag and the four new
// prefixes in prefs are refused, said without the value.
func TestTheSettingsRefuseACredentialAPasswordedAddressATagsKeyAndTheirPrefixesInPrefs(t *testing.T) {
f := newFake(t)
f.declareConfiguration(declaredConfiguration, declaredTags)
c, err := f.readConfiguration()
if err != nil || len(c.Identities) != 2 || len(c.Calendars) != 3 || len(c.AddressBooks) != 2 || len(c.Tags) != 2 {
t.Fatalf("the declared shapes are taken: %+v %v", c, err)
}
if c.Calendars["home"].Color != "#3366cc" || c.Tags["todo"].Color != "#0000FF" {
t.Errorf("colours are said as Betterbird keeps them: %+v %+v", c.Calendars["home"], c.Tags["todo"])
}
for _, bad := range []struct{ conf, tags, says string }{
{`{"calendars": {"home": {"name": "H", "url": "https://dav.example.org/c/", "password": "hunter2"}}}`, `{}`, "shaped like a credential"},
{`{"calendars": {"home": {"name": "H", "url": "https://me:hunter2@dav.example.org/c/"}}}`, `{}`, "carries a password"},
{`{"address-books": {"b": {"name": "B", "url": "https://me:hunter2@dav.example.org/b/"}}}`, `{}`, "carries a password"},
{`{}`, `{"tags": {"todo": {"name": "To do", "key": "x"}}}`, "does not hold the shape"},
{`{"tags": {"todo": {"name": "To do"}}}`, `{}`, "holds no tags"},
{`{"identities": {"x": {"account": "mail", "name": "Me"}}}`, `{}`, "default identity"},
{`{"filters": {"mail": [{"name": "F", "enabled": true, "when": ["manual"], "match": "all", "actions": [{"action": "Delete"}]}]}}`, `{}`, "needs a condition"},
{`{"filters": {"mail": [{"name": "F", "enabled": true, "when": ["manual"], "match": "every-message", "actions": [{"action": "Move to folder", "folder": "imap://x@y/INBOX"}]}]}}`, `{}`, "path within the account"},
} {
f.declareConfiguration(bad.conf, bad.tags)
_, err := f.readConfiguration()
if err == nil || !strings.Contains(err.Error(), bad.says) || strings.Contains(err.Error(), "hunter2") {
t.Errorf("%s %s: %v", bad.conf, bad.tags, err)
}
}
// A key is an identity's, a calendar's or an address book's.
f.declareConfiguration(`{"identities": {"x": {"account": "mail", "address": "x@mail.example.org", "key": "id9"}},
"calendars": {"c": {"name": "C", "url": "https://dav.example.org/c/", "key": "c9"}},
"address-books": {"b": {"name": "B", "url": "https://dav.example.org/b/", "key": "ab9"}}}`, `{}`)
if _, err := f.readConfiguration(); err != nil {
t.Fatal(err)
}
// prefs refuses the four prefixes the new settings own, and the account's and a credential's as before.
for _, name := range []string{"calendar.registry.c1.color", "calendar.list.sortOrder", "ldap_2.servers.ab1.description",
"mailnews.tags.$label1.color", "mail.identity.id1.htmlSigText", "signon.rememberSignons"} {
body, _ := json.Marshal(map[string]any{"prefs": map[string]any{name: "x"}})
if _, err := parseDeclaredPrefs(body); err == nil || !strings.Contains(err.Error(), name) {
t.Errorf("%s: %v", name, err)
}
}
got, err := parseDeclaredPrefs([]byte(`{"prefs": {"mail.uifontsize": 14, "mail.chat.enabled": false, "calendar.alarms.show": null,
"intl.date_time.pattern_override.date_short": "yyyy-MM-dd"}}`))
if err != nil || !reflect.DeepEqual(got, map[string]string{"mail.uifontsize": "14", "mail.chat.enabled": "false",
"intl.date_time.pattern_override.date_short": `"yyyy-MM-dd"`}) {
t.Fatalf("one entry per preference, null taking one out: %v %v", got, err)
}
if _, err := parseDeclaredPrefs([]byte(`{"prefs": {"mail.uifontsize": 14.5}}`)); err == nil {
t.Fatal("a number Betterbird does not keep is refused")
}
}
// §7, §9: each kind is adopted, created or refused as accounts are; the declared fields go into the user.js, and
// what Betterbird rewrites waits for it to close; leaving the setting releases or removes, never events.
func TestEachKindIsAdoptedOrCreatedAndWhatBetterbirdRewritesWaitsForItToClose(t *testing.T) {
f := newFake(t)
f.profileWithAccounts()
f.profileWithConfiguration()
f.noPackage()
links(t, nil)
f.declare("// none")
f.declareAccounts(adoptable)
f.declareConfiguration(declaredConfiguration, declaredTags)
before := f.read(profileDir + "/prefs.js")
// Betterbird runs: the user.js is placed, prefs.js and the filter files wait (§9).
u, err := f.userJS(true)
if err != nil {
t.Fatal(err)
}
by := entriesBy(u)
for want, how := range map[string][2]string{
"identities/mail": {adopted, "id1"}, "identities/mailalias": {created, "ident_mesh_mailalias"},
"calendars/home": {adopted, "c1"}, "calendars/work": {created, "mesh_work"}, "calendars/shared": {refused, ""},
"address-books/contacts": {adopted, "ab1"}, "address-books/team": {created, "mesh_team"},
"tags/$label1": {adopted, "$label1"}, "tags/todo": {created, "todo"},
} {
if e := by[want]; e.How != how[0] || e.Key != how[1] {
t.Errorf("%s: %+v, want %v", want, e, how)
}
}
if why := by["calendars/shared"].Why; !strings.Contains(why, "2 entries") || !strings.Contains(why, "c2, c3") ||
!strings.Contains(why, "key in this node's layer") {
t.Errorf("an ambiguous match is refused with the keys and the remedy: %s", why)
}
placed := f.read(profileDir + "/user.js")
for _, want := range []string{
`user_pref("mail.identity.id1.htmlSigText", "-- \nOther");`,
`user_pref("mail.identity.id1.reply_on_top", 1);`,
`user_pref("mail.identity.id1.sig_bottom", false);`,
`user_pref("mail.identity.ident_mesh_mailalias.useremail", "alias@mail.example.org");`,
`user_pref("mail.identity.ident_mesh_mailalias.smtpServer", "smtp1");`,
`user_pref("calendar.registry.c1.color", "#3366cc");`,
`user_pref("calendar.registry.c1.imip.identity.key", "id1");`,
`user_pref("calendar.registry.mesh_work.type", "caldav");`,
`user_pref("ldap_2.servers.ab1.carddav.username", "other");`,
`user_pref("ldap_2.servers.mesh_team.dirType", 102);`,
`user_pref("mailnews.tags.todo.color", "#0000FF");`,
`user_pref("mailnews.tags.todo.ordinal", "1002");`,
} {
if !strings.Contains(placed, want) {
t.Errorf("the user.js lacks %s", want)
}
}
for _, never := range []string{"calendar.registry.c2.", "calendar.registry.c3.", "calendar.registry.c4.", "calendar.registry.c5.",
"ldap_2.servers.pab.", "mailnews.tags.personal.", "calendar.list.sortOrder"} {
if strings.Contains(placed, never) {
t.Errorf("the user.js writes %s, which nothing declares or which is refused", never)
}
}
if !strings.HasPrefix(u.Closed, "not written") || !strings.Contains(u.Closed, "running") ||
u.Waiting[kindIdentities] != 1 || u.Waiting[kindCalendars] != 1 {
t.Fatalf("while Betterbird runs prefs.js is not written, and the waiting changes are counted: %q %v", u.Closed, u.Waiting)
}
if f.read(profileDir+"/prefs.js") != before {
t.Fatal("prefs.js changed while Betterbird runs")
}
ch, _ := f.Check()
waits := false
for _, x := range ch.Findings {
waits = waits || (strings.Contains(x.What, "change(s) to calendars are not written yet") && strings.Contains(x.Do, "betterbird_restart"))
}
if !waits {
t.Fatalf("the check counts the waiting changes per kind: %+v", ch.Findings)
}
// A key in the node's layer settles the ambiguous calendar; one the profile does not hold is refused.
f.declareConfiguration(strings.Replace(declaredConfiguration, `"url": "https://dav.example.org/cal/shared/"}`,
`"url": "https://dav.example.org/cal/shared/", "key": "c3"}`, 1), declaredTags)
if u, _ = f.userJS(false); entriesBy(u)["calendars/shared"].Key != "c3" || entriesBy(u)["calendars/shared"].How != adopted {
t.Fatalf("the named key is adopted: %+v", entriesBy(u)["calendars/shared"])
}
f.declareConfiguration(strings.Replace(declaredConfiguration, `"url": "https://dav.example.org/cal/shared/"}`,
`"url": "https://dav.example.org/cal/shared/", "key": "c9"}`, 1), declaredTags)
if u, _ = f.userJS(false); entriesBy(u)["calendars/shared"].How != refused || !strings.Contains(entriesBy(u)["calendars/shared"].Why, "c9") {
t.Fatalf("a key the profile does not hold is refused: %+v", entriesBy(u)["calendars/shared"])
}
f.declareConfiguration(strings.Replace(declaredConfiguration, `"url": "https://dav.example.org/cal/shared/"}`,
`"url": "https://dav.example.org/cal/shared/", "key": "c1"}`, 1), declaredTags)
if u, _ = f.userJS(false); entriesBy(u)["calendars/shared"].How != refused {
t.Fatalf("a key whose entry does not match is refused: %+v", entriesBy(u)["calendars/shared"])
}
f.declareConfiguration(declaredConfiguration, declaredTags)
// Closed: the lists gain the module's own keys at their end, every other line as it was.
stopBetterbird(t, f)
f.write(profileDir+"/lock", "")
if err := f.Prepare(); err == nil || !strings.Contains(err.Error(), "locked") {
t.Fatalf("the profile's lock holds the writes back: %v", err)
}
os.Remove(filepath.Join(f.Root, profileDir, "lock"))
if err := f.Prepare(); err != nil && !strings.Contains(err.Error(), "calendars shared is refused") {
t.Fatal(err)
}
after := f.read(profileDir + "/prefs.js")
for _, want := range []string{`user_pref("mail.account.account1.identities", "id1,ident_mesh_mailalias");`,
`user_pref("calendar.list.sortOrder", "c1 c2 c3 c4 c5 mesh_work");`} {
if !strings.Contains(after, want) {
t.Errorf("prefs.js lacks %s:\n%s", want, after)
}
}
bl, al := strings.Split(before, "\n"), strings.Split(after, "\n")
for i := range bl {
if bl[i] != al[i] && !strings.Contains(bl[i], ".identities") && !strings.Contains(bl[i], "sortOrder") {
t.Errorf("line %d changed: %q to %q", i+1, bl[i], al[i])
}
}
// Leaving: an adopted entry is released (its lines go, the profile keeps it), a created one's preferences are
// removed; the calendar's events stay.
f.write(profileDir+"/prefs.js", f.read(profileDir+"/prefs.js")+
`user_pref("calendar.registry.mesh_work.calendarSyncState", "x");`+"\n"+`user_pref("mailnews.tags.todo.tag", "To do");`+"\n")
f.declareConfiguration(`{"identities": {"mail": {"account": "mail"}}}`, `{"tags": {}}`)
if err := f.Prepare(); err != nil {
t.Fatal(err)
}
final := f.read(profileDir + "/prefs.js")
for _, gone := range []string{"mesh_work", "ident_mesh_mailalias", "mailnews.tags.todo."} {
if strings.Contains(final, gone) {
t.Errorf("what the module created and the setting left is removed: %s is still there", gone)
}
}
for _, kept := range []string{`calendar.registry.c1.uri`, `ldap_2.servers.ab1.carddav.url`, `mailnews.tags.$label1.tag`,
`calendar.registry.c4.uri`} {
if !strings.Contains(final, kept) {
t.Errorf("released or undeclared, %s stays the person's", kept)
}
}
if f.read(profileDir+"/calendar-data/cache.sqlite") != "events" {
t.Error("a calendar's events are never touched")
}
if placed := f.read(profileDir + "/user.js"); strings.Contains(placed, "calendar.registry.c1.") || strings.Contains(placed, "ab1") {
t.Errorf("a released entry's lines leave the user.js:\n%s", placed)
}
}
const declaredFilters = `{"filters": {"mail": [
{"name": "Receipts", "enabled": true, "when": ["new-mail", "manual"], "match": "all",
"conditions": [{"field": "subject", "op": "contains", "value": "receipt"}],
"actions": [{"action": "Move to folder", "folder": "INBOX/Receipts"}, {"action": "Forward", "value": "archive@mail.example.org"}]},
{"name": "Everything", "enabled": false, "when": ["manual"], "match": "every-message", "actions": [{"action": "Mark read"}]}
]}}`
const renderedFilters = "version=\"9\"\nlogging=\"no\"\n" +
"name=\"Receipts\"\nenabled=\"yes\"\ntype=\"17\"\naction=\"Move to folder\"\n" +
"actionValue=\"imap://other2@imap.example.org/INBOX/Receipts\"\naction=\"Forward\"\nactionValue=\"archive@mail.example.org\"\n" +
"condition=\"AND (subject,contains,receipt)\"\n" +
"name=\"Everything\"\nenabled=\"no\"\ntype=\"16\"\naction=\"Mark read\"\ncondition=\"ALL\"\n"
// §8: an account named in filters has its msgFilterRules.dat written whole from the list, the replaced file kept;
// an account never named keeps its file byte for byte; one taken out keeps the file it last had.
func TestFiltersAreOwnedWholePerDeclaredAccount(t *testing.T) {
f := newFake(t)
f.profileWithAccounts()
f.profileWithConfiguration()
f.noPackage()
links(t, nil)
f.declare("// none")
f.declareAccounts(adoptable)
f.declareConfiguration(declaredFilters, `{"tags": {}}`)
filters := profileDir + "/ImapMail/imap.example.org/" + filterFileName
local := profileDir + "/Mail/Local Folders/" + filterFileName
old, oldLocal := f.read(filters), f.read(local)
u, err := f.userJS(true)
if err != nil {
t.Fatal(err)
}
if u.Waiting[kindFilters] != 1 || f.read(filters) != old {
t.Fatalf("while Betterbird runs the filter file waits: %v", u.Waiting)
}
stopBetterbird(t, f)
if err := f.Prepare(); err != nil {
t.Fatal(err)
}
if got := f.read(filters); got != renderedFilters {
t.Fatalf("the file is the declared list, its folder this node's address:\n%s", got)
}
keptPath := testHome + "/.config/betterbird/kept/60016jsp.default-default/mail." + filterFileName
if f.read(keptPath) != old {
t.Fatal("the replaced file is kept in the module's own directory")
}
if f.read(local) != oldLocal {
t.Fatal("an account never named keeps its file byte for byte")
}
if parsed := parseFilters(renderedFilters, "imap://other2@imap.example.org"); len(parsed.NotDeclarable) != 0 ||
mustJSON(parsed.Filters) != mustJSON(mustConfig(t, declaredFilters).Filters["mail"]) {
t.Fatalf("the file reads back as the declared list: %+v", parsed)
}
// A filter the person adds in Betterbird is drift first, and gone at the next apply; the kept file stays the first.
f.write(filters, renderedFilters+"name=\"Mine\"\nenabled=\"yes\"\ntype=\"16\"\naction=\"Delete\"\ncondition=\"ALL\"\n")
d, _ := f.Drift(kindFilters)
if len(d.Kinds) != 1 || len(d.Kinds[0].Declared) != 1 || d.Kinds[0].Declared[0].State != "differs" ||
!strings.Contains(mustJSON(d.Kinds[0].Declared[0].Differs), "Mine") {
t.Fatalf("drift shows the added filter: %+v", d)
}
if err := f.Prepare(); err != nil {
t.Fatal(err)
}
if f.read(filters) != renderedFilters || f.read(keptPath) != old {
t.Fatal("the added filter is gone, and the kept file is still the one first replaced")
}
// Taken out of filters: the module stops writing, the last file stays the person's, the kept copy is named.
f.declareConfiguration(`{"filters": {}}`, `{"tags": {}}`)
f.write(filters, renderedFilters+"name=\"Later\"\nenabled=\"yes\"\ntype=\"16\"\naction=\"Delete\"\ncondition=\"ALL\"\n")
lastHad := f.read(filters)
if err := f.Prepare(); err != nil {
t.Fatal(err)
}
if f.read(filters) != lastHad || f.read(keptPath) != old {
t.Fatal("an account taken out keeps its file byte for byte, and the kept copy stays")
}
l, _ := f.List(kindFilters)
if l.Kept["mail"] != "~/.config/betterbird/kept/60016jsp.default-default/mail."+filterFileName {
t.Fatalf("betterbird_filters says where the kept copy is: %+v", l.Kept)
}
d, _ = f.Drift(kindFilters)
if len(d.Kinds[0].Declared) != 0 || len(d.Kinds[0].Undeclared) != 0 {
t.Fatalf("drift no longer counts that account's filters: %+v", d.Kinds[0])
}
}
func mustConfig(t *testing.T, body string) Configuration {
t.Helper()
var c Configuration
if err := decodeInto([]byte(body), configurationFile, &c); err != nil {
t.Fatal(err)
}
return c
}
// §11: per kind and entry, the field that differs with both values; an undeclared entry is listed; once the module's
// user.js is placed, what it declares holds.
func TestDriftNamesEachDifferingFieldAndTheUndeclaredEntries(t *testing.T) {
f := newFake(t)
f.profileWithAccounts()
f.profileWithConfiguration()
f.noPackage()
links(t, nil)
f.declare("// none")
f.declareAccounts(adoptable)
f.declareConfiguration(strings.Replace(declaredConfiguration, `"work": {"name": "Team", "url": "https://dav.example.org/cal/team/"},`, "", 1), declaredTags)
d, err := f.Drift("")
if err != nil {
t.Fatal(err)
}
by := map[string]KindDrift{}
for _, k := range d.Kinds {
by[k.Kind] = k
}
field := func(kind, name, f string) *fieldDiff {
for _, e := range by[kind].Declared {
if e.Name == name {
for _, x := range e.Differs {
if x.Field == f {
return &x
}
}
}
}
return nil
}
if x := field(kindIdentities, "mail", "signature"); x == nil || mustJSON(x.Profile) != `["Regards"]` {
t.Errorf("the signature differs, with both values: %+v", by[kindIdentities])
}
if x := field(kindCalendars, "home", "color"); x == nil || x.Declared != "#3366cc" || x.Profile != "#3366ff" {
t.Errorf("the colour differs: %+v", by[kindCalendars])
}
if x := field(kindAddressBooks, "contacts", "user"); x == nil || x.Profile != "me" {
t.Errorf("the user differs: %+v", by[kindAddressBooks])
}
states := map[string]string{}
for _, k := range d.Kinds {
for _, e := range k.Declared {
states[k.Kind+"/"+e.Name] = e.State
}
}
if states["tags/todo"] != "lacks" || states["calendars/shared"] != refused || states["tags/$label1"] != "holds" ||
states["address-books/team"] != "lacks" {
t.Errorf("lacks, refused and holds: %v", states)
}
undeclared := mustJSON(d)
for _, want := range []string{`"key":"personal"`, `"key":"c2"`, `"key":"c3"`} {
if !strings.Contains(undeclared, want) {
t.Errorf("the undeclared entries are listed: lacks %s", want)
}
}
if _, err := f.userJS(true); err != nil {
t.Fatal(err)
}
d, _ = f.Drift("")
for _, k := range d.Kinds {
for _, e := range k.Declared {
if e.State == "differs" {
t.Errorf("once the user.js is placed, %s %s holds: %+v", k.Kind, e.Name, e)
}
}
}
}
// §1, §2, §13: the listing tools answer the operator's own values, Owl's marked, an ICS calendar by its name only,
// and never a credential: through the tools, a password in an address and a field named for one are withheld.
func TestTheListingsShowTheValuesOwlsMarkedAnICSByNameAndNoCredential(t *testing.T) {
f := newFake(t)
f.profileWithAccounts()
f.profileWithConfiguration()
f.write(profileDir+"/prefs.js", f.read(profileDir+"/prefs.js")+
`user_pref("calendar.registry.c6.type", "caldav");`+"\n"+
`user_pref("calendar.registry.c6.uri", "https://other:hunter2@dav.example.org/cal/x/");`+"\n"+
`user_pref("calendar.registry.c6.name", "Old");`+"\n")
f.write(profileDir+"/logins.json", `{"logins":[{"hostname":"https://dav.example.org","encryptedPassword":"hunter3"}]}`)
links(t, nil)
f.declare("// none")
f.declareAccounts(adoptable)
f.declareConfiguration(declaredConfiguration, declaredTags)
old := machine
machine = f.Machine
t.Cleanup(func() { machine = old })
answers := map[string]string{}
for _, tool := range tools() {
if strings.HasPrefix(tool.Name, "betterbird_") && contains([]string{"betterbird_identities", "betterbird_calendars",
"betterbird_address_books", "betterbird_filters", "betterbird_tags", "betterbird_accounts", "betterbird_drift"}, tool.Name) {
out, err := tool.Run(map[string]any{})
if err != nil {
t.Fatalf("%s: %v", tool.Name, err)
}
answers[tool.Name] = mustJSON(out)
}
}
cals := answers["betterbird_calendars"]
for _, shown := range []string{"https://dav.example.org/cal/home/", `"name":"Work","url":"owl://work@corp.example.com/calendar"`,
`"owl":true`, `"name":"Holidays"`} {
if !strings.Contains(cals, shown) {
t.Errorf("betterbird_calendars shows %s: %s", shown, cals)
}
}
if strings.Contains(cals, "secret-token") {
t.Error("an ICS calendar is listed by its name only")
}
for name, a := range answers {
if strings.Contains(a, "hunter2") || strings.Contains(a, "hunter3") {
t.Errorf("%s carries a credential: %s", name, a)
}
}
if !strings.Contains(cals, "https://other:‹withheld›@dav.example.org/cal/x/") {
t.Errorf("an address's password is shown as ‹withheld›: %s", cals)
}
for _, shown := range []string{`"signature":["Regards"]`, "other@mail.example.org"} {
if !strings.Contains(answers["betterbird_identities"], shown) {
t.Errorf("betterbird_identities shows %s", shown)
}
}
if got := mustJSON(credentialChecked(map[string]any{"name": "x", "password": "p", "nested": []any{map[string]any{"oauthToken": "t"}}})); got !=
`{"name":"x","nested":[{"oauthToken":"‹withheld›"}],"password":"‹withheld›"}` {
t.Errorf("a field named for a credential is withheld: %s", got)
}
// The start's words reach the journal: a refusal names kinds, names and keys, never an address.
stopBetterbird(t, f)
err := f.Prepare()
if err == nil || !strings.Contains(err.Error(), "calendars shared is refused") || strings.Contains(err.Error(), "example.org") {
t.Fatalf("%v", err)
}
}
@@ -0,0 +1,700 @@
package main
// The five settings derived from the desktop nodes' profiles and proposed for the operator's Approve, one layer
// at a time (novox/hq ADR 0329 §10, §12).
//
// betterbird_derive asks every desktop node that holds Betterbird for what its profile holds (its own derive with
// here), and lays the answers out per layer: an entry found on every such node goes into the mesh-wide layer, with
// the value of the node the derive was called on, and the answer lists every field another node holds differently,
// which then shows as drift there; an entry found on some nodes only goes into each of those nodes' layers, so
// nothing is added to another node until the operator moves it. With propose it proposes the next layer through
// the controller's settings verb: first the mesh-wide one, then each node's that needs one, each only once no
// proposal of the module's is open, against the layer as it then stands, which it carries whole beside the derived
// kinds (a layer is replaced whole). The ask shows the values in their shape (ADR 0277 §6); the operator reads them
// whole with `settings proposals <id>` at their desk.
import (
"encoding/json"
"errors"
"fmt"
"regexp"
"sort"
"strings"
)
// derivedEntry is one entry a node's profile holds, in the setting's shape. Match is what makes two nodes' entries
// the same one: an identity's account and address, a calendar's or an address book's address, a tag's key, an
// account's name for its filters.
type derivedEntry struct {
Match string `json:"match"`
Name string `json:"name"`
Value json.RawMessage `json:"value"`
}
// nodeDerivation is what one node's profile holds of the five kinds.
type nodeDerivation struct {
Node string `json:"node"`
Kinds map[string][]derivedEntry `json:"kinds"`
// NotDerived names what a setting does not hold, by name and why: Owl's, an ICS calendar, a local address
// book, an address carrying a password, a filter a setting cannot say.
NotDerived []string `json:"not_derived,omitempty"`
}
var nameChars = regexp.MustCompile(`[^a-z0-9]`)
// entryNameFor is a name of the operator's kind for an entry: lower-case letters and digits from its words,
// unique among those used.
func entryNameFor(words string, fallback string, used map[string]bool) string {
base := nameChars.ReplaceAllString(strings.ToLower(words), "")
if base == "" || base[0] < 'a' || base[0] > 'z' {
base = fallback + base
}
if len(base) > 28 {
base = base[:28]
}
name := base
for i := 2; used[name]; i++ {
name = fmt.Sprintf("%s%d", base, i)
}
used[name] = true
return name
}
func raw(v any) json.RawMessage {
b, _ := json.Marshal(v)
return b
}
// deriveHere reads this node's profile into the five kinds.
func (m *Machine) deriveHere(kinds []string) (nodeDerivation, error) {
d := nodeDerivation{Node: nodeName(), Kinds: map[string][]derivedEntry{}}
p, err := m.profile()
if err != nil {
return d, err
}
prefs := mergedPrefs(p.path)
accounts, _, err := m.readAccounts()
if err != nil {
return d, err
}
nameOf := map[string]string{}
serverOf := map[string]string{}
for _, r := range resolveAccounts(accounts, prefs, m.readRecord(p.Dir)) {
if r.How != refused {
nameOf[r.Keys.Account], serverOf[r.name] = r.name, r.Keys.Server
}
}
want := map[string]bool{}
for _, k := range kinds {
want[k] = true
}
// Identities, by their account's name; the default carries no name or address (the setting accounts' own).
identityName := map[string]string{}
used := map[string]bool{}
byAccount := map[string]bool{}
for _, pi := range m.profileIdentities(prefs) {
account, declared := nameOf[pi.AccountKey]
switch {
case pi.Owl:
if pi.Default {
d.NotDerived = append(d.NotDerived, fmt.Sprintf("identities: %s <%s> is Owl's (novox/hq ADR 0316 §13)",
pi.Value.Name, pi.Value.Address))
}
continue
case !declared:
if !byAccount[pi.AccountKey] {
byAccount[pi.AccountKey] = true
d.NotDerived = append(d.NotDerived, fmt.Sprintf("identities: those of the profile's account %s, which the "+
"setting accounts does not hold", pi.AccountKey))
}
continue
}
v := pi.Value
v.Account = account
e := derivedEntry{}
if pi.Default {
v.Address, v.Name = "", ""
e.Match, e.Name = account+"|", entryNameFor(account, "id", used)
} else {
local, _, _ := strings.Cut(v.Address, "@")
e.Match, e.Name = account+"|"+strings.ToLower(v.Address), entryNameFor(account+local, "id", used)
}
identityName[pi.Key] = e.Name
e.Value = raw(v)
if want[kindIdentities] {
d.Kinds[kindIdentities] = append(d.Kinds[kindIdentities], e)
}
}
// Calendars: CalDAV only; Owl's are listed, an ICS one by its name only, a local one by its name (§13).
used = map[string]bool{}
for _, pc := range profileCalendars(prefs) {
if !want[kindCalendars] {
break
}
switch {
case pc.Kind == "owl" || strings.HasPrefix(pc.Kind, "owl"):
d.NotDerived = append(d.NotDerived, fmt.Sprintf("calendars: %s (%s) is Owl's", pc.Value.Name, pc.Value.URL))
continue
case pc.Kind == "ics":
d.NotDerived = append(d.NotDerived, fmt.Sprintf("calendars: %s is subscribed by an ICS address, which is "+
"often itself the secret: listed by its name only, never declared (novox/hq ADR 0329 §13)", pc.Value.Name))
continue
case pc.Kind != "caldav":
d.NotDerived = append(d.NotDerived, fmt.Sprintf("calendars: %s is %s, kept in the profile: its events are never declared",
pc.Value.Name, pc.Kind))
continue
case addressPassword(pc.Value.URL):
d.NotDerived = append(d.NotDerived, fmt.Sprintf("calendars: %s at %s carries a password in its address: refused "+
"(novox/hq ADR 0329 §2)", pc.Value.Name, withoutPassword(pc.Value.URL)))
continue
}
v := pc.Value
v.Invitations = identityName[pc.IdentityKey]
d.Kinds[kindCalendars] = append(d.Kinds[kindCalendars], derivedEntry{Match: strings.ToLower(strings.TrimRight(v.URL, "/")),
Name: entryNameFor(v.Name, "calendar", used), Value: raw(v)})
}
// Address books: CardDAV only.
used = map[string]bool{}
for _, b := range profileAddressBooks(prefs) {
if !want[kindAddressBooks] {
break
}
switch {
case !b.CardDAV:
d.NotDerived = append(d.NotDerived, fmt.Sprintf("address-books: %s is %s: its contacts are never declared", b.Value.Name, b.Kind))
continue
case addressPassword(b.Value.URL):
d.NotDerived = append(d.NotDerived, fmt.Sprintf("address-books: %s at %s carries a password in its address: "+
"refused (novox/hq ADR 0329 §2)", b.Value.Name, withoutPassword(b.Value.URL)))
continue
}
d.Kinds[kindAddressBooks] = append(d.Kinds[kindAddressBooks], derivedEntry{
Match: strings.ToLower(strings.TrimRight(b.Value.URL, "/")), Name: entryNameFor(b.Value.Name, "book", used), Value: raw(b.Value)})
}
// Tags, by their own key.
if want[kindTags] {
for _, t := range profileTags(prefs) {
d.Kinds[kindTags] = append(d.Kinds[kindTags], derivedEntry{Match: t.Key, Name: t.Key, Value: raw(t.Value)})
}
}
// Filters of each declared account whose file holds one or more: an account with none keeps its file as the
// person's until a setting names it.
if want[kindFilters] {
for _, name := range sortedKeys(serverOf) {
dir := m.accountDirectory(p.path, prefs, serverOf[name])
if dir == "" {
continue
}
text, err := readBounded(dir + "/" + filterFileName)
if err != nil {
continue
}
parsed := parseFilters(string(text), folderBase(prefs, serverOf[name]))
for _, why := range parsed.NotDeclarable {
d.NotDerived = append(d.NotDerived, "filters of "+name+": "+why)
}
if len(parsed.Filters) > 0 {
d.Kinds[kindFilters] = append(d.Kinds[kindFilters], derivedEntry{Match: name, Name: name, Value: raw(parsed.Filters)})
}
}
}
return d, nil
}
// difference is one field another node holds differently from the value the mesh-wide layer gets.
type difference struct {
Kind string `json:"kind"`
Entry string `json:"entry"`
Node string `json:"node"`
Field string `json:"field"`
MeshWide any `json:"mesh_wide"`
There any `json:"there"`
}
// DeriveAnswer is what betterbird_derive answers its caller: the values it derived, which ADR 0329 §10 lets it show.
type DeriveAnswer struct {
// Here is this node's own derivation, when asked with here.
Here *nodeDerivation `json:"here,omitempty"`
// From are the desktop nodes derived from, and NotHeard those holding Betterbird that did not answer.
From []string `json:"derived_from,omitempty"`
NotHeard []string `json:"not_heard,omitempty"`
// Layers are the derived values per layer: "mesh-wide", then each node's own.
Layers map[string]map[string]any `json:"layers,omitempty"`
Differences []difference `json:"differences,omitempty"`
NotDerived map[string][]string `json:"not_derived,omitempty"`
// Next is the layer a propose proposes now, or why none.
Next string `json:"next,omitempty"`
Proposed string `json:"proposed,omitempty"`
Note string `json:"note,omitempty"`
}
const meshWide = "mesh-wide"
// Derive derives the kinds asked for (all five when none) from the desktop nodes, and with propose proposes the
// next layer. prefs are values for the setting prefs to carry in the mesh-wide layer (issue 586: the operator's
// setup moved out of the catalogue); layer names the layer to propose instead of the next in order.
func (m *Machine) Derive(kinds []string, here, propose bool, prefs map[string]any, layer string) (DeriveAnswer, error) {
var a DeriveAnswer
if len(kinds) == 0 {
kinds = allKinds
}
for _, k := range kinds {
if !contains(allKinds, k) {
return a, fmt.Errorf("kinds are %s; %q is not one", strings.Join(allKinds, ", "), k)
}
}
if here {
d, err := m.deriveHere(kinds)
if err != nil {
return a, err
}
a.Here = &d
return a, nil
}
self := nodeName()
if self == "" {
return a, errors.New("this runtime does not say which machine it serves (MESH_NODE), so the node derived on is unknown")
}
holders, err := holdersOfBetterbird()
if err != nil {
return a, err
}
derived := map[string]nodeDerivation{}
for _, n := range holders {
if n == self {
d, err := m.deriveHere(kinds)
if err != nil {
return a, fmt.Errorf("this node's profile: %v", err)
}
derived[n] = d
continue
}
d, err := deriveOn(n, kinds)
if err != nil {
a.NotHeard = append(a.NotHeard, n+": "+err.Error())
continue
}
derived[n] = d
}
if _, ok := derived[self]; !ok {
return a, fmt.Errorf("%s does not hold Betterbird as the controller says: derive on a desktop node", self)
}
a.From = sortedKeys(derived)
a.Layers, a.Differences = layOut(self, derived, kinds)
a.NotDerived = map[string][]string{}
for n, d := range derived {
if len(d.NotDerived) > 0 {
a.NotDerived[n] = d.NotDerived
}
}
if len(a.NotDerived) == 0 {
a.NotDerived = nil
}
if len(prefs) > 0 {
a.Layers[meshWide]["prefs"] = prefs
}
a.Note = "the values each layer would get; an entry on every desktop node is mesh-wide with this node's value, one " +
"on some nodes only is in those nodes' own layers. With propose, the next layer is proposed for the operator's " +
"Approve, carrying the layer as it stands beside these kinds; nothing changes until then (novox/hq ADR 0329 §10)"
if !propose {
return a, nil
}
if len(a.NotHeard) > 0 {
return a, fmt.Errorf("nothing was proposed: %d desktop node(s) holding Betterbird did not answer (%s), and a "+
"layer derived without them would put on every node what only some hold; derive again when they are awake",
len(a.NotHeard), strings.Join(a.NotHeard, "; "))
}
if open, err := openProposals(); err != nil {
return a, fmt.Errorf("nothing was proposed: whether a proposal of the module's is open cannot be read: %v", err)
} else if len(open) > 0 {
return a, fmt.Errorf("nothing was proposed: the module's proposal %s waits for the operator; the next layer is "+
"proposed once it is set or declined (novox/hq ADR 0329 §10, ADR 0277 §10)", strings.Join(open, ", "))
}
order := []string{meshWide}
for _, n := range sortedKeys(a.Layers) {
if n != meshWide {
order = append(order, n)
}
}
if layer != "" {
if _, ok := a.Layers[layer]; !ok {
return a, fmt.Errorf("nothing was proposed: layer is %s", strings.Join(order, " or "))
}
order = []string{layer}
}
for _, l := range order {
node := l
if l == meshWide {
node = ""
}
current, err := readLayer(node)
if err != nil {
return a, fmt.Errorf("nothing was proposed: %v", err)
}
next := withKinds(current, a.Layers[l])
if l == meshWide {
if prefs, ok := next["prefs"]; ok {
body, _ := json.Marshal(map[string]any{"prefs": prefs})
if _, err := parseDeclaredPrefs(body); err != nil {
return a, fmt.Errorf("nothing was proposed: %v", err)
}
}
}
if err := proposable(next, l == meshWide); err != nil {
return a, fmt.Errorf("nothing was proposed for %s: %v", l, err)
}
if mustJSON(next) == mustJSON(current) {
continue
}
a.Next = l
said, err := proposeLayer(node, next, true)
a.Proposed = m.maskValue(said)
if err != nil {
return a, err
}
a.Note = "proposed the " + l + " layer for the operator's Approve; the next layer is proposed by calling again " +
"once this one is set or declined"
return a, nil
}
a.Next = "none: every layer already holds what was derived"
return a, nil
}
// proposable checks a layer's derived kinds as the module would read them, so the operator is never asked to set
// a value the module then refuses. A key to adopt names one node's profile, so the mesh-wide layer holds none (§4).
func proposable(layer map[string]any, mesh bool) error {
conf := map[string]any{}
for _, k := range []string{kindIdentities, kindCalendars, kindAddressBooks, kindFilters} {
if v, ok := layer[k]; ok {
conf[k] = v
}
}
var c Configuration
if err := decodeInto([]byte(mustJSON(conf)), configurationFile, &c); err != nil {
return err
}
if v, ok := layer[kindTags]; ok {
if err := decodeInto([]byte(mustJSON(map[string]any{kindTags: v})), tagsFile, &c); err != nil {
return err
}
}
// A node's layer may name an identity the mesh-wide layer holds for a calendar's invitations: that is judged
// where the layers meet, on the node.
var problems []string
if mesh {
for name, id := range c.Identities {
if id != nil && id.Key != "" {
problems = append(problems, "identities "+name+": a key is given in a node's own layer only")
}
}
for name, cal := range c.Calendars {
if cal != nil && cal.Key != "" {
problems = append(problems, "calendars "+name+": a key is given in a node's own layer only")
}
}
for name, ab := range c.AddressBooks {
if ab != nil && ab.Key != "" {
problems = append(problems, "address-books "+name+": a key is given in a node's own layer only")
}
}
}
for _, p := range configurationProblems(c) {
if !strings.Contains(p, "invitations names") {
problems = append(problems, p)
}
}
if len(problems) > 0 {
return errors.New(strings.Join(problems, "; "))
}
return nil
}
// layOut puts each kind's entries into the layers (§10).
func layOut(self string, derived map[string]nodeDerivation, kinds []string) (map[string]map[string]any, []difference) {
layers := map[string]map[string]any{meshWide: {}}
var diffs []difference
nodes := sortedKeys(derived)
for _, kind := range kinds {
type seen struct {
by map[string]derivedEntry
}
groups := map[string]*seen{}
var order []string
for _, n := range append([]string{self}, nodes...) {
for _, e := range derived[n].Kinds[kind] {
g := groups[e.Match]
if g == nil {
g = &seen{by: map[string]derivedEntry{}}
groups[e.Match] = g
order = append(order, e.Match)
}
if _, already := g.by[n]; !already {
g.by[n] = e
}
}
}
mesh := map[string]any{}
used := map[string]bool{}
for _, match := range order {
g := groups[match]
if len(g.by) == len(nodes) {
mine := g.by[self]
name := uniqueName(mine.Name, used)
var v any
_ = json.Unmarshal(mine.Value, &v)
mesh[name] = v
for _, n := range nodes {
if n == self {
continue
}
diffs = append(diffs, fieldDifferences(kind, name, n, mine.Value, g.by[n].Value)...)
}
}
}
for _, match := range order {
g := groups[match]
if len(g.by) == len(nodes) {
continue
}
for _, n := range sortedKeys(g.by) {
e := g.by[n]
if layers[n] == nil {
layers[n] = map[string]any{}
}
nodeKind, _ := layers[n][kind].(map[string]any)
if nodeKind == nil {
nodeKind = map[string]any{}
layers[n][kind] = nodeKind
}
var v any
_ = json.Unmarshal(e.Value, &v)
nodeKind[uniqueName(e.Name, used)] = v
}
}
if len(mesh) > 0 {
layers[meshWide][kind] = mesh
}
}
return layers, diffs
}
func uniqueName(name string, used map[string]bool) string {
n := name
for i := 2; used[n]; i++ {
n = fmt.Sprintf("%s%d", name, i)
}
used[n] = true
return n
}
// fieldDifferences are the fields a node holds differently from the mesh-wide value of one entry.
func fieldDifferences(kind, entry, node string, mesh, there json.RawMessage) []difference {
var a, b map[string]any
if json.Unmarshal(mesh, &a) != nil || json.Unmarshal(there, &b) != nil {
if mustJSON(json.RawMessage(mesh)) != mustJSON(json.RawMessage(there)) {
var x, y any
_ = json.Unmarshal(mesh, &x)
_ = json.Unmarshal(there, &y)
return []difference{{Kind: kind, Entry: entry, Node: node, Field: "the whole list", MeshWide: x, There: y}}
}
return nil
}
var out []difference
for _, f := range sortedKeys(mergeKeys(a, b)) {
if mustJSON(a[f]) != mustJSON(b[f]) {
out = append(out, difference{Kind: kind, Entry: entry, Node: node, Field: f, MeshWide: a[f], There: b[f]})
}
}
return out
}
func mergeKeys(a, b map[string]any) map[string]bool {
out := map[string]bool{}
for k := range a {
out[k] = true
}
for k := range b {
out[k] = true
}
return out
}
// withKinds is a layer with the derived kinds in place of its own, every other key kept.
func withKinds(layer map[string]any, kinds map[string]any) map[string]any {
out := map[string]any{}
for k, v := range layer {
out[k] = v
}
for k, v := range kinds {
if k == "prefs" {
merged := map[string]any{}
if was, ok := out["prefs"].(map[string]any); ok {
for n, x := range was {
merged[n] = x
}
}
for n, x := range v.(map[string]any) {
merged[n] = x
}
out[k] = merged
continue
}
out[k] = v
}
return out
}
func contains(list []string, s string) bool {
for _, x := range list {
if x == s {
return true
}
}
return false
}
// readLayer is the module's layer as the controller holds it: the mesh-wide one for node "", else that node's.
func readLayer(node string) (map[string]any, error) {
args := map[string]any{"module": "betterbird"}
if node != "" {
args["node"] = node
}
raw, err := asker("seat:mesh-controller.settings", args)
if err != nil {
return nil, fmt.Errorf("the controller's settings verb did not answer: %v", err)
}
said, ok := verbSaid(raw)
if !ok {
return nil, fmt.Errorf("the controller did not answer the layer: %s", said)
}
// A missing layer is said in exactly these words; anything else must read as the layer's JSON. A proposal
// replaces the layer whole, so an answer read wrongly as empty would drop what the layer holds: refused, loudly.
text := strings.TrimSpace(said)
where := "the whole mesh"
if node != "" {
where = node
}
if strings.HasPrefix(text, "betterbird on "+where+": no layer") {
return map[string]any{}, nil
}
out := map[string]any{}
if err := json.NewDecoder(strings.NewReader(text)).Decode(&out); err != nil || !strings.HasPrefix(text, "{") {
first, _, _ := strings.Cut(text, "\n")
return nil, fmt.Errorf("the controller's answer for the layer of betterbird on %s cannot be read as the layer "+
"(it begins %q), so nothing is proposed: a layer is replaced whole and would lose what it holds", where, first)
}
return out, nil
}
// proposeLayer proposes a layer whole for the operator's Approve (ADR 0277); replace says a key the layer had
// and the proposal does not hold is meant to go, which a re-derive needs when an entry left every profile.
func proposeLayer(node string, values map[string]any, replace bool) (string, error) {
body, err := json.Marshal(values)
if err != nil {
return "", err
}
args := map[string]any{"module": "betterbird", "values": string(body), "propose": "true"}
if replace {
args["replace"] = "true"
}
if node != "" {
args["node"] = node
}
raw, err := asker("seat:mesh-controller.settings", args)
if err != nil {
return "", fmt.Errorf("the controller's settings verb did not answer: %v", err)
}
said, ok := verbSaid(raw)
if !ok {
return said, fmt.Errorf("the controller did not take the proposal: %s", said)
}
return said, nil
}
var (
proposalHead = regexp.MustCompile(`^(\S+)\s+(?:set|clear) betterbird on (.+?)\s+\(`)
waitingLine = regexp.MustCompile(`^\s+waiting for the operator until`)
)
// openProposals are the module's proposals that wait for the operator, by id and layer.
func openProposals() ([]string, error) {
raw, err := asker("seat:mesh-controller.settings", map[string]any{"proposals": "true"})
if err != nil {
return nil, err
}
said, ok := verbSaid(raw)
if !ok {
return nil, errors.New(said)
}
var out []string
current := ""
for _, line := range strings.Split(said, "\n") {
if mm := proposalHead.FindStringSubmatch(line); mm != nil {
current = mm[1] + " (" + mm[2] + ")"
continue
}
if !strings.HasPrefix(line, " ") {
current = ""
}
if current != "" && waitingLine.MatchString(line) {
out = append(out, current)
current = ""
}
}
return out, nil
}
// holdersOfBetterbird are the nodes the controller has assigned the module to: the desktop nodes.
func holdersOfBetterbird() ([]string, error) {
raw, err := asker("seat:mesh-controller.nodes", map[string]any{})
if err != nil {
return nil, fmt.Errorf("the controller's nodes verb did not answer: %v", err)
}
said, _ := verbSaid(raw)
var nodes []struct {
Name string `json:"name"`
}
if err := json.Unmarshal([]byte(said), &nodes); err != nil {
return nil, fmt.Errorf("the controller's nodes cannot be read: %v", err)
}
var out []string
for _, n := range nodes {
raw, err := asker("seat:mesh-controller.node", map[string]any{"node": n.Name})
if err != nil {
return nil, fmt.Errorf("the controller's node verb did not answer for %s: %v", n.Name, err)
}
said, _ := verbSaid(raw)
for _, line := range strings.Split(said, "\n") {
if rest, ok := strings.CutPrefix(strings.TrimSpace(line), "assigned:"); ok {
for _, mod := range strings.Split(rest, ",") {
if strings.TrimSpace(mod) == "betterbird" {
out = append(out, n.Name)
}
}
}
}
}
sort.Strings(out)
if len(out) == 0 {
return nil, errors.New("the controller names no node the module is assigned to")
}
return out, nil
}
// deriveOn asks another node's betterbird for what its profile holds.
func deriveOn(node string, kinds []string) (nodeDerivation, error) {
raw, err := asker("betterbird.betterbird_derive@"+node, map[string]any{"here": true, "kinds": kinds})
if err != nil {
return nodeDerivation{}, err
}
said, ok := verbSaid(raw)
if !ok {
return nodeDerivation{}, errors.New(said)
}
var a DeriveAnswer
if err := json.Unmarshal([]byte(said), &a); err != nil || a.Here == nil {
return nodeDerivation{}, fmt.Errorf("its answer cannot be read")
}
a.Here.Node = node
return *a.Here, nil
}
@@ -0,0 +1,263 @@
package main
import (
"encoding/json"
"errors"
"fmt"
"reflect"
"strings"
"testing"
)
// ADR 0329 §10, §12: derived from every desktop node's profile, laid out per layer, and proposed one layer at a
// time, carrying the layer as it stands.
// derivingMesh is two desktop nodes, g14 (this one, f) and shanks (f2), and a server that holds no Betterbird, as the
// controller answers them; layers are the controller's layers, open the proposals that wait.
type derivingMesh struct {
t *testing.T
shanks *fake
layers map[string]map[string]any
open bool
proposed []map[string]any
asked []string
asleep bool
// garbled answers the layer in words that are not its JSON.
garbled bool
}
func (d *derivingMesh) ask(key string, body any) (json.RawMessage, error) {
d.asked = append(d.asked, key)
args, _ := body.(map[string]any)
answer := func(ok bool, text string) (json.RawMessage, error) {
b, _ := json.Marshal(map[string]any{"ok": ok, "output": text})
return b, nil
}
switch {
case key == "seat:mesh-controller.nodes":
return answer(true, `[{"name": "g14"}, {"name": "novox"}, {"name": "shanks"}]`)
case key == "seat:mesh-controller.node":
if args["node"] == "novox" {
return answer(true, "novox\n assigned: postgres, nats\n")
}
return answer(true, fmt.Sprintf("%s\n last heard from here\n assigned: adwaita, betterbird, dunst\n", args["node"]))
case key == "betterbird.betterbird_derive@shanks":
if d.asleep {
return nil, errors.New("nats: no responders available for request")
}
here, err := d.shanks.deriveHere(stringsOf(args["kinds"]))
if err != nil {
d.t.Fatal(err)
}
text, _ := json.Marshal(credentialChecked(DeriveAnswer{Here: &here}))
return json.Marshal(map[string]any{"content": []any{map[string]any{"type": "text", "text": string(text)}}})
case key == "seat:mesh-controller.settings" && args["proposals"] == "true":
if d.open {
return answer(true, "sab12 set betterbird on the whole mesh (accounts, identities)\n by g14.node-tools at 2026-10-13 01:00; fingerprint ab12\n waiting for the operator until 2026-10-14 01:00\n"+
"scd34 set mounts on shanks (sources)\n by g14.node-tools at 2026-10-11 00:05; fingerprint dd0b\n chosen: done\n")
}
return answer(true, "scd34 set mounts on shanks (sources)\n by g14.node-tools; fingerprint dd0b\n chosen: done\n")
case key == "seat:mesh-controller.settings" && args["propose"] == "true":
d.proposed = append(d.proposed, args)
return answer(true, "proposal sab12: Set the settings of betterbird?\n fingerprint ab12")
case key == "seat:mesh-controller.settings":
node, _ := args["node"].(string)
if d.garbled {
return answer(true, "Settings of betterbird on "+node+":\n accounts = {…}\n")
}
layer := d.layers[node]
if layer == nil {
where := node
if where == "" {
where = "the whole mesh"
}
return answer(true, "betterbird on "+where+": no layer — the module's definition says\nbetterbird on "+where+", every value")
}
b, _ := json.MarshalIndent(layer, "", " ")
return answer(true, string(b)+"\n approved by the operator\nbetterbird on the whole mesh, every value and where it comes from:\n")
}
d.t.Fatalf("asked %s %v", key, body)
return nil, nil
}
func stringsOf(v any) []string {
switch x := v.(type) {
case []string:
return x
case []any:
var out []string
for _, s := range x {
out = append(out, s.(string))
}
return out
}
return nil
}
func TestDeriveLaysEachEntryOutPerLayerAndProposesOneLayerAtATime(t *testing.T) {
f := newFake(t)
f.profileWithAccounts()
f.profileWithConfiguration()
f.write(profileDir+"/prefs.js", f.read(profileDir+"/prefs.js")+
`user_pref("calendar.registry.c6.type", "caldav");`+"\n"+
`user_pref("calendar.registry.c6.uri", "https://other:hunter2@dav.example.org/cal/x/");`+"\n"+
`user_pref("calendar.registry.c6.name", "Old");`+"\n")
f.declareAccounts(adoptable)
links(t, nil)
f2 := newFake(t)
f2.profileWithAccounts()
f2.profileWithConfiguration()
// shanks holds the same accounts and calendars, another signature, another tag colour and a calendar of its own.
f2.write(profileDir+"/prefs.js", strings.NewReplacer(`"Regards"`, `"Cheers"`, `"#FF0000"`, `"#CC0000"`).Replace(f2.read(profileDir+"/prefs.js"))+
`user_pref("calendar.registry.s1.type", "caldav");`+"\n"+
`user_pref("calendar.registry.s1.uri", "https://dav.example.org/cal/standup/");`+"\n"+
`user_pref("calendar.registry.s1.name", "Standup");`+"\n")
f2.declareAccounts(adoptable)
mesh := &derivingMesh{t: t, shanks: f2, layers: map[string]map[string]any{"": {"accounts": map[string]any{"mail": "as approved"}}}}
oldNode, oldAsk := nodeName, asker
t.Cleanup(func() { nodeName, asker = oldNode, oldAsk })
nodeName = func() string { return "g14" }
asker = mesh.ask
a, err := f.Derive(nil, false, false, nil, "")
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(a.From, []string{"g14", "shanks"}) || len(a.NotHeard) != 0 {
t.Fatalf("derived from the nodes holding Betterbird: %v %v", a.From, a.NotHeard)
}
whole := mustJSON(a)
mw := a.Layers[meshWide]
ids, _ := mw[kindIdentities].(map[string]any)
if mail, _ := ids["mail"].(map[string]any); mail == nil || mustJSON(mail["signature"]) != `["Regards"]` || mail["address"] != nil {
t.Errorf("an identity on both nodes is mesh-wide with this node's value, the default without its address: %v", ids)
}
cals, _ := mw[kindCalendars].(map[string]any)
if cals["home"] == nil || cals["standup"] != nil {
t.Errorf("a calendar both hold is mesh-wide, one only shanks holds is not: %v", cals)
}
if sh := mustJSON(a.Layers["shanks"]); !strings.Contains(sh, "https://dav.example.org/cal/standup/") {
t.Errorf("an entry on one node only goes into that node's layer: %s", sh)
}
tags, _ := mw[kindTags].(map[string]any)
if l1, _ := tags["$label1"].(map[string]any); l1 == nil || l1["color"] != "#FF0000" {
t.Errorf("the tag is mesh-wide with this node's colour: %v", tags)
}
diffs := map[string]difference{}
for _, d := range a.Differences {
diffs[d.Kind+"/"+d.Entry+"/"+d.Field] = d
}
if d := diffs["identities/mail/signature"]; d.Node != "shanks" || mustJSON(d.There) != `["Cheers"]` {
t.Errorf("the answer lists the field shanks holds differently: %+v", a.Differences)
}
if d := diffs["tags/$label1/color"]; d.There != "#CC0000" || d.MeshWide != "#FF0000" {
t.Errorf("%+v", a.Differences)
}
for _, never := range []string{"secret-token", "hunter2", "owl://", `"Work"`} {
if strings.Contains(mustJSON(a.Layers), never) {
t.Errorf("no ICS address, credential or Owl calendar is derived: %s in %s", never, mustJSON(a.Layers))
}
}
if !strings.Contains(whole, "Holidays") || !strings.Contains(whole, "‹withheld›") || strings.Contains(mustJSON(credentialChecked(a)), "hunter2") {
t.Errorf("what is left out is named, an ICS calendar by its name and a password as ‹withheld›: %v", a.NotDerived)
}
if len(mesh.proposed) != 0 {
t.Fatal("without propose nothing is proposed")
}
// Propose: the mesh-wide layer first, as it stands with the derived kinds, and the operator's setup of prefs.
prefs := map[string]any{}
for n, v := range shippedDefaultsWant {
prefs[n], _ = prefValue(v)
}
a, err = f.Derive(nil, false, true, prefs, "")
if err != nil {
t.Fatal(err)
}
if len(mesh.proposed) != 1 || mesh.proposed[0]["node"] != nil || mesh.proposed[0]["replace"] != "true" || a.Next != meshWide {
t.Fatalf("the mesh-wide layer is proposed first: %v %s", mesh.proposed, a.Next)
}
var values map[string]any
_ = json.Unmarshal([]byte(mesh.proposed[0]["values"].(string)), &values)
if values["accounts"] == nil || values[kindIdentities] == nil || values[kindTags] == nil || values["prefs"] == nil {
t.Fatalf("the layer as it stands, with every derived kind and prefs: %v", values)
}
body, _ := json.Marshal(map[string]any{"prefs": values["prefs"]})
if got, err := parseDeclaredPrefs(body); err != nil || !reflect.DeepEqual(got, shippedDefaultsWant) {
t.Fatalf("the operator's 21 preferences, one entry each: %v %v", got, err)
}
if !strings.Contains(a.Proposed, "sab12") {
t.Fatalf("%s", a.Proposed)
}
// While it waits nothing more is proposed.
mesh.open = true
if _, err := f.Derive(nil, false, true, nil, ""); err == nil || !strings.Contains(err.Error(), "sab12 (the whole mesh)") || len(mesh.proposed) != 1 {
t.Fatalf("one open proposal of the module's at a time: %v", err)
}
// Approved: the mesh-wide layer holds it, so the next layer is shanks's own.
mesh.open = false
mesh.layers[""] = values
a, err = f.Derive(nil, false, true, prefs, "")
if err != nil {
t.Fatal(err)
}
if len(mesh.proposed) != 2 || mesh.proposed[1]["node"] != "shanks" || a.Next != "shanks" ||
!strings.Contains(mesh.proposed[1]["values"].(string), "standup") {
t.Fatalf("then the node's own layer: %v", mesh.proposed)
}
var shanks map[string]any
if err := json.Unmarshal([]byte(mesh.proposed[1]["values"].(string)), &shanks); err != nil {
t.Fatal(err)
}
mesh.layers["shanks"] = shanks
if a, err = f.Derive(nil, false, true, prefs, ""); err != nil || len(mesh.proposed) != 2 || !strings.HasPrefix(a.Next, "none") {
t.Fatalf("nothing left to propose: %v %s", err, a.Next)
}
// A desktop node that does not answer: derived without it, never proposed.
mesh.asleep = true
a, err = f.Derive(nil, false, true, nil, meshWide)
if err == nil || len(a.NotHeard) != 1 || !strings.Contains(err.Error(), "did not answer") || len(mesh.proposed) != 2 {
t.Fatalf("a layer is never proposed without a desktop node: %v %v", err, a.NotHeard)
}
// A key to adopt is a node's: the mesh-wide layer's proposal refuses one.
if err := proposable(map[string]any{kindCalendars: map[string]any{"c": map[string]any{"name": "C",
"url": "https://dav.example.org/c/", "key": "c3"}}}, true); err == nil || !strings.Contains(err.Error(), "node's own layer") {
t.Fatalf("%v", err)
}
}
// Review 121: a layer the controller answers in any other words than its JSON or "no layer" refuses the proposal,
// because a proposal replaces the layer whole and would drop accounts.
func TestALayerThatCannotBeReadRefusesTheProposal(t *testing.T) {
f := newFake(t)
f.profileWithAccounts()
f.profileWithConfiguration()
f.declareAccounts(adoptable)
links(t, nil)
f2 := newFake(t)
f2.profileWithAccounts()
f2.declareAccounts(adoptable)
mesh := &derivingMesh{t: t, shanks: f2, garbled: true}
oldNode, oldAsk := nodeName, asker
t.Cleanup(func() { nodeName, asker = oldNode, oldAsk })
nodeName = func() string { return "g14" }
asker = mesh.ask
if _, err := f.Derive(nil, false, true, nil, ""); err == nil || !strings.Contains(err.Error(), "cannot be read as the layer") ||
len(mesh.proposed) != 0 {
t.Fatalf("the derive refuses: %v %v", err, mesh.proposed)
}
if _, err := f.ProfileAccounts(true); err == nil || !strings.Contains(err.Error(), "cannot be read as the layer") ||
len(mesh.proposed) != 0 {
t.Fatalf("betterbird_accounts refuses: %v %v", err, mesh.proposed)
}
if _, err := readLayer(""); err == nil {
t.Fatal("an unreadable layer is an error, never an empty layer")
}
mesh.garbled = false
if got, err := readLayer("shanks"); err != nil || len(got) != 0 {
t.Fatalf("no layer, in the controller's words, is an empty layer: %v %v", got, err)
}
}
@@ -0,0 +1,271 @@
package main
// An account's filters, owned whole per declared account (novox/hq ADR 0329 §8).
//
// Betterbird keeps an account's filters in msgFilterRules.dat in the account's own directory: a header, then per
// filter its name, whether it is on, when it runs (a bit mask), its actions and one condition line. For an account
// the setting filters names, the module writes that file from the declared list, naming each folder by its path
// within the account and turning it into the folder's address on this node; a filter in the file the setting does
// not hold is gone after the apply. Before its first write on a node the module keeps the file it replaces, in its
// own directory in the home (ADR 0182). It writes only while Betterbird does not run, because Betterbird rewrites
// the file itself (§9).
import (
"bufio"
"fmt"
"net/url"
"path/filepath"
"regexp"
"sort"
"strconv"
"strings"
)
// filterFileName is the file an account's filters are in.
const filterFileName = "msgFilterRules.dat"
// accountDirectory is an account's directory in the profile, as its server's preferences name it, or "".
func accountDirectory(profile string, prefs map[string]string, server string) string {
p := "mail.server." + server + "."
if rel := unquote(prefs[p+"directory-rel"]); strings.HasPrefix(rel, "[ProfD]") {
dir := filepath.Join(profile, filepath.FromSlash(strings.TrimPrefix(rel, "[ProfD]")))
if strings.HasPrefix(filepath.Clean(dir), filepath.Clean(profile)+string(filepath.Separator)) {
return dir
}
return ""
}
return ""
}
// folderBase is the address of an account's root folder on this node: imap://user@host for IMAP, mailbox://
// for POP3 and local folders, the user's @ written %40 as Betterbird writes it.
func folderBase(prefs map[string]string, server string) string {
p := "mail.server." + server + "."
kind := unquote(prefs[p+"type"])
host := unquote(prefs[p+"hostname"])
user := unquote(prefs[p+"userName"])
scheme := "mailbox"
switch kind {
case "imap":
scheme = "imap"
case "none":
host, user = "Local Folders", "nobody"
}
return scheme + "://" + strings.ReplaceAll(url.PathEscape(user), "@", "%40") + "@" + url.PathEscape(host)
}
// folderAddress is a folder's address on this node from its path within the account.
func folderAddress(base, path string) string {
parts := strings.Split(strings.Trim(path, "/"), "/")
for i, p := range parts {
parts[i] = url.PathEscape(p)
}
return base + "/" + strings.Join(parts, "/")
}
// folderPath is a folder's path within the account from its address, when the address is the account's.
func folderPath(base, address string) (string, bool) {
rest, ok := strings.CutPrefix(address, base+"/")
if !ok {
return "", false
}
parts := strings.Split(rest, "/")
for i, p := range parts {
u, err := url.PathUnescape(p)
if err != nil {
return "", false
}
parts[i] = u
}
return strings.Join(parts, "/"), true
}
// whenOf is a filter's type mask from its words; whenWords the reverse, or false when the mask holds a bit no
// word says (a news filter's).
func whenOf(words []string) int {
n := 0
for _, w := range words {
n |= filterWhen[w]
}
return n
}
func whenWords(mask int) ([]string, bool) {
var out []string
left := mask
for _, w := range []string{"new-mail", "manual", "after-junk", "sending", "archiving", "periodic"} {
if mask&filterWhen[w] != 0 {
out = append(out, w)
left &^= filterWhen[w]
}
}
return out, left == 0
}
// renderFilters is the file Betterbird reads for a declared list of filters on this node.
func renderFilters(filters []Filter, base string) string {
var b strings.Builder
b.WriteString("version=\"9\"\nlogging=\"no\"\n")
for _, f := range filters {
fmt.Fprintf(&b, "name=\"%s\"\n", f.Name)
fmt.Fprintf(&b, "enabled=\"%s\"\n", map[bool]string{true: "yes", false: "no"}[f.Enabled])
fmt.Fprintf(&b, "type=\"%d\"\n", whenOf(f.When))
for _, a := range f.Actions {
fmt.Fprintf(&b, "action=\"%s\"\n", a.Action)
switch {
case a.Folder != "":
fmt.Fprintf(&b, "actionValue=\"%s\"\n", folderAddress(base, a.Folder))
case a.Value != "":
fmt.Fprintf(&b, "actionValue=\"%s\"\n", a.Value)
}
}
fmt.Fprintf(&b, "condition=\"%s\"\n", conditionText(f))
}
return b.String()
}
func conditionText(f Filter) string {
if f.Match == "every-message" {
return "ALL"
}
join := "AND"
if f.Match == "any" {
join = "OR"
}
var parts []string
for _, c := range f.Conditions {
parts = append(parts, fmt.Sprintf("%s (%s,%s,%s)", join, c.Field, c.Op, c.Value))
}
return strings.Join(parts, " ")
}
var (
filterLine = regexp.MustCompile(`^([A-Za-z]+)="((?:[^"\\]|\\.)*)"$`)
clause = regexp.MustCompile(`(AND|OR) \(([^,()]+),([^,()]+),([^()]*)\)`)
)
// parsedFilters is a filter file read into the setting's shape. NotDeclarable names, by the filter's name, the
// filters a setting cannot hold as they are (a folder of another account, an action of an add-on, a news filter).
type parsedFilters struct {
Filters []Filter
NotDeclarable []string
}
// parseFilters reads a filter file for an account whose folders are under base.
func parseFilters(text, base string) parsedFilters {
var out parsedFilters
var cur *Filter
bad := ""
finish := func() {
if cur == nil {
return
}
if bad != "" {
out.NotDeclarable = append(out.NotDeclarable, cur.Name+": "+bad)
} else if p := filterProblems(*cur); len(p) > 0 {
out.NotDeclarable = append(out.NotDeclarable, cur.Name+": "+p[0])
} else {
out.Filters = append(out.Filters, *cur)
}
cur, bad = nil, ""
}
s := bufio.NewScanner(strings.NewReader(text))
s.Buffer(make([]byte, 64<<10), 4<<20)
for s.Scan() {
mm := filterLine.FindStringSubmatch(strings.TrimSpace(s.Text()))
if mm == nil {
continue
}
field, value := mm[1], strings.NewReplacer(`\"`, `"`, `\\`, `\`).Replace(mm[2])
if field == "name" {
finish()
cur = &Filter{Name: value}
continue
}
if cur == nil {
continue
}
switch field {
case "enabled":
cur.Enabled = value == "yes"
case "type":
n, _ := strconv.Atoi(value)
words, ok := whenWords(n)
if !ok && bad == "" {
bad = "it runs on news or at a time no setting word says"
}
cur.When = words
case "action":
cur.Actions = append(cur.Actions, Action{Action: value})
case "actionValue":
if len(cur.Actions) == 0 {
continue
}
a := &cur.Actions[len(cur.Actions)-1]
if filterActions[a.Action] == "folder" {
path, ok := folderPath(base, value)
if !ok && bad == "" {
bad = "it moves or copies to a folder of another account"
}
a.Folder = path
} else {
a.Value = value
}
case "condition":
switch {
case value == "ALL":
cur.Match = "every-message"
default:
all := clause.FindAllStringSubmatch(value, -1)
if len(all) == 0 && bad == "" {
bad = "its condition cannot be read"
}
for _, c := range all {
if c[1] == "OR" {
cur.Match = "any"
} else if cur.Match == "" {
cur.Match = "all"
}
cur.Conditions = append(cur.Conditions, Condition{Field: c[2], Op: c[3], Value: c[4]})
}
}
}
}
finish()
return out
}
// filterDifferences says how a file's filters differ from the declared list, by place and name: missing,
// not declared, or differing.
func filterDifferences(declared, have []Filter) []string {
var out []string
for i := 0; i < len(declared) || i < len(have); i++ {
switch {
case i >= len(have):
out = append(out, fmt.Sprintf("filter %d (%s) is declared and not in the file", i+1, declared[i].Name))
case i >= len(declared):
out = append(out, fmt.Sprintf("filter %d (%s) is in the file and not declared: removed at the next apply", i+1, have[i].Name))
default:
d, h := mustJSON(declared[i]), mustJSON(have[i])
if d != h {
out = append(out, fmt.Sprintf("filter %d: declared %s, the file holds %s", i+1, d, h))
}
}
}
return out
}
// keptFilterFile is where the module keeps the file it replaced at its first write of an account's filters.
func (m *Machine) keptFilterFile(profileDir, account string) string {
base := filepath.Base(profileDir)
return m.home(".config/betterbird/kept", base, account+"."+filterFileName)
}
func sortedAccountsOf(fs map[string][]Filter) []string {
out := make([]string, 0, len(fs))
for k := range fs {
out = append(out, k)
}
sort.Strings(out)
return out
}
@@ -0,0 +1,480 @@
package main
// What the tools show of the five kinds (novox/hq ADR 0329 §1, §11–§13): the profile's entries beside the declared
// ones, with the operator's own values, and how a node differs from what is declared. Never a credential (§2):
// every answer passes credentialChecked, which withholds a field whose name says password, token, secret or cookie
// and the password part of any address.
import (
"encoding/json"
"fmt"
"strings"
)
// configView is the profile and the settings, read once for an answer.
type configView struct {
p Profile
prefs map[string]string
accounts map[string]Account
ra []resolvedAccount
conf Configuration
problem string
plan configPlan
rec configRecord
}
func (m *Machine) view() (configView, error) {
var v configView
p, err := m.profile()
if err != nil {
return v, err
}
v.p, v.prefs = p, mergedPrefs(p.path)
accounts, _, aerr := m.readAccounts()
if aerr != nil {
v.problem = aerr.Error()
}
v.accounts = accounts
v.ra = resolveAccounts(accounts, v.prefs, m.readRecord(p.Dir))
conf, cerr := m.readConfiguration()
if cerr != nil {
v.problem = strings.TrimPrefix(v.problem+"; "+cerr.Error(), "; ")
}
v.conf = conf
v.rec = m.readConfigRecord(p.Dir)
v.plan = m.planConfiguration(p, conf, accounts, v.ra, v.prefs)
return v, nil
}
// listedEntry is one entry of the profile, with its values: Owl's marked, an ICS calendar by its name only (§13).
type listedEntry struct {
Key string `json:"key"`
Kind string `json:"kind,omitempty"`
// Account is an identity's account key in the profile, or the account name filters are listed under.
Account string `json:"account,omitempty"`
Owl bool `json:"owl,omitempty"`
// DeclaredAs is the declared entry this one is, or empty: the person's own.
DeclaredAs string `json:"declared_as,omitempty"`
Value any `json:"value,omitempty"`
Note string `json:"note,omitempty"`
}
// declaredEntry is one declared entry, its value and how it resolved on this node.
type declaredEntry struct {
Name string `json:"name"`
Value any `json:"value"`
How string `json:"how,omitempty"`
Key string `json:"key,omitempty"`
Why string `json:"why,omitempty"`
}
// KindListing is what a listing tool answers.
type KindListing struct {
Profile string `json:"profile"`
Kind string `json:"kind"`
InProfile []listedEntry `json:"in_the_profile"`
Declared []declaredEntry `json:"declared"`
// Kept says where the module keeps the filter files it replaced, per account (§8).
Kept map[string]string `json:"kept_filter_files,omitempty"`
Problem string `json:"problem,omitempty"`
}
func (v configView) resolved(kind string) map[string]resolvedEntry {
out := map[string]resolvedEntry{}
for _, e := range v.plan.Entries {
if e.Kind == kind {
out[e.Name] = e
}
}
return out
}
// declaredBy is, per profile key, the declared entry of a kind adopted or created under it.
func (v configView) declaredBy(kind string) map[string]string {
out := map[string]string{}
for _, e := range v.plan.Entries {
if e.Kind == kind && e.How != refused {
out[e.Key] = e.Name
}
}
return out
}
// List answers one kind: the profile's entries beside the declared ones.
func (m *Machine) List(kind string) (KindListing, error) {
v, err := m.view()
a := KindListing{Kind: kind, InProfile: []listedEntry{}, Declared: []declaredEntry{}}
if err != nil {
return a, err
}
a.Profile, a.Problem = v.p.Dir, v.problem
res := v.resolved(kind)
by := v.declaredBy(kind)
declare := func(name string, value any) {
r := res[name]
a.Declared = append(a.Declared, declaredEntry{Name: name, Value: value, How: r.How, Key: r.Key, Why: r.Why})
}
switch kind {
case kindIdentities:
for _, pi := range m.profileIdentities(v.prefs) {
a.InProfile = append(a.InProfile, listedEntry{Key: pi.Key, Account: pi.AccountKey, Owl: pi.Owl,
DeclaredAs: by[pi.Key], Value: pi.Value})
}
for _, name := range sortedKeys(v.conf.Identities) {
declare(name, v.conf.Identities[name])
}
case kindCalendars:
for _, pc := range profileCalendars(v.prefs) {
e := listedEntry{Key: pc.Key, Kind: pc.Kind, DeclaredAs: by[pc.Key], Value: pc.Value}
switch {
case strings.HasPrefix(pc.Kind, "owl"):
e.Owl, e.Value = true, map[string]string{"name": pc.Value.Name, "url": pc.Value.URL}
e.Note = "Owl's: never declared (novox/hq ADR 0316 §13)"
case pc.Kind == "ics":
e.Value = map[string]string{"name": pc.Value.Name}
e.Note = "subscribed by an ICS address, which is often itself the secret: its name only, never declared (novox/hq ADR 0329 §13)"
case pc.Kind != "caldav":
e.Note = "kept in the profile: its events are never declared"
}
a.InProfile = append(a.InProfile, e)
}
for _, name := range sortedKeys(v.conf.Calendars) {
declare(name, v.conf.Calendars[name])
}
case kindAddressBooks:
for _, b := range profileAddressBooks(v.prefs) {
e := listedEntry{Key: b.Key, Kind: b.Kind, DeclaredAs: by[b.Key], Value: b.Value}
if !b.CardDAV {
e.Value, e.Note = map[string]string{"name": b.Value.Name}, "its contacts are kept in the profile and never declared"
}
a.InProfile = append(a.InProfile, e)
}
for _, name := range sortedKeys(v.conf.AddressBooks) {
declare(name, v.conf.AddressBooks[name])
}
case kindTags:
for _, t := range profileTags(v.prefs) {
a.InProfile = append(a.InProfile, listedEntry{Key: t.Key, DeclaredAs: by[t.Key], Value: t.Value})
}
for _, key := range sortedKeys(v.conf.Tags) {
declare(key, v.conf.Tags[key])
}
case kindFilters:
for _, f := range m.profileFilters(v) {
a.InProfile = append(a.InProfile, f)
}
for _, account := range v.conf.sortedFilterAccounts() {
d := declaredEntry{Name: account, Value: v.conf.Filters[account], How: "owned whole"}
for _, fp := range v.plan.filters {
if fp.Account == account {
d.Key, d.Why = fp.File, strings.TrimSpace(fp.Refused+" "+fp.Waiting)
if fp.Refused != "" {
d.How = refused
}
}
}
a.Declared = append(a.Declared, d)
}
a.Kept = map[string]string{}
for _, account := range sortedKeys(v.rec.Filters) {
if r := v.rec.Filters[account]; r.Kept != "" {
a.Kept[account] = r.Kept
}
}
if len(a.Kept) == 0 {
a.Kept = nil
}
default:
return a, fmt.Errorf("a kind is one of %s", strings.Join(allKinds, ", "))
}
return a, nil
}
// profileFilters are the filters of each declared account in the profile, by the account's name.
func (m *Machine) profileFilters(v configView) []listedEntry {
var out []listedEntry
for _, r := range v.ra {
if r.How == refused {
continue
}
dir := m.accountDirectory(v.p.path, v.prefs, r.Keys.Server)
if dir == "" {
continue
}
text, err := readBounded(dir + "/" + filterFileName)
if err != nil {
continue
}
parsed := parseFilters(string(text), folderBase(v.prefs, r.Keys.Server))
e := listedEntry{Key: m.tilde(strings.TrimPrefix(dir+"/"+filterFileName, m.Root)), Account: r.name, Value: parsed.Filters}
if _, declared := v.conf.Filters[r.name]; declared {
e.DeclaredAs = r.name
}
if len(parsed.NotDeclarable) > 0 {
e.Note = "not declarable as they are: " + strings.Join(parsed.NotDeclarable, "; ")
}
out = append(out, e)
}
return out
}
// fieldDiff is one field a profile holds differently from the declared value.
type fieldDiff struct {
Field string `json:"field"`
Declared any `json:"declared"`
Profile any `json:"profile"`
}
// DriftEntry is one declared entry against the profile: holds, differs, lacks or refused (§11).
type DriftEntry struct {
Name string `json:"name"`
State string `json:"state"`
Key string `json:"key,omitempty"`
Differs []fieldDiff `json:"differs,omitempty"`
Why string `json:"why,omitempty"`
}
// KindDrift is one kind's drift, and the profile's entries nothing declares.
type KindDrift struct {
Kind string `json:"kind"`
Declared []DriftEntry `json:"declared"`
Undeclared []listedEntry `json:"undeclared,omitempty"`
}
// DriftAnswer is what betterbird_drift answers.
type DriftAnswer struct {
Profile string `json:"profile"`
Kinds []KindDrift `json:"kinds"`
Note string `json:"note"`
Problem string `json:"problem,omitempty"`
}
// Drift answers, per kind, how the profile differs from what is declared: the profile as Betterbird reads it at
// its next start (prefs.js with user.js over it), so a declared value the module placed and Betterbird has not
// started with yet holds; the drift then is only what the module did not place.
func (m *Machine) Drift(kind string) (DriftAnswer, error) {
v, err := m.view()
a := DriftAnswer{Kinds: []KindDrift{}}
if err != nil {
return a, err
}
a.Profile, a.Problem = v.p.Dir, v.problem
a.Note = "the profile as Betterbird reads it at its next start (prefs.js, the module's user.js over it): differs " +
"names each field with the declared value and the profile's; lacks is an entry the profile does not hold yet " +
"(betterbird_user_js with apply, betterbird_restart); undeclared entries stay the person's"
kinds := allKinds
if kind != "" {
if !contains(allKinds, kind) {
return a, fmt.Errorf("a kind is one of %s", strings.Join(allKinds, ", "))
}
kinds = []string{kind}
}
for _, k := range kinds {
a.Kinds = append(a.Kinds, m.kindDrift(v, k))
}
return a, nil
}
func (m *Machine) kindDrift(v configView, kind string) KindDrift {
d := KindDrift{Kind: kind, Declared: []DriftEntry{}}
res := v.resolved(kind)
by := v.declaredBy(kind)
entry := func(name string, declared any, profile func(key string) (any, bool)) {
r := res[name]
e := DriftEntry{Name: name, Key: r.Key}
switch {
case r.How == refused:
e.State, e.Why = refused, r.Why
default:
have, ok := profile(r.Key)
if !ok {
e.State = "lacks"
break
}
e.Differs = differing(declared, have)
e.State = map[bool]string{true: "holds", false: "differs"}[len(e.Differs) == 0]
}
d.Declared = append(d.Declared, e)
}
switch kind {
case kindIdentities:
ids := map[string]profileIdentity{}
for _, pi := range m.profileIdentities(v.prefs) {
ids[pi.Key] = pi
if by[pi.Key] == "" && !pi.Owl {
d.Undeclared = append(d.Undeclared, listedEntry{Key: pi.Key, Account: pi.AccountKey, Value: pi.Value})
}
}
for _, name := range sortedKeys(v.conf.Identities) {
want := *v.conf.Identities[name]
want.Key = ""
entry(name, want, func(key string) (any, bool) {
pi, ok := ids[key]
if !ok {
return nil, false
}
have := pi.Value
have.Account = want.Account
if want.Address == "" {
have.Address, have.Name = "", ""
}
return have, true
})
}
case kindCalendars:
cals := map[string]profileCalendar{}
for _, pc := range profileCalendars(v.prefs) {
cals[pc.Key] = pc
if by[pc.Key] == "" && pc.Kind == "caldav" {
d.Undeclared = append(d.Undeclared, listedEntry{Key: pc.Key, Kind: pc.Kind, Value: pc.Value})
}
}
idName := v.declaredBy(kindIdentities)
for _, name := range sortedKeys(v.conf.Calendars) {
want := *v.conf.Calendars[name]
want.Key = ""
entry(name, want, func(key string) (any, bool) {
pc, ok := cals[key]
if !ok {
return nil, false
}
have := pc.Value
have.Invitations = idName[pc.IdentityKey]
if have.Reminders == nil && want.Reminders != nil && *want.Reminders {
on := true
have.Reminders = &on
}
return have, true
})
}
case kindAddressBooks:
books := map[string]profileAddressBook{}
for _, b := range profileAddressBooks(v.prefs) {
books[b.Key] = b
if by[b.Key] == "" && b.CardDAV {
d.Undeclared = append(d.Undeclared, listedEntry{Key: b.Key, Kind: b.Kind, Value: b.Value})
}
}
for _, name := range sortedKeys(v.conf.AddressBooks) {
want := *v.conf.AddressBooks[name]
want.Key = ""
entry(name, want, func(key string) (any, bool) {
b, ok := books[key]
return b.Value, ok
})
}
case kindTags:
tags := map[string]profileTag{}
for _, t := range profileTags(v.prefs) {
tags[t.Key] = t
if by[t.Key] == "" {
d.Undeclared = append(d.Undeclared, listedEntry{Key: t.Key, Value: t.Value})
}
}
for _, key := range sortedKeys(v.conf.Tags) {
want := *v.conf.Tags[key]
entry(key, want, func(k string) (any, bool) {
t, ok := tags[k]
have := t.Value
if want.Order == 0 {
have.Order = 0
}
return have, ok
})
}
case kindFilters:
files := map[string]listedEntry{}
for _, f := range m.profileFilters(v) {
files[f.Account] = f
released := v.rec.Filters[f.Account].Released
if f.DeclaredAs == "" && !released {
if list, _ := f.Value.([]Filter); len(list) > 0 {
d.Undeclared = append(d.Undeclared, f)
}
}
}
for _, fp := range v.plan.filters {
e := DriftEntry{Name: fp.Account, Key: fp.File}
switch {
case fp.Refused != "":
e.State, e.Why = refused, fp.Refused
case fp.path == "" || !fp.exists:
e.State, e.Why = "lacks", strings.TrimSpace(fp.Waiting)
default:
have, _ := files[fp.Account].Value.([]Filter)
for _, line := range filterDifferences(v.conf.Filters[fp.Account], have) {
e.Differs = append(e.Differs, fieldDiff{Field: line})
}
if fp.have != fp.want && len(e.Differs) == 0 {
e.Differs = append(e.Differs, fieldDiff{Field: "the file's text, beyond the filters' own words"})
}
e.State = map[bool]string{true: "holds", false: "differs"}[len(e.Differs) == 0]
}
d.Declared = append(d.Declared, e)
}
}
return d
}
// differing are the fields of two values of the same shape that differ.
func differing(declared, have any) []fieldDiff {
var a, b map[string]any
_ = json.Unmarshal([]byte(mustJSON(declared)), &a)
_ = json.Unmarshal([]byte(mustJSON(have)), &b)
var out []fieldDiff
for _, f := range sortedKeys(mergeKeys(a, b)) {
if mustJSON(a[f]) != mustJSON(b[f]) {
out = append(out, fieldDiff{Field: f, Declared: a[f], Profile: b[f]})
}
}
return out
}
// driftCounts are, per kind, how many declared entries do not hold.
func (m *Machine) driftCounts(v configView) map[string]int {
out := map[string]int{}
for _, k := range allKinds {
for _, e := range m.kindDrift(v, k).Declared {
if e.State != "holds" {
out[k]++
}
}
}
return out
}
// credentialChecked is an answer as it may leave the tool (§2): a field whose name says password, token, secret
// or cookie withheld, and the password part of any address as ‹withheld›.
func credentialChecked(v any) any {
b, err := json.Marshal(v)
if err != nil {
return v
}
var tree any
if json.Unmarshal(b, &tree) != nil {
return v
}
return scrubTree(tree)
}
func scrubTree(n any) any {
switch x := n.(type) {
case map[string]any:
for k, v := range x {
if secretName.MatchString(k) {
x[k] = "‹withheld›"
continue
}
x[k] = scrubTree(v)
}
return x
case []any:
for i, v := range x {
x[i] = scrubTree(v)
}
return x
case string:
return withoutPassword(x)
}
return n
}
+131 -31
View File
@@ -2,14 +2,17 @@
// mail and calendar client, in the operator's session, served by the node's runtime as the operator
// account. The module holds no seat, so every tool is its own.
//
// The operator's mail, calendars, accounts and credentials are never read: a tool reads processes,
// profiles.ini, and of the profile only prefs.js, user.js and extensions.json, and answers no
// preference that can hold a credential or names an account.
// The operator's mail, calendar events, contacts and credentials are never read: a tool reads processes,
// profiles.ini, and of the profile its prefs.js, user.js, extensions.json, each account's msgFilterRules.dat and
// of logins.json only the server each kept sign-in is for. The tools show the operator's own configuration, an
// account's, an identity's, a calendar's, an address book's, a filter's and a tag's values, and never a credential
// (novox/hq ADR 0329 §1, §2): every answer passes credentialChecked.
package main
import (
"fmt"
"os"
"strings"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
@@ -34,6 +37,31 @@ func main() {
var machine = NewMachine()
func tools() []stdio.Tool {
list := toolList()
for i := range list {
run := list[i].Run
list[i].Run = func(args map[string]any) (any, error) {
out, err := run(args)
return credentialChecked(out), err
}
}
return list
}
// listTool is the listing tool of one of the five kinds (novox/hq ADR 0329 §12).
func listTool(kind, what string) stdio.Tool {
return stdio.Tool{
Name: "betterbird_" + strings.ReplaceAll(kind, "-", "_"),
Description: "The " + what + " of the profile Betterbird uses beside the ones the module's setting " + kind +
" declares (novox/hq ADR 0329): each of the profile's with its key, its values and the declared entry it is " +
"(none: the person's own), Owl's marked; each declared one with its value and how it resolved on this node " +
"(adopted, created or refused with why, and its key). The operator's own values, never a password, token " +
"or cookie. betterbird_drift says how they differ. (r)",
Run: func(map[string]any) (any, error) { return machine.List(kind) },
}
}
func toolList() []stdio.Tool {
return []stdio.Tool{
{
Name: "betterbird_status",
@@ -42,9 +70,9 @@ func tools() []stdio.Tool {
"or the community package before the switch), its version and build beside the newest published build; " +
"whether the enterprise policy is placed; Owl's version against the mesh's pin and whether it is active; " +
"where its output goes; the profile it uses and how that was found; the module's user.js (declared, " +
"placed, in effect); the declared accounts by count (in the profile's list, other accounts, not signed " +
"in yet), never an address, host or name; the add-ons in the profile; and the preferences that decide " +
"how its windows and notifications look. (r)",
"placed, in effect, with every declared identity, calendar, address book and tag as it resolved); the " +
"declared accounts by count (in the profile's list, other accounts, not signed in yet); the add-ons in the " +
"profile; and the preferences that decide how its windows and notifications look. (r)",
Run: func(map[string]any) (any, error) { return machine.Status() },
},
{
@@ -69,13 +97,13 @@ func tools() []stdio.Tool {
},
{
Name: "betterbird_config",
Description: "The configuration the module ships, preference by preference: the value its policy locks, " +
"its default (mesh-defaults.js, or Betterbird's own where the module keeps that), the node layer (the " +
"setting prefs on this machine), the profile's own saved value and the value in effect. It flags drift: " +
"a value the profile saved that beats the module's default without a node layer saying so. With keep " +
"(names), it proposes a node layer of the setting prefs for this machine holding the profile's values of " +
"those preferences, through the controller's settings verb, for the operator's Approve; it never writes " +
"user.js. (r; a with keep)",
Description: "Betterbird's preferences through the module's layers, preference by preference: the value its " +
"policy locks, the setting prefs (the operator's setup, mesh-wide with this node's layer over it, one entry " +
"per preference), Betterbird's own default where the module keeps that, the profile's own saved value and " +
"the value in effect. It flags drift: a declared value the profile's user.js does not hold yet, or a value " +
"the profile saved where the module keeps Betterbird's default. With keep (names), it proposes this node's " +
"layer of the setting prefs holding the profile's values of those preferences beside what the layer holds, " +
"through the controller's settings verb, for the operator's Approve; it never writes user.js. (r; a with keep)",
Input: map[string]any{
"keep": map[string]any{"type": "array", "items": map[string]any{"type": "string"},
"description": "managed preferences whose profile value this machine keeps: proposed as a node layer, e.g. [\"mail.uifontsize\"]"},
@@ -108,15 +136,16 @@ func tools() []stdio.Tool {
},
{
Name: "betterbird_user_js",
Description: "The module's user.js — the preferences the module declares (setting prefs), then the declared " +
"accounts' preferences under the module's own keys (setting accounts) — against the profile's user.js: " +
"whether it is placed, and whether the running Betterbird started after it was. With apply, place it in " +
"the profile (only where the profile has no user.js or the module's own; an operator's file is never " +
"overwritten). A declared account the profile already holds at its address is adopted under its own keys; " +
"one it does not hold is created under the module's keys, which are added to the account lists when " +
"Betterbird is not running; an ambiguous one is refused alone, with the remedy (novox/hq ADR 0317). " +
"Accounts' values are withheld in the answer. Betterbird reads it at its next start: betterbird_restart. " +
"(r; a with apply)",
Description: "The module's user.js — the setting prefs, then the declared accounts' preferences (setting " +
"accounts), then the declared identities, calendars, address books and tags — against the profile's " +
"user.js: whether it is placed, and whether the running Betterbird started after it was. With apply, place " +
"it in the profile (only where the profile has no user.js or the module's own; an operator's file is never " +
"overwritten). A declared entry the profile already holds is adopted under its own keys; one it does not " +
"hold is created under the module's keys; an ambiguous one is refused alone, with the remedy (novox/hq ADR " +
"0317, ADR 0329 §7). What Betterbird rewrites itself — the account and identity lists, the calendars' " +
"order, the removals and each declared account's msgFilterRules.dat — is written only while Betterbird is " +
"closed, and the answer counts what waits (§8, §9). Accounts' values are withheld in the user.js shown. " +
"Betterbird reads it at its next start: betterbird_restart. (r; a with apply)",
Input: map[string]any{
"apply": map[string]any{"type": "boolean", "description": "place the declared user.js in the profile (default false)"},
},
@@ -179,19 +208,20 @@ func tools() []stdio.Tool {
"behind the newest published build; the enterprise policy placed and the module's defaults with it; no " +
"managed preference drifting from them; Owl at its pin and active; one " +
"Betterbird runs; its profile is found; the module's user.js on the machine, valid, placed and in " +
"effect; the declared accounts in the profile's list, none refused, each signed in once; its output " +
"read. Answers ok and each finding with what to do, never an address, host or name. (r)",
"effect; the declared accounts in the profile's list, none refused, each signed in once; per kind, no " +
"declared identity, calendar, address book, filter or tag drifting or refused, and no change waiting for " +
"Betterbird to close; its output read. Answers ok and each finding with what to do. (r)",
Run: func(map[string]any) (any, error) { return machine.Check() },
},
{
Name: "betterbird_accounts",
Description: "The accounts of the profile Betterbird uses, by count and kind: the IMAP and POP3 accounts a " +
"declaration can hold, the accounts Owl keeps (never declared), and the rest. Never an address, a host " +
"or a name, and never anything of the login store. With propose, it builds the module's setting " +
"accounts from them (every account but Owl's, without any password or token) and proposes it through " +
"the controller's settings verb for the operator's Approve on a channel that proves who answers " +
"(novox/hq ADR 0317, ADR 0277): nothing changes until then, and the operator reads the value whole at " +
"their desk. (r; a with propose)",
Description: "The accounts of the profile Betterbird uses with their values (name, address, user, servers' " +
"hosts and ports) and the declared account each is, Owl's marked by name and address (never declared), " +
"with counts by kind. Never a password, a token or anything of the login store (novox/hq ADR 0329 §1, §2). " +
"With propose, it builds the module's setting accounts from them (every account but Owl's) and proposes " +
"it in the mesh-wide layer as it stands, through the controller's settings verb, for the operator's " +
"Approve on a channel that proves who answers (novox/hq ADR 0317, ADR 0277); the answer shows the value it " +
"proposed. (r; a with propose)",
Input: map[string]any{
"propose": map[string]any{"type": "boolean", "description": "build the setting accounts from the profile and propose it to the operator (default false)"},
},
@@ -203,6 +233,76 @@ func tools() []stdio.Tool {
return machine.ProfileAccounts(propose)
},
},
listTool(kindIdentities, "identities (each account's default and its other identities, with name, address, reply-to, organisation and signature)"),
listTool(kindCalendars, "calendars (CalDAV with name, address, colour and options; an ICS one by its name only)"),
listTool(kindAddressBooks, "address books (CardDAV with name, address and user; a local one by its name)"),
listTool(kindFilters, "filters (per declared account: its msgFilterRules.dat as filters, folders by their path within the account; where the module keeps a file it replaced)"),
listTool(kindTags, "tags (key, name, colour, place)"),
{
Name: "betterbird_derive",
Description: "Derive the module's settings identities, calendars, address-books, filters and tags (or the kinds " +
"given) from the profiles of every desktop node holding Betterbird, laid out per layer: an entry on every " +
"such node goes into the mesh-wide layer with this node's value, and the answer lists each field another " +
"node holds differently; one on some nodes only goes into those nodes' own layers. Owl's, ICS calendars, " +
"local address books and any address carrying a password are left out and named. prefs puts values of the " +
"setting prefs into the mesh-wide layer. With propose, the next layer (the mesh-wide first, then each " +
"node's) is proposed for the operator's Approve through the controller's settings verb, carrying the layer " +
"as it stands beside the derived kinds; refused while a proposal of the module's waits, or while a desktop " +
"node does not answer (novox/hq ADR 0329 §10, ADR 0277). here answers this node's profile only, as the " +
"other nodes are asked. The values are the operator's own; never a credential. (r; a with propose)",
Input: map[string]any{
"kinds": map[string]any{"type": "array", "items": map[string]any{"type": "string", "enum": allKinds},
"description": "the kinds to derive (default all five)"},
"here": map[string]any{"type": "boolean", "description": "this node's profile only (default false)"},
"propose": map[string]any{"type": "boolean", "description": "propose the next layer to the operator (default false)"},
"prefs": map[string]any{"type": "object", "description": "values of the setting prefs for the mesh-wide " +
"layer, by preference name, e.g. {\"mail.uifontsize\": 14}"},
"layer": map[string]any{"type": "string", "description": "the layer to propose instead of the next: mesh-wide or a node's name"},
},
Run: func(args map[string]any) (any, error) {
kinds, err := texts(args, "kinds")
if err != nil {
return nil, err
}
here, err := flag(args, "here", false)
if err != nil {
return nil, err
}
propose, err := flag(args, "propose", false)
if err != nil {
return nil, err
}
layer, err := text(args, "layer", false)
if err != nil {
return nil, err
}
var prefs map[string]any
if v, ok := args["prefs"]; ok && v != nil {
if prefs, ok = v.(map[string]any); !ok {
return nil, fmt.Errorf("prefs is an object of preference names and values, not %T", v)
}
}
return machine.Derive(kinds, here, propose, prefs, layer)
},
},
{
Name: "betterbird_drift",
Description: "How this node's profile differs from the declared identities, calendars, address books, " +
"filters and tags (novox/hq ADR 0329 §11): per declared entry whether the profile holds it, holds another " +
"value (each differing field with the declared value and the profile's), lacks it, or it is refused; and " +
"the profile's entries nothing declares, which stay the person's. The profile as Betterbird reads it at " +
"its next start. (r)",
Input: map[string]any{
"kind": map[string]any{"type": "string", "enum": allKinds, "description": "one kind (default every kind)"},
},
Run: func(args map[string]any) (any, error) {
kind, err := text(args, "kind", false)
if err != nil {
return nil, err
}
return machine.Drift(kind)
},
},
{
Name: "betterbird_reminder_test",
Description: "OPENS WINDOWS ON THE OPERATOR'S SCREEN: tell the operator before calling it. Shows a calendar " +
@@ -13,8 +13,8 @@ import (
)
// betterbird's shape (novox/hq issue 345, ADR 0208, ADR 0316): the X display on its own machine; its directory
// in the home with the user.js the setting prefs fills and the accounts file the setting accounts fills, both
// trusted; the community package declared absent before anything is placed in /opt/betterbird; Betterbird from
// in the home with the mergeable files the settings fill, each naming the keys it takes (ADR 0326): prefs, accounts
// and configuration (identities, calendars, address-books, filters) trusted, tags not (ADR 0329 §5); the community package declared absent before anything is placed in /opt/betterbird; Betterbird from
// its pinned upstream release, Owl pinned by its vendor's digest, the policy and the dictionaries' default
// unpacked into the installation; the tools' program for the start's preparation, the start and the launcher;
// every new build waiting for the operator's push; and the Go bundle serving exactly the listed tools.
@@ -84,7 +84,7 @@ func TestItPlacesBetterbirdWhereTheCommunityPackageDidAndOnlyAfterItIsGone(t *te
for _, r := range m.Resources {
ids = append(ids, r["id"].(string))
}
want := "configuration-dir,user-js,accounts,community-package,installation,owl,policies,spelling,preparation,start,launcher"
want := "configuration-dir,prefs,accounts,configuration,tags,community-package,installation,owl,policies,spelling,preparation,start,launcher"
if strings.Join(ids, ",") != want {
t.Fatalf("the resources, in the order the node-engine applies them: %v, want %s", ids, want)
}
@@ -102,7 +102,7 @@ func TestItPlacesBetterbirdWhereTheCommunityPackageDidAndOnlyAfterItIsGone(t *te
}
}
for _, never := range []string{".thunderbird", "prefs.js", "logins.json", "key4.db"} {
if strings.Contains(raw, never) {
if regexp.MustCompile(regexp.QuoteMeta(never) + `\b`).MatchString(raw) {
t.Errorf("module.json names %q: the profile is Betterbird's", never)
}
}
@@ -114,29 +114,58 @@ func TestItPlacesBetterbirdWhereTheCommunityPackageDidAndOnlyAfterItIsGone(t *te
}
}
func TestTheUserJSAndTheAccountsAreTrustedFilesOfTheModulesOwn(t *testing.T) {
// ADR 0329 §4–§6, ADR 0326, issue 586: every setting reaches the machine as a mergeable file of the account's that
// names the keys it takes; four kinds and accounts and prefs are trusted, tags is not; no setting has a default in the
// definition, since each is the operator's own value; and each file's default declares nothing.
func TestTheSettingsAreMergeableFilesThatNameTheirKeysAndTagsAloneIsUntrusted(t *testing.T) {
m, _ := readManifest(t)
file := resource(t, m, "user-js")
content := file["content"].(string)
if !strings.HasPrefix(content, mark) || !strings.Contains(content, "${setting:prefs}") || file["trusted"] != true {
t.Fatalf("the user.js begins with the mark, holds the setting, and is trusted (only the terminal or the operator's warrant sets it): %v", file)
if len(m.Settings) != 0 {
t.Fatalf("no setting has a default in the definition: the operator's setup is the settings' mesh-wide layer (issue 586, ADR 0262): %v", m.Settings)
}
// The default is a comment, so a machine without the setting places a valid, empty user.js.
def := m.Settings["prefs"].(map[string]any)["default"].(string)
if err := validUserJS(strings.ReplaceAll(content, "${setting:prefs}", def)); err != nil {
t.Fatal(err)
files := map[string]struct {
path, mode string
takes []any
trusted bool
}{
"prefs": {prefsFile, "0644", []any{"prefs"}, true},
"accounts": {accountsFile, "0600", []any{"accounts"}, true},
"configuration": {configurationFile, "0600", []any{"identities", "calendars", "address-books", "filters"}, true},
"tags": {tagsFile, "0644", []any{"tags"}, false},
}
acc := resource(t, m, "accounts")
if acc["path"] != "${machine:account-home}/"+accountsFile || acc["merge"] != "json" || acc["trusted"] != true ||
acc["mode"] != "0600" || acc["owner"] != "${machine:account}" {
t.Fatalf("the setting accounts reaches the machine as a trusted mergeable file of the account's (ADR 0316 §10, §15): %v", acc)
merging := 0
for _, r := range m.Resources {
if r["merge"] != nil {
merging++
}
}
got, err := parseAccounts([]byte(acc["content"].(string)))
if merging != len(files) {
t.Fatalf("%d mergeable files, want %d", merging, len(files))
}
for id, want := range files {
r := resource(t, m, id)
if r["path"] != "${machine:account-home}/"+want.path || r["merge"] != "json" || r["trusted"] != want.trusted ||
r["mode"] != want.mode || r["owner"] != "${machine:account}" || !reflect.DeepEqual(r["takes-settings"], want.takes) {
t.Errorf("%s: %v", id, r)
}
var content map[string]any
if err := json.Unmarshal([]byte(r["content"].(string)), &content); err != nil || len(content) != len(want.takes) {
t.Errorf("%s's default holds each key it takes, empty: %v %v", id, content, err)
}
}
got, err := parseAccounts([]byte(resource(t, m, "accounts")["content"].(string)))
if err != nil || len(got) != 0 {
t.Fatalf("its default declares no account: %v %v", got, err)
}
if _, declared := m.Settings["accounts"]; declared {
t.Fatal("accounts is the operator's value and has no default in the definition (ADR 0262)")
prefs, err := parseDeclaredPrefs([]byte(resource(t, m, "prefs")["content"].(string)))
if err != nil || len(prefs) != 0 {
t.Fatalf("its default declares no preference: %v %v", prefs, err)
}
var c Configuration
if err := decodeInto([]byte(resource(t, m, "configuration")["content"].(string)), configurationFile, &c); err != nil {
t.Fatal(err)
}
if err := decodeInto([]byte(resource(t, m, "tags")["content"].(string)), tagsFile, &c); err != nil {
t.Fatal(err)
}
}
@@ -278,8 +307,10 @@ func TestTheToolsAgreeWithTheManifest(t *testing.T) {
if b["kind"] != "bundle" || b["language"] != "go" || b["from"] != "cmd/betterbird-tools" || b["binary"] != "betterbird-tools" {
t.Errorf("the Go tools bundle: %v", b)
}
// betterbird_accounts proposes the setting accounts through the controller's settings verb (ADR 0317 §9).
if !reflect.DeepEqual(m.Invokes, []string{"seat:mesh-controller.settings"}) {
t.Errorf("the module invokes the controller's settings verb and nothing else: %v", m.Invokes)
// The tools propose through the controller's settings verb (ADR 0317 §9), and betterbird_derive asks the controller
// which nodes hold the module and each of them for its profile (ADR 0329 §10).
if !reflect.DeepEqual(m.Invokes, []string{"seat:mesh-controller.settings", "seat:mesh-controller.nodes",
"seat:mesh-controller.node", "betterbird.betterbird_derive"}) {
t.Errorf("the module invokes the controller's settings, nodes and node verbs and its own derive: %v", m.Invokes)
}
}
@@ -0,0 +1,749 @@
package main
// How the five settings of ADR 0329 reach the profile (§7–§9).
//
// Identities, calendars, address books and tags are matched as accounts are (ADR 0317): an entry the module
// recorded on this node is kept by its recorded key; else an entry under the module's own key is its creation;
// else a key a node's layer names is adopted when the profile holds it and it matches; else the profile's
// entries that match (an identity by its address within its account, a calendar and an address book by their
// address, a tag by its key): one is adopted, none means the entry is created under the module's own key, more
// than one is refused with the remedy. Their declared fields are written into the user.js the module places,
// where they win at every start. An entry leaving the setting releases what was adopted and removes the
// preferences of what was created; never a calendar's events, an address book's contacts or a cache.
//
// What Betterbird rewrites itself is written only while it does not run (§9): the lists in prefs.js (an adopted
// account's identities, the calendars' order) and each account's msgFilterRules.dat. Until then the module's
// check counts, per kind, the changes that wait.
import (
"bufio"
"bytes"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"regexp"
"strconv"
"strings"
)
// configRecordFile is where the module records, per profile, what it adopted or created of the five kinds and
// which filter files it wrote (ADR 0329 §7, §8, ADR 0317 §3).
const configRecordFile = ".config/betterbird/configuration-record.json"
// resolvedEntry is one declared entry as it resolved on this node.
type resolvedEntry struct {
Kind string `json:"kind"`
Name string `json:"name"`
How string `json:"how"`
Key string `json:"key,omitempty"`
Why string `json:"why,omitempty"`
}
type recordedEntry struct {
How string `json:"how"`
Key string `json:"key"`
}
// filterRecord is what the module did with one account's filter file on a profile.
type filterRecord struct {
// File is the account's filter file as the module last wrote it, and Kept the file it replaced at its first write.
File string `json:"file"`
Kept string `json:"kept,omitempty"`
// Released says the account left the setting filters: the module no longer writes its file (§8).
Released bool `json:"released,omitempty"`
}
type configRecord struct {
Entries map[string]map[string]recordedEntry `json:"entries,omitempty"`
Filters map[string]filterRecord `json:"filters,omitempty"`
}
func (m *Machine) readConfigRecord(profileDir string) configRecord {
all := map[string]configRecord{}
if raw, err := readBounded(m.home(configRecordFile)); err == nil {
_ = json.Unmarshal(raw, &all)
}
r := all[profileDir]
if r.Entries == nil {
r.Entries = map[string]map[string]recordedEntry{}
}
if r.Filters == nil {
r.Filters = map[string]filterRecord{}
}
return r
}
func (m *Machine) writeConfigRecord(profileDir string, r configRecord) error {
all := map[string]configRecord{}
if raw, err := readBounded(m.home(configRecordFile)); err == nil {
_ = json.Unmarshal(raw, &all)
}
all[profileDir] = r
body, err := json.MarshalIndent(all, "", " ")
if err != nil {
return err
}
return writeAtomic(m.home(configRecordFile), append(body, '\n'), 0o600)
}
// writeAtomic writes a file beside itself and renames it into place.
func writeAtomic(path string, data []byte, mode os.FileMode) error {
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
return err
}
tmp := path + ".mesh-new"
if err := writeSynced(tmp, data, mode); err != nil {
os.Remove(tmp)
return err
}
if err := os.Rename(tmp, path); err != nil {
os.Remove(tmp)
return err
}
return nil
}
// The module's own keys for what it creates: each names the entry.
func ownIdentityKey(name string) string { return "ident_mesh_" + name }
func ownCalendarKey(name string) string { return "mesh_" + name }
func ownBookKey(name string) string { return "mesh_" + name }
var (
ownIdentity = regexp.MustCompile(`^ident_mesh_[a-z][a-z0-9]{0,30}$`)
ownMeshKey = regexp.MustCompile(`^mesh_[a-z][a-z0-9]{0,30}$`)
)
// configPlan is what an apply does for the five kinds on this profile.
type configPlan struct {
Entries []resolvedEntry
// lines are the user.js statements, a comment before each entry's.
lines []string
// identityLists are, per adopted account key, the identity keys of the module's own it wants in its list.
identityLists map[string][]string
// calendarOrder are the calendar keys the module created, for calendar.list.sortOrder.
calendarOrder []string
// removals are the preference prefixes of what the module created and the setting no longer holds.
removals []string
filters []filterPlan
// released are the accounts that left the setting filters since the module last wrote their file.
released []string
record configRecord
problem string
}
// filterPlan is one declared account's filter file on this node.
type filterPlan struct {
Account string `json:"account"`
File string `json:"file,omitempty"`
// Waiting says why the file cannot be written yet, when it cannot.
Waiting string `json:"waiting,omitempty"`
Refused string `json:"refused,omitempty"`
want string
have string
exists bool
path string
}
func (p filterPlan) changes() bool { return p.Refused == "" && p.path != "" && p.want != p.have }
// planConfiguration resolves every declared entry against the profile and says what an apply writes.
func (m *Machine) planConfiguration(profile Profile, c Configuration, accounts map[string]Account,
ra []resolvedAccount, prefs map[string]string) configPlan {
rec := m.readConfigRecord(profile.Dir)
plan := configPlan{identityLists: map[string][]string{},
record: configRecord{Entries: map[string]map[string]recordedEntry{}, Filters: map[string]filterRecord{}}}
byName := map[string]resolvedAccount{}
for _, r := range ra {
byName[r.name] = r
}
have := profileAccounts(prefs)
listOfAccount := map[string][]string{}
for _, a := range have {
listOfAccount[a.Key] = a.Identities
}
identityKeyOf := map[string]string{}
// Identities.
taken := map[string]bool{}
for _, name := range sortedKeys(c.Identities) {
id := c.Identities[name]
e := resolvedEntry{Kind: kindIdentities, Name: name}
acct, declared := byName[id.Account]
switch {
case !declared:
e.How, e.Why = refused, fmt.Sprintf("its account %s is not in the setting accounts", id.Account)
case acct.How == refused:
e.How, e.Why = refused, fmt.Sprintf("its account %s is refused: betterbird_check says why", id.Account)
case id.Address == "":
e.How, e.Key = acct.How, acct.Keys.Identity
if taken[e.Key] {
e.How, e.Key, e.Why = refused, "", fmt.Sprintf("account %s has its default identity declared twice", id.Account)
}
default:
list := listOfAccount[acct.Keys.Account]
var others []string
if len(list) > 1 {
others = list[1:]
}
present := func(k string) bool {
for _, x := range others {
if x == k {
return true
}
}
return false
}
matches := func(k string) bool {
return strings.EqualFold(unquote(prefs["mail.identity."+k+".useremail"]), id.Address)
}
e = resolveEntry(kindIdentities, name, id.Key, ownIdentityKey(name), rec.Entries[kindIdentities][name],
others, present, matches, taken, "an identity of its account with its address",
"give the one to adopt as key in this node's layer of the setting identities")
if e.How == created {
plan.identityLists[acct.Keys.Account] = append(plan.identityLists[acct.Keys.Account], e.Key)
}
}
if e.How != refused {
taken[e.Key] = true
identityKeyOf[name] = e.Key
plan.lines = append(plan.lines, fmt.Sprintf("// identity %s: %s under %s", name, e.How, e.Key))
plan.lines = append(plan.lines, identityLines(e, id, acct)...)
}
plan.Entries = append(plan.Entries, e)
}
// A created account's identity list is the module's whole: its default and the identities it created.
for _, name := range sortedKeys(byName) {
r := byName[name]
if r.How == created && len(plan.identityLists[r.Keys.Account]) > 0 {
list := append([]string{r.Keys.Identity}, plan.identityLists[r.Keys.Account]...)
plan.lines = append(plan.lines, fmt.Sprintf("user_pref(%s, %s);", quoted("mail.account."+r.Keys.Account+".identities"),
quoted(strings.Join(list, ","))))
delete(plan.identityLists, r.Keys.Account)
}
}
// Adopted accounts keep their list in prefs.js: the module's own keys are written into it while Betterbird is
// closed. An adopted account whose list holds none of them and wants none is left out.
for _, a := range have {
if _, wanted := plan.identityLists[a.Key]; wanted {
continue
}
for _, k := range a.Identities {
if ownIdentity.MatchString(k) {
plan.identityLists[a.Key] = []string{}
break
}
}
}
// Calendars.
cals := profileCalendars(prefs)
calKeys := map[string]profileCalendar{}
var caldav []string
for _, pc := range cals {
calKeys[pc.Key] = pc
if pc.Kind == "caldav" {
caldav = append(caldav, pc.Key)
}
}
taken = map[string]bool{}
for _, name := range sortedKeys(c.Calendars) {
cal := c.Calendars[name]
present := func(k string) bool { pc, ok := calKeys[k]; return ok && pc.Kind == "caldav" }
matches := func(k string) bool { return sameURL(calKeys[k].Value.URL, cal.URL) }
e := resolveEntry(kindCalendars, name, cal.Key, ownCalendarKey(name), rec.Entries[kindCalendars][name],
caldav, present, matches, taken, "a CalDAV calendar at its address",
"remove the duplicate in Betterbird, or give the one to adopt as key in this node's layer of the setting calendars")
if e.How != refused {
taken[e.Key] = true
if e.How == created {
plan.calendarOrder = append(plan.calendarOrder, e.Key)
}
plan.lines = append(plan.lines, fmt.Sprintf("// calendar %s: %s under %s", name, e.How, e.Key))
plan.lines = append(plan.lines, calendarLines(e, cal, identityKeyOf[cal.Invitations])...)
}
plan.Entries = append(plan.Entries, e)
}
// Address books.
books := profileAddressBooks(prefs)
bookKeys := map[string]profileAddressBook{}
var carddav []string
for _, b := range books {
bookKeys[b.Key] = b
if b.CardDAV {
carddav = append(carddav, b.Key)
}
}
taken = map[string]bool{}
for _, name := range sortedKeys(c.AddressBooks) {
ab := c.AddressBooks[name]
present := func(k string) bool { b, ok := bookKeys[k]; return ok && b.CardDAV }
matches := func(k string) bool { return sameURL(bookKeys[k].Value.URL, ab.URL) }
e := resolveEntry(kindAddressBooks, name, ab.Key, ownBookKey(name), rec.Entries[kindAddressBooks][name],
carddav, present, matches, taken, "a CardDAV address book at its address",
"remove the duplicate in Betterbird, or give the one to adopt as key in this node's layer of the setting address-books")
if e.How != refused {
taken[e.Key] = true
plan.lines = append(plan.lines, fmt.Sprintf("// address book %s: %s under %s", name, e.How, e.Key))
plan.lines = append(plan.lines, addressBookLines(e, ab, name)...)
}
plan.Entries = append(plan.Entries, e)
}
// Tags: a tag's key is its own, unique in a profile, so a match is never ambiguous.
tagKeys := map[string]bool{}
for _, t := range profileTags(prefs) {
tagKeys[t.Key] = true
}
for _, key := range sortedKeys(c.Tags) {
t := c.Tags[key]
e := resolvedEntry{Kind: kindTags, Name: key, Key: key, How: adopted}
if was, ok := rec.Entries[kindTags][key]; ok && was.How == created {
e.How = created
} else if !tagKeys[key] {
e.How = created
}
plan.lines = append(plan.lines, fmt.Sprintf("// tag %s: %s", key, e.How))
plan.lines = append(plan.lines, tagLines(key, t)...)
plan.Entries = append(plan.Entries, e)
}
// What the setting no longer holds: an adopted entry is released (its lines go), a created one's preferences
// are removed from prefs.js while Betterbird is closed.
declaredNow := map[string]map[string]bool{}
for _, e := range plan.Entries {
if declaredNow[e.Kind] == nil {
declaredNow[e.Kind] = map[string]bool{}
}
declaredNow[e.Kind][e.Name] = true
if e.How != refused {
if plan.record.Entries[e.Kind] == nil {
plan.record.Entries[e.Kind] = map[string]recordedEntry{}
}
plan.record.Entries[e.Kind][e.Name] = recordedEntry{How: e.How, Key: e.Key}
}
}
for _, kind := range []string{kindIdentities, kindCalendars, kindAddressBooks, kindTags} {
for _, name := range sortedKeys(rec.Entries[kind]) {
was := rec.Entries[kind][name]
if declaredNow[kind][name] || was.How != created {
continue
}
switch kind {
case kindIdentities:
plan.removals = append(plan.removals, "mail.identity."+was.Key+".")
case kindCalendars:
plan.removals = append(plan.removals, "calendar.registry."+was.Key+".")
case kindAddressBooks:
plan.removals = append(plan.removals, "ldap_2.servers."+was.Key+".")
case kindTags:
plan.removals = append(plan.removals, "mailnews.tags."+was.Key+".")
}
}
}
// Filters, owned whole per declared account (§8).
for _, account := range sortedAccountsOf(c.Filters) {
fp := filterPlan{Account: account}
r, ok := byName[account]
switch {
case !ok:
fp.Refused = fmt.Sprintf("its account %s is not in the setting accounts", account)
case r.How == refused:
fp.Refused = fmt.Sprintf("its account %s is refused: betterbird_check says why", account)
default:
dir := m.accountDirectory(profile.path, prefs, r.Keys.Server)
if dir == "" {
fp.Waiting = "Betterbird has not made the account's directory yet: it does at its first start with the account"
break
}
fp.path = filepath.Join(dir, filterFileName)
fp.File = m.tilde(strings.TrimPrefix(fp.path, m.Root))
fp.want = renderFilters(c.Filters[account], folderBase(prefs, r.Keys.Server))
raw, err := readBounded(fp.path)
fp.exists = err == nil
fp.have = string(raw)
}
was := rec.Filters[account]
was.Released = false
if fp.path != "" {
was.File = fp.File
}
if was.File != "" || was.Kept != "" {
plan.record.Filters[account] = was
}
plan.filters = append(plan.filters, fp)
}
for _, account := range sortedKeys(rec.Filters) {
if _, still := c.Filters[account]; still {
continue
}
was := rec.Filters[account]
if !was.Released {
plan.released = append(plan.released, account)
}
was.Released = true
plan.record.Filters[account] = was
}
return plan
}
// resolveEntry resolves an identity, a calendar or an address book as ADR 0317 resolves an account (§7).
func resolveEntry(kind, name, layerKey, own string, was recordedEntry, candidates []string, present, matches func(string) bool,
taken map[string]bool, what, remedy string) resolvedEntry {
e := resolvedEntry{Kind: kind, Name: name}
refuse := func(format string, args ...any) resolvedEntry {
e.How, e.Key, e.Why = refused, "", fmt.Sprintf(format, args...)
return e
}
// 1. Recorded, while the profile still holds it.
if was.Key != "" && !taken[was.Key] && (present(was.Key) || was.How == created) && (layerKey == "" || layerKey == was.Key) {
e.How, e.Key = was.How, was.Key
return e
}
// 2. Under the module's own key: its creation.
if present(own) {
e.How, e.Key = created, own
return e
}
// 3. The key a node's layer names.
if layerKey != "" {
switch {
case !present(layerKey):
return refuse("the key %s this node's layer names is not in the profile", layerKey)
case taken[layerKey]:
return refuse("the key %s is already another declared entry's", layerKey)
case !matches(layerKey):
return refuse("the key %s this node's layer names is not %s", layerKey, what)
}
e.How, e.Key = adopted, layerKey
return e
}
// 4. By the match.
var found []string
for _, k := range candidates {
if !taken[k] && !ownMeshKey.MatchString(k) && !ownIdentity.MatchString(k) && matches(k) {
found = append(found, k)
}
}
switch len(found) {
case 0:
e.How, e.Key = created, own
case 1:
e.How, e.Key = adopted, found[0]
default:
return refuse("%d entries in the profile are %s (%s): %s (novox/hq ADR 0329 §7)", len(found), what,
strings.Join(found, ", "), remedy)
}
return e
}
// sameURL compares two addresses without a trailing slash's difference.
func sameURL(a, b string) bool {
return a != "" && strings.EqualFold(strings.TrimRight(a, "/"), strings.TrimRight(b, "/"))
}
func pref(name, value string) string { return fmt.Sprintf("user_pref(%s, %s);", quoted(name), value) }
func boolWord(b bool) string { return strconv.FormatBool(b) }
// identityLines are an identity's declared fields under its key. The default identity's name and address are
// the setting accounts' (§4), so only another identity carries them here.
func identityLines(e resolvedEntry, id *Identity, acct resolvedAccount) []string {
p := "mail.identity." + e.Key + "."
var out []string
if id.Address != "" {
out = append(out, pref(p+"fullName", jsQuoted(id.Name)), pref(p+"useremail", jsQuoted(id.Address)))
if e.How == created {
out = append(out, pref(p+"valid", "true"), pref(p+"smtpServer", quoted(acct.Keys.SMTP)))
}
}
above := 0
if id.SignatureAboveQuote {
above = 1
}
out = append(out,
pref(p+"reply_to", jsQuoted(id.ReplyTo)),
pref(p+"organization", jsQuoted(id.Organization)),
pref(p+"htmlSigText", jsQuoted(strings.Join(id.Signature, "\n"))),
pref(p+"htmlSigFormat", boolWord(id.SignatureHTML)),
pref(p+"attach_signature", "false"),
pref(p+"reply_on_top", strconv.Itoa(above)),
pref(p+"sig_bottom", boolWord(!id.SignatureAboveQuote)))
return out
}
func calendarLines(e resolvedEntry, cal *Calendar, identityKey string) []string {
p := "calendar.registry." + e.Key + "."
var out []string
if e.How == created {
out = append(out, pref(p+"type", quoted("caldav")), pref(p+"cache.enabled", "true"),
pref(p+"calendar-main-in-composite", "true"))
}
reminders := cal.Reminders == nil || *cal.Reminders
out = append(out, pref(p+"uri", jsQuoted(cal.URL)), pref(p+"name", jsQuoted(cal.Name)))
if cal.Color != "" {
out = append(out, pref(p+"color", jsQuoted(strings.ToLower(cal.Color))))
}
out = append(out, pref(p+"readOnly", boolWord(cal.ReadOnly)), pref(p+"suppressAlarms", boolWord(!reminders)),
pref(p+"username", jsQuoted(cal.User)), pref(p+"imip.identity.key", jsQuoted(identityKey)))
return out
}
func addressBookLines(e resolvedEntry, ab *AddressBook, name string) []string {
p := "ldap_2.servers." + e.Key + "."
var out []string
if e.How == created {
out = append(out, pref(p+"dirType", "102"), pref(p+"filename", quoted("abook-mesh_"+name+".sqlite")))
}
return append(out, pref(p+"description", jsQuoted(ab.Name)), pref(p+"carddav.url", jsQuoted(ab.URL)),
pref(p+"carddav.username", jsQuoted(ab.User)))
}
func tagLines(key string, t *Tag) []string {
p := "mailnews.tags." + key + "."
out := []string{pref(p+"tag", jsQuoted(t.Name))}
if t.Color != "" {
out = append(out, pref(p+"color", jsQuoted(strings.ToUpper(t.Color))))
}
if t.Order > 0 {
out = append(out, pref(p+"ordinal", quoted(tagOrdinal(t.Order))))
}
return out
}
// configBlock is the user.js block of the five kinds.
func (p configPlan) configBlock() string {
if len(p.lines) == 0 {
return ""
}
return "\n// The identities, calendars, address books and tags the settings declare (novox/hq ADR 0329), under the\n" +
"// keys each was adopted or created under. No password, token or cookie.\n" + strings.Join(p.lines, "\n") + "\n"
}
// closedFor says why the profile's own files may not be written now, or "": Betterbird runs, or the profile's
// lock says one holds it (§9).
func (m *Machine) closedFor(profile string) string {
if ps := m.mainProcs(); len(ps) > 0 {
return fmt.Sprintf("Betterbird is running (pid %d), and it rewrites these files when it quits", ps[0].PID)
}
if _, err := os.Lstat(filepath.Join(profile, "lock")); err == nil {
return "the profile is locked (its lock file is there): a Betterbird holds it, or one ended without quitting"
}
return ""
}
// prefsChanges is prefs.js as the plan wants it, and how many changes each kind has in it.
func (p configPlan) prefsChanges(text string) (string, map[string]int) {
counts := map[string]int{}
var lines []string
done := map[string]bool{}
s := bufio.NewScanner(strings.NewReader(text))
s.Buffer(make([]byte, 64<<10), 4<<20)
for s.Scan() {
line := s.Text()
mm := prefLine.FindStringSubmatch(line)
if mm == nil {
lines = append(lines, line)
continue
}
name := mm[1]
removed := false
for _, prefix := range p.removals {
if strings.HasPrefix(name, prefix) {
removed = true
counts[kindOfPrefix(prefix)]++
break
}
}
if removed {
continue
}
if acct, ok := strings.CutPrefix(name, "mail.account."); ok && strings.HasSuffix(acct, ".identities") {
key := strings.TrimSuffix(acct, ".identities")
if want, ok := p.identityLists[key]; ok {
done[name] = true
next := listedWithOwn(listOf(mm[2]), want, ownIdentity.MatchString, ",")
if next != strings.Join(listOf(mm[2]), ",") {
line = pref(name, quoted(next))
counts[kindIdentities]++
}
}
}
if name == "calendar.list.sortOrder" {
done[name] = true
have := strings.Fields(unquote(mm[2]))
next := listedWithOwn(have, p.calendarOrder, ownMeshKey.MatchString, " ")
if next != strings.Join(have, " ") {
line = pref(name, quoted(next))
counts[kindCalendars]++
}
}
lines = append(lines, line)
}
if !done["calendar.list.sortOrder"] && len(p.calendarOrder) > 0 {
lines = append(lines, pref("calendar.list.sortOrder", quoted(strings.Join(p.calendarOrder, " "))))
counts[kindCalendars]++
}
out := strings.Join(lines, "\n")
if text == "" || strings.HasSuffix(text, "\n") {
out += "\n"
}
return out, counts
}
func kindOfPrefix(prefix string) string {
switch {
case strings.HasPrefix(prefix, "mail.identity."):
return kindIdentities
case strings.HasPrefix(prefix, "calendar."):
return kindCalendars
case strings.HasPrefix(prefix, "ldap_2."):
return kindAddressBooks
}
return kindTags
}
// listedWithOwn is a list with the module's own keys made the wanted ones, every other key in its place.
func listedWithOwn(have, wanted []string, own func(string) bool, sep string) string {
want := map[string]bool{}
for _, k := range wanted {
want[k] = true
}
var out []string
there := map[string]bool{}
for _, k := range have {
if own(k) && !want[k] {
continue
}
out = append(out, k)
there[k] = true
}
for _, k := range wanted {
if !there[k] {
out = append(out, k)
}
}
return strings.Join(out, sep)
}
// Waiting counts, per kind, the changes of what Betterbird rewrites that wait for it to close (§9).
func (m *Machine) waiting(profile string, p configPlan, ra []resolvedAccount) map[string]int {
raw, _ := readBounded(filepath.Join(profile, "prefs.js"))
_, counts := p.prefsChanges(string(raw))
var wantAccounts, wantSMTP []string
for _, r := range ra {
if r.How == created {
wantAccounts, wantSMTP = append(wantAccounts, r.Keys.Account), append(wantSMTP, r.Keys.SMTP)
}
}
if _, changed := withListed(string(raw), map[string][]string{
"mail.accountmanager.accounts": wantAccounts, "mail.smtpservers": wantSMTP}); changed {
counts["accounts"]++
}
for _, f := range p.filters {
if f.changes() {
counts[kindFilters]++
}
}
for k, v := range counts {
if v == 0 {
delete(counts, k)
}
}
return counts
}
// writeClosed writes what Betterbird rewrites itself, only while it is closed: prefs.js's lists and the
// removals, and each declared account's filter file, keeping the file it replaces at its first write. It
// answers what it wrote, or why it waits.
func (m *Machine) writeClosed(profile Profile, p *configPlan) (string, error) {
var said []string
path := filepath.Join(profile.path, "prefs.js")
raw, err := readBounded(path)
if err != nil && !errors.Is(err, os.ErrNotExist) {
return "", err
}
next, counts := p.prefsChanges(string(raw))
pending := next != string(raw)
for _, f := range p.filters {
pending = pending || f.changes()
}
if !pending {
return "unchanged", nil
}
if why := m.closedFor(profile.path); why != "" {
return "", errors.New(why + ": the identities' and calendars' lists, the removals and the filter files are " +
"written at its next start from the launcher, or by betterbird_restart")
}
if next != string(raw) {
mode := os.FileMode(0o600)
if st, err := os.Stat(path); err == nil {
mode = st.Mode().Perm()
}
if err := writeAtomic(path, []byte(next), mode); err != nil {
return "", err
}
said = append(said, fmt.Sprintf("prefs.js: %d change(s) to the lists and removals", sum(counts)))
}
for i, f := range p.filters {
if !f.changes() {
continue
}
was := p.record.Filters[f.Account]
if was.Kept == "" && f.exists {
kept := m.keptFilterFile(profile.Dir, f.Account)
if err := writeAtomic(kept, []byte(f.have), 0o600); err != nil {
return strings.Join(said, "; "), fmt.Errorf("the filter file of %s was not written: the file it replaces could not be kept: %v", f.Account, err)
}
was.Kept = m.tilde(strings.TrimPrefix(kept, m.Root))
}
mode := os.FileMode(0o644)
if st, err := os.Stat(f.path); err == nil {
mode = st.Mode().Perm()
}
if err := writeAtomic(f.path, []byte(f.want), mode); err != nil {
return strings.Join(said, "; "), err
}
was.File = f.File
p.record.Filters[f.Account] = was
p.filters[i].have, p.filters[i].exists = f.want, true
said = append(said, "the filters of "+f.Account)
}
if len(said) == 0 {
return "unchanged", nil
}
return "written: " + strings.Join(said, "; "), nil
}
func sum(m map[string]int) int {
n := 0
for _, v := range m {
n += v
}
return n
}
// accountDirectory is an account's directory in the profile, as its server's preferences name it, or "" when
// they name none within the profile.
func (m *Machine) accountDirectory(profile string, prefs map[string]string, server string) string {
if dir := accountDirectory(profile, prefs, server); dir != "" {
return dir
}
abs := unquote(prefs["mail.server."+server+".directory"])
if abs == "" {
return ""
}
dir := m.path(abs)
if strings.HasPrefix(filepath.Clean(dir), filepath.Clean(profile)+string(filepath.Separator)) {
return dir
}
return ""
}
func mustJSON(v any) string {
b, _ := json.Marshal(v)
return string(bytes.TrimSpace(b))
}
@@ -0,0 +1,228 @@
package main
// The setting prefs, one entry per preference (novox/hq issue 586, ADR 0329; ADR 0316 §11 for the user.js).
//
// Until issue 586 the setting was one line of user_pref(…) statements, so a node's layer replaced it whole and
// the operator's own choices were shipped in the catalogue as the module's defaults file. Now the setting is a
// map from a preference's name to its value, and reaches the machine as ~/.config/betterbird/prefs.json, a
// file the node-engine merges from the module's layers key by key: the mesh-wide layer holds the operator's
// setup, and a node's layer overrides one preference, or takes one out with null. The module itself ships only
// its policy's locks and Betterbird's own defaults (mesh-defaults.js names the preferences it manages there).
//
// The tools turn the setting into the head of the user.js they place in the profile, where Betterbird reads it
// at its start and lets it win over what it saved itself. A preference that can hold a credential, or that
// another setting owns (an account's, an identity's, a calendar's, an address book's, a tag's), is refused, so
// each value has one owner (ADR 0316 §11, ADR 0329 §6).
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"math"
"os"
"regexp"
"sort"
"strconv"
"strings"
)
// prefsFile is the setting prefs on the machine, written by the node-engine.
const prefsFile = ".config/betterbird/prefs.json"
// configurationOwned are the preferences the settings of ADR 0329 own: the setting prefs refuses them (§6).
var configurationOwned = []string{"calendar.registry.", "calendar.list.", "ldap_2.servers.", "mailnews.tags."}
// prefName is a preference's name as the setting may give it.
var prefName = regexp.MustCompile(`^[A-Za-z0-9_][A-Za-z0-9_.\-\[\]@{}]{0,199}$`)
// prefOwner says why the setting prefs may not set a preference, or "" when it may.
func prefOwner(name string) string {
if passwordSecret.MatchString(name) {
return "a preference that can hold a credential: not through the mesh"
}
for _, p := range accountOwned {
if strings.HasPrefix(name, p) {
return "an account's preference: an account is declared in the setting accounts, never in prefs (novox/hq ADR 0316 §11)"
}
}
for _, p := range configurationOwned {
if strings.HasPrefix(name, p) {
return "owned by the setting calendars, address-books or tags, never by prefs (novox/hq ADR 0329 §6)"
}
}
return ""
}
// readDeclaredPrefs reads the setting prefs on the machine: each preference's value as user.js writes it. A
// missing file says the module is not pushed here in this form; a file that names a preference another setting
// owns, or holds a value user.js cannot carry, is refused whole, naming the preference.
func (m *Machine) readDeclaredPrefs() (map[string]string, bool, error) {
raw, err := readBounded(m.home(prefsFile))
if errors.Is(err, os.ErrNotExist) {
return map[string]string{}, false, nil
}
if err != nil {
return nil, false, err
}
prefs, err := parseDeclaredPrefs(raw)
return prefs, true, err
}
func parseDeclaredPrefs(raw []byte) (map[string]string, error) {
var top map[string]json.RawMessage
if err := json.Unmarshal(raw, &top); err != nil {
return nil, fmt.Errorf("~/%s is not a JSON object: %v", prefsFile, err)
}
out := map[string]string{}
body, ok := top["prefs"]
if !ok || string(body) == "null" {
return out, nil
}
var values map[string]any
dec := json.NewDecoder(bytes.NewReader(body))
dec.UseNumber()
if err := dec.Decode(&values); err != nil {
return nil, fmt.Errorf("the setting prefs is a map from a preference's name to its value: %v", err)
}
var problems []string
for _, name := range sortedKeys(values) {
if !prefName.MatchString(name) {
problems = append(problems, fmt.Sprintf("%q is not a preference's name", name))
continue
}
if why := prefOwner(name); why != "" {
problems = append(problems, name+": "+why)
continue
}
lit, err := prefLiteral(values[name])
if err != nil {
problems = append(problems, name+": "+err.Error())
continue
}
if lit != "" {
out[name] = lit
}
}
if len(problems) > 0 {
return nil, fmt.Errorf("the setting prefs is refused: %s", strings.Join(problems, "; "))
}
return out, nil
}
// prefLiteral is a setting's value as user.js writes it: true or false, a whole number, or a string. null is
// no value: a node's layer takes the mesh-wide one out with it, and Betterbird's own default holds there.
func prefLiteral(v any) (string, error) {
switch x := v.(type) {
case nil:
return "", nil
case bool:
return strconv.FormatBool(x), nil
case json.Number:
n, err := x.Int64()
if err != nil || n > math.MaxInt32 || n < math.MinInt32 {
return "", fmt.Errorf("%s is not a whole number Betterbird keeps", x)
}
return strconv.FormatInt(n, 10), nil
case float64:
if x != math.Trunc(x) || x > math.MaxInt32 || x < math.MinInt32 {
return "", fmt.Errorf("%v is not a whole number Betterbird keeps", x)
}
return strconv.FormatInt(int64(x), 10), nil
case string:
if strings.ContainsAny(x, lineEnds) {
return "", errors.New("a string preference is one line")
}
return jsQuoted(x), nil
}
return "", fmt.Errorf("a value is true, false, a whole number or a string, not %T", v)
}
// prefValue is a preference's literal as the setting holds it: the reverse of prefLiteral.
func prefValue(lit string) (any, bool) {
switch {
case lit == "true":
return true, true
case lit == "false":
return false, true
}
if n, err := strconv.ParseInt(lit, 10, 32); err == nil {
return n, true
}
if strings.HasPrefix(lit, `"`) {
if s, err := strconv.Unquote(lit); err == nil {
return s, true
}
}
return nil, false
}
// lineEnds are the characters a reader may take as a line's end: a setting holds none (novox/hq issue 339).
const lineEnds = "\n\r\v\f\x00\u0085

"
// jsQuoted is a string as a preference file writes it: quote, backslash and the control characters escaped the
// way Gecko's preference parser reads them.
func jsQuoted(s string) string {
var b strings.Builder
b.WriteByte('"')
for _, r := range s {
switch {
case r == '"':
b.WriteString(`\"`)
case r == '\\':
b.WriteString(`\\`)
case r == '\n':
b.WriteString(`\n`)
case r == '\r':
b.WriteString(`\r`)
case r == '\t':
b.WriteString(`\t`)
case r < 0x20 || r == 0x7f || r == 0x2028 || r == 0x2029 || r == 0x85:
fmt.Fprintf(&b, `\u%04x`, r)
default:
b.WriteRune(r)
}
}
b.WriteByte('"')
return b.String()
}
// declaredText is the head of the user.js the module places: its mark and the setting prefs, one statement a
// line, by name. present says the setting is on this machine; err says why it is refused.
func (m *Machine) declaredText() (string, bool, error) {
prefs, present, err := m.readDeclaredPrefs()
if err != nil || !present {
return "", present, err
}
return composeDeclared(prefs), true, nil
}
// composeDeclared is the user.js head for a set of declared preferences.
func composeDeclared(prefs map[string]string) string {
var b strings.Builder
b.WriteString(mark + ", novox/hq issue 345). Owned by the mesh: the tools write it into the profile from the\n" +
"// module's settings at Betterbird's start (betterbird_user_js with apply, betterbird_restart, the launcher), and\n" +
"// Betterbird reads it at its start, where it wins over what Betterbird saved. Change the settings, never this file.\n" +
"//\n// The setting prefs (novox/hq issue 586): the operator's setup, mesh-wide with a node's own layer over it.\n")
names := make([]string, 0, len(prefs))
for n := range prefs {
names = append(names, n)
}
sort.Strings(names)
for _, n := range names {
fmt.Fprintf(&b, "user_pref(%s, %s);\n", quoted(n), prefs[n])
}
if len(names) == 0 {
b.WriteString("// none\n")
}
return b.String()
}
func sortedKeys[V any](m map[string]V) []string {
out := make([]string, 0, len(m))
for k := range m {
out = append(out, k)
}
sort.Strings(out)
return out
}
@@ -0,0 +1,178 @@
package main
// The profile's identities, calendars, address books and tags, read from its preferences into the setting's
// shape (novox/hq ADR 0329). The profile is read as Betterbird starts with it: prefs.js, with user.js over it.
// Nothing here reads the login store, a calendar's events or an address book's contacts (§13).
import (
"path/filepath"
"sort"
"strconv"
"strings"
)
// profileIdentity is one identity the profile holds.
type profileIdentity struct {
Key string
AccountKey string
// Default says it is its account's first identity.
Default bool
Owl bool
Value Identity
}
// profileIdentities reads every identity of every account in the profile's list, the default first.
func (m *Machine) profileIdentities(prefs map[string]string) []profileIdentity {
var out []profileIdentity
for _, a := range profileAccounts(prefs) {
for i, id := range a.Identities {
out = append(out, profileIdentity{Key: id, AccountKey: a.Key, Default: i == 0, Owl: a.Owl,
Value: m.readIdentity(prefs, id)})
}
}
return out
}
// readIdentity reads one identity's fields. A signature the profile reads from a file is read as its text, since
// a file's path differs per node (§10).
func (m *Machine) readIdentity(prefs map[string]string, key string) Identity {
p := "mail.identity." + key + "."
text := func(name string) string { return unquote(prefs[p+name]) }
id := Identity{Address: text("useremail"), Name: text("fullName"), ReplyTo: text("reply_to"),
Organization: text("organization"), SignatureHTML: prefs[p+"htmlSigFormat"] == "true",
SignatureAboveQuote: prefs[p+"reply_on_top"] == "1" && prefs[p+"sig_bottom"] == "false"}
sig := text("htmlSigText")
if prefs[p+"attach_signature"] == "true" {
if file := text("sig_file"); file != "" {
sig = m.signatureFile(file)
}
}
if sig = strings.TrimRight(strings.ReplaceAll(sig, "\r\n", "\n"), "\n"); sig != "" {
id.Signature = strings.Split(sig, "\n")
}
return id
}
// signatureFile is a signature file's text, read only within the home and bounded.
func (m *Machine) signatureFile(file string) string {
clean := filepath.Clean(file)
if !strings.HasPrefix(clean, strings.TrimSuffix(m.Home, "/")+"/") {
return ""
}
raw, err := readBounded(m.path(clean))
if err != nil || len(raw) > 64<<10 {
return ""
}
return string(raw)
}
// keysUnder are the distinct second-level keys of the preferences under a prefix: calendar.registry.<key>.….
func keysUnder(prefs map[string]string, prefix string) []string {
seen := map[string]bool{}
for name := range prefs {
if rest, ok := strings.CutPrefix(name, prefix); ok {
if key, _, found := strings.Cut(rest, "."); found && key != "" {
seen[key] = true
}
}
}
return sortedKeys(seen)
}
// profileCalendar is one calendar of the profile: Kind is Betterbird's (caldav, ics, storage, owl …).
type profileCalendar struct {
Key string
Kind string
// IdentityKey is the identity that answers its invitations.
IdentityKey string
Value Calendar
}
func profileCalendars(prefs map[string]string) []profileCalendar {
var out []profileCalendar
for _, key := range keysUnder(prefs, "calendar.registry.") {
p := "calendar.registry." + key + "."
text := func(name string) string { return unquote(prefs[p+name]) }
c := profileCalendar{Key: key, Kind: text("type"), IdentityKey: text("imip.identity.key"),
Value: Calendar{Name: text("name"), URL: text("uri"), Color: strings.ToLower(text("color")),
ReadOnly: prefs[p+"readOnly"] == "true", User: text("username")}}
if prefs[p+"suppressAlarms"] == "true" {
off := false
c.Value.Reminders = &off
}
if c.Kind == "" && c.Value.URL == "" {
continue
}
out = append(out, c)
}
return out
}
// profileAddressBook is one address book of the profile: CardDAV is the kind a setting holds.
type profileAddressBook struct {
Key string
CardDAV bool
Kind string
Value AddressBook
}
// Betterbird's kinds of address book (dirType).
var addressBookKinds = map[string]string{"0": "LDAP", "101": "local", "102": "CardDAV"}
func profileAddressBooks(prefs map[string]string) []profileAddressBook {
var out []profileAddressBook
for _, key := range keysUnder(prefs, "ldap_2.servers.") {
p := "ldap_2.servers." + key + "."
if _, has := prefs[p+"dirType"]; !has && prefs[p+"description"] == "" {
continue
}
kind := addressBookKinds[prefs[p+"dirType"]]
if kind == "" {
kind = "other"
}
out = append(out, profileAddressBook{Key: key, CardDAV: kind == "CardDAV", Kind: kind,
Value: AddressBook{Name: unquote(prefs[p+"description"]), URL: unquote(prefs[p+"carddav.url"]),
User: unquote(prefs[p+"carddav.username"])}})
}
return out
}
// profileTag is one tag of the profile.
type profileTag struct {
Key string
ordinal string
Value Tag
}
// profileTags reads the tags, each with its place as Betterbird orders them.
func profileTags(prefs map[string]string) []profileTag {
var out []profileTag
for _, key := range keysUnder(prefs, "mailnews.tags.") {
p := "mailnews.tags." + key + "."
if _, has := prefs[p+"tag"]; !has {
continue
}
out = append(out, profileTag{Key: key, ordinal: unquote(prefs[p+"ordinal"]),
Value: Tag{Name: unquote(prefs[p+"tag"]), Color: strings.ToUpper(unquote(prefs[p+"color"]))}})
}
// Betterbird orders the tags by their ordinal, a tag without one by its key in its place.
at := func(t profileTag) string {
if t.ordinal != "" {
return t.ordinal
}
return t.Key
}
sort.SliceStable(out, func(i, j int) bool {
if at(out[i]) != at(out[j]) {
return at(out[i]) < at(out[j])
}
return out[i].Key < out[j].Key
})
for i := range out {
out[i].Value.Order = i + 1
}
return out
}
// tagOrdinal is the ordinal a tag's place is written as, so Betterbird's order by text is the declared one.
func tagOrdinal(order int) string { return strconv.Itoa(1000 + order) }
@@ -165,8 +165,8 @@ func (m *Machine) ReminderTest(after time.Duration, carry []string, stop bool) (
// The module's user.js first, then what is carried, then the test's own preferences: the last
// statement wins, so neither can take the test's calendar or its quiet start away.
user := ""
if mine, err := readBounded(m.home(declared)); err == nil && validUserJS(string(mine)) == nil {
user = string(mine) + "\n"
if mine, present, err := m.declaredText(); err == nil && present && validUserJS(mine) == nil {
user = mine + "\n"
}
user += strings.Join(carried, "\n") + "\n" + testPrefs(strings.TrimPrefix(calendar, m.Root))
if err := os.WriteFile(filepath.Join(dir, "profile", "user.js"), []byte(user), 0o600); err != nil {
@@ -1,85 +1,20 @@
// Written by the mesh (module betterbird, novox/hq issue 345): the operator's setup of Betterbird, taken from
// the laptop's profile and approved by the operator on 2026-10-12. Betterbird reads this file at its start as
// a default: a value the profile saved in its prefs.js (a choice made in Betterbird's settings window) wins
// over it, a node's layer of the setting prefs (user.js) wins over both, and the policy's locks
// (distribution/policies.json) win over everything. betterbird_config shows the layers and the drift;
// betterbird_config_apply brings a profile back to these values. Change it in the catalogue, never here.
// Written by the mesh (module betterbird, novox/hq issue 345, issue 586): the preferences the module manages at
// Betterbird's own default. It holds no value of the operator's: the operator's setup is the module's setting
// prefs, mesh-wide with a node's own layer over it, which the tools place in the profile's user.js, where it wins
// at every start; the policy's locks (distribution/policies.json) win over everything. betterbird_config shows the
// layers and the drift. Change it in the catalogue, never here.
//
// A line "keeps Betterbird's default" names a preference the module manages without writing a value: the
// operator's setup is Betterbird's own default there, and a value the profile saved is drift.
// --- Theme and display ---
// A line "keeps Betterbird's default" names a preference the module manages without writing a value: Betterbird's
// own default holds there, and a value the profile saved is drift, which betterbird_config_apply clears.
// keeps Betterbird's default: extensions.activeThemeID
// The default theme (default-theme@mozilla.org) follows the desktop's dark mode; compactness comes from
// mail.uidensity, not a theme. Betterbird keeps the active theme in its add-on database, so another theme is
// changed back in the add-on manager, never by clearing prefs.js.
// The default theme (default-theme@mozilla.org) follows the desktop's dark mode. Betterbird keeps the active theme
// in its add-on database, so another theme is changed back in the add-on manager, never by clearing prefs.js.
// keeps Betterbird's default: browser.display.use_system_colors
// Betterbird's own colours, which the default theme is drawn for.
// keeps Betterbird's default: layout.css.always_underline_links
// Links are underlined only where the message's own style says so.
// Compact density: more of the message list fits on a laptop screen (Betterbird's default is 1, normal).
pref("mail.uidensity", 0);
// 14-point interface text, the operator's choice; a larger desk screen sets its own size in a node layer.
pref("mail.uifontsize", 14);
// Message bodies are shown as their sender wrote them, not darkened.
pref("mail.dark-reader.enabled", false);
// --- Layout ---
// Vertical view: folders, the message list and the message side by side (Betterbird's default is 0, classic).
pref("mail.pane_config.dynamic", 2);
// Each row of the message list on several lines, so sender, subject and date all show in the narrow column.
pref("mail.pane_config.multiline_all", true);
// The message list as cards (Betterbird's default is 1, the table).
pref("mail.threadpane.listview", 0);
// Threads grouped in every folder by default (Betterbird's default is 0, unthreaded).
pref("mailnews.default_view_flags", 1);
// --- Mail handling and compose ---
// No confirmation before Shift+Delete removes a message for good.
pref("mail.warn_on_shift_delete", false);
// keeps Betterbird's default: mail.spam.manualMark
// Marking a message as junk by hand does nothing more than mark it.
// keeps Betterbird's default: mail.compose.show_attachment_pane
// The composer shows its attachment pane only when a message has attachments.
// --- Notifications ---
// No pop-up for new mail: the operator reads mail when they choose to.
pref("mail.biff.show_alert", false);
// No sound for new mail.
pref("mail.biff.play_sound", false);
// No tray icon for new mail.
pref("mail.biff.show_tray_icon", false);
// If a node's layer turns alerts back on, they go through the desktop's notification daemon.
pref("mail.biff.use_system_alert", true);
// --- Calendar ---
// The calendar's working day starts at 09:00 (Betterbird's default is 08:00).
pref("calendar.view.daystarthour", 9);
// and ends at 20:00 (Betterbird's default is 17:00).
pref("calendar.view.dayendhour", 20);
// A calendar item is deleted without asking.
pref("calendar.item.promptDelete", false);
// The calendar follows the node's time zone, because the laptop travels.
pref("calendar.timezone.useSystemTimezone", true);
// Reminders show in their window (Betterbird's default, made explicit: the reminder is why this module exists).
pref("calendar.alarms.show", true);
// Reminders missed while the laptop slept show at the next start (Betterbird's default is false).
pref("calendar.alarms.showmissed", true);
// --- Chat ---
// No chat button: the operator uses no chat account in Betterbird (Betterbird's default is true).
pref("mail.chat.enabled", false);
// --- Privacy ---
// No remote content in a message unless its sender is allowed (Betterbird's default, made explicit).
pref("mailnews.message_display.disable_remote_image", true);
// Ask sites not to track. A default only: the policy cannot lock a privacy. preference.
pref("privacy.donottrackheader.enabled", true);
+48 -12
View File
@@ -9,15 +9,11 @@
"x11-display"
],
"invokes": [
"seat:mesh-controller.settings"
"seat:mesh-controller.settings",
"seat:mesh-controller.nodes",
"seat:mesh-controller.node",
"betterbird.betterbird_derive"
],
"settings": {
"prefs": {
"kind": "preference",
"default": "// none",
"why": "Betterbird runs with its own defaults until a machine's assignment names a preference: one line of user_pref(\"name\", value); statements, which the module's user.js holds and Betterbird reads at its start (novox/hq issue 345). An account's preferences (mail.account., mail.identity., mail.server., mail.smtpserver. and the account lists) are refused here: an account is declared in the setting accounts, so it has one owner (novox/hq ADR 0316 §11)"
}
},
"tools": [
"betterbird_status",
"betterbird_prefs",
@@ -28,6 +24,13 @@
"betterbird_restart",
"betterbird_check",
"betterbird_accounts",
"betterbird_identities",
"betterbird_calendars",
"betterbird_address_books",
"betterbird_filters",
"betterbird_tags",
"betterbird_derive",
"betterbird_drift",
"betterbird_reminder_test"
],
"resources": [
@@ -39,12 +42,16 @@
"mode": "0755"
},
{
"id": "user-js",
"id": "prefs",
"type": "file",
"path": "${machine:account-home}/.config/betterbird/user.js",
"path": "${machine:account-home}/.config/betterbird/prefs.json",
"owner": "${machine:account}",
"mode": "0644",
"content": "// Betterbird's user.js (module betterbird, novox/hq issue 345). Owned by the mesh: the node-engine\n// writes it here at every push, and the module's tools copy it whole into Betterbird's profile\n// (betterbird_user_js with apply, betterbird_restart), where Betterbird reads it at its start and lets it\n// win over the preferences Betterbird saves itself. Change the module's setting prefs, never this file or the profile's copy.\n//\n// Only comments and user_pref(\"name\", value); statements; the tools refuse anything else, and any\n// preference whose name can hold a credential.\n${setting:prefs}\n",
"merge": "json",
"takes-settings": [
"prefs"
],
"content": "{\n \"prefs\": {}\n}\n",
"trusted": true
},
{
@@ -60,6 +67,35 @@
"content": "{\n \"accounts\": {}\n}\n",
"trusted": true
},
{
"id": "configuration",
"type": "file",
"path": "${machine:account-home}/.config/betterbird/configuration.json",
"owner": "${machine:account}",
"mode": "0600",
"merge": "json",
"takes-settings": [
"identities",
"calendars",
"address-books",
"filters"
],
"content": "{\n \"identities\": {},\n \"calendars\": {},\n \"address-books\": {},\n \"filters\": {}\n}\n",
"trusted": true
},
{
"id": "tags",
"type": "file",
"path": "${machine:account-home}/.config/betterbird/tags.json",
"owner": "${machine:account}",
"mode": "0644",
"merge": "json",
"takes-settings": [
"tags"
],
"content": "{\n \"tags\": {}\n}\n",
"trusted": false
},
{
"id": "community-package",
"type": "package",
@@ -101,7 +137,7 @@
"type": "file",
"path": "/usr/local/bin/betterbird",
"mode": "0755",
"content": "#!/bin/sh\n# Betterbird's start, written by the mesh (module betterbird, novox/hq ADR 0316). Replaced at every push.\n#\n# Before Betterbird starts, the module's tools place its user.js in the profile (the setting prefs, and the\n# accounts the setting accounts declares) and add the declared accounts to the profile's account list. They do\n# it only while no Betterbird of this account runs, because Betterbird rewrites its saved preferences when it quits; a\n# second start while one runs only hands over to it. What the preparation could not do it says here, and\n# betterbird_check says it too. Betterbird starts either way: its configuration never holds the mail back.\n/usr/local/lib/mesh-betterbird/betterbird-tools prepare || echo \"betterbird: the mesh's preparation did not complete; betterbird_check says why\" >&2\nexec /opt/betterbird/betterbird \"$@\"\n"
"content": "#!/bin/sh\n# Betterbird's start, written by the mesh (module betterbird, novox/hq ADR 0316). Replaced at every push.\n#\n# Before Betterbird starts, the module's tools place its user.js in the profile (the settings prefs, accounts, identities, calendars,\n# address-books and tags) and write what Betterbird rewrites itself: the account lists and each declared account's filters. They do\n# it only while no Betterbird of this account runs, because Betterbird rewrites its saved preferences when it quits; a\n# second start while one runs only hands over to it. What the preparation could not do it says here, and\n# betterbird_check says it too. Betterbird starts either way: its configuration never holds the mail back.\n/usr/local/lib/mesh-betterbird/betterbird-tools prepare || echo \"betterbird: the mesh's preparation did not complete; betterbird_check says why\" >&2\nexec /opt/betterbird/betterbird \"$@\"\n"
},
{
"id": "launcher",
+16 -2
View File
@@ -7,9 +7,23 @@ machine that is off builds nothing and blocks nothing. The controller asks the r
the outcome names the machine that built it.
What a holding machine needs is what the builder always needed, said here once: a container runtime
(the socket is mounted), the artifact store and the package registry as provisions, a workspace, and
the bus credential. The code is `cmd/mesh-builder` in the mesh-controller repository, compiled from
(the socket is mounted), the artifact store as a provision, a workspace, and the bus credential. The code is `cmd/mesh-builder` in the mesh-controller repository, compiled from
that repository's main (`build.artifacts[].context`); this module ships the packaging.
Assign it to every machine with a container runtime. It replaces `builder`, the one-holder form of the
same thing; retire that once this is assigned where it was.
## The package registry (novox/hq ADR 0336)
The package registry (`npm-package-registry`, provided by the gitea module) is an optional requirement.
Genesis assigns the build agent before anything provides it, so the agent is assignable on a mesh with no
registry, and `mesh-controller.plan` and `node` list the requirement as unbound until the gitea module is
assigned; it then binds with no further act.
Until it binds, the agent on genesis's node reads the `genesis-package-registry` own-secret, which the
installer seals to it at the packages step: the forge it raised, the port, the npm path, the account
`mesh-builder` and its password. Issued outside the mesh, so no other machine is given one. A real binding
wins over it, and each build's log says which the build resolved from. A build that needs the registry (a
`package` artifact, an image whose Dockerfile names `.npmrc`, a TypeScript artifact with an `@novox/`
dependency) on an agent with neither fails before building, naming the machine. Go artifacts and images
that do not name `.npmrc` build without it. `manifest_test.go` holds the manifest to this.
+3
View File
@@ -0,0 +1,3 @@
module build-agent
go 1.22
+117
View File
@@ -0,0 +1,117 @@
// The build agent's manifest (novox/hq ADR 0336, issue 121). The build agent is assigned at genesis's
// step 13, before anything provides the package registry, so that requirement is optional
// (ADR 0330); until it binds, the agent on genesis's node reads the genesis document the installer
// seals to it at the packages step. The program is mesh-controller's cmd/mesh-builder; only the
// packaging lives here, and this is its test.
package buildagent
import (
"encoding/json"
"os"
"strings"
"testing"
)
type manifest struct {
Requires []json.RawMessage `json:"requires"`
Binds map[string]string `json:"binds"`
Secrets map[string]string `json:"secrets"`
OwnSecrets map[string]json.RawMessage `json:"own-secrets"`
Resources []map[string]any `json:"resources"`
}
func read(t *testing.T) manifest {
t.Helper()
raw, err := os.ReadFile("module.json")
if err != nil {
t.Fatal(err)
}
var m manifest
if err := json.Unmarshal(raw, &m); err != nil {
t.Fatal(err)
}
return m
}
// requirement is one entry of `requires`: its name, and whether it is optional.
func requirements(t *testing.T, m manifest) map[string]bool {
t.Helper()
out := map[string]bool{}
for _, raw := range m.Requires {
var name string
if json.Unmarshal(raw, &name) == nil {
out[name] = false
continue
}
var long struct {
Name string `json:"name"`
Optional bool `json:"optional"`
}
if err := json.Unmarshal(raw, &long); err != nil {
t.Fatalf("requires: %s: %v", raw, err)
}
out[long.Name] = long.Optional
}
return out
}
func TestThePackageRegistryIsOptionalAndTheArtifactStoreStaysHard(t *testing.T) {
r := requirements(t, read(t))
optional, has := r["npm-package-registry"]
if !has || !optional {
t.Errorf("npm-package-registry is not an optional requirement (ADR 0336 §1): %v", r)
}
if optional, has := r["artifact-store"]; !has || optional {
t.Errorf("artifact-store is not a hard requirement: %v", r)
}
}
// The binding and the credential of the registry are delivered only while it is bound: ADR 0330 §6
// composes neither for an absent requirement, so no file of the module may name them through a
// placeholder unless it is itself only-when-bound.
func TestTheRegistrysBindsAndSecretsAreOnlyWhenBound(t *testing.T) {
m := read(t)
if m.Binds["npm-package-registry"] == "" || m.Secrets["npm-package-registry"] == "" {
t.Fatalf("the agent is not told where the registry is once it binds: binds %v, secrets %v", m.Binds, m.Secrets)
}
for _, r := range m.Resources {
content, _ := r["content"].(string)
if strings.Contains(content, "${bound:npm-package-registry:") || strings.Contains(content, "${secret:npm-package-registry}") {
if r["only-when-bound"] != "npm-package-registry" {
t.Errorf("%v names the registry's binding and is delivered while it is absent", r["id"])
}
}
}
}
// The genesis document is an own-secret issued outside the mesh — by the installer at genesis's packages
// step (ADR 0336 §3) — so the mesh never makes one of its own on any other machine, and the agent is
// told where it lands.
func TestTheGenesisDocumentIsDeclaredAndIssuedOutside(t *testing.T) {
m := read(t)
raw, has := m.OwnSecrets["genesis-package-registry"]
if !has {
t.Fatalf("the build agent declares no genesis-package-registry own-secret: %v", m.OwnSecrets)
}
var own struct {
Path string `json:"path"`
IssuedBy string `json:"issued-by"`
}
if err := json.Unmarshal(raw, &own); err != nil || own.IssuedBy != "outside" {
t.Fatalf("the genesis document is not issued outside the mesh: %s (%v)", raw, err)
}
if !strings.HasPrefix(own.Path, "${dir:mesh-state}/") {
t.Fatalf("the genesis document is not where the container reads /run/mesh: %s", own.Path)
}
inside := "/run/mesh/" + strings.TrimPrefix(own.Path, "${dir:mesh-state}/")
for _, r := range m.Resources {
if r["id"] == "agent-env" {
content, _ := r["content"].(string)
if !strings.Contains(content, "MESH_GENESIS_PACKAGE_REGISTRY="+inside+"\n") {
t.Errorf("the agent is not told where the genesis document is (%s):\n%s", inside, content)
}
return
}
}
t.Fatal("the build agent has no environment file")
}
+10 -3
View File
@@ -14,7 +14,10 @@
],
"requires": [
"artifact-store",
"npm-package-registry"
{
"name": "npm-package-registry",
"optional": true
}
],
"binds": {
"npm-package-registry": "${dir:mesh-state}/package-registry.json"
@@ -23,7 +26,11 @@
"npm-package-registry": "${dir:mesh-state}/package-registry.secret"
},
"own-secrets": {
"broker": "${dir:mesh-state}/broker"
"broker": "${dir:mesh-state}/broker",
"genesis-package-registry": {
"path": "${dir:mesh-state}/genesis-package-registry.json",
"issued-by": "outside"
}
},
"data": {
"own": [
@@ -52,7 +59,7 @@
"type": "file",
"path": "${dir:mesh-state}/build-agent.env",
"mode": "0600",
"content": "MESH_BROKER_FILE=/run/mesh/broker\nMESH_NODE=${machine:name}\nMESH_REGISTRY=${bound:artifact-store:at}:${bound:artifact-store:port}\nMESH_PACKAGE_BINDING=/run/mesh/package-registry.json\nMESH_NPM_TOKEN_FILE=/run/mesh/package-registry.secret\nMESH_WORKSPACE=${dir:workspace}\n"
"content": "MESH_BROKER_FILE=/run/mesh/broker\nMESH_NODE=${machine:name}\nMESH_REGISTRY=${bound:artifact-store:at}:${bound:artifact-store:port}\nMESH_PACKAGE_BINDING=/run/mesh/package-registry.json\nMESH_NPM_TOKEN_FILE=/run/mesh/package-registry.secret\nMESH_GENESIS_PACKAGE_REGISTRY=/run/mesh/genesis-package-registry.json\nMESH_WORKSPACE=${dir:workspace}\n"
},
{
"id": "server",
@@ -17,6 +17,7 @@ import (
"errors"
"fmt"
"net"
"net/netip"
"os"
"os/exec"
"path/filepath"
@@ -147,6 +148,8 @@ type JailSettings struct {
// Fail2ban is the daemon as this machine has it, through its own client.
type Fail2ban struct {
Run Runner
// Root is where the machine's files are read from: "" on the machine, a directory in a test.
Root string
}
func (f Fail2ban) client(ctx context.Context, args ...string) (string, error) {
@@ -229,6 +232,13 @@ type BanOutcome struct {
}
// Ban bans one address in one jail now. The daemon's own answer is how many addresses it added.
//
// **Never one of the operator's own addresses, nor one the jail spares** (novox/hq ADR 0339 §3, narrowing
// ADR 0179 §4): an address on the setting `own-addresses`, or on any entry of the jail's own ignore list — the
// private ranges ADR 0186 spares among them — is refused, naming the entry it lies on, and nothing reaches the
// daemon. fail2ban's `banip` bans whatever it is given, ignore list or not: the 2026-10-04 lockout of the
// operator's home (issue 238) needed only a jail; this verb could have done it in one call. When the setting
// cannot be read the ban is refused too: whether the address is the operator's cannot then be told.
func (f Fail2ban) Ban(ctx context.Context, ip, jail string) (*BanOutcome, error) {
if err := address(ip); err != nil {
return nil, err
@@ -236,6 +246,9 @@ func (f Fail2ban) Ban(ctx context.Context, ip, jail string) (*BanOutcome, error)
if err := jailName(jail); err != nil {
return nil, err
}
if err := f.spared(ctx, ip, jail); err != nil {
return nil, err
}
out, err := f.client(ctx, "set", jail, "banip", ip)
if err != nil {
return nil, err
@@ -255,6 +268,38 @@ func (f Fail2ban) Ban(ctx context.Context, ip, jail string) (*BanOutcome, error)
return outcome, nil
}
// spared refuses an address the operator owns or the jail ignores, saying which entry it lies on.
func (f Fail2ban) spared(ctx context.Context, ip, jail string) error {
a, err := netip.ParseAddr(ip)
if err != nil {
return fmt.Errorf("%q is not an address", ip)
}
own, err := ReadOwn(f.Root)
if err != nil {
return fmt.Errorf("refused, and nothing was banned: %v, so whether %s is one of them cannot be told", err, ip)
}
if on := own.Lies(a); on != "" {
return fmt.Errorf("refused, and nothing was banned: %s lies on %s, one of the operator's own addresses "+
"(the setting %s), which no jail bans (novox/hq ADR 0339)", ip, on, SettingKey)
}
out, err := f.client(ctx, "get", jail, "ignoreip")
if err != nil {
return err
}
var entries []netip.Prefix
for _, word := range listed(out) {
// A file of entries (the operator's own, read above) or a name the daemon resolves is not an entry here.
if p, err := entry(word); err == nil {
entries = append(entries, p)
}
}
if on := liesOn(a, entries); on != "" {
return fmt.Errorf("refused, and nothing was banned: %s lies on %s, which the jail %s never bans — the "+
"machine itself, the mesh's own range or a private one (novox/hq ADR 0186, ADR 0339)", ip, on, jail)
}
return nil
}
// Released is how many bans the daemon let go, of which address, from where.
type Released struct {
Released int `json:"released"`
@@ -108,7 +108,8 @@ func TestBannedGathersEveryJailsBansSoonestToEndFirst(t *testing.T) {
func TestBanAsksByJailAndAnswersTheBanAsHeldRefusingANonAddressFirst(t *testing.T) {
var calls [][]string
f := Fail2ban{Run: fake(map[string]string{
f := Fail2ban{Root: t.TempDir(), Run: fake(map[string]string{
"get recidive ignoreip": ignoring,
"set recidive banip 198.51.100.7": "1\n",
"get recidive banip --with-time": withTime + "198.51.100.7 \t2026-10-02 17:00:00 + 604800 = 2026-10-09 17:00:00\n",
}, &calls)}
@@ -119,8 +120,8 @@ func TestBanAsksByJailAndAnswersTheBanAsHeldRefusingANonAddressFirst(t *testing.
if r.Added != 1 || r.Banned == nil || r.Banned.Until != "2026-10-09 17:00:00" {
t.Errorf("%+v", r)
}
if !reflect.DeepEqual(calls[0], []string{"fail2ban-client", "set", "recidive", "banip", "198.51.100.7"}) {
t.Errorf("first call %v", calls[0])
if !reflect.DeepEqual(calls[1], []string{"fail2ban-client", "set", "recidive", "banip", "198.51.100.7"}) {
t.Errorf("the ban %v", calls)
}
if _, err := f.Ban(ctx, "not-an-ip", "recidive"); err == nil || !strings.Contains(err.Error(), "is not an address") {
t.Errorf("a non-address: %v", err)
@@ -128,7 +129,7 @@ func TestBanAsksByJailAndAnswersTheBanAsHeldRefusingANonAddressFirst(t *testing.
if _, err := f.Ban(ctx, "198.51.100.7", "a jail; rm"); err == nil || !strings.Contains(err.Error(), "is not a jail's name") {
t.Errorf("a non-name: %v", err)
}
if len(calls) != 2 {
if len(calls) != 3 {
t.Errorf("a refused ban reached the daemon: %v", calls)
}
}
@@ -189,7 +190,7 @@ func TestTheSeatsVerbsAndTheModulesOwnToolAreServed(t *testing.T) {
names = append(names, tool.Name)
}
want := []string{"node-intrusion-prevention.status", "node-intrusion-prevention.banned", "node-intrusion-prevention.ban",
"node-intrusion-prevention.unban", "fail2ban_settings"}
"node-intrusion-prevention.unban", "fail2ban_settings", "fail2ban_health"}
if !reflect.DeepEqual(names, want) {
t.Errorf("%v", names)
}
@@ -5,6 +5,10 @@
// by the mesh from the modules a machine runs and written as declared resources; these touch only
// what the running daemon holds.
//
// Started as `fail2ban-tools watch` it is the module's long-running process, as root (watch.go, novox/hq
// ADR 0339): it writes the operator's own addresses into the list every jail ignores, checks that every jail
// names that list, and keeps every failure a jail ignored from one of them, for the health tool to say.
//
// stdout is the MCP channel; everything this module says, it says on stderr.
package main
@@ -12,7 +16,10 @@ import (
"context"
"fmt"
"os"
"os/signal"
"strings"
"syscall"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
)
@@ -21,6 +28,20 @@ import (
const Seat = "node-intrusion-prevention"
func main() {
if len(os.Args) > 1 {
if os.Args[1] != "watch" || len(os.Args) != 2 {
fmt.Fprintln(os.Stderr, "usage: fail2ban-tools [watch]")
os.Exit(2)
}
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, os.Interrupt)
defer stop()
w := &Watcher{F: Fail2ban{Run: execRunner}, Now: time.Now}
if err := w.Watch(ctx); err != nil {
fmt.Fprintf(os.Stderr, "[fail2ban] %v\n", err)
os.Exit(1)
}
return
}
if err := stdio.Serve("", tools(Fail2ban{Run: execRunner})); err != nil {
fmt.Fprintf(os.Stderr, "[fail2ban] %v\n", err)
os.Exit(1)
@@ -60,5 +81,9 @@ func tools(f Fail2ban) []stdio.Tool {
Description: "One jail's effective settings on this machine: ban time, window, tries, the addresses it never bans, its actions and what it reads.",
Input: map[string]any{"jail": str("the jail")},
Run: func(a map[string]any) (any, error) { return f.Settings(ctx, arg(a, "jail")) }},
{Name: "fail2ban_health",
Description: "Whether every jail on this machine ignores the operator's own addresses (the mesh-wide setting own-addresses, novox/hq ADR 0339) and none of them failed in the last day: each jail and address with the failures it ignored, how many and when the last was, and what the module's watch last saw. Not healthy while one did, so the mesh raises it as a warning; it clears a day after the last.",
Input: map[string]any{},
Run: func(map[string]any) (any, error) { return ReadHealth(f.Root, time.Now()), nil }},
}
}
+186
View File
@@ -0,0 +1,186 @@
package main
// The operator's own addresses (novox/hq ADR 0339, issue 238): the public addresses and ranges the operator
// names as theirs, in the mesh-wide setting `own-addresses`. Every jail ignores them beside the private ranges
// ADR 0186 spares, the ban verb refuses them, and a failure a jail counted from one is said as a warning.
//
// **The setting reaches the machine as a mergeable file, so it may be unset.** A `${setting:own-addresses}`
// would leave the module out of every machine until the operator gave a value (ADR 0262: an operator's value
// has no default), and an empty list is what the record says the mesh starts from. The mesh writes the
// setting into SettingFile, whose own content is the empty list; the watch, as root, writes the entries
// into ListFile, one per line, and every jail reads that through `ignoreip = … file:<ListFile>`, which fail2ban
// (1.1 and later) reloads within a second of the file changing — no restart, no ban lost. A failure ignored
// because of it is logged as `Ignore <address> by file:<ListFile>`, which is how the watch tells the
// operator's own addresses from the private ranges, which are logged `by ip`.
import (
"encoding/json"
"errors"
"fmt"
"io/fs"
"net/netip"
"os"
"path/filepath"
"sort"
"strings"
)
// The files, as the machine has them.
const (
// SettingFile is the setting as the mesh writes it: {"own-addresses": […]} or a string of entries.
SettingFile = "/etc/fail2ban/mesh-own-addresses.json"
// StateDir is the module's own directory, where the watch writes.
StateDir = "/var/lib/mesh-fail2ban"
// ListFile is the entries one per line, as fail2ban reads them.
ListFile = StateDir + "/own-addresses"
// StateFile is what the watch saw, for the health tool to read.
StateFile = StateDir + "/watch.json"
// LogFile is the daemon's log, where it says what it ignored.
LogFile = "/var/log/fail2ban.log"
// SettingKey is the setting's name (ADR 0339 §1).
SettingKey = "own-addresses"
// IgnoredBy is how every jail's ignore list names the operator's own addresses, and how the daemon's log
// names them as the reason it ignored a failure.
IgnoredBy = "file:" + ListFile
)
// Own is the setting read: its entries, and each one that is not an address or a range, with why.
type Own struct {
Entries []netip.Prefix `json:"-"`
Listed []string `json:"entries"`
Problems []string `json:"problems,omitempty"`
}
// ReadOwn reads the setting from SettingFile under root. A file not written yet is no entry: the mesh has
// not written the setting there, and the record starts from none. Any other failure to read it is an error,
// so a caller that refuses on it fails closed.
func ReadOwn(root string) (Own, error) {
raw, err := os.ReadFile(filepath.Join(root, SettingFile))
if errors.Is(err, fs.ErrNotExist) {
return Own{Listed: []string{}}, nil
}
if err != nil {
return Own{}, fmt.Errorf("the operator's own addresses cannot be read from %s: %w", SettingFile, err)
}
var doc map[string]any
if err := json.Unmarshal(raw, &doc); err != nil {
return Own{}, fmt.Errorf("the operator's own addresses in %s are not JSON: %w", SettingFile, err)
}
return ParseOwn(doc[SettingKey])
}
// ParseOwn reads the setting's value: a list of entries, or one string of entries separated by spaces or
// commas, as `settings set` may give either. Each entry is an address or a range in either family; anything
// else is a problem, named, and left out — a typo never widens what is spared.
func ParseOwn(value any) (Own, error) {
var words []string
switch v := value.(type) {
case nil:
case string:
words = strings.FieldsFunc(v, func(r rune) bool { return r == ',' || r == ' ' || r == '\t' || r == '\n' })
case []any:
for _, item := range v {
s, ok := item.(string)
if !ok {
return Own{}, fmt.Errorf("%s holds %v, which is not text: each entry is an address or a range", SettingKey, item)
}
words = append(words, strings.TrimSpace(s))
}
default:
return Own{}, fmt.Errorf("%s is %T: it is a list of addresses and ranges, or one line of them", SettingKey, value)
}
own := Own{Listed: []string{}}
seen := map[netip.Prefix]bool{}
for _, w := range words {
if w == "" {
continue
}
p, err := entry(w)
if err != nil {
own.Problems = append(own.Problems, err.Error())
continue
}
if seen[p] {
continue
}
seen[p] = true
own.Entries = append(own.Entries, p)
}
sort.Slice(own.Entries, func(i, j int) bool { return own.Entries[i].String() < own.Entries[j].String() })
for _, p := range own.Entries {
own.Listed = append(own.Listed, show(p))
}
return own, nil
}
// entry is one address or range; an address is the range of itself.
func entry(word string) (netip.Prefix, error) {
if strings.Contains(word, "/") {
p, err := netip.ParsePrefix(word)
if err != nil {
return netip.Prefix{}, fmt.Errorf("%q is not an address or a range", word)
}
return p.Masked(), nil
}
a, err := netip.ParseAddr(word)
if err != nil {
return netip.Prefix{}, fmt.Errorf("%q is not an address or a range", word)
}
a = a.Unmap()
return netip.PrefixFrom(a, a.BitLen()), nil
}
// show is an entry as a person reads it: a single address without its length.
func show(p netip.Prefix) string {
if p.IsSingleIP() {
return p.Addr().String()
}
return p.String()
}
// Lies is the entry ip lies on, or "".
func (o Own) Lies(ip netip.Addr) string {
return liesOn(ip, o.Entries)
}
func liesOn(ip netip.Addr, entries []netip.Prefix) string {
ip = ip.Unmap()
for _, p := range entries {
if p.Contains(ip) {
return show(p)
}
}
return ""
}
// ListText is the entries as fail2ban reads ListFile: one per line, under a line saying whose it is.
func (o Own) ListText() string {
var b strings.Builder
b.WriteString("# The operator's own addresses (novox/hq ADR 0339), written by the fail2ban module's watch from the\n")
b.WriteString("# mesh-wide setting own-addresses. Every jail ignores them. Replaced on every change; edit nothing here.\n")
for _, l := range o.Listed {
b.WriteString(l + "\n")
}
return b.String()
}
// WriteList writes ListFile under root when its content differs, by a rename so the daemon never reads half
// of it. Says whether it wrote.
func WriteList(root string, o Own) (bool, error) {
path := filepath.Join(root, ListFile)
want := o.ListText()
if have, err := os.ReadFile(path); err == nil && string(have) == want {
return false, nil
}
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
return false, err
}
tmp := path + ".new"
if err := os.WriteFile(tmp, []byte(want), 0o644); err != nil {
return false, err
}
if err := os.Rename(tmp, path); err != nil {
return false, err
}
return true, nil
}
@@ -0,0 +1,305 @@
package main
// The operator's own addresses (novox/hq ADR 0339, issue 238): the setting read, every jail's ignore list
// naming it, the ban verb refusing them, the watch keeping what a jail ignored, and the health saying it.
import (
"context"
"encoding/json"
"net/netip"
"os"
"path/filepath"
"reflect"
"strings"
"testing"
"time"
)
// ignoring is a jail's ignore list as fail2ban-client 1.1 prints it, with the base the module writes.
const ignoring = "These IP addresses/networks are ignored:\n|- 127.0.0.0/8\n|- 10.10.0.0/24\n|- 10.0.0.0/8\n" +
"|- 172.16.0.0/12\n|- 192.168.0.0/16\n|- 169.254.0.0/16\n|- fc00::/7\n|- fe80::/10\n" +
"|- file:/var/lib/mesh-fail2ban/own-addresses\n`- ::1\n"
// home is the operator's home uplink in these tests (RFC 5737).
const home = "203.0.113.5"
func writeSetting(t *testing.T, root, content string) {
t.Helper()
path := filepath.Join(root, SettingFile)
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
t.Fatal(err)
}
}
func TestTheSettingIsAListOrALineAndAnEntryThatIsNoAddressIsLeftOutAndSaid(t *testing.T) {
list, err := ParseOwn([]any{"203.0.113.5", "198.51.100.0/24", "2001:db8::/32", "203.0.113.5", "not-an-address"})
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(list.Listed, []string{"198.51.100.0/24", "2001:db8::/32", "203.0.113.5"}) {
t.Errorf("listed %v", list.Listed)
}
if len(list.Problems) != 1 || !strings.Contains(list.Problems[0], "not-an-address") {
t.Errorf("problems %v", list.Problems)
}
line, err := ParseOwn("203.0.113.5, 198.51.100.7")
if err != nil || !reflect.DeepEqual(line.Listed, []string{"198.51.100.7", "203.0.113.5"}) {
t.Errorf("a line: %v %v", line.Listed, err)
}
if on := list.Lies(netip.MustParseAddr("198.51.100.77")); on != "198.51.100.0/24" {
t.Errorf("an address in a range lies on %q", on)
}
if on := list.Lies(netip.MustParseAddr("192.0.2.1")); on != "" {
t.Errorf("a stranger lies on %q", on)
}
if _, err := ParseOwn(3.0); err == nil {
t.Error("a number was taken as the setting")
}
}
func TestTheSettingNotYetWrittenIsNoEntryAndAnUnreadableOneIsAnError(t *testing.T) {
root := t.TempDir()
own, err := ReadOwn(root)
if err != nil || len(own.Entries) != 0 {
t.Fatalf("no file: %+v %v", own, err)
}
writeSetting(t, root, "{\n \"own-addresses\": []\n}\n")
if own, err := ReadOwn(root); err != nil || len(own.Entries) != 0 {
t.Fatalf("the module's own content: %+v %v", own, err)
}
writeSetting(t, root, "{not json")
if _, err := ReadOwn(root); err == nil {
t.Fatal("a broken file read as no entry")
}
}
// The verb of 2026-10-04's remedy, used the other way: fail2ban's banip bans what it is given, ignore list or not.
func TestBanRefusesTheOperatorsOwnAddressAndWhatTheJailSparesBeforeTheDaemonIsAsked(t *testing.T) {
root := t.TempDir()
writeSetting(t, root, `{"own-addresses": ["203.0.113.5", "198.51.100.0/24"]}`)
var calls [][]string
f := Fail2ban{Root: root, Run: fake(map[string]string{"get gitea ignoreip": ignoring}, &calls)}
for ip, on := range map[string]string{home: home, "198.51.100.9": "198.51.100.0/24", "192.168.1.1": "192.168.0.0/16",
"10.10.0.4": "10.10.0.0/24"} {
_, err := f.Ban(ctx, ip, "gitea")
if err == nil || !strings.Contains(err.Error(), "refused") || !strings.Contains(err.Error(), "lies on "+on) {
t.Errorf("%s: %v", ip, err)
}
}
for _, c := range calls {
if len(c) > 2 && c[1] == "set" {
t.Fatalf("a refused ban reached the daemon: %v", calls)
}
}
if _, err := f.Ban(ctx, home, "gitea"); err == nil || !strings.Contains(err.Error(), "own-addresses") {
t.Errorf("the refusal does not name the setting: %v", err)
}
writeSetting(t, root, "{broken")
if _, err := f.Ban(ctx, "192.0.2.1", "gitea"); err == nil || !strings.Contains(err.Error(), "cannot be told") {
t.Errorf("an unreadable setting let a ban through: %v", err)
}
}
func TestTheWatchWritesTheListEveryJailIgnoresAndRewritesItOnlyOnAChange(t *testing.T) {
root := t.TempDir()
own, _ := ParseOwn([]any{home})
wrote, err := WriteList(root, own)
if err != nil || !wrote {
t.Fatalf("%v %v", wrote, err)
}
raw, _ := os.ReadFile(filepath.Join(root, ListFile))
var entries []string
for _, l := range strings.Split(string(raw), "\n") {
if l != "" && !strings.HasPrefix(l, "#") {
entries = append(entries, l)
}
}
if !reflect.DeepEqual(entries, []string{home}) {
t.Errorf("the list fail2ban reads: %q", raw)
}
if wrote, _ := WriteList(root, own); wrote {
t.Error("the same list was written again")
}
none, _ := ParseOwn(nil)
if wrote, _ := WriteList(root, none); !wrote {
t.Error("an emptied setting did not empty the list")
}
}
// logAt is a line as fail2ban 1.1 logs a failure it ignored.
func logAt(at time.Time, jail, ip, by string) string {
return at.Format("2006-01-02 15:04:05") + ",123 fail2ban.filter [1234]: INFO [" + jail + "] Ignore " + ip + " by " + by + "\n"
}
func appendLog(t *testing.T, root, text string) {
t.Helper()
path := filepath.Join(root, LogFile)
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
t.Fatal(err)
}
f, err := os.OpenFile(path, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o640)
if err != nil {
t.Fatal(err)
}
defer f.Close()
if _, err := f.WriteString(text); err != nil {
t.Fatal(err)
}
}
func daemon(missing bool) Runner {
gitea := ignoring
if missing {
gitea = strings.Replace(ignoring, "|- file:/var/lib/mesh-fail2ban/own-addresses\n", "", 1)
}
return fake(map[string]string{"status": "Status\n|- Number of jail:\t2\n`- Jail list:\tgitea, recidive\n",
"get gitea ignoreip": gitea, "get recidive ignoreip": ignoring}, nil)
}
// The 2026-10-04 loop as it now runs: forty failures from the operator's home, ignored and never banned, and
// said by the health as one jail, one address and a count; a private address ignored is not the operator's.
func TestAFailureIgnoredFromTheOperatorsOwnAddressIsSaidUntilADayAfterTheLast(t *testing.T) {
root := t.TempDir()
writeSetting(t, root, `{"own-addresses": ["203.0.113.5"]}`)
now := time.Date(2026, 10, 13, 10, 0, 0, 0, time.Local)
clock := now
w := &Watcher{F: Fail2ban{Run: daemon(false)}, Root: root, Now: func() time.Time { return clock }}
appendLog(t, root, logAt(now.Add(-48*time.Hour), "gitea", home, IgnoredBy)+
logAt(now.Add(-time.Minute), "sshd", "192.168.1.1", "ip"))
s := w.Look(context.Background())
if err := WriteState(root, s); err != nil {
t.Fatal(err)
}
if h := ReadHealth(root, now); !h.Healthy || !strings.Contains(h.Why, "every jail (2)") {
t.Fatalf("nothing of the operator's in the last day, yet: %+v", h)
}
if !reflect.DeepEqual(s.Own, []string{home}) {
t.Errorf("own %v", s.Own)
}
var loop strings.Builder
for i := 0; i < 40; i++ {
loop.WriteString(logAt(now.Add(time.Duration(i)*15*time.Second), "gitea", home, IgnoredBy))
}
appendLog(t, root, loop.String()[:loop.Len()-10]) // the daemon is still writing the last line
clock = now.Add(10 * time.Minute)
s = w.Look(context.Background())
if len(s.Ignored) != 1 || s.Ignored[0].Count != 39 {
t.Fatalf("a line still being written was counted, or the loop was not: %+v", s.Ignored)
}
appendLog(t, root, loop.String()[loop.Len()-10:])
s = w.Look(context.Background())
want := IgnoredFailures{Jail: "gitea", Address: home, Count: 40, First: now, Last: now.Add(39 * 15 * time.Second)}
if len(s.Ignored) != 1 || s.Ignored[0] != want {
t.Fatalf("ignored %+v", s.Ignored)
}
if err := WriteState(root, s); err != nil {
t.Fatal(err)
}
h := ReadHealth(root, clock)
if h.Healthy || !strings.Contains(h.Why, "the jail gitea counted 40 failure(s) from 203.0.113.5") ||
!strings.Contains(h.Why, "banned none") {
t.Fatalf("the loop was not said: %+v", h)
}
// A watch started again reads the log whole and still knows the last day.
again := &Watcher{F: Fail2ban{Run: daemon(false)}, Root: root, Now: func() time.Time { return clock }}
if s := again.Look(context.Background()); len(s.Ignored) != 1 || s.Ignored[0].Count != 40 {
t.Errorf("after a start: %+v", s.Ignored)
}
// A day after the last, it clears.
clock = now.Add(39*15*time.Second + Remembered + time.Second)
s = w.Look(context.Background())
if err := WriteState(root, s); err != nil {
t.Fatal(err)
}
if h := ReadHealth(root, clock); !h.Healthy {
t.Errorf("a day after the last it still says: %+v", h)
}
}
func TestAJailThatLeavesTheListOutABadEntryAndAStoppedWatchAreSaid(t *testing.T) {
root := t.TempDir()
writeSetting(t, root, `{"own-addresses": "203.0.113.5 nonsense"}`)
now := time.Date(2026, 10, 13, 10, 0, 0, 0, time.Local)
appendLog(t, root, "")
w := &Watcher{F: Fail2ban{Run: daemon(true)}, Root: root, Now: func() time.Time { return now }}
if err := WriteState(root, w.Look(context.Background())); err != nil {
t.Fatal(err)
}
h := ReadHealth(root, now)
if h.Healthy || !strings.Contains(h.Why, "the jail(s) gitea do not ignore") || !strings.Contains(h.Why, `"nonsense"`) {
t.Errorf("%+v", h)
}
if h := ReadHealth(root, now.Add(StaleAfter+time.Second)); h.Healthy || !strings.Contains(h.Why, "not looking") {
t.Errorf("a stopped watch: %+v", h)
}
if h := ReadHealth(t.TempDir(), now); h.Healthy {
t.Errorf("no watch at all: %+v", h)
}
}
// The manifest: every jail's base ignore list names the list, the setting reaches the machine without a
// default and only from the controller's terminal, and the watch is judged by the health tool.
func TestTheManifestGivesEveryJailTheListAndTheWatchItsHealth(t *testing.T) {
raw, err := os.ReadFile("../../module.json")
if err != nil {
t.Fatal(err)
}
var m struct {
Tools []string `json:"tools"`
Resources []map[string]any `json:"resources"`
}
if err := json.Unmarshal(raw, &m); err != nil {
t.Fatal(err)
}
byID := map[string]map[string]any{}
for _, r := range m.Resources {
byID[r["id"].(string)] = r
}
base, _ := byID["jail-local"]["content"].(string)
var ignoreip []string
for _, l := range strings.Split(base, "\n") {
if strings.HasPrefix(l, "ignoreip") {
ignoreip = append(ignoreip, l)
}
}
if len(ignoreip) != 1 || !strings.HasSuffix(ignoreip[0], " "+IgnoredBy) {
t.Errorf("the base ignore list: %q", ignoreip)
}
for id, r := range byID {
if content, _ := r["content"].(string); id != "jail-local" && strings.Contains(content, "ignoreip") {
t.Errorf("%s sets an ignore list of its own, which leaves the operator's own addresses out", id)
}
}
setting := byID["own-addresses"]
if setting["path"] != SettingFile || setting["merge"] != "json" || setting["trusted"] != true ||
!reflect.DeepEqual(setting["takes-settings"], []any{SettingKey}) {
t.Errorf("the setting's file: %v", setting)
}
var content map[string]any
if err := json.Unmarshal([]byte(setting["content"].(string)), &content); err != nil ||
!reflect.DeepEqual(content, map[string]any{SettingKey: []any{}}) {
t.Errorf("the setting starts from %v (%v)", content, err)
}
watch := byID["watch"]
health, _ := watch["health"].(map[string]any)
if watch["type"] != "process" || !reflect.DeepEqual(watch["run"], []any{"./fail2ban-tools", "watch"}) ||
health["kind"] != "tool" || health["tool"] != "fail2ban_health" || watch["user"] != nil {
t.Errorf("the watch: %v", watch)
}
if list := byID["own-addresses-list"]; list["path"] != ListFile || list["create-once"] != true {
t.Errorf("the list is there, empty, before the daemon first reads it: %v", list)
}
if byID["state-dir"]["path"] != StateDir {
t.Errorf("the watch's directory: %v", byID["state-dir"])
}
if !contains(m.Tools, "fail2ban_health") {
t.Errorf("tools %v", m.Tools)
}
}
@@ -0,0 +1,297 @@
package main
// The watch (novox/hq ADR 0339 §2 and §4, issue 238): the module's long-running process, as root. Every look it
//
// - writes the operator's own addresses from the setting into the list every jail ignores (own.go);
// - asks the daemon whether every jail's ignore list names that list, so no jail leaves it out;
// - reads what the daemon logged since the last look, and keeps every failure a jail ignored because the
// address is one of the operator's own, for a day.
//
// What it saw goes to StateFile, which the health tool reads as the operator's account. A failure from one
// of the operator's own addresses is never banned — that is the point — and never silent: the module's health
// says it, so the controller raises it as a warning naming the node, the jail, the address and the count, and
// it clears a day after the last one. A loop like the one of 2026-10-04 is then seen within minutes.
import (
"bufio"
"context"
"encoding/json"
"fmt"
"io"
"os"
"path/filepath"
"regexp"
"sort"
"strings"
"time"
)
// The watch's timing.
var (
// LookEvery is how often the watch looks.
LookEvery = 30 * time.Second
// Remembered is how long an ignored failure is said: a day, as the record has it.
Remembered = 24 * time.Hour
// StaleAfter is how old the watch's last look may be before the health says it stopped looking.
StaleAfter = 3 * time.Minute
// keptPerAddress bounds what is remembered of one address in one jail; the count says when it met it.
keptPerAddress = 10000
)
// IgnoredFailures is the failures one jail ignored from one of the operator's own addresses, in the last day.
type IgnoredFailures struct {
Jail string `json:"jail"`
Address string `json:"address"`
Count int `json:"count"`
First time.Time `json:"first"`
Last time.Time `json:"last"`
// AtLeast says the count met what is remembered of one address and is a floor.
AtLeast bool `json:"at-least,omitempty"`
}
// State is what the watch saw at its last look.
type State struct {
Looked time.Time `json:"looked"`
// Own is the operator's own addresses as written into the list, and Problems the entries left out.
Own []string `json:"own"`
Problems []string `json:"problems,omitempty"`
// Unread is why something the watch reads could not be read at this look.
Unread []string `json:"unread,omitempty"`
// Jails is how many jails the daemon runs; Missing those whose ignore list does not name the list.
Jails int `json:"jails"`
Missing []string `json:"missing,omitempty"`
// Ignored is every jail and address of the operator's own with a failure ignored in the last day.
Ignored []IgnoredFailures `json:"ignored"`
}
// ignoredLine is the daemon's line for a failure it ignored: `<time>,<ms> fail2ban.filter [<pid>]: INFO
// [<jail>] Ignore <address> by <source>` (fail2ban 1.1, server/filter.py logIgnoreIp).
var ignoredLine = regexp.MustCompile(`^(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}),\d+ .*?\]:\s+\S+\s+\[([^\]]+)\] Ignore (\S+) by (\S+)\s*$`)
// Watcher is the watch's memory between looks: where it is in the log and what it saw there.
type Watcher struct {
F Fail2ban
Root string
Now func() time.Time
logFile os.FileInfo
offset int64
partial string
seen map[[2]string][]time.Time
}
// Look is one look: the list written, the jails asked, the log read, the state answered.
func (w *Watcher) Look(ctx context.Context) State {
now := w.Now()
s := State{Looked: now, Own: []string{}, Ignored: []IgnoredFailures{}}
own, err := ReadOwn(w.Root)
if err != nil {
s.Unread = append(s.Unread, err.Error())
} else {
s.Own, s.Problems = own.Listed, own.Problems
if _, err := WriteList(w.Root, own); err != nil {
s.Unread = append(s.Unread, fmt.Sprintf("the list every jail ignores (%s) cannot be written: %v", ListFile, err))
}
}
if jails, err := w.F.Jails(ctx); err != nil {
s.Unread = append(s.Unread, fmt.Sprintf("the daemon cannot be asked for its jails: %v", err))
} else {
s.Jails = len(jails)
for _, j := range jails {
out, err := w.F.client(ctx, "get", j, "ignoreip")
if err != nil {
s.Unread = append(s.Unread, fmt.Sprintf("the jail %s's ignore list cannot be read: %v", j, err))
continue
}
if !contains(listed(out), IgnoredBy) {
s.Missing = append(s.Missing, j)
}
}
}
if err := w.readLog(now); err != nil {
s.Unread = append(s.Unread, err.Error())
}
s.Ignored = w.ignored(now)
return s
}
func contains(list []string, want string) bool {
for _, l := range list {
if l == want {
return true
}
}
return false
}
// readLog reads what the daemon logged since the last look. A log rotated away is read from its start; the
// first look reads the whole of it, so a watch started again still knows the last day.
func (w *Watcher) readLog(now time.Time) error {
path := filepath.Join(w.Root, LogFile)
f, err := os.Open(path)
if err != nil {
return fmt.Errorf("the daemon's log %s cannot be read: %w", LogFile, err)
}
defer f.Close()
info, err := f.Stat()
if err != nil {
return fmt.Errorf("the daemon's log %s cannot be read: %w", LogFile, err)
}
if w.logFile == nil || !os.SameFile(w.logFile, info) || info.Size() < w.offset {
w.offset, w.partial = 0, ""
}
w.logFile = info
if _, err := f.Seek(w.offset, io.SeekStart); err != nil {
return fmt.Errorf("the daemon's log %s cannot be read: %w", LogFile, err)
}
r := bufio.NewReader(f)
for {
line, err := r.ReadString('\n')
w.offset += int64(len(line))
if err != nil {
// A line the daemon is still writing is kept for the next look.
w.partial += line
break
}
w.take(w.partial+strings.TrimRight(line, "\n"), now)
w.partial = ""
}
return nil
}
// take keeps one line of the log when it is a failure ignored because of the operator's own addresses.
func (w *Watcher) take(line string, now time.Time) {
m := ignoredLine.FindStringSubmatch(line)
if m == nil || m[4] != IgnoredBy {
return
}
at, err := time.ParseInLocation("2006-01-02 15:04:05", m[1], time.Local)
if err != nil || now.Sub(at) > Remembered {
return
}
if w.seen == nil {
w.seen = map[[2]string][]time.Time{}
}
key := [2]string{m[2], m[3]}
times := append(w.seen[key], at)
if len(times) > keptPerAddress {
times = times[len(times)-keptPerAddress:]
}
w.seen[key] = times
}
// ignored is what was seen in the last day, by jail and address; anything older is forgotten.
func (w *Watcher) ignored(now time.Time) []IgnoredFailures {
out := []IgnoredFailures{}
for key, times := range w.seen {
kept := times[:0]
for _, at := range times {
if now.Sub(at) <= Remembered {
kept = append(kept, at)
}
}
if len(kept) == 0 {
delete(w.seen, key)
continue
}
w.seen[key] = kept
sort.Slice(kept, func(i, j int) bool { return kept[i].Before(kept[j]) })
out = append(out, IgnoredFailures{Jail: key[0], Address: key[1], Count: len(kept), First: kept[0],
Last: kept[len(kept)-1], AtLeast: len(kept) >= keptPerAddress})
}
sort.Slice(out, func(i, j int) bool {
if out[i].Jail != out[j].Jail {
return out[i].Jail < out[j].Jail
}
return out[i].Address < out[j].Address
})
return out
}
// WriteState writes the state for the health tool, readable by the operator's account, by a rename.
func WriteState(root string, s State) error {
raw, err := json.MarshalIndent(s, "", " ")
if err != nil {
return err
}
path := filepath.Join(root, StateFile)
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
return err
}
tmp := path + ".new"
if err := os.WriteFile(tmp, append(raw, '\n'), 0o644); err != nil {
return err
}
return os.Rename(tmp, path)
}
// Watch looks until it is stopped.
func (w *Watcher) Watch(ctx context.Context) error {
for {
if err := WriteState(w.Root, w.Look(ctx)); err != nil {
fmt.Fprintf(os.Stderr, "[fail2ban] what the watch saw cannot be written to %s: %v\n", StateFile, err)
}
select {
case <-ctx.Done():
return nil
case <-time.After(LookEvery):
}
}
}
// Health is the module's health as the node-engine reads it (novox/hq ADR 0240): healthy, and why.
type Health struct {
Healthy bool `json:"healthy"`
Why string `json:"why"`
State *State `json:"seen,omitempty"`
}
// ReadHealth judges what the watch last saw. Not healthy when the watch is not looking, when something it
// reads cannot be read, when the setting holds an entry that is not an address, when a jail's ignore list
// leaves the operator's own addresses out, and when a jail ignored a failure from one in the last day — each
// said, so the controller's warning names the jail, the address and the count (ADR 0339 §4).
func ReadHealth(root string, now time.Time) Health {
raw, err := os.ReadFile(filepath.Join(root, StateFile))
if err != nil {
return Health{Why: fmt.Sprintf("the watch has written nothing this tool can read (%v): it is not looking", err)}
}
var s State
if err := json.Unmarshal(raw, &s); err != nil {
return Health{Why: fmt.Sprintf("what the watch wrote is not its state: %v", err)}
}
if age := now.Sub(s.Looked); age > StaleAfter {
return Health{State: &s, Why: fmt.Sprintf("the watch last looked at %s, %s ago: it is not looking",
s.Looked.Local().Format("2006-01-02 15:04:05"), age.Round(time.Second))}
}
var wrong []string
wrong = append(wrong, s.Unread...)
if len(s.Problems) > 0 {
wrong = append(wrong, fmt.Sprintf("the setting %s holds what is not an address or a range, left out: %s",
SettingKey, strings.Join(s.Problems, ", ")))
}
if len(s.Missing) > 0 {
wrong = append(wrong, fmt.Sprintf("the jail(s) %s do not ignore the operator's own addresses: their ignore "+
"list lacks %s (novox/hq ADR 0339 §2)", strings.Join(s.Missing, ", "), IgnoredBy))
}
for _, i := range s.Ignored {
count := fmt.Sprint(i.Count)
if i.AtLeast {
count = "at least " + count
}
wrong = append(wrong, fmt.Sprintf("the jail %s counted %s failure(s) from %s, one of the operator's own "+
"addresses, in the last day, the last at %s, and banned none (novox/hq ADR 0339); it clears a day after "+
"the last", i.Jail, count, i.Address, i.Last.Local().Format("2006-01-02 15:04:05")))
}
if len(wrong) > 0 {
return Health{State: &s, Why: strings.Join(wrong, "; ")}
}
return Health{Healthy: true, State: &s, Why: fmt.Sprintf("every jail (%d) ignores the operator's own addresses "+
"(%d entr%s), and none failed in the last day", s.Jails, len(s.Own), plural(len(s.Own)))}
}
func plural(n int) string {
if n == 1 {
return "y"
}
return "ies"
}
+51 -3
View File
@@ -17,7 +17,8 @@
}
],
"tools": [
"fail2ban_settings"
"fail2ban_settings",
"fail2ban_health"
],
"jailing": {
"into": "/etc/fail2ban/jail.d/mesh.conf",
@@ -29,6 +30,20 @@
"type": "package",
"package": "fail2ban"
},
{
"id": "state-dir",
"type": "directory",
"path": "/var/lib/mesh-fail2ban",
"mode": "0755"
},
{
"id": "own-addresses-list",
"type": "file",
"path": "/var/lib/mesh-fail2ban/own-addresses",
"mode": "0644",
"create-once": true,
"content": ""
},
{
"id": "jail-d",
"type": "directory",
@@ -58,7 +73,7 @@
"type": "file",
"path": "/etc/fail2ban/jail.local",
"mode": "0644",
"content": "[INCLUDES]\n\nbefore = paths-arch.conf\n\n[DEFAULT]\n\n# Never act on the machine itself or on a tunnel peer: the mesh's private range is\n# ${machine:mesh-range}, named here rather than written as a value the module cannot\n# know (novox/hq ADR 0112). Without this, fail2ban could ban the mesh's own nodes.\n# **A ban list never holds a neighbour.** The mesh's own range is named rather than written\n# (novox/hq ADR 0112), and every private range beside it: a source on one is somebody's own\n# network, not the internet. On a machine behind a router that reflects local traffic, every\n# client in the house arrives as the gateway's address — so one mistyped local request banned\n# 192.168.1.1 on the home server and would have cut the whole house off from it (ADR 0186).\nignoreip = 127.0.0.1/8 ::1 ${machine:mesh-range} 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 169.254.0.0/16 fc00::/7 fe80::/10\n\n# Three failures in a day ban for a day (novox/hq ADR 0179). The attackers this mesh sees pace\n# themselves at one try every ten minutes, under any ten-minute window; a day's window counts\n# them, and a day's ban costs a person who mistyped three times once, from one address, while\n# the mesh's own range is never banned at all.\nbantime = 1d\nfindtime = 1d\nmaxretry = 3\n\n# Ban through iptables, not through a firewall front-end the machine may not have. ufw is\n# installed on two of this mesh's machines and absent on the other two, and fail2ban finds out\n# only at ban time: the service reports healthy, the jail counts the attempt, the ban command\n# exits 127, and nothing is blocked. Proven on 2026-09-28 -- 'ufw: command not found' on a\n# machine the mesh reported as protected.\n#\n# The action below is this module's own, already used by the recidive jail on every machine\n# here, and it bans in DOCKER-USER as well as INPUT, so a container's published port is\n# covered too.\nbanaction = iptables-allports-dualchain\nbanaction_allports = iptables-allports-dualchain\n\n[sshd]\nenabled = true\nport = ssh\nlogpath = %(sshd_log)s\nbackend = %(sshd_backend)s\n"
"content": "[INCLUDES]\n\nbefore = paths-arch.conf\n\n[DEFAULT]\n\n# Never act on the machine itself or on a tunnel peer: the mesh's private range is\n# ${machine:mesh-range}, named here rather than written as a value the module cannot\n# know (novox/hq ADR 0112). Without this, fail2ban could ban the mesh's own nodes.\n# **A ban list never holds a neighbour.** The mesh's own range is named rather than written\n# (novox/hq ADR 0112), and every private range beside it: a source on one is somebody's own\n# network, not the internet. On a machine behind a router that reflects local traffic, every\n# client in the house arrives as the gateway's address — so one mistyped local request banned\n# 192.168.1.1 on the home server and would have cut the whole house off from it (ADR 0186).\n#\n# **Nor the operator's own addresses** (novox/hq ADR 0339, issue 238): the public addresses the operator names\n# as theirs, in the mesh-wide setting own-addresses. An agent polling the forge from the operator's home banned\n# that home for four weeks on 2026-10-04, and nothing in the house reached the control-node until it was found\n# among 400 bans. The file is written by this module's watch from the setting, one entry per line; fail2ban\n# reads it again within a second of a change, so a new value needs no restart. Empty until the operator\n# fills the setting. A jail that sets its own ignoreip leaves all of this out; the watch says so.\nignoreip = 127.0.0.1/8 ::1 ${machine:mesh-range} 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 169.254.0.0/16 fc00::/7 fe80::/10 file:/var/lib/mesh-fail2ban/own-addresses\n\n# Three failures in a day ban for a day (novox/hq ADR 0179). The attackers this mesh sees pace\n# themselves at one try every ten minutes, under any ten-minute window; a day's window counts\n# them, and a day's ban costs a person who mistyped three times once, from one address, while\n# the mesh's own range is never banned at all.\nbantime = 1d\nfindtime = 1d\nmaxretry = 3\n\n# Ban through iptables, not through a firewall front-end the machine may not have. ufw is\n# installed on two of this mesh's machines and absent on the other two, and fail2ban finds out\n# only at ban time: the service reports healthy, the jail counts the attempt, the ban command\n# exits 127, and nothing is blocked. Proven on 2026-09-28 -- 'ufw: command not found' on a\n# machine the mesh reported as protected.\n#\n# The action below is this module's own, already used by the recidive jail on every machine\n# here, and it bans in DOCKER-USER as well as INPUT, so a container's published port is\n# covered too.\nbanaction = iptables-allports-dualchain\nbanaction_allports = iptables-allports-dualchain\n\n[sshd]\nenabled = true\nport = ssh\nlogpath = %(sshd_log)s\nbackend = %(sshd_backend)s\n"
},
{
"id": "jail-sshd",
@@ -67,6 +82,36 @@
"mode": "0644",
"content": "[sshd]\nenabled = true\nport = ssh\nlogpath = %(sshd_log)s\nbackend = %(sshd_backend)s\nmaxretry = 3\nfindtime = 1d\nbantime = 1d\n"
},
{
"id": "own-addresses",
"type": "file",
"path": "/etc/fail2ban/mesh-own-addresses.json",
"mode": "0644",
"merge": "json",
"takes-settings": [
"own-addresses"
],
"trusted": true,
"content": "{\n \"own-addresses\": []\n}\n"
},
{
"id": "watch",
"type": "process",
"name": "fail2ban-watch",
"artifact": "tools",
"run": [
"./fail2ban-tools",
"watch"
],
"health": {
"kind": "tool",
"tool": "fail2ban_health",
"interval": "60s",
"timeout": "10s",
"looks": 2,
"grace": "90s"
}
},
{
"id": "log",
"type": "file",
@@ -108,7 +153,10 @@
"jail-recidive",
"action-dualchain",
"composed-jails"
]
],
"health": {
"kind": "unit"
}
}
],
"build": {
+15 -2
View File
@@ -138,6 +138,17 @@ function meshConfig(file?: string): Record<string, string> {
catch { return {}; }
}
/**
* A password, token or API key is the vault's, never a setting (novox/hq issue 342): a value for one of
* them in the settings file is ignored, and that is said once — by its key, never with its value.
*/
const saidIgnored = new Set<string>();
function ignoreSecretSetting(cfg: Record<string, unknown>, key: string): void {
if (cfg[key] === undefined || saidIgnored.has(key)) return;
saidIgnored.add(key);
console.error(`the setting ${key} is ignored: it comes from the vault`);
}
export class GiteaClient {
readonly baseUrl: string;
private readonly tokens: TokenSource;
@@ -151,7 +162,8 @@ export class GiteaClient {
/**
* Build from the module's resolved environment. The URL comes from MESH_GITEA_URL (the mesh's own
* name), falling back to the bare GITEA_URL and to the forge's loopback port. The token, in order:
* one configured in settings or the environment (MESH_GITEA_TOKEN / GITEA_TOKEN), which wins; else
* one configured in the environment (MESH_GITEA_TOKEN / GITEA_TOKEN), which wins — never a setting, which is
* ignored and said (hq issue 342); else
* one the module mints for itself with the admin account the vault delivered and keeps in its own
* state (token.ts; hq issue 100). Throws only when neither is possible, naming what is missing,
* rather than hand back a client that fails on first use.
@@ -159,7 +171,8 @@ export class GiteaClient {
static fromEnv(env: NodeJS.ProcessEnv = process.env): GiteaClient {
const cfg = meshConfig(env.MESH_GITEA_CONFIG_FILE);
const url = cfg.url ?? env.MESH_GITEA_URL ?? env.GITEA_URL ?? `http://127.0.0.1:${env.GITEA_PORT ?? "3000"}`;
const configured = cfg.token ?? env.MESH_GITEA_TOKEN ?? env.GITEA_TOKEN;
ignoreSecretSetting(cfg, "token");
const configured = env.MESH_GITEA_TOKEN ?? env.GITEA_TOKEN;
if (configured) return new GiteaClient(url, new ConfiguredToken(configured));
return new GiteaClient(url, MintedToken.fromEnv(url, env));
}
+41
View File
@@ -0,0 +1,41 @@
// A password, a token or an API key is the vault's, never a setting (novox/hq issue 342): a value for it in
// the settings file the mesh merges is ignored, the vault's is used, and that is said once, by the key alone.
// Run against the compiled module, the way the runtime loads it.
import { test, mock } from "node:test";
import assert from "node:assert/strict";
import { mkdtempSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { GiteaClient } from "../dist/client.js";
import { ConfiguredToken, MintedToken } from "../dist/token.js";
const SETTING = "one-a-setting-carried";
const VAULT = "the-vault-made-this";
function delivered(settings: Record<string, string>): Record<string, string> {
const dir = mkdtempSync(join(tmpdir(), "gitea-342-"));
writeFileSync(join(dir, "config.json"), JSON.stringify(settings));
writeFileSync(join(dir, "vault.secret"), VAULT + "\n");
return { dir, config: join(dir, "config.json"), vault: join(dir, "vault.secret") };
}
test("the vault's token is used, a setting of it is ignored and said once, without its value", () => {
const said: string[] = [];
const err = mock.method(console, "error", (...args: unknown[]) => { said.push(args.join(" ")); });
try {
const at = delivered({ token: SETTING });
const env = { MESH_GITEA_CONFIG_FILE: at.config, MESH_GITEA_URL: "http://127.0.0.1:1", MESH_GITEA_ADMIN_USER: "mesh-admin", MESH_GITEA_ADMIN_PASSWORD_FILE: at.vault, MESH_GITEA_STATE_DIR: at.dir };
const first = GiteaClient.fromEnv(env);
const second = GiteaClient.fromEnv(env);
// No token from the setting: the module mints its own with the vault's admin account.
assert.ok((first as any).tokens instanceof MintedToken);
assert.ok((second as any).tokens instanceof MintedToken);
assert.ok(!((first as any).tokens instanceof ConfiguredToken));
assert.deepEqual(said, ["the setting token is ignored: it comes from the vault"]);
assert.ok(!said.join("\n").includes(SETTING), "the setting's value is never said");
} finally {
err.mock.restore();
}
});
+20 -6
View File
@@ -44,6 +44,17 @@ function meshConfig(file?: string): Record<string, string> {
catch { return {}; }
}
/**
* A password, token or API key is the vault's, never a setting (novox/hq issue 342): a value for one of
* them in the settings file is ignored, and that is said once — by its key, never with its value.
*/
const saidIgnored = new Set<string>();
function ignoreSecretSetting(cfg: Record<string, unknown>, key: string): void {
if (cfg[key] === undefined || saidIgnored.has(key)) return;
saidIgnored.add(key);
console.error(`the setting ${key} is ignored: it comes from the vault`);
}
export class GrafanaClient {
readonly baseUrl: string;
private readonly authHeader: string;
@@ -54,21 +65,24 @@ export class GrafanaClient {
}
/**
* Build from the module's resolved environment. Auth is a service-account/API token
* (MESH_GRAFANA_TOKEN, sent as Bearer) when present, else HTTP basic with the admin password the
* module keeps as its own secret (MESH_GRAFANA_PASSWORD, user MESH_GRAFANA_USER, default admin).
* Throws when neither is configured — the module then contributes nothing rather than failing.
* Build from the module's resolved environment. Auth is HTTP basic with the vault's admin password,
* which the runtime config carries as `password` (kept protected, so no setting replaces it) or
* MESH_GRAFANA_PASSWORD (user MESH_GRAFANA_USER, default admin); else a service-account/API token from
* the environment (MESH_GRAFANA_TOKEN, sent as Bearer). A `token` in the settings file is never used: a
* credential is the vault's, never a setting (hq issue 342). Throws when neither is configured — the
* module then contributes nothing rather than failing.
*/
static fromEnv(env: NodeJS.ProcessEnv = process.env): GrafanaClient {
const cfg = meshConfig(env.MESH_GRAFANA_CONFIG_FILE);
const url = cfg.url ?? env.MESH_GRAFANA_URL ?? `http://127.0.0.1:${env.GRAFANA_PORT ?? "3000"}`;
const token = cfg.token ?? env.MESH_GRAFANA_TOKEN;
if (token) return new GrafanaClient(url, `Bearer ${token}`);
ignoreSecretSetting(cfg, "token");
const password = cfg.password ?? env.MESH_GRAFANA_PASSWORD;
if (password) {
const user = cfg.user ?? env.MESH_GRAFANA_USER ?? "admin";
return new GrafanaClient(url, `Basic ${Buffer.from(`${user}:${password}`).toString("base64")}`);
}
const token = env.MESH_GRAFANA_TOKEN;
if (token) return new GrafanaClient(url, `Bearer ${token}`);
throw new Error("no Grafana auth — set MESH_GRAFANA_TOKEN or MESH_GRAFANA_PASSWORD");
}
+28 -8
View File
@@ -71,20 +71,36 @@
"mode": "0644",
"content": "GF_SERVER_ROOT_URL=https://${bound:route:name}\nGF_AUTH_GENERIC_OAUTH_ENABLED=true\nGF_AUTH_GENERIC_OAUTH_NAME=Keycloak\nGF_AUTH_GENERIC_OAUTH_CLIENT_ID=${bound:oidc-client:as}\nGF_AUTH_GENERIC_OAUTH_CLIENT_SECRET__FILE=/run/secrets/oidc-client\nGF_AUTH_GENERIC_OAUTH_SCOPES=openid email profile roles\nGF_AUTH_GENERIC_OAUTH_AUTH_URL=${bound:oidc-client:issuer}${bound:oidc-client:authorization-path}\nGF_AUTH_GENERIC_OAUTH_TOKEN_URL=${bound:oidc-client:issuer}${bound:oidc-client:token-path}\nGF_AUTH_GENERIC_OAUTH_API_URL=${bound:oidc-client:issuer}${bound:oidc-client:userinfo-path}\nGF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH=contains(roles[*], 'admin') && 'Admin' || contains(realm_access.roles[*], 'admin') && 'Admin' || 'Viewer'\nGF_AUTH_GENERIC_OAUTH_USE_PKCE=true\nGF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=true\nGF_AUTH_GENERIC_OAUTH_ALLOW_ASSIGN_GRAFANA_ADMIN=true\n"
},
{
"id": "influxdb-datasources",
"type": "directory",
"path": "${dir:state}/datasources",
"mode": "0755",
"owner": "472:472"
},
{
"id": "influxdb-credential",
"type": "directory",
"path": "${dir:state}/influxdb",
"mode": "0700",
"owner": "472:472"
},
{
"id": "influxdb-secret",
"type": "file",
"path": "${dir:state}/influxdb-api.secret",
"path": "${dir:influxdb-credential}/influxdb-api",
"mode": "0400",
"owner": "472:472",
"content": "${secret:influxdb-api}"
"content": "${secret:influxdb-api}",
"only-when-bound": "influxdb-api"
},
{
"id": "influxdb-datasource",
"type": "file",
"path": "${dir:state}/datasource-influxdb.yaml",
"path": "${dir:influxdb-datasources}/mesh-influxdb.yaml",
"mode": "0644",
"content": "apiVersion: 1\n# Written by the mesh from grafana's influxdb-api binding; grafana reads it at start. Its own name and\n# uid, so a data source somebody made in the UI is never overwritten, and read-only in the UI because\n# the mesh resets it. The password is read from the file the mesh delivers, never written here.\ndatasources:\n - name: InfluxDB (mesh)\n uid: mesh-influxdb-api\n type: influxdb\n access: proxy\n url: ${bound:influxdb-api:scheme}://${bound:influxdb-api:at}:${bound:influxdb-api:port}\n user: ${bound:influxdb-api:as}\n isDefault: false\n editable: false\n jsonData:\n dbName: ${bound:influxdb-api:bucket}\n httpMode: POST\n secureJsonData:\n password: $__file{/run/secrets/influxdb-api}\n"
"content": "apiVersion: 1\n# Written by the mesh from grafana's influxdb-api binding, only while it is bound (novox/hq ADR 0330):\n# without influxdb, grafana runs with no mesh data source. grafana reads it at start. Its own name and\n# uid, so a data source somebody made in the UI is never overwritten, and read-only in the UI because\n# the mesh resets it. The password is read from the file the mesh delivers, never written here.\ndatasources:\n - name: InfluxDB (mesh)\n uid: mesh-influxdb-api\n type: influxdb\n access: proxy\n url: ${bound:influxdb-api:scheme}://${bound:influxdb-api:at}:${bound:influxdb-api:port}\n user: ${bound:influxdb-api:as}\n isDefault: false\n editable: false\n jsonData:\n dbName: ${bound:influxdb-api:bucket}\n httpMode: POST\n secureJsonData:\n password: $__file{/run/secrets/influxdb/influxdb-api}\n",
"only-when-bound": "influxdb-api"
},
{
"id": "server",
@@ -103,8 +119,8 @@
"${dir:data}:/var/lib/grafana",
"${dir:state}/admin.secret:/run/secrets/admin:ro",
"${dir:state}/oidc-client.secret:/run/secrets/oidc-client:ro",
"${dir:state}/influxdb-api.secret:/run/secrets/influxdb-api:ro",
"${dir:state}/datasource-influxdb.yaml:/etc/grafana/provisioning/datasources/mesh-influxdb.yaml:ro"
"${dir:influxdb-credential}:/run/secrets/influxdb:ro",
"${dir:influxdb-datasources}:/etc/grafana/provisioning/datasources:ro"
],
"env": {
"GF_SECURITY_ADMIN_PASSWORD__FILE": "/run/secrets/admin"
@@ -125,13 +141,17 @@
"path": "${dir:mesh-state}/config.json",
"mode": "0600",
"content": "{\n \"user\": \"admin\",\n \"password\": \"${secret:admin}\"\n}\n",
"merge": "json"
"merge": "json",
"protected": ["password", "token"]
}
],
"requires": [
"route",
"oidc-client",
"influxdb-api"
{
"name": "influxdb-api",
"optional": true
}
],
"contributes": {
"route": {
+39
View File
@@ -0,0 +1,39 @@
// grafana runs without a data source, so its influxdb-api is optional (novox/hq ADR 0330, issue 166):
// a mesh without influxdb can assign grafana, and influxdb is never installed for it. While absent, the
// data source and its credential are not delivered, and grafana mounts their directories empty.
import { test } from "node:test";
import assert from "node:assert/strict";
import { readFileSync } from "node:fs";
type Requirement = string | { name: string; optional?: boolean };
type Resource = { id: string; type: string; path?: string; content?: string; volumes?: string[]; "only-when-bound"?: string };
const manifest = JSON.parse(readFileSync(new URL("../module.json", import.meta.url), "utf8")) as {
requires: Requirement[];
resources: Resource[];
};
const optional = (name: string) =>
manifest.requires.some((r) => typeof r !== "string" && r.name === name && r.optional === true);
test("influxdb-api is optional, and route and oidc-client stay hard", () => {
assert.ok(optional("influxdb-api"));
assert.ok(manifest.requires.includes("route"));
assert.ok(manifest.requires.includes("oidc-client"));
});
test("every file naming influxdb-api is delivered only while it is bound", () => {
for (const r of manifest.resources) {
if (r.type === "file" && /\$\{(bound:influxdb-api:|secret:influxdb-api\})/.test(r.content ?? "")) {
assert.equal(r["only-when-bound"], "influxdb-api", `${r.id} names influxdb-api and is delivered while it is absent`);
}
}
});
test("the container mounts the data source's directories, never a file that may be absent", () => {
// A bind mount of a missing file makes a directory in its place, which the file could then never be written over.
const server = manifest.resources.find((r) => r.id === "server");
for (const v of server?.volumes ?? []) {
assert.ok(!/influxdb.*\.(yaml|secret)|influxdb-api:/.test(v.split(":")[0]), `${v} mounts an only-when-bound file`);
}
});
@@ -0,0 +1,40 @@
// A password, a token or an API key is the vault's, never a setting (novox/hq issue 342): grafana logs in
// with the vault's admin password, a `token` in the settings file is ignored, and that is said once, by the
// key alone. The runtime config keeps both keys protected, so the controller refuses a layer setting either.
import { test, mock } from "node:test";
import assert from "node:assert/strict";
import { mkdtempSync, readFileSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { GrafanaClient } from "../dist/client.js";
const SETTING = "one-a-setting-carried";
const VAULT = "the-vault-made-this";
test("the vault's admin password is used, a token setting is ignored and said once, without its value", () => {
const said: string[] = [];
const err = mock.method(console, "error", (...args: unknown[]) => { said.push(args.join(" ")); });
try {
const dir = mkdtempSync(join(tmpdir(), "grafana-342-"));
// What the runtime config holds once the vault filled it, with a token a layer carried beside it.
writeFileSync(join(dir, "config.json"), JSON.stringify({ user: "admin", password: VAULT, token: SETTING }));
const env = { MESH_GRAFANA_CONFIG_FILE: join(dir, "config.json") };
const basic = "Basic " + Buffer.from(`admin:${VAULT}`).toString("base64");
assert.equal((GrafanaClient.fromEnv(env) as any).authHeader, basic);
assert.equal((GrafanaClient.fromEnv(env) as any).authHeader, basic);
assert.deepEqual(said, ["the setting token is ignored: it comes from the vault"]);
assert.ok(!said.join("\n").includes(SETTING), "the setting's value is never said");
} finally {
err.mock.restore();
}
});
test("the runtime config keeps password and token protected", () => {
const manifest = JSON.parse(readFileSync(new URL("../module.json", import.meta.url), "utf8")) as {
resources: { id: string; protected?: string[] }[];
};
const runtime = manifest.resources.find((r) => r.id === "runtime-config");
assert.deepEqual(runtime?.protected, ["password", "token"]);
});
+13 -1
View File
@@ -27,6 +27,17 @@ function meshConfig(file?: string): Record<string, string> {
catch { return {}; }
}
/**
* A password, token or API key is the vault's, never a setting (novox/hq issue 342): a value for one of
* them in the settings file is ignored, and that is said once — by its key, never with its value.
*/
const saidIgnored = new Set<string>();
function ignoreSecretSetting(cfg: Record<string, unknown>, key: string): void {
if (cfg[key] === undefined || saidIgnored.has(key)) return;
saidIgnored.add(key);
console.error(`the setting ${key} is ignored: it comes from the vault`);
}
/** Read a secret the mesh mounted at a file path (an own-secret delivered by `secret accept`);
* absent or unreadable yields undefined so callers fall back rather than crash. */
function readSecret(file?: string): string | undefined {
@@ -53,7 +64,8 @@ export class HomeAssistantClient {
static fromEnv(env: NodeJS.ProcessEnv = process.env): HomeAssistantClient {
const cfg = meshConfig(env.MESH_HOMEASSISTANT_CONFIG_FILE);
const url = cfg.url ?? env.MESH_HOMEASSISTANT_URL ?? `http://127.0.0.1:${env.HOMEASSISTANT_PORT ?? "8123"}`;
const token = cfg.token ?? readSecret(env.MESH_HOMEASSISTANT_TOKEN_FILE) ?? env.MESH_HOMEASSISTANT_TOKEN;
ignoreSecretSetting(cfg, "token");
const token = readSecret(env.MESH_HOMEASSISTANT_TOKEN_FILE) ?? env.MESH_HOMEASSISTANT_TOKEN;
if (!token) throw new Error("no Home Assistant token — set MESH_HOMEASSISTANT_TOKEN");
return new HomeAssistantClient(url, token);
}
+16 -4
View File
@@ -123,11 +123,23 @@
}
],
"requires": [
"lidarr-api",
"mqtt-topic",
"radarr-api",
{
"name": "lidarr-api",
"optional": true
},
{
"name": "mqtt-topic",
"optional": true
},
{
"name": "radarr-api",
"optional": true
},
"route",
"sonarr-api"
{
"name": "sonarr-api",
"optional": true
}
],
"contributes": {
"mqtt-topic": {
+1 -1
View File
@@ -7,7 +7,7 @@
"scripts": {
"build": "tsc client.ts index.ts tools/index.ts provisions/hass.ts provisions/probe.ts provisions/connections.ts provisions/mesh.ts provisions/index.ts --module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist",
"typecheck": "tsc -p tsconfig.json",
"test": "node --test --experimental-strip-types 'test/*.test.ts'"
"test": "npm run build && node --test --experimental-strip-types 'test/*.test.ts'"
},
"dependencies": {
"@novox/mesh-sdk": "^0.1.0"
@@ -0,0 +1,20 @@
// Home Assistant runs without the Servarr apps and without a broker, so those requirements are optional
// (novox/hq ADR 0330, issue 166): a mesh without lidarr or an MQTT broker can assign home-assistant, and
// nothing is installed for it. Its provisions step already reads a binding that is not there as absent.
import { test } from "node:test";
import assert from "node:assert/strict";
import { readFileSync } from "node:fs";
type Requirement = string | { name: string; optional?: boolean };
const manifest = JSON.parse(readFileSync(new URL("../module.json", import.meta.url), "utf8")) as { requires: Requirement[] };
test("the Servarr APIs and mqtt-topic are optional, and route stays hard", () => {
for (const name of ["sonarr-api", "radarr-api", "lidarr-api", "mqtt-topic"]) {
assert.ok(
manifest.requires.some((r) => typeof r !== "string" && r.name === name && r.optional === true),
`${name} is not optional`,
);
}
assert.ok(manifest.requires.includes("route"));
});
@@ -0,0 +1,38 @@
// A password, a token or an API key is the vault's, never a setting (novox/hq issue 342): a value for it in
// the settings file the mesh merges is ignored, the vault's is used, and that is said once, by the key alone.
// Run against the compiled module, the way the runtime loads it.
import { test, mock } from "node:test";
import assert from "node:assert/strict";
import { mkdtempSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { HomeAssistantClient } from "../dist/client.js";
const SETTING = "one-a-setting-carried";
const VAULT = "the-vault-made-this";
function delivered(settings: Record<string, string>): Record<string, string> {
const dir = mkdtempSync(join(tmpdir(), "home-assistant-342-"));
writeFileSync(join(dir, "config.json"), JSON.stringify(settings));
writeFileSync(join(dir, "vault.secret"), VAULT + "\n");
return { dir, config: join(dir, "config.json"), vault: join(dir, "vault.secret") };
}
test("the vault's token is used, a setting of it is ignored and said once, without its value", () => {
const said: string[] = [];
const err = mock.method(console, "error", (...args: unknown[]) => { said.push(args.join(" ")); });
try {
const at = delivered({ token: SETTING });
const env = { MESH_HOMEASSISTANT_CONFIG_FILE: at.config, MESH_HOMEASSISTANT_TOKEN_FILE: at.vault };
const first = HomeAssistantClient.fromEnv(env);
const second = HomeAssistantClient.fromEnv(env);
assert.equal((first as any).token, VAULT);
assert.equal((second as any).token, VAULT);
assert.deepEqual(said, ["the setting token is ignored: it comes from the vault"]);
assert.ok(!said.join("\n").includes(SETTING), "the setting's value is never said");
} finally {
err.mock.restore();
}
});
+21 -1
View File
@@ -42,6 +42,24 @@ function meshConfig(file?: string): Record<string, string> {
catch { return {}; }
}
/**
* A password, token or API key is the vault's, never a setting (novox/hq issue 342): a value for one of
* them in the settings file is ignored, and that is said once — by its key, never with its value.
*/
const saidIgnored = new Set<string>();
function ignoreSecretSetting(cfg: Record<string, unknown>, key: string): void {
if (cfg[key] === undefined || saidIgnored.has(key)) return;
saidIgnored.add(key);
console.error(`the setting ${key} is ignored: it comes from the vault`);
}
/** A secret delivered as a file, trimmed; undefined when there is none. */
function readSecret(path: string | undefined): string | undefined {
if (!path) return undefined;
try { return readFileSync(path, "utf8").trim() || undefined; }
catch { return undefined; }
}
export class IcecastClient {
readonly baseUrl: string;
private readonly authHeader?: string;
@@ -57,7 +75,9 @@ export class IcecastClient {
static fromEnv(env: NodeJS.ProcessEnv = process.env): IcecastClient {
const cfg = meshConfig(env.MESH_ICECAST_CONFIG_FILE);
const url = cfg.url ?? (env.MESH_ICECAST_URL ?? `http://127.0.0.1:${env.ICECAST_PORT ?? "8000"}`);
return new IcecastClient(url, cfg.user ?? env.MESH_ICECAST_ADMIN_USER, cfg.password ?? env.MESH_ICECAST_ADMIN_PASSWORD);
ignoreSecretSetting(cfg, "password");
const password = readSecret(env.MESH_ICECAST_ADMIN_PASSWORD_FILE) ?? env.MESH_ICECAST_ADMIN_PASSWORD;
return new IcecastClient(url, cfg.user ?? env.MESH_ICECAST_ADMIN_USER ?? "admin", password);
}
async getStatus(): Promise<IcecastStatus> {
+2 -1
View File
@@ -120,7 +120,8 @@
],
"env": {
"MESH_ICECAST_URL": "http://127.0.0.1:${port:8000}",
"MESH_ICECAST_CONFIG_FILE": "${dir:mesh-state}/config.json"
"MESH_ICECAST_CONFIG_FILE": "${dir:mesh-state}/config.json",
"MESH_ICECAST_ADMIN_PASSWORD_FILE": "${dir:state}/admin.secret"
}
}
]
@@ -0,0 +1,39 @@
// A password, a token or an API key is the vault's, never a setting (novox/hq issue 342): a value for it in
// the settings file the mesh merges is ignored, the vault's is used, and that is said once, by the key alone.
// Run against the compiled module, the way the runtime loads it.
import { test, mock } from "node:test";
import assert from "node:assert/strict";
import { mkdtempSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { IcecastClient } from "../dist/client.js";
const SETTING = "one-a-setting-carried";
const VAULT = "the-vault-made-this";
function delivered(settings: Record<string, string>): Record<string, string> {
const dir = mkdtempSync(join(tmpdir(), "icecast-342-"));
writeFileSync(join(dir, "config.json"), JSON.stringify(settings));
writeFileSync(join(dir, "vault.secret"), VAULT + "\n");
return { dir, config: join(dir, "config.json"), vault: join(dir, "vault.secret") };
}
test("the vault's password is used, a setting of it is ignored and said once, without its value", () => {
const said: string[] = [];
const err = mock.method(console, "error", (...args: unknown[]) => { said.push(args.join(" ")); });
try {
const at = delivered({ password: SETTING });
const env = { MESH_ICECAST_CONFIG_FILE: at.config, MESH_ICECAST_ADMIN_PASSWORD_FILE: at.vault };
const first = IcecastClient.fromEnv(env);
const second = IcecastClient.fromEnv(env);
const basic = "Basic " + Buffer.from(`admin:${VAULT}`).toString("base64");
assert.equal((first as any).authHeader, basic);
assert.equal((second as any).authHeader, basic);
assert.deepEqual(said, ["the setting password is ignored: it comes from the vault"]);
assert.ok(!said.join("\n").includes(SETTING), "the setting's value is never said");
} finally {
err.mock.restore();
}
});
+13 -1
View File
@@ -43,6 +43,17 @@ function meshConfig(file?: string): Record<string, string> {
catch { return {}; }
}
/**
* A password, token or API key is the vault's, never a setting (novox/hq issue 342): a value for one of
* them in the settings file is ignored, and that is said once — by its key, never with its value.
*/
const saidIgnored = new Set<string>();
function ignoreSecretSetting(cfg: Record<string, unknown>, key: string): void {
if (cfg[key] === undefined || saidIgnored.has(key)) return;
saidIgnored.add(key);
console.error(`the setting ${key} is ignored: it comes from the vault`);
}
/** A secret delivered as a file, trimmed; undefined when there is none, so the caller can fall back. */
function tokenFromFile(file?: string): string | undefined {
if (!file) return undefined;
@@ -71,7 +82,8 @@ export class InfluxDBClient {
const url = cfg.url ?? env.MESH_INFLUXDB_URL ?? `http://127.0.0.1:${env.INFLUXDB_PORT ?? "8086"}`;
// The token reaches the process as a file (novox/hq ADR 0086); the environment variable stays
// only for a workstation running the tools by hand.
const token = cfg.token ?? tokenFromFile(env.MESH_INFLUXDB_TOKEN_FILE) ?? env.MESH_INFLUXDB_TOKEN;
ignoreSecretSetting(cfg, "token");
const token = tokenFromFile(env.MESH_INFLUXDB_TOKEN_FILE) ?? env.MESH_INFLUXDB_TOKEN;
if (!token) throw new Error("no InfluxDB token — set MESH_INFLUXDB_TOKEN_FILE");
const org = cfg.org ?? env.MESH_INFLUXDB_ORG ?? "mesh";
return new InfluxDBClient(url, token, org);
@@ -0,0 +1,38 @@
// A password, a token or an API key is the vault's, never a setting (novox/hq issue 342): a value for it in
// the settings file the mesh merges is ignored, the vault's is used, and that is said once, by the key alone.
// Run against the compiled module, the way the runtime loads it.
import { test, mock } from "node:test";
import assert from "node:assert/strict";
import { mkdtempSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { InfluxDBClient } from "../dist/client.js";
const SETTING = "one-a-setting-carried";
const VAULT = "the-vault-made-this";
function delivered(settings: Record<string, string>): Record<string, string> {
const dir = mkdtempSync(join(tmpdir(), "influxdb-342-"));
writeFileSync(join(dir, "config.json"), JSON.stringify(settings));
writeFileSync(join(dir, "vault.secret"), VAULT + "\n");
return { dir, config: join(dir, "config.json"), vault: join(dir, "vault.secret") };
}
test("the vault's token is used, a setting of it is ignored and said once, without its value", () => {
const said: string[] = [];
const err = mock.method(console, "error", (...args: unknown[]) => { said.push(args.join(" ")); });
try {
const at = delivered({ token: SETTING });
const env = { MESH_INFLUXDB_CONFIG_FILE: at.config, MESH_INFLUXDB_TOKEN_FILE: at.vault };
const first = InfluxDBClient.fromEnv(env);
const second = InfluxDBClient.fromEnv(env);
assert.equal((first as any).token, VAULT);
assert.equal((second as any).token, VAULT);
assert.deepEqual(said, ["the setting token is ignored: it comes from the vault"]);
assert.ok(!said.join("\n").includes(SETTING), "the setting's value is never said");
} finally {
err.mock.restore();
}
});
+1 -1
View File
@@ -112,7 +112,7 @@ for p in postgres-provisioner objectstore-provisioner route-proxy; do
(cd ${shellQuote(dir)}/mesh-controller && CGO_ENABLED=0 go build -o ${shellQuote(bin)}/$p ./examples/$p)
done
export MESH_LAB_HOST_BINARY=${shellQuote(join(dir, "mesh-host", "mesh-host"))}
export MESH_LAB_BUNDLE=${shellQuote(join(dir, "mesh-host", "examples", "foundation-first-node-nats.lock"))}
export MESH_LAB_BUNDLE=${shellQuote(join(dir, "mesh-host", "examples", "foundation-first-node.lock"))}
export MESH_LAB_MODULES=${shellQuote(join(dir, "mesh-controller", "examples", "modules"))}
export MESH_LAB_BUILDER=${shellQuote(join(dir, "mesh-controller", "build", "mesh-builder"))}
export MESH_LAB_BOOTSTRAP_BINARY=${shellQuote(join(dir, "mesh-host", "mesh-bootstrap"))}
+2 -1
View File
@@ -114,7 +114,8 @@
"path": "${dir:mesh-state}/config.json",
"mode": "0600",
"content": "{\n \"password\": \"${secret:server-password}\"\n}\n",
"merge": "json"
"merge": "json",
"protected": ["password"]
}
],
"build": {
+13 -1
View File
@@ -52,6 +52,17 @@ function meshConfig(file?: string): Record<string, string> {
catch { return {}; }
}
/**
* A password, token or API key is the vault's, never a setting (novox/hq issue 342): a value for one of
* them in the settings file is ignored, and that is said once — by its key, never with its value.
*/
const saidIgnored = new Set<string>();
function ignoreSecretSetting(cfg: Record<string, unknown>, key: string): void {
if (cfg[key] === undefined || saidIgnored.has(key)) return;
saidIgnored.add(key);
console.error(`the setting ${key} is ignored: it comes from the vault`);
}
/** Read a secret from the file the mesh mounted it at (an own-secret), if the pointing env is set. */
function readSecret(path: string | undefined): string | undefined {
if (!path) return undefined;
@@ -81,7 +92,8 @@ export class MailuClient {
static fromEnv(env: NodeJS.ProcessEnv = process.env): MailuClient {
const cfg = meshConfig(env.MESH_MAILU_CONFIG_FILE);
const url = cfg.url ?? env.MESH_MAILU_URL;
const apiKey = cfg.apiKey ?? readSecret(env.MESH_MAILU_API_KEY_FILE) ?? env.MESH_MAILU_API_KEY;
ignoreSecretSetting(cfg, "apiKey");
const apiKey = readSecret(env.MESH_MAILU_API_KEY_FILE) ?? env.MESH_MAILU_API_KEY;
if (!url || !apiKey) {
throw new Error("Mailu is not configured — set MESH_MAILU_URL and the API token own-secret");
}
+38
View File
@@ -0,0 +1,38 @@
// A password, a token or an API key is the vault's, never a setting (novox/hq issue 342): a value for it in
// the settings file the mesh merges is ignored, the vault's is used, and that is said once, by the key alone.
// Run against the compiled module, the way the runtime loads it.
import { test, mock } from "node:test";
import assert from "node:assert/strict";
import { mkdtempSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { MailuClient } from "../dist/client.js";
const SETTING = "one-a-setting-carried";
const VAULT = "the-vault-made-this";
function delivered(settings: Record<string, string>): Record<string, string> {
const dir = mkdtempSync(join(tmpdir(), "mailu-342-"));
writeFileSync(join(dir, "config.json"), JSON.stringify(settings));
writeFileSync(join(dir, "vault.secret"), VAULT + "\n");
return { dir, config: join(dir, "config.json"), vault: join(dir, "vault.secret") };
}
test("the vault's apiKey is used, a setting of it is ignored and said once, without its value", () => {
const said: string[] = [];
const err = mock.method(console, "error", (...args: unknown[]) => { said.push(args.join(" ")); });
try {
const at = delivered({ apiKey: SETTING });
const env = { MESH_MAILU_CONFIG_FILE: at.config, MESH_MAILU_URL: "http://127.0.0.1:1/api/v1", MESH_MAILU_API_KEY_FILE: at.vault };
const first = MailuClient.fromEnv(env);
const second = MailuClient.fromEnv(env);
assert.equal((first as any).apiKey, VAULT);
assert.equal((second as any).apiKey, VAULT);
assert.deepEqual(said, ["the setting apiKey is ignored: it comes from the vault"]);
assert.ok(!said.join("\n").includes(SETTING), "the setting's value is never said");
} finally {
err.mock.restore();
}
});
@@ -0,0 +1,150 @@
package main
import (
"sort"
"strings"
"testing"
"time"
)
// novox/hq issue 611: mesh-controller #293 changed one test file. Its check's plan, made while another walk built
// build-agent and route-proxy, read them whole and named them; at the cut their recorded build sources held none of
// its files, so its walk was done with no tier, "moved nothing the mesh holds: nothing to walk". The stranded rule
// (issue 474) read that as "ended without building build-agent, route-proxy", stopped it, and the operator was asked
// about a change already on every machine. The controller now says per module that the build matched what runs
// (the walk's delivery.unchanged), and such a delivery is delivered: its merge moves nothing (ADR 0270 §1). A walk
// that says neither — not built, not unchanged — still stops it.
const (
head293 = "94c928b4c7e1d2aa"
merge293 = "8f1aec0f2b9d6e11"
walk293 = "plan-1791867648768608552"
)
var at293 = time.Date(2026, 10, 13, 5, 10, 0, 0, time.UTC)
// matched is the why the controller gives a module it says unchanged.
const matched = "its build source in novox/mesh-controller holds none of the 1 changed file(s) " +
"(e.g. internal/builder/replay363_test.go): the build the mesh holds is its build"
// replay293 is #293 checked moving build-agent and route-proxy, merged, and its walk done with no tier, saying the
// modules named unchanged.
func replay293(t *testing.T, unchanged map[string]string) *world {
t.Helper()
w := newWorldOver(t, newMemStore(), newFakeController(), newFakeForge(), at293.Add(-time.Minute))
p := pr(ctl474, 293, head293, "fix/363-replay", "internal/builder/replay363_test.go")
w.h.PullUpdated(p)
w.h.Checked(verdictMoving(ctl474, 293, head293, "build-agent", "route-proxy"))
w.later(time.Minute)
w.h.PullMerged(mergedAt(p, merge293, at293))
w.later(90 * time.Second)
note := nothingNote
if len(unchanged) > 0 {
note += "; unchanged, the build the mesh holds is theirs: " + strings.Join(sortedKeys(unchanged), ", ")
}
w.h.WalkMoved(w.ctl.put(Walk{ID: walk293, Repository: ctl474, Branch: "main", Commit: merge293, Created: w.now,
Updated: w.now, State: walkDone, Tiers: [][]string{}, Modules: map[string]*WalkModule{}, Note: note,
Delivery: &WalkDelivery{Awaits: "mesh-delivery", Merges: []WalkMerge{{Repository: ctl474, Commit: merge293}},
Unchanged: unchanged}}))
w.settleAll()
return w
}
// The live case: every module its plan named, the walk said matched what runs. Delivered at once, naming them, a
// success on the pull request, and nothing listed by stalled or asked of the operator, then or past the grace.
func TestAMergeWhoseModulesMatchWhatRunsIsDelivered(t *testing.T) {
w := replay293(t, map[string]string{"build-agent": matched, "route-proxy": matched})
id := IDOf(ctl474, head293)
d := w.delivery(id)
if d.State != Delivered {
t.Fatalf("#293, its modules said unchanged by its walk, is %s, waiting for %q", d.State, w.h.waitsFor(d))
}
why := d.Transitions[len(d.Transitions)-1].Why
for _, want := range []string{walk293, "build-agent, route-proxy", "matched what runs"} {
if !strings.Contains(why, want) {
t.Fatalf("delivered as %q, not saying %q", why, want)
}
}
if d.Stranded != "" {
t.Fatalf("it was read as stranded: %q", d.Stranded)
}
status := w.forge.statuses["novox/mesh-controller@"+head293+" mesh/delivery"]
if !strings.HasPrefix(status, "success ") {
t.Fatalf("mesh/delivery says %q", status)
}
w.later(strandedAfter + time.Minute)
w.settleAll()
if s := w.state(id); s != Delivered {
t.Fatalf("past the grace it is %s", s)
}
if lines := w.h.Stalled(); len(lines) != 0 {
t.Fatalf("stalled lists %+v", lines)
}
}
// A walk that said only one of its modules unchanged left the other unbuilt: it is still stopped past the grace,
// naming only the module nothing delivered.
func TestAMergeWithAModuleNeitherBuiltNorUnchangedIsStillStopped(t *testing.T) {
w := replay293(t, map[string]string{"build-agent": matched})
id := IDOf(ctl474, head293)
if s := w.state(id); s != Published {
t.Fatalf("within the grace it is %s", s)
}
w.later(strandedAfter + time.Minute)
w.settleAll()
d := w.delivery(id)
if d.State != Stopped {
t.Fatalf("route-proxy neither built nor unchanged, it is %s", d.State)
}
why := d.Transitions[len(d.Transitions)-1].Why
if !strings.Contains(why, "without building route-proxy (") {
t.Fatalf("its stop says %q", why)
}
}
// A walk that says nothing unchanged — a controller from before, or one that built nothing at all — stops it as
// before (issue 474).
func TestAMergeWhoseWalkSaysNothingUnchangedIsStillStopped(t *testing.T) {
w := replay293(t, nil)
w.later(strandedAfter + time.Minute)
w.settleAll()
if d := w.delivery(IDOf(ctl474, head293)); d.State != Stopped ||
!strings.Contains(d.Stranded, "without building build-agent, route-proxy") {
t.Fatalf("it is %s: %q", d.State, d.Stranded)
}
}
// Batched behind a later merge of its branch (issue 474's shape), a merge whose module the walk said unchanged moves
// nothing at either commit: delivered, not covered by a later merge that delivered nothing for it.
func TestABatchedMergeWhoseModuleMatchesWhatRunsIsDelivered(t *testing.T) {
w := newWorldOver(t, newMemStore(), newFakeController(), newFakeForge(), at215.Add(-time.Minute))
p215 := pr(ctl474, 215, head215, "fix/431-make-check-runs-what-the-gate-runs", "Makefile")
p217 := pr(ctl474, 217, head217, "fix/432-a-test-reads-what-it-carries", "cmd/mesh-controller/check_test.go")
w.h.PullUpdated(p215)
w.h.PullUpdated(p217)
w.h.Checked(verdictMoving(ctl474, 215, head215, "build-agent"))
w.h.Checked(verdictMoving(ctl474, 217, head217))
w.later(time.Minute)
w.h.PullMerged(mergedAt(p215, merge215, at215))
w.later(time.Minute)
w.h.PullMerged(mergedAt(p217, merge217, at215.Add(56*time.Second)))
w.later(90 * time.Second)
w.h.WalkMoved(w.ctl.put(Walk{ID: batchWalk, Repository: ctl474, Branch: "main", Commit: merge217, Created: w.now,
Updated: w.now, State: walkDone, Tiers: [][]string{}, Modules: map[string]*WalkModule{}, Note: nothingNote,
Delivery: &WalkDelivery{Awaits: "mesh-delivery", Merges: []WalkMerge{
{Repository: ctl474, Commit: merge215, Carried: merge217}, {Repository: ctl474, Commit: merge217}},
Unchanged: map[string]string{"build-agent": "its build source holds none of the 2 changed file(s)"}}}))
w.settleAll()
if d := w.delivery(IDOf(ctl474, head215)); d.State != Delivered {
t.Fatalf("#215, build-agent said unchanged by its batch's walk, is %s, waiting for %q", d.State, w.h.waitsFor(d))
}
}
func sortedKeys(m map[string]string) []string {
var out []string
for k := range m {
out = append(out, k)
}
sort.Strings(out)
return out
}
@@ -3,6 +3,7 @@ package main
import (
"encoding/json"
"fmt"
"maps"
"regexp"
"sort"
"strings"
@@ -460,6 +461,11 @@ type WalkDelivery struct {
// cut, is the walk under the same id: its next record is `building`. A merge this owner reads in a batch is
// not held as unwalked while the batch is heard of.
Batch *WalkBatch `json:"batch,omitempty"`
// Unchanged are the modules the walk did not build because their build source holds none of the files its
// merges changed, each with why (novox/hq issue 611, ADR 0267 §5): the build the mesh holds is theirs. A delivery
// plan that read them whole names them; this says they matched what runs, not that the walk left them unbuilt.
// Absent from a controller that does not say it.
Unchanged map[string]string `json:"unchanged,omitempty"`
}
// WalkMerge is one merge a walk answers.
@@ -583,6 +589,9 @@ type WalkSeen struct {
// from them.
Moved []string `json:"moved,omitempty"`
MovedSaid bool `json:"moved_said,omitempty"`
// Unchanged are the modules the walk said matched what runs, with why (novox/hq issue 611): not built, and
// nothing for it to deliver.
Unchanged map[string]string `json:"unchanged,omitempty"`
}
// Seen is a walk as a delivery keeps it.
@@ -594,6 +603,9 @@ func (w Walk) Seen() *WalkSeen {
s.Waits = d.Awaits != "" && d.Go == nil
s.GoBy = d.By
s.StoppedBy, s.StoppedWhy = d.Stopped, d.StoppedWhy
if len(d.Unchanged) > 0 {
s.Unchanged = maps.Clone(d.Unchanged)
}
}
s.Branch = w.Branch
s.Created = w.Created
@@ -644,9 +656,10 @@ func (d *Delivery) stoppedWords() string {
return s
}
// walkMovedItsModules is whether the walk it follows built every module its plan moves, and whether that is known:
// from the modules the walk's record names, or, for a record kept before they were, from a walk of no tier, which
// moved nothing. Unknown when its plan is not known.
// walkMovedItsModules is whether the walk it follows answered every module its plan moves, and whether that is
// known: each built, or said unchanged — its build source holds none of the files the walk's merges changed, so the
// build the mesh holds is its build (novox/hq issue 611) — from the modules the walk's record names, or, for a
// record kept before they were, from a walk of no tier, which moved nothing. Unknown when its plan is not known.
func (d *Delivery) walkMovedItsModules() (moved, known bool) {
if d.Walk == nil || d.Plan == nil {
return false, false
@@ -654,28 +667,46 @@ func (d *Delivery) walkMovedItsModules() (moved, known bool) {
if !d.Walk.MovedSaid && d.Walk.Tiers > 0 {
return false, false
}
for _, m := range d.Plan.Moved {
if !containsString(d.Walk.Moved, m) {
return false, true
}
}
return true, true
return len(d.notMovedBy()) == 0, true
}
// notMovedBy is the modules its plan moves that the walk it follows did not build.
// notMovedBy is the modules its plan moves that the walk it follows neither built nor said unchanged: what a walk
// that ended without building them left undelivered.
func (d *Delivery) notMovedBy() []string {
var out []string
if d.Walk == nil || d.Plan == nil {
return nil
}
for _, m := range d.Plan.Moved {
if !containsString(d.Walk.Moved, m) {
if _, unchanged := d.Walk.Unchanged[m]; !containsString(d.Walk.Moved, m) && !unchanged {
out = append(out, m)
}
}
return out
}
// unchangedOfIts is the modules its plan moves that the walk it follows said matched what runs, sorted (novox/hq
// issue 611).
func (d *Delivery) unchangedOfIts() []string {
var out []string
if d.Walk == nil || d.Plan == nil {
return nil
}
for _, m := range d.Plan.Moved {
if _, unchanged := d.Walk.Unchanged[m]; unchanged && !containsString(d.Walk.Moved, m) {
out = append(out, m)
}
}
sort.Strings(out)
return out
}
// builtNoneOfIts is whether the walk it follows built none of the modules its plan moves and said each unchanged:
// its merge moves nothing, whichever commit the walk carried (ADR 0270 §1).
func (d *Delivery) builtNoneOfIts() bool {
return d.Plan != nil && len(d.Plan.Moved) > 0 && len(d.unchangedOfIts()) == len(d.Plan.Moved)
}
// Started is whether the walk is past its wait: it waited for nobody, or was let go — whatever came of it
// since. A walk superseded or stopped while it waited never started.
func (s *WalkSeen) Started() bool { return s != nil && !s.Waits && (s.WaitedFor == "" || s.GoBy != "") }
@@ -249,19 +249,31 @@ var Table = []Row{
}},
{From: []State{Published}, Event: EvDone, To: Delivered,
Guard: "nothing for a walk to move; or its walk, carrying its own commit, ended done before its word having built " +
"every module it moves (novox/hq issue 474)",
"every module it moves or said it unchanged (novox/hq issue 474); or its walk said every module it moves " +
"unchanged, its build source holding none of the files changed, so it moves nothing (novox/hq issue 611, " +
"ADR 0270 §1)",
Holds: func(d *Delivery, f Facts) error {
if d.Walk == nil && d.MovesNothing() {
return nil
}
moved, _ := d.walkMovedItsModules()
return unless(d.endedUnstarted() && !d.carriedLater() && moved, "a walk moves it")
return unless(d.endedUnstarted() && moved && (!d.carriedLater() || d.builtNoneOfIts()), "a walk moves it")
},
Says: func(d *Delivery) string {
if d.Walk == nil {
return "nothing for a walk to move"
}
return fmt.Sprintf("its walk %s ended done before its word, having built every module it moves", d.Walk.ID)
unchanged := d.unchangedOfIts()
switch {
case len(unchanged) == 0:
return fmt.Sprintf("its walk %s ended done before its word, having built every module it moves", d.Walk.ID)
case d.builtNoneOfIts():
return fmt.Sprintf("its walk %s moved nothing: %s matched what runs, its build source holding none of "+
"the files changed, so the build the mesh holds is its build (novox/hq issue 611)", d.Walk.ID,
strings.Join(unchanged, ", "))
}
return fmt.Sprintf("its walk %s ended done before its word, having built every module it moves but %s, "+
"which matched what runs (novox/hq issue 611)", d.Walk.ID, strings.Join(unchanged, ", "))
}},
{From: []State{Published, Ready}, Event: EvHold, To: Held,
Guard: "something holds it for a person: its group's composed check did not pass for the heads that merged, " +
+13 -1
View File
@@ -32,6 +32,17 @@ function meshConfig(file?: string): Record<string, string> {
catch { return {}; }
}
/**
* A password, token or API key is the vault's, never a setting (novox/hq issue 342): a value for one of
* them in the settings file is ignored, and that is said once — by its key, never with its value.
*/
const saidIgnored = new Set<string>();
function ignoreSecretSetting(cfg: Record<string, unknown>, key: string): void {
if (cfg[key] === undefined || saidIgnored.has(key)) return;
saidIgnored.add(key);
console.error(`the setting ${key} is ignored: it comes from the vault`);
}
export class NextcloudClient {
constructor(
private readonly container: string,
@@ -53,7 +64,8 @@ export class NextcloudClient {
const ocsUrl = cfg.url ?? env.MESH_NEXTCLOUD_URL ?? `http://127.0.0.1:${env.NEXTCLOUD_PORT ?? "80"}`;
const adminUser = cfg.user ?? env.MESH_NEXTCLOUD_ADMIN_USER ?? "admin";
const passwordFile = env.MESH_NEXTCLOUD_ADMIN_PASSWORD_FILE;
const adminPassword = cfg.password ?? env.MESH_NEXTCLOUD_ADMIN_PASSWORD
ignoreSecretSetting(cfg, "password");
const adminPassword = env.MESH_NEXTCLOUD_ADMIN_PASSWORD
?? (passwordFile ? readFileSync(passwordFile, "utf8").trim() : undefined);
if (!adminPassword) {
throw new Error("no Nextcloud admin password — set MESH_NEXTCLOUD_ADMIN_PASSWORD_FILE " +
@@ -0,0 +1,38 @@
// A password, a token or an API key is the vault's, never a setting (novox/hq issue 342): a value for it in
// the settings file the mesh merges is ignored, the vault's is used, and that is said once, by the key alone.
// Run against the compiled module, the way the runtime loads it.
import { test, mock } from "node:test";
import assert from "node:assert/strict";
import { mkdtempSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { NextcloudClient } from "../dist/client.js";
const SETTING = "one-a-setting-carried";
const VAULT = "the-vault-made-this";
function delivered(settings: Record<string, string>): Record<string, string> {
const dir = mkdtempSync(join(tmpdir(), "nextcloud-342-"));
writeFileSync(join(dir, "config.json"), JSON.stringify(settings));
writeFileSync(join(dir, "vault.secret"), VAULT + "\n");
return { dir, config: join(dir, "config.json"), vault: join(dir, "vault.secret") };
}
test("the vault's password is used, a setting of it is ignored and said once, without its value", () => {
const said: string[] = [];
const err = mock.method(console, "error", (...args: unknown[]) => { said.push(args.join(" ")); });
try {
const at = delivered({ password: SETTING });
const env = { MESH_NEXTCLOUD_CONFIG_FILE: at.config, MESH_NEXTCLOUD_ADMIN_PASSWORD_FILE: at.vault };
const first = NextcloudClient.fromEnv(env);
const second = NextcloudClient.fromEnv(env);
assert.equal((first as any).adminPassword, VAULT);
assert.equal((second as any).adminPassword, VAULT);
assert.deepEqual(said, ["the setting password is ignored: it comes from the vault"]);
assert.ok(!said.join("\n").includes(SETTING), "the setting's value is never said");
} finally {
err.mock.restore();
}
});
+14 -1
View File
@@ -17,7 +17,20 @@ another node requires.
(`/etc/nfs-server/machines`, rendered by the mesh), never one a consumer gives, and it must lie inside
the private network's range. The control node, or any node, gets a share only when both name it. A
want that is not granted, and a grant nobody wants, are said by `exports`. Port 2049 is opened to the
mesh only; the home network is never named.
mesh by default. The machine's local network is reached only by a grant to `local-network` (below).
- **The local network, only when granted and admitted** (novox/hq ADR 0341, after ADR 0340). `grants` may
name `local-network` in a node's place: `library=ws:rw,local-network:ro`. The process then reads the
machine's links (`ip -j`) and takes the subnets on the links carrying its default route that lie wholly
in a private range (RFC 1918, fc00::/7); a public subnet, the private network's link and container
bridges are never counted, and no range is ever typed in a setting. It exports to such a subnet only
when the mesh's packet filter (table `inet mesh`, chain `input`, read with `nft -j list ruleset`)
admits 2049/tcp from all of it, which it does once the assignment binds 2049 with the reach
`local-network`. A grant the filter does not admit is not exported, said by `exports`, and makes the
module unhealthy, naming that act. Every device on the local network then acts as the folder's owner,
as the hand-written export it replaces already allowed.
- **A dataset's `sharenfs` is cleared only when nobody loses the folder.** `adopt` refuses while the
property, by its value or the file ZFS writes for it, exports to a client the mesh's live export does
not cover: for a property to the home network, the module's `local-network` export.
- **Every client is the folder's owner.** NFS with `sec=sys` believes the user id a client sends, and one
id names different accounts on different machines. Every line is `all_squash` with `anonuid` and
`anongid` read from the folder's owner on this machine, so a file written from any node lands owned by
@@ -11,7 +11,7 @@ import (
func TestStrayFindsAnExportOfTheModulesThatNoFileCarries(t *testing.T) {
live := []Entry{{Path: "/srv/library", Client: "10.0.0.4"}}
if got := Stray(live, nil, nil, meshRange); len(got) != 1 {
if got := Stray(live, nil, nil, meshRange, nil); len(got) != 1 {
t.Errorf("an export inside the mesh range that no file carries is the module's leftover: %+v", got)
}
}
@@ -21,7 +21,7 @@ func TestStrayLeavesAnExportAnotherFileCarriesEvenInsideTheMeshRange(t *testing.
// ZFS's own file exports the library to a client inside the mesh's range.
foreign := ParseExports("/srv/library 10.0.0.9(rw,no_subtree_check)\n", "/etc/exports.d/zfs.exports")
live := []Entry{{Path: "/srv/library", Client: "10.0.0.9", Options: "rw"}}
if got := Stray(live, nil, foreign, meshRange); len(got) != 0 {
if got := Stray(live, nil, foreign, meshRange, nil); len(got) != 0 {
t.Errorf("what ZFS's file exports is not the module's, whatever its client: %+v", got)
}
}
@@ -30,7 +30,7 @@ func TestStrayLeavesAnExportAnotherFileCarriesEvenInsideTheMeshRange(t *testing.
func TestStrayLeavesAClientOutsideTheMeshRange(t *testing.T) {
live := []Entry{{Path: "/srv/library", Client: "192.0.2.0/24"}, {Path: "/srv/library", Client: "192.0.2.7"},
{Path: "/srv/library", Client: "10.0.0.0/16"}}
if got := Stray(live, nil, nil, meshRange); len(got) != 0 {
if got := Stray(live, nil, nil, meshRange, nil); len(got) != 0 {
t.Errorf("a client the module never exports to (outside or wider than the mesh's range) is not its leftover: %+v", got)
}
}
@@ -0,0 +1,417 @@
package main
import (
"context"
"encoding/json"
"fmt"
"net/netip"
"sort"
"strings"
)
// A share granted to the machine's local network (novox/hq ADR 0341 clauses 8 and 9, after ADR 0340).
//
// **The grant names `local-network`, never a range.** `grants` may name `local-network` for a share beside
// node names: `library=ws:rw,local-network:ro`. The export goes to the subnets this machine's own local
// network is, by ADR 0340's rules: the networks on the links that carry its default route (the uplink),
// each wholly inside a private range (RFC 1918, or fc00::/7), never a public or link-local one, never the
// private network's own link or a container bridge. A range typed in a setting is never used.
//
// **A grant the filter does not admit is not exported, and is said.** ADR 0340 lets an assignment bind a
// port with the reach `local-network`, and the filter then admits it from those subnets on that port's
// rule alone. The module does not take the reach on trust: every pass it reads the mesh's filter (the
// table `inet mesh`, chain `input`) and exports to a local subnet only when 2049/tcp is admitted from all
// of it. Otherwise the share is not exported there, and the status and the health say so, naming the act
// (bind 2049 with the reach local-network) — a grant never reads as served while the filter drops it.
//
// **What this does not promise.** NFS with sec=sys trusts the client, so every device on the local
// network acts as the share's owner (all_squash, as for a node). That is what the hand-written export it
// replaces already allowed.
// LocalNetwork is the name a grant gives the machine's local network.
const LocalNetwork = "local-network"
// NFSPort is the port the filter must admit from a local subnet before a share is exported to it.
const NFSPort = 2049
// private are the ranges a local network must lie wholly inside (ADR 0340 §2).
var private = []netip.Prefix{
netip.MustParsePrefix("10.0.0.0/8"),
netip.MustParsePrefix("172.16.0.0/12"),
netip.MustParsePrefix("192.168.0.0/16"),
netip.MustParsePrefix("fc00::/7"),
}
// wholly says whether a network lies wholly inside one of the private ranges.
func wholly(p netip.Prefix) bool {
for _, r := range private {
if p.Bits() >= r.Bits() && r.Contains(p.Addr()) {
return true
}
}
return false
}
// ipRoute is one route as `ip -j route show default` answers it.
type ipRoute struct {
Dev string `json:"dev"`
}
// ipLink is one link as `ip -j addr show` answers it.
type ipLink struct {
Name string `json:"ifname"`
AddrInfo []struct {
Family string `json:"family"`
Local string `json:"local"`
Prefixlen int `json:"prefixlen"`
Scope string `json:"scope"`
} `json:"addr_info"`
}
// LocalNetworksFrom is the machine's local networks from what `ip` answered: the links carrying a default
// route, and their addresses. A network not wholly private, or not global in scope, is left out, and why.
func LocalNetworksFrom(routes4, routes6, addrs string) ([]string, []string) {
uplinks := map[string]bool{}
for _, text := range []string{routes4, routes6} {
var rs []ipRoute
if strings.TrimSpace(text) == "" {
continue
}
if err := json.Unmarshal([]byte(text), &rs); err != nil {
continue
}
for _, r := range rs {
if r.Dev != "" {
uplinks[r.Dev] = true
}
}
}
var links []ipLink
if strings.TrimSpace(addrs) != "" {
_ = json.Unmarshal([]byte(addrs), &links)
}
seen := map[string]bool{}
var out, left []string
for _, l := range links {
if !uplinks[l.Name] {
continue
}
for _, a := range l.AddrInfo {
if a.Scope != "global" {
continue
}
addr, err := netip.ParseAddr(a.Local)
if err != nil {
continue
}
p, err := addr.Prefix(a.Prefixlen)
if err != nil {
continue
}
if !wholly(p) {
left = append(left, fmt.Sprintf("%s on %s is not wholly inside a private range, so it is not the local network", p, l.Name))
continue
}
if !seen[p.String()] {
seen[p.String()] = true
out = append(out, p.String())
}
}
}
sort.Strings(out)
return out, left
}
// LocalNetworks is this machine's local networks now, read with `ip`, and what was left out or could not
// be read.
func (m *Machine) LocalNetworks(ctx context.Context) ([]string, []string) {
r4 := m.Run(ctx, "ip", "-j", "-4", "route", "show", "default")
r6 := m.Run(ctx, "ip", "-j", "-6", "route", "show", "default")
ad := m.Run(ctx, "ip", "-j", "addr", "show")
var notes []string
if r4.Code != 0 || ad.Code != 0 {
notes = append(notes, "this machine's links could not be read with ip: "+firstLine(r4.Err+" "+ad.Err))
}
nets, left := LocalNetworksFrom(r4.Out, r6.Out, ad.Out)
return nets, append(notes, left...)
}
// nftRule is one rule of `nft -j list ruleset`.
type nftRule struct {
Family string `json:"family"`
Table string `json:"table"`
Chain string `json:"chain"`
Expr []json.RawMessage `json:"expr"`
}
// nftChain is one chain of `nft -j list ruleset`.
type nftChain struct {
Family string `json:"family"`
Table string `json:"table"`
Name string `json:"name"`
Policy string `json:"policy"`
}
// Admitted says, for each subnet, whether the mesh's filter (table inet mesh, chain input) accepts tcp to
// the port from all of it, read from `nft -j list ruleset`. A rule counts only when every condition it has
// is one this reads — the protocol, the destination port and the source address — and its verdict is
// accept; a rule with any other condition (an interface, a connection state) is left out, so nothing is
// called admitted that may not be. With no such table, nothing is admitted, and why.
func Admitted(ruleset string, port int, subnets []string) (map[string]bool, string) {
out := map[string]bool{}
var doc struct {
Nftables []map[string]json.RawMessage `json:"nftables"`
}
if err := json.Unmarshal([]byte(ruleset), &doc); err != nil {
return out, "the packet filter's rules could not be read (nft -j list ruleset): " + err.Error()
}
var rules []nftRule
found := false
policy := ""
for _, item := range doc.Nftables {
if raw, ok := item["chain"]; ok {
var c nftChain
if json.Unmarshal(raw, &c) == nil && c.Family == "inet" && c.Table == "mesh" && c.Name == "input" {
found, policy = true, c.Policy
}
}
if raw, ok := item["rule"]; ok {
var r nftRule
if json.Unmarshal(raw, &r) == nil && r.Family == "inet" && r.Table == "mesh" && r.Chain == "input" {
rules = append(rules, r)
}
}
}
if !found {
return out, "the mesh's packet filter (table inet mesh, chain input) is not there, so no local network is admitted"
}
if policy == "accept" {
for _, s := range subnets {
out[s] = true
}
return out, ""
}
for _, r := range rules {
sources, ok := admitsPort(r.Expr, port)
if !ok {
continue
}
for _, s := range subnets {
p, err := netip.ParsePrefix(s)
if err != nil {
continue
}
if sources == nil {
out[s] = true
continue
}
for _, src := range sources {
if src.Bits() <= p.Bits() && src.Contains(p.Addr()) {
out[s] = true
}
}
}
}
return out, ""
}
// admitsPort reads one rule: whether it accepts tcp to the port, and from which sources (nil: from
// anywhere). ok is false for a rule that does not accept that port, or that has a condition this does not
// read.
func admitsPort(exprs []json.RawMessage, port int) (sources []netip.Prefix, ok bool) {
accepts, portMatched, anywhere := false, false, true
for _, raw := range exprs {
var e map[string]json.RawMessage
if json.Unmarshal(raw, &e) != nil {
return nil, false
}
if _, is := e["accept"]; is {
accepts = true
continue
}
if _, is := e["counter"]; is {
continue
}
m, is := e["match"]
if !is {
return nil, false
}
var match struct {
Op string `json:"op"`
Left json.RawMessage `json:"left"`
Right json.RawMessage `json:"right"`
}
if json.Unmarshal(m, &match) != nil || (match.Op != "==" && match.Op != "in") {
return nil, false
}
var left struct {
Payload *struct {
Protocol string `json:"protocol"`
Field string `json:"field"`
} `json:"payload"`
Meta *struct {
Key string `json:"key"`
} `json:"meta"`
}
if json.Unmarshal(match.Left, &left) != nil {
return nil, false
}
switch {
case left.Meta != nil && left.Meta.Key == "l4proto":
var proto string
if json.Unmarshal(match.Right, &proto) != nil || proto != "tcp" {
return nil, false
}
case left.Payload != nil && left.Payload.Protocol == "tcp" && left.Payload.Field == "dport":
if !hasPort(match.Right, port) {
return nil, false
}
portMatched = true
case left.Payload != nil && (left.Payload.Protocol == "ip" || left.Payload.Protocol == "ip6") && left.Payload.Field == "saddr":
anywhere = false
sources = append(sources, prefixesOf(match.Right)...)
default:
return nil, false
}
}
if !accepts || !portMatched {
return nil, false
}
if anywhere {
return nil, true
}
if sources == nil {
sources = []netip.Prefix{}
}
return sources, true
}
// hasPort reads a match's right side: a number, or a set holding it.
func hasPort(raw json.RawMessage, port int) bool {
var n int
if json.Unmarshal(raw, &n) == nil {
return n == port
}
var set struct {
Set []json.RawMessage `json:"set"`
}
if json.Unmarshal(raw, &set) == nil {
for _, item := range set.Set {
if json.Unmarshal(item, &n) == nil && n == port {
return true
}
}
}
return false
}
// prefixesOf reads a match's right side as sources: an address, a prefix, or a set of either.
func prefixesOf(raw json.RawMessage) []netip.Prefix {
var s string
if json.Unmarshal(raw, &s) == nil {
if a, err := netip.ParseAddr(s); err == nil {
return []netip.Prefix{netip.PrefixFrom(a, a.BitLen())}
}
if p, err := netip.ParsePrefix(s); err == nil {
return []netip.Prefix{p.Masked()}
}
return nil
}
var pre struct {
Prefix *struct {
Addr string `json:"addr"`
Len int `json:"len"`
} `json:"prefix"`
Set []json.RawMessage `json:"set"`
}
if json.Unmarshal(raw, &pre) != nil {
return nil
}
if pre.Prefix != nil {
if a, err := netip.ParseAddr(pre.Prefix.Addr); err == nil {
if p, err := a.Prefix(pre.Prefix.Len); err == nil {
return []netip.Prefix{p}
}
}
return nil
}
var out []netip.Prefix
for _, item := range pre.Set {
out = append(out, prefixesOf(item)...)
}
return out
}
// LocalNetworkAdmitted is what the process learns each pass: the machine's local subnets, which of them the
// filter admits 2049 from, and what it could not learn.
func (m *Machine) LocalNetworkAdmitted(ctx context.Context) (nets []string, admitted map[string]bool, notes []string) {
nets, notes = m.LocalNetworks(ctx)
if len(nets) == 0 {
return nets, map[string]bool{}, notes
}
r := m.Run(ctx, "nft", "-j", "list", "ruleset")
if r.Code != 0 {
return nets, map[string]bool{}, append(notes, "the packet filter's rules could not be read (nft -j list ruleset): "+firstLine(r.Err))
}
admitted, note := Admitted(r.Out, NFSPort, nets)
if note != "" {
notes = append(notes, note)
}
return nets, admitted, notes
}
// Covered says whether a client a hand-made export names is covered by the mesh's live export of the same
// path: the same address, or a network inside one the mesh exports to. "*" and a name are never covered.
func Covered(client string, mine []Grantee) bool {
var want netip.Prefix
if a, err := netip.ParseAddr(client); err == nil {
want = netip.PrefixFrom(a, a.BitLen())
} else if p, err := netip.ParsePrefix(client); err == nil {
want = p.Masked()
} else {
return false
}
for _, g := range mine {
var have netip.Prefix
if a, err := netip.ParseAddr(g.Address); err == nil {
have = netip.PrefixFrom(a, a.BitLen())
} else if p, err := netip.ParsePrefix(g.Address); err == nil {
have = p.Masked()
} else {
continue
}
if have.Bits() <= want.Bits() && have.Contains(want.Addr()) {
return true
}
}
return false
}
// PropertyClients is who a dataset's sharenfs property exports to: every `@network` or address its value
// names, and "*" for a bare `on` or a value naming nobody, which nothing the mesh exports ever covers.
func PropertyClients(sharenfs string) []string {
v := strings.TrimSpace(sharenfs)
if v == "on" || v == "" {
return []string{"*"}
}
var out []string
for _, opt := range strings.Split(v, ",") {
key, list, ok := strings.Cut(strings.TrimSpace(opt), "=")
if key != "rw" && key != "ro" {
continue
}
if !ok {
return []string{"*"} // rw or ro alone: to everyone
}
for _, c := range strings.Split(list, ":") {
c = strings.TrimPrefix(strings.TrimSpace(c), "@")
if c != "" {
out = append(out, c)
}
}
}
if len(out) == 0 {
return []string{"*"}
}
return out
}
@@ -0,0 +1,184 @@
package main
import (
"context"
"reflect"
"strings"
"testing"
)
// Defends novox/hq ADR 0341 clauses 8 to 10: a share granted to local-network is exported to the machine's
// own private subnets on its uplink, only those the mesh's filter admits 2049 from; a grant the filter does
// not admit is said and unhealthy; and sharenfs is cleared only once the mesh's export covers every client
// the property exported to.
const homeNet = "192.168.1.0/24"
// The home server's links, as `ip -j` answers them: the uplink with a private and a public address, the
// private network's link, and a container bridge.
const (
routes4 = `[{"dst":"default","gateway":"192.168.1.1","dev":"enp6s0"}]`
addrs = `[
{"ifname":"lo","addr_info":[{"family":"inet","local":"127.0.0.1","prefixlen":8,"scope":"host"}]},
{"ifname":"enp6s0","addr_info":[
{"family":"inet","local":"192.168.1.135","prefixlen":24,"scope":"global"},
{"family":"inet","local":"198.51.100.7","prefixlen":24,"scope":"global"},
{"family":"inet6","local":"fe80::1","prefixlen":64,"scope":"link"}]},
{"ifname":"mesh0","addr_info":[{"family":"inet","local":"10.0.0.2","prefixlen":32,"scope":"global"}]},
{"ifname":"docker0","addr_info":[{"family":"inet","local":"172.17.0.1","prefixlen":16,"scope":"global"}]}]`
)
// rulesetAdmitting is the mesh's filter with 2049 admitted from the nodes and, when local is set, from it.
func rulesetAdmitting(local string) string {
src := `{"set":["10.0.0.1","10.0.0.3"]}`
if local != "" {
addr, bits, _ := strings.Cut(local, "/")
src = `{"set":["10.0.0.1","10.0.0.3",{"prefix":{"addr":"` + addr + `","len":` + bits + `}}]}`
}
return `{"nftables":[{"metainfo":{"version":"1.1"}},
{"table":{"family":"inet","name":"mesh"}},
{"chain":{"family":"inet","table":"mesh","name":"input","type":"filter","hook":"input","prio":0,"policy":"drop"}},
{"rule":{"family":"inet","table":"mesh","chain":"input","expr":[{"match":{"op":"!=","left":{"meta":{"key":"iifname"}},"right":{"set":["enp6s0","mesh0"]}}},{"accept":null}]}},
{"rule":{"family":"inet","table":"mesh","chain":"input","expr":[{"match":{"op":"==","left":{"payload":{"protocol":"ip","field":"saddr"}},"right":` + src + `}},{"match":{"op":"==","left":{"payload":{"protocol":"tcp","field":"dport"}},"right":2049}},{"accept":null}]}},
{"rule":{"family":"inet","table":"mesh","chain":"input","expr":[{"match":{"op":"==","left":{"payload":{"protocol":"tcp","field":"dport"}},"right":445}},{"accept":null}]}}]}`
}
func (f *fake) homeLinks(ruleset string) {
f.answers["ip -j -4 route show default"] = Ran{Out: routes4}
f.answers["ip -j -6 route show default"] = Ran{Out: "[]"}
f.answers["ip -j addr show"] = Ran{Out: addrs}
f.answers["nft -j list ruleset"] = Ran{Out: ruleset}
}
func TestTheLocalNetworkIsTheUplinksPrivateSubnetsOnly(t *testing.T) {
nets, left := LocalNetworksFrom(routes4, "", addrs)
if !reflect.DeepEqual(nets, []string{homeNet}) {
t.Errorf("local networks %v; the uplink's private subnet alone, never the public one, the mesh's link or a bridge", nets)
}
if len(left) != 1 || !strings.Contains(left[0], "198.51.100.0/24") {
t.Errorf("the public subnet is not said as left out: %v", left)
}
if nets, _ := LocalNetworksFrom("", "", addrs); len(nets) != 0 {
t.Errorf("with no default route there is no uplink and no local network: %v", nets)
}
}
func TestAdmittedReadsThePortsOwnRuleOnly(t *testing.T) {
got, note := Admitted(rulesetAdmitting(homeNet), 2049, []string{homeNet})
if note != "" || !got[homeNet] {
t.Errorf("2049 admitted from the home network is not read as admitted: %v %q", got, note)
}
got, _ = Admitted(rulesetAdmitting(""), 2049, []string{homeNet})
if got[homeNet] {
t.Error("2049 admitted from the nodes only is read as admitted from the home network")
}
// A rule open to every source on 445 says nothing of 2049, and the interface rule is not read as one.
if got, _ := Admitted(rulesetAdmitting(""), 445, []string{homeNet}); !got[homeNet] {
t.Error("a port open from anywhere is not admitted")
}
if got, note := Admitted(`{"nftables":[]}`, 2049, []string{homeNet}); got[homeNet] || note == "" {
t.Errorf("with no mesh filter a subnet is admitted, or nothing is said: %v %q", got, note)
}
}
// grantedLocal is the library granted to ws and to the local network, ws wanting it.
func grantedLocal(f *fake, grant string) {
f.mountinfo("/", "/srv/library")
f.dir("/srv/library")
f.file(ConfigFile, grantsConfig("library=/srv/library", "library=ws:rw,"+grant))
f.serve("library")
f.kernelReads()
}
func TestAShareGrantedToTheLocalNetworkGoesToItsSubnetWhenTheFilterAdmitsIt(t *testing.T) {
f := newFake(t)
grantedLocal(f, "local-network:ro")
f.homeLinks(rulesetAdmitting(homeNet))
s := f.machine().Pass(context.Background(), nil)
text := f.read(ExportFile)
if !strings.Contains(text, " "+homeNet+"(ro,mp,sync,secure,no_subtree_check,all_squash,") {
t.Errorf("the library is not exported read-only to the home network, squashed:\n%s", text)
}
if strings.Contains(text, "198.51.100") || strings.Contains(text, "172.17") {
t.Errorf("a public subnet or a bridge is exported to:\n%s", text)
}
if len(s.Unreached) != 0 || !s.Shares[0].Live || len(s.Shares[0].To) != 2 {
t.Errorf("status: %+v", s)
}
if h := f.machine().Health(); !h.Healthy || !strings.Contains(h.Why, "local-network ("+homeNet+", ro)") {
t.Errorf("health: %+v", h)
}
}
func TestAGrantTheFilterDoesNotAdmitIsNotExportedAndIsUnhealthy(t *testing.T) {
f := newFake(t)
grantedLocal(f, "local-network:rw")
f.homeLinks(rulesetAdmitting(""))
s := f.machine().Pass(context.Background(), nil)
if strings.Contains(f.read(ExportFile), homeNet) {
t.Errorf("the library is exported to a subnet the filter drops:\n%s", f.read(ExportFile))
}
if len(s.Unreached) != 1 || !strings.Contains(s.Unreached[0], "reach local-network") {
t.Errorf("the unreached grant is not said with its act: %v", s.Unreached)
}
if h := f.machine().Health(); h.Healthy || !strings.Contains(h.Why, "does not admit 2049/tcp") {
t.Errorf("health: %+v", h)
}
}
func TestWithNoPrivateNetworkOnTheUplinkALocalGrantGoesNowhereAndIsSaid(t *testing.T) {
f := newFake(t)
grantedLocal(f, "local-network:rw")
s := f.machine().Pass(context.Background(), nil) // ip answers nothing
if strings.Contains(f.read(ExportFile), "192.168") {
t.Errorf("exported to a local network that was not found:\n%s", f.read(ExportFile))
}
if !strings.Contains(strings.Join(s.Said, "\n"), "reports no private network on its uplink") {
t.Errorf("not said: %v", s.Said)
}
}
func TestAdoptWaitsUntilTheLocalNetworkExportCoversTheProperty(t *testing.T) {
f := newFake(t)
f.uid = 1000
zfsWith(f, "rw=@"+homeNet)
f.file("/etc/exports.d/zfs.exports", "/srv/library "+homeNet+"(sec=sys,rw,no_subtree_check,mountpoint)\n")
grantedLocal(f, "local-network:rw")
f.homeLinks(rulesetAdmitting("")) // the filter does not admit the home network yet
f.machine().Pass(context.Background(), nil)
if _, err := f.machine().Adopt("/srv/library", false, ""); err == nil || !strings.Contains(err.Error(), "does not cover") {
t.Fatalf("adopt while the home network is not covered: %v", err)
}
f.homeLinks(rulesetAdmitting(homeNet))
f.machine().Pass(context.Background(), nil)
f.ran = nil
dry, err := f.machine().Adopt("/srv/library", false, "")
if err != nil || !strings.Contains(dry.Said, "would set sharenfs=off") {
t.Fatalf("dry run once covered: %+v %v", dry, err)
}
for _, r := range f.ran {
if strings.Contains(r, "zfs set") {
t.Fatalf("a dry run changed something: %v", f.ran)
}
}
}
func TestPropertyClientsAndCovered(t *testing.T) {
for v, want := range map[string]string{
"rw=@192.168.1.0/24": "192.168.1.0/24",
"rw=@192.168.1.0/24:@10.0.0.3": "192.168.1.0/24 10.0.0.3",
"on": "*",
"rw,no_root_squash": "*",
"sec=sys,ro=@10.0.0.3": "10.0.0.3",
} {
if got := strings.Join(PropertyClients(v), " "); got != want {
t.Errorf("PropertyClients(%q) = %q, not %q", v, got, want)
}
}
mine := []Grantee{{Node: "ws", Address: "10.0.0.3"}, {Node: LocalNetwork, Address: homeNet}}
for client, want := range map[string]bool{"10.0.0.3": true, homeNet: true, "192.168.1.7": true, "192.168.0.0/16": false, "*": false, "10.0.0.4": false} {
if Covered(client, mine) != want {
t.Errorf("Covered(%q) is not %v", client, want)
}
}
}
@@ -259,7 +259,9 @@ func TestAdoptIsADryRunWithoutConfirmAndRefusesWhileTheMeshExportIsNotLive(t *te
f := newFake(t)
f.uid = 1000
f.dir("/srv/library")
zfsWith(f, "rw=@192.0.2.0/24")
// A property the mesh's export covers once live: the one node. One to the home network is
// localnet_test.go's.
zfsWith(f, "rw=@"+wsAddress)
f.file(ConfigFile, config("library=/srv/library"))
f.serve("library")
// The kernel has not read the mesh's file: no exportfs effect.
+45 -7
View File
@@ -31,6 +31,11 @@ type Status struct {
// Warnings make the module no less healthy but are said in its health: what it could not learn, and
// how old what it relies on is.
Warnings []string `json:"warnings,omitempty"`
// Local is the machine's local subnets this pass (ADR 0341): what a grant to local-network may reach.
Local []string `json:"local_networks,omitempty"`
// Unreached is each grant to the local network the packet filter does not admit: not exported, and the
// module is unhealthy until it is.
Unreached []string `json:"unreached,omitempty"`
// Reloaded is when the process last had the kernel read the export files again, and why.
Reloaded *time.Time `json:"reloaded,omitempty"`
ReloadWhy string `json:"reload_why,omitempty"`
@@ -61,8 +66,17 @@ func (m *Machine) Pass(ctx context.Context, last *Status) Status {
if _, note := m.Backup(); note != "" {
s.Warnings = append(s.Warnings, note)
}
if grantsLocal(c) {
nets, admitted, notes := m.LocalNetworkAdmitted(ctx)
c.Local = &Local{Nets: nets, Admitted: admitted}
s.Local = nets
s.Said = append(s.Said, notes...)
}
m.setWalk(ctx)
exports, refused, said := m.Judge(c, roster, wants)
if c.Local != nil {
s.Unreached = c.Local.Unreached
}
m.setWalk(nil)
if ctx.Err() != nil {
// Stopping: write nothing and reload nothing, so Stopped starts at once.
@@ -100,7 +114,7 @@ func (m *Machine) Pass(ctx context.Context, last *Status) Status {
}
}
if reload == "" {
if stray := Stray(live, mine, foreign, c.Range); len(stray) > 0 {
if stray := Stray(live, mine, foreign, c.Range, s.Local); len(stray) > 0 {
reload = fmt.Sprintf("the kernel exports %s to %s, which the export file no longer carries", stray[0].Path, stray[0].Client)
}
}
@@ -116,7 +130,7 @@ func (m *Machine) Pass(ctx context.Context, last *Status) Status {
if note != "" && len(exports) > 0 {
s.Problems = append(s.Problems, note)
}
for _, e := range Stray(live, mine, foreign, c.Range) {
for _, e := range Stray(live, mine, foreign, c.Range, s.Local) {
s.Problems = append(s.Problems, fmt.Sprintf("the kernel still exports %s to %s, which the module's export file "+
"no longer carries: a reload has to take it away (exportfs -ra as root)", e.Path, e.Client))
}
@@ -256,21 +270,34 @@ func loud(what string) {
}
// Stray is what the kernel exports that is the module's and that its file no longer carries: an entry to a
// client inside the private network's range — the only clients the module ever exports to — that neither
// the module's file nor any other export file (/etc/exports, ZFS's own) has. What another file exports is
// not the module's, whatever its client.
func Stray(live, mine, foreign []Entry, rng string) []Entry {
// client inside the private network's range, or one of the machine's local subnets — the only clients the
// module ever exports to — that neither the module's file nor any other export file (/etc/exports, ZFS's
// own) has. What another file exports is not the module's, whatever its client.
func Stray(live, mine, foreign []Entry, rng string, local []string) []Entry {
prefix, err := netip.ParsePrefix(rng)
if err != nil {
return nil
}
ours := []netip.Prefix{prefix}
for _, n := range local {
if p, err := netip.ParsePrefix(n); err == nil {
ours = append(ours, p.Masked())
}
}
has := map[string]bool{}
for _, e := range append(append([]Entry{}, mine...), foreign...) {
has[path.Clean(e.Path)+" "+e.Client] = true
}
var out []Entry
for _, e := range live {
if has[path.Clean(e.Path)+" "+e.Client] || !insideRange(e.Client, prefix) {
if has[path.Clean(e.Path)+" "+e.Client] {
continue
}
inside := false
for _, p := range ours {
inside = inside || insideRange(e.Client, p)
}
if !inside {
continue
}
out = append(out, e)
@@ -288,3 +315,14 @@ func insideRange(client string, prefix netip.Prefix) bool {
}
return false
}
// grantsLocal says whether any share is granted to the local network, so the process looks at the links
// and the filter only when a grant asks for them.
func grantsLocal(c Config) bool {
for _, nodes := range c.Grants {
if _, ok := nodes[LocalNetwork]; ok {
return true
}
}
return false
}
+34 -3
View File
@@ -28,7 +28,8 @@ import (
// write to WantsFile). The access is the lower of the two: a grant of ro is never widened by a want. The
// address is the mesh's own, from the roster the mesh renders (RosterFile), never one a consumer gave,
// and it must lie in the private network's range. A share nobody is granted and wants is exported to
// nobody. The home network is never named.
// nobody. The local network is named only by a grant to `local-network`, and then only the subnets the
// packet filter admits 2049 from (localnet.go, novox/hq ADR 0341).
//
// **Never the backups, never root's.** A share is judged at its real path (symlinks resolved). It is
// refused when it is the machine's root, is not there, is not a folder, is owned by root or root's group
@@ -86,6 +87,17 @@ type Config struct {
// Grants is, per share, the nodes this machine lets mount it and with which access.
Grants map[string]map[string]string
Range string
// Local is the machine's local network as the process found it this pass (localnet.go): nil when it was
// not looked at, and a grant to local-network then goes nowhere.
Local *Local
}
// Local is the machine's local subnets, which of them the filter admits 2049 from, and — written by Judge —
// each local-network grant the filter does not admit.
type Local struct {
Nets []string
Admitted map[string]bool
Unreached []string
}
// Want is one share a node's mounts module wants of this machine, as it put it on the bus.
@@ -172,7 +184,8 @@ func ParseShares(value string) ([]Share, []string) {
}
// ParseGrants reads the `grants` setting: `share=node:rw|ro[,node:rw|ro]`, entries separated by spaces;
// `none` grants nothing.
// `none` grants nothing. `local-network` in a node's place grants the machine's local network
// (localnet.go).
func ParseGrants(value string) (map[string]map[string]string, []string) {
out := map[string]map[string]string{}
seen := map[string]bool{}
@@ -457,8 +470,25 @@ func (m *Machine) Judge(c Config, roster map[string]string, wants map[string][]W
e.To = append(e.To, Grantee{Node: node, Address: addr, Access: access})
}
}
if g, ok := granted[LocalNetwork]; ok {
switch {
case c.Local == nil || len(c.Local.Nets) == 0:
said = append(said, fmt.Sprintf("%s is granted to the local network, and this machine reports no private network on its uplink, so it is exported to no local network", s.Name))
default:
for _, n := range c.Local.Nets {
if !c.Local.Admitted[n] {
c.Local.Unreached = append(c.Local.Unreached, fmt.Sprintf("the share %s is granted to the local network %s, and the packet filter does not admit %d/tcp from it, so it is not exported there: bind nfs-server's %d with the reach local-network (novox/hq ADR 0340, ADR 0341)", s.Name, n, NFSPort, NFSPort))
continue
}
e.To = append(e.To, Grantee{Node: LocalNetwork, Address: n, Access: lower(g, s.Access)})
}
}
}
grantedNodes := make([]string, 0, len(granted))
for n := range granted {
if n == LocalNetwork {
continue
}
if !wanting[n] {
grantedNodes = append(grantedNodes, n)
}
@@ -474,7 +504,7 @@ func (m *Machine) Judge(c Config, roster map[string]string, wants map[string][]W
e.Clients = strings.Join(addrs, ",")
e.Line = ExportLine(e)
if len(e.To) == 0 {
said = append(said, fmt.Sprintf("%s is exported to no node: none is both granted it and wants it", s.Name))
said = append(said, fmt.Sprintf("%s is exported to no node: none is both granted it and wants it, and no local network is granted and admitted", s.Name))
}
out = append(out, e)
}
@@ -517,6 +547,7 @@ func ExportFileText(exports []Exported) string {
b.WriteString("# Written by the mesh's module nfs-server (novox/hq ADR 0263). Rendered by the module's own process\n")
b.WriteString("# from its assignment's `shares` and `grants` and the nodes that want each share; edit the settings,\n")
b.WriteString("# never this file. One share per folder, to each granted and wanting node's private address only,\n")
b.WriteString("# and to a local subnet only when local-network is granted and the filter admits 2049 from it;\n")
b.WriteString("# every client squashed to the folder's owner.\n")
for _, e := range exports {
if e.Line == "" {
+29 -1
View File
@@ -127,7 +127,7 @@ func (m *Machine) Exports() ExportsAnswer {
}
a.Notes = append(a.Notes, st.Problems...)
a.Notes = append(a.Notes, st.Warnings...)
a.Unmatched = st.Said
a.Unmatched = append(append([]string{}, st.Said...), st.Unreached...)
}
entries, sets, notes := m.Foreign()
a.Notes = append(a.Notes, notes...)
@@ -380,6 +380,31 @@ func (m *Machine) Adopt(p string, confirm bool, why string) (AdoptAnswer, error)
if lnote != "" || !SharedLive(live, *mine) {
return a, fmt.Errorf("the mesh's export of %s to %s is not live in the kernel yet, so taking over would unexport it; wait for the module's process, or call reload", p, grantees(mine.To))
}
// Every client the property exported to must be covered by the mesh's live export, so no client loses
// the folder in the take-over (novox/hq ADR 0341 clause 10): for a property to the home network, the
// module's local-network export of it. Who the property exports to is read from its value and from the
// file ZFS writes for it.
clients := PropertyClients(ds.ShareNFS)
entries, _, _ := m.Foreign()
for _, e := range entries {
if strings.Contains(e.From, "zfs") && path.Clean(e.Path) == p {
clients = append(clients, e.Client)
}
}
var uncovered []string
seenClient := map[string]bool{}
for _, c := range clients {
if seenClient[c] {
continue
}
seenClient[c] = true
if !Covered(c, mine.To) {
uncovered = append(uncovered, c)
}
}
if len(uncovered) > 0 {
return a, fmt.Errorf("the dataset's sharenfs exports %s to %s, which the mesh's export (to %s) does not cover, so taking over would unexport it for them: grant the share to local-network (and bind %d with the reach local-network) until the mesh's export covers them, or have a person decide they lose it", p, strings.Join(uncovered, ", "), grantees(mine.To), NFSPort)
}
plan := fmt.Sprintf("set sharenfs=off on the dataset %s (now %q) and run exportfs -ra; %s stays exported as the share %s to %s, every client as %d:%d",
ds.Name, ds.ShareNFS, p, mine.Name, grantees(mine.To), mine.UID, mine.GID)
if !confirm {
@@ -428,6 +453,9 @@ func (m *Machine) Health() Health {
for _, r := range st.Refused {
found = append(found, r.Why)
}
for _, u := range st.Unreached {
found = append(found, sentence(u))
}
if st.ReloadError != "" {
found = append(found, sentence("the kernel could not read the export files again: "+st.ReloadError))
}
+3 -3
View File
@@ -55,7 +55,7 @@
"grants": {
"kind": "preference",
"default": "none",
"why": "which nodes may mount which share, and how, is this machine's to say (hq ADR 0263, review of mesh-catalog #136): share=node:rw|ro[,node:rw|ro], entries separated by spaces; none exports to no node, whatever a node wants"
"why": "which nodes may mount which share, and how, is this machine's to say (hq ADR 0263, review of mesh-catalog #136): share=node:rw|ro[,node:rw|ro], entries separated by spaces; none exports to no node, whatever a node wants. local-network in a node's place grants this machine's local network, the private subnets on its uplink (hq ADR 0341)"
}
},
"facts": {
@@ -74,7 +74,7 @@
"protocol": "tcp",
"from": "mesh",
"fixed": true,
"why": "the shares, to the mesh's machines only (hq ADR 0263): NFS version 4 alone, which needs no other port, and never the home network, where a device that is not a node could claim any user id"
"why": "the shares, to the mesh's machines (hq ADR 0263): NFS version 4 alone, which needs no other port. A machine whose grants name local-network binds this port with the reach local-network (hq ADR 0340, ADR 0341), and the module exports to a local subnet only while the filter admits it from there"
}
],
"resources": [
@@ -154,7 +154,7 @@
"type": "file",
"path": "/etc/nfs-server/shares.conf",
"mode": "0644",
"content": "# Written by the mesh (module nfs-server, novox/hq ADR 0263) from this machine's assignment.\n# Replaced on every push; change the `shares` setting, never this file.\n#\n# The shares: name=folder, or name=folder:ro, one share per folder. The grants: share=node:rw|ro[,…],\n# which nodes may mount each and how. The module's process exports a share to a node's private address\n# only when it is granted here and that node's mounts module wants it, every client mapped to the\n# folder's owner, and only to addresses inside the range below.\nshares=${setting:shares}\ngrants=${setting:grants}\nrange=${machine:mesh-range}\n"
"content": "# Written by the mesh (module nfs-server, novox/hq ADR 0263) from this machine's assignment.\n# Replaced on every push; change the `shares` setting, never this file.\n#\n# The shares: name=folder, or name=folder:ro, one share per folder. The grants: share=node:rw|ro[,…],\n# which nodes may mount each and how; local-network in a node's place grants this machine's local\n# network (novox/hq ADR 0341). The module's process exports a share to a node's private address only\n# when it is granted here and that node's mounts module wants it, and to a local subnet only when the\n# packet filter admits 2049 from it; every client mapped to the folder's owner.\nshares=${setting:shares}\ngrants=${setting:grants}\nrange=${machine:mesh-range}\n"
}
],
"build": {
+288
View File
@@ -0,0 +1,288 @@
package main
import (
"fmt"
"path"
"sort"
"strings"
)
// The shares and logins, from the assignment, the hand-written configuration as found, and the module's
// rendering (novox/hq ADR 0341 clauses 2 to 4).
//
// **One share per folder, every client as its owner.** A share is a name and a path in the `shares`
// setting (`library=/srv/library`, `:ro` for read-only). It is served with `force user` and `force group`
// read from the real folder's owner, never written as numbers in a setting, so a file written by any client
// lands owned by the folder's owner. The refusals are ADR 0263's: the machine's root, a folder not there or
// not a folder, a folder owned by root or root's group, and a folder inside or holding a backup repository.
//
// **SMB 2.1 and later, on 445 alone.** NetBIOS is off (`disable netbios`, `smb ports = 445`), so 137, 138 and
// 139 are never served; the minimum protocol is SMB2_10. No guest: `map to guest = never`.
// Share is one share the operator asked for.
type Share struct {
Name string `json:"name"`
Path string `json:"path"`
Access string `json:"access"`
}
// Config is what the mesh rendered from the assignment.
type Config struct {
Shares []Share
Users []string
// Passwords is the folder the mesh places the logins' secrets in (smb-password-<user>.secret).
Passwords string
}
// ParseConfig reads the config file: `shares=`, `users=` and `passwords=` lines.
func ParseConfig(text string) (Config, []string) {
var c Config
var problems []string
for _, line := range strings.Split(text, "\n") {
line = strings.TrimSpace(line)
if line == "" || strings.HasPrefix(line, "#") {
continue
}
key, value, ok := strings.Cut(line, "=")
if !ok {
problems = append(problems, fmt.Sprintf("the line %q of %s says nothing this module reads", line, ConfigFile))
continue
}
switch strings.TrimSpace(key) {
case "shares":
shares, bad := ParseShares(value)
c.Shares, problems = shares, append(problems, bad...)
case "users":
users, bad := ParseUsers(value)
c.Users, problems = users, append(problems, bad...)
case "passwords":
c.Passwords = strings.TrimSpace(value)
default:
problems = append(problems, fmt.Sprintf("%s names %q, which this module does not read", ConfigFile, key))
}
}
return c, problems
}
// ParseShares reads the `shares` setting: `name=path` or `name=path:ro`, separated by spaces.
func ParseShares(value string) ([]Share, []string) {
var shares []Share
var problems []string
seen := map[string]bool{}
for _, item := range strings.Fields(value) {
if item == "none" {
continue
}
name, p, ok := strings.Cut(item, "=")
if !ok || name == "" || p == "" {
problems = append(problems, fmt.Sprintf("the share %q is not name=path", item))
continue
}
if !validName(name) {
problems = append(problems, fmt.Sprintf("the share name %q is not lower-case letters, digits and dashes", name))
continue
}
if seen[name] {
problems = append(problems, fmt.Sprintf("the share %q is named twice", name))
continue
}
seen[name] = true
s := Share{Name: name, Path: p, Access: "rw"}
if strings.HasSuffix(p, ":ro") {
s.Path, s.Access = strings.TrimSuffix(p, ":ro"), "ro"
} else if strings.HasSuffix(p, ":rw") {
s.Path = strings.TrimSuffix(p, ":rw")
}
shares = append(shares, s)
}
sort.Slice(shares, func(i, j int) bool { return shares[i].Name < shares[j].Name })
return shares, problems
}
// ParseUsers reads the `users` setting: login names separated by spaces; `none` is none.
func ParseUsers(value string) ([]string, []string) {
var out, problems []string
seen := map[string]bool{}
for _, u := range strings.Fields(value) {
if u == "none" {
continue
}
if !validName(u) {
problems = append(problems, fmt.Sprintf("the login %q is not lower-case letters, digits and dashes", u))
continue
}
if !seen[u] {
seen[u] = true
out = append(out, u)
}
}
sort.Strings(out)
return out, problems
}
func validName(n string) bool {
if n == "" || n[0] == '-' {
return false
}
for _, r := range n {
if !(r >= 'a' && r <= 'z' || r >= '0' && r <= '9' || r == '-') {
return false
}
}
return true
}
// Section is one section of a Samba configuration: its name and its parameters, keys lower-cased with their
// spaces folded.
type Section struct {
Name string `json:"name"`
Params map[string]string `json:"params"`
}
// ParseSamba reads a Samba configuration's sections, in order. Comments (# and ;) and continuation lines are
// handled as Samba does.
func ParseSamba(text string) []Section {
var out []Section
var cur *Section
text = strings.ReplaceAll(text, "\\\n", " ")
for _, line := range strings.Split(text, "\n") {
line = strings.TrimSpace(line)
if line == "" || line[0] == '#' || line[0] == ';' {
continue
}
if strings.HasPrefix(line, "[") && strings.HasSuffix(line, "]") {
out = append(out, Section{Name: strings.TrimSpace(line[1 : len(line)-1]), Params: map[string]string{}})
cur = &out[len(out)-1]
continue
}
k, v, ok := strings.Cut(line, "=")
if !ok || cur == nil {
continue
}
cur.Params[foldKey(k)] = strings.TrimSpace(v)
}
return out
}
func foldKey(k string) string { return strings.Join(strings.Fields(strings.ToLower(k)), " ") }
// special are the sections that are not shares.
var special = map[string]bool{"global": true, "homes": true, "printers": true, "print$": true}
// HandShare is one share of the hand-written configuration, and the setting's share that takes its place.
type HandShare struct {
Name string `json:"name"`
Path string `json:"path"`
// ValidUsers is who may log in to it, as the hand-written file said.
ValidUsers string `json:"valid_users,omitempty"`
ReadOnly bool `json:"read_only"`
// Counterpart is the setting's share of the same name and folder; empty when the settings name none.
Counterpart string `json:"counterpart,omitempty"`
Why string `json:"why,omitempty"`
}
// HandShares are the hand-written configuration's shares, each matched to the setting's share of the same
// name serving the same folder (a trailing slash aside).
func HandShares(sections []Section, shares []Share) []HandShare {
byName := map[string]Share{}
for _, s := range shares {
byName[strings.ToLower(s.Name)] = s
}
var out []HandShare
for _, sec := range sections {
if special[strings.ToLower(sec.Name)] {
continue
}
h := HandShare{Name: sec.Name, Path: sec.Params["path"], ValidUsers: sec.Params["valid users"], ReadOnly: readOnly(sec.Params)}
s, ok := byName[strings.ToLower(sec.Name)]
switch {
case !ok:
h.Why = fmt.Sprintf("the setting shares names no share %s", sec.Name)
case cleanPath(s.Path) != cleanPath(h.Path):
h.Why = fmt.Sprintf("the setting's share %s serves %s, and the hand-written one serves %s", s.Name, s.Path, h.Path)
default:
h.Counterpart = s.Name
}
out = append(out, h)
}
return out
}
func cleanPath(p string) string {
if p == "" {
return ""
}
return path.Clean(p)
}
// readOnly is Samba's reading of a share's access: read only = yes unless writable/writeable/write ok says
// otherwise, as Samba defaults it.
func readOnly(p map[string]string) bool {
yes := func(v string) bool {
v = strings.ToLower(strings.TrimSpace(v))
return v == "yes" || v == "true" || v == "1"
}
if v, ok := p["read only"]; ok {
return yes(v)
}
for _, k := range []string{"writable", "writeable", "write ok"} {
if v, ok := p[k]; ok {
return !yes(v)
}
}
return true
}
// Served is a share as the module serves it.
type Served struct {
Name string `json:"name"`
Path string `json:"path"`
Access string `json:"access"`
UID int `json:"uid"`
GID int `json:"gid"`
User string `json:"force_user"`
Group string `json:"force_group"`
Users []string `json:"valid_users"`
// Live is whether Samba serves it now, as testparm reads the configuration in use.
Live bool `json:"live"`
}
// Refused is a share that is not served, and why.
type Refused struct {
Name string `json:"name"`
Path string `json:"path"`
Why string `json:"why"`
}
// Render is Samba's whole configuration from the shares the module serves.
func Render(shares []Served) string {
var b strings.Builder
b.WriteString(Marker + "\n")
b.WriteString("# Rendered by the module's own process from its assignment's settings shares and users; change the\n")
b.WriteString("# settings, never this file. The hand-written configuration it replaced is kept beside it, dated.\n")
b.WriteString("[global]\n")
for _, kv := range [][2]string{
{"server role", "standalone server"},
{"security", "user"},
{"passdb backend", "tdbsam"},
{"map to guest", "never"},
{"server min protocol", "SMB2_10"},
{"disable netbios", "yes"},
{"smb ports", "445"},
{"load printers", "no"},
{"printing", "bsd"},
{"printcap name", "/dev/null"},
{"disable spoolss", "yes"},
} {
fmt.Fprintf(&b, " %s = %s\n", kv[0], kv[1])
}
for _, s := range shares {
ro := "no"
if s.Access == "ro" {
ro = "yes"
}
fmt.Fprintf(&b, "\n[%s]\n path = %s\n read only = %s\n browseable = yes\n guest ok = no\n", s.Name, s.Path, ro)
fmt.Fprintf(&b, " valid users = %s\n force user = %s\n force group = %s\n", strings.Join(s.Users, " "), s.User, s.Group)
}
return b.String()
}
@@ -0,0 +1,279 @@
package main
import (
"bytes"
"context"
"errors"
"fmt"
"io/fs"
"os"
"os/exec"
"path"
"path/filepath"
"strconv"
"strings"
"syscall"
"time"
)
// The machine as the module sees it (novox/hq ADR 0341).
//
// **Two processes, two accounts.** The tools run as the operator account (ADR 0175 §4) and read; what only
// root may do (`smbcontrol`, `smbstatus`) goes through `sudo -n`. The module's long-running process runs as
// root: it judges the shares, and once a person adopted the configuration it writes Samba's configuration
// file whole, sets the logins' passwords from their secrets, and stops NetBIOS.
//
// **It writes its own files and nothing else.** Samba's configuration (only after adoption), its dated copy
// of the hand-written one, the status, and the record of which passwords it set. Nothing here can create,
// chown, chmod or remove anything under a shared path: the machine offers no such call, and Write refuses
// every path but the module's (novox/hq ADR 0051).
// The module's files on the machine.
const (
// ConfigFile is what the mesh renders from the assignment: the shares, the logins, where the secrets are.
ConfigFile = "/etc/smb-server/shares.conf"
// SambaConf is Samba's configuration: the hand-written one until adoption, the module's after.
SambaConf = "/etc/samba/smb.conf"
// Candidate is the module's rendering, checked with testparm before it takes SambaConf's place.
Candidate = "/etc/samba/smb.conf.mesh-new"
// HandCopyPrefix begins the dated copy of the hand-written configuration kept at adoption.
HandCopyPrefix = "/etc/samba/smb.conf.hand-written-"
// StatusFile is the process's last look, for the tools.
StatusFile = "/run/smb-server/status.json"
// AdoptDir is where the adopt verb, as the operator account, leaves a person's word for the process.
AdoptDir = "/var/lib/smb-server/adopt"
// AdoptFile is that word: when, and why.
AdoptFile = AdoptDir + "/adopt.json"
// SetFile is which password the process last set for each login, as a digest, readable by root only.
SetFile = "/var/lib/smb-server/passwords.json"
// Unit is Samba's service; NMBUnit its NetBIOS name service, stopped once the module takes over.
Unit = "smb.service"
NMBUnit = "nmb.service"
// Port is the one port SMB is served on.
Port = 445
)
// Marker is the first line of a configuration the module wrote: how it tells its own from a person's.
const Marker = "# Written by the mesh (module smb-server, novox/hq ADR 0341)."
// writable is every file the module may write.
func writable(p string) bool {
switch p {
case SambaConf, Candidate, StatusFile, AdoptFile, SetFile:
return true
}
return strings.HasPrefix(p, HandCopyPrefix) && !strings.Contains(strings.TrimPrefix(p, HandCopyPrefix), "/")
}
// CommandTimeout bounds every command.
const CommandTimeout = 20 * time.Second
// Ran is what one command did.
type Ran struct {
Out string
Err string
Code int
}
// Runner runs one command, with what it reads on stdin; injected, so everything is tested over recorded
// answers.
type Runner func(ctx context.Context, input, name string, args ...string) Ran
// ExecRunner runs a command on the machine, bounded by CommandTimeout.
func ExecRunner(ctx context.Context, input, name string, args ...string) Ran {
ctx, cancel := context.WithTimeout(ctx, CommandTimeout)
defer cancel()
cmd := exec.CommandContext(ctx, name, args...)
if input != "" {
cmd.Stdin = strings.NewReader(input)
}
var stdout, stderr bytes.Buffer
cmd.Stdout, cmd.Stderr = &stdout, &stderr
err := cmd.Run()
r := Ran{Out: stdout.String(), Err: strings.TrimSpace(stderr.String())}
if err != nil {
var exit *exec.ExitError
if errors.As(err, &exit) {
r.Code = exit.ExitCode()
} else {
r.Code = -1
if r.Err == "" {
r.Err = err.Error()
}
}
}
return r
}
// Owner is who owns a path, and whether it is a directory.
type Owner struct {
UID, GID int
Dir bool
}
// Machine is a filesystem root (the real one, or a test's tree), a runner, who this process is, and a clock.
type Machine struct {
Root string
Run Runner
UID int
Now func() time.Time
// ownerOf reads a file's owner from its inode; tests set it so their folders stand for an owner.
ownerOf func(info os.FileInfo) (uid, gid int)
}
// Here is this machine.
func Here() *Machine {
return &Machine{Root: "/", Run: ExecRunner, UID: os.Getuid(), Now: time.Now, ownerOf: inodeOwner}
}
func (m *Machine) at(p string) string { return filepath.Join(m.Root, p) }
// Read reads a file.
func (m *Machine) Read(p string) (string, error) {
b, err := os.ReadFile(m.at(p))
return string(b), err
}
// Exists says whether a path is there.
func (m *Machine) Exists(p string) bool {
_, err := os.Lstat(m.at(p))
return err == nil
}
// Stat is a path's owner and whether it is a folder.
func (m *Machine) Stat(p string) (Owner, error) {
info, err := os.Stat(m.at(p))
if err != nil {
return Owner{}, err
}
of := m.ownerOf
if of == nil {
of = inodeOwner
}
uid, gid := of(info)
return Owner{UID: uid, GID: gid, Dir: info.IsDir()}, nil
}
// Real is a path with every symbolic link followed, as the machine sees it.
func (m *Machine) Real(p string) (string, error) {
real, err := filepath.EvalSymlinks(m.at(p))
if err != nil {
return "", err
}
rel, err := filepath.Rel(m.Root, real)
if err != nil || strings.HasPrefix(rel, "..") {
return "", fmt.Errorf("%s leads outside the machine", p)
}
return path.Join("/", filepath.ToSlash(rel)), nil
}
// Children are a folder's subfolders, by name; symbolic links are not followed.
func (m *Machine) Children(p string) []string {
entries, err := os.ReadDir(m.at(p))
if err != nil {
return nil
}
var out []string
for _, e := range entries {
if e.Type()&fs.ModeSymlink == 0 && e.IsDir() {
out = append(out, e.Name())
}
}
return out
}
// Write writes one of the module's own files, and refuses every other path.
func (m *Machine) Write(p, content string, mode os.FileMode) error {
if !writable(p) {
return fmt.Errorf("%s is not one of the module's files; it writes nothing else", p)
}
if err := os.MkdirAll(filepath.Dir(m.at(p)), 0o755); err != nil {
return err
}
tmp := m.at(p) + ".tmp"
if err := os.WriteFile(tmp, []byte(content), mode); err != nil {
return err
}
return os.Rename(tmp, m.at(p))
}
// Place moves the checked candidate into Samba's configuration's place.
func (m *Machine) Place() error { return os.Rename(m.at(Candidate), m.at(SambaConf)) }
// NameOf is the account or group name for an id, read from the machine's passwd or group file.
func (m *Machine) NameOf(file string, id int) string {
text, _ := m.Read(file)
for _, line := range strings.Split(text, "\n") {
f := strings.Split(line, ":")
if len(f) >= 3 && f[2] == strconv.Itoa(id) {
return f[0]
}
}
return ""
}
// HasAccount says whether the machine has an account of that name.
func (m *Machine) HasAccount(name string) bool {
text, _ := m.Read("/etc/passwd")
for _, line := range strings.Split(text, "\n") {
if f := strings.SplitN(line, ":", 2); len(f) == 2 && f[0] == name {
return true
}
}
return false
}
// Escalated is a command as the operator account runs what only root may: through sudo without a prompt.
func (m *Machine) Escalated(name string, args ...string) (string, []string) {
if m.UID == 0 {
return name, args
}
return "sudo", append([]string{"-n", name}, args...)
}
// AsRoot runs what only root may do, and names a refusal by how it failed.
func (m *Machine) AsRoot(ctx context.Context, input, name string, args ...string) (Ran, error) {
program, argv := m.Escalated(name, args...)
r := m.Run(ctx, input, program, argv...)
if r.Code != 0 {
why := r.Err
if why == "" {
why = strings.TrimSpace(r.Out)
}
return r, fmt.Errorf("%s %s: exit %d: %s", name, strings.Join(args, " "), r.Code, why)
}
return r, nil
}
// UnitState is what the service manager says of a unit: active, inactive, failed, …
func (m *Machine) UnitState(unit string) string {
r := m.Run(context.Background(), "", "systemctl", "is-active", unit)
if s := strings.TrimSpace(r.Out); s != "" {
return s
}
return "unknown"
}
// UnitEnabled is what the service manager says of a unit's boot: enabled, disabled, …
func (m *Machine) UnitEnabled(unit string) string {
r := m.Run(context.Background(), "", "systemctl", "is-enabled", unit)
if s := strings.TrimSpace(r.Out); s != "" {
return s
}
return "unknown"
}
func inodeOwner(info os.FileInfo) (uid, gid int) {
if st, ok := info.Sys().(*syscall.Stat_t); ok {
return int(st.Uid), int(st.Gid)
}
return -1, -1
}
func firstLine(s string) string {
s = strings.TrimSpace(s)
if i := strings.IndexByte(s, '\n'); i >= 0 {
return s[:i]
}
return s
}
+38
View File
@@ -0,0 +1,38 @@
// The smb-server module's Go bundle (novox/hq ADR 0341): a node serves folders over SMB to the devices on its
// own network, one share per folder, every client as the folder's owner, SMB 2.1 and later on 445 alone.
//
// Started with no arguments it is served by the node's runtime as the operator account, over MCP on stdio
// (ADR 0188, ADR 0193): the node-smb-server seat's verbs and the module's health. Started as
// `smb-server config` it is the module's long-running process, as root (process.go). stdout is the MCP
// channel; everything else is said on stderr.
package main
import (
"context"
"fmt"
"os"
"os/signal"
"syscall"
stdio "git.novox.be/novox/mesh-sdk/go"
)
func main() {
if len(os.Args) > 1 {
if os.Args[1] != "config" || len(os.Args) != 2 {
fmt.Fprintln(os.Stderr, "usage: smb-server [config]")
os.Exit(2)
}
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, os.Interrupt)
defer stop()
if err := Here().RunConfig(ctx); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
return
}
if err := stdio.Serve("", Tools(Here())); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
@@ -0,0 +1,397 @@
package main
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"path"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/health"
)
// The module's process, as root: every Every it judges the shares, and — only once a person adopted the
// configuration — writes Samba's configuration whole, checks it with testparm before Samba reads it, sets the
// logins' passwords from their secrets and stops NetBIOS (novox/hq ADR 0341 clauses 3 to 7). Before adoption
// it reads the hand-written configuration, reports it, and writes nothing but its status.
// Every is how often the process looks.
const Every = 30 * time.Second
// Adoption is a person's word to take over, left by the adopt verb.
type Adoption struct {
At string `json:"at"`
Why string `json:"why"`
// Copy is the dated copy of the hand-written configuration, once the process kept it.
Copy string `json:"copy,omitempty"`
}
// Login is one login of the `users` setting, as the process found it.
type Login struct {
Name string `json:"name"`
// Password is "set" (from its secret), "kept as found" (in Samba's database, no secret given),
// "waits" (no secret, not in Samba's database) or "refused" (why says).
Password string `json:"password"`
Why string `json:"why,omitempty"`
}
// Status is the process's last look, as the tools read it.
type Status struct {
At string `json:"at"`
// Mode is "found" (the hand-written configuration serves, kept as found) or "adopted" (the module's).
Mode string `json:"mode"`
Shares []Served `json:"shares"`
Refused []Refused `json:"refused,omitempty"`
Hand []HandShare `json:"hand_written,omitempty"`
Logins []Login `json:"logins,omitempty"`
Problems []string `json:"problems,omitempty"`
Waits []health.Wait `json:"waits,omitempty"`
Adopted *Adoption `json:"adopted,omitempty"`
// Written is when the process last wrote Samba's configuration, and Reloaded when Samba last read it.
Written string `json:"written,omitempty"`
Reloaded string `json:"reloaded,omitempty"`
NetBIOS string `json:"netbios,omitempty"`
}
// WalkDepth and WalkLimit bound the look below a share for a backup repository (ADR 0263 §3).
const (
WalkDepth = 4
WalkLimit = 200000
)
// isRepository says whether a folder is a restic repository: `config` beside `keys`, `snapshots`, `data`.
func (m *Machine) isRepository(dir string) bool {
if o, err := m.Stat(path.Join(dir, "config")); err != nil || o.Dir {
return false
}
for _, sub := range []string{"keys", "snapshots", "data"} {
if o, err := m.Stat(path.Join(dir, sub)); err != nil || !o.Dir {
return false
}
}
return true
}
// repositoryNear finds a backup repository at, above or within WalkDepth levels below a folder; complete is
// false when the look below stopped at its bound.
func (m *Machine) repositoryNear(real string) (found string, complete bool) {
for p := real; ; p = path.Dir(p) {
if m.isRepository(p) {
return p, true
}
if p == "/" {
break
}
}
seen, level := 0, []string{real}
for depth := 1; depth <= WalkDepth && len(level) > 0; depth++ {
var next []string
for _, d := range level {
for _, c := range m.Children(d) {
if seen++; seen > WalkLimit {
return "", false
}
p := path.Join(d, c)
if m.isRepository(p) {
return p, true
}
next = append(next, p)
}
}
level = next
}
return "", true
}
// Judge turns the asked-for shares into served shares, or refusals.
func (m *Machine) Judge(c Config) ([]Served, []Refused) {
var out []Served
var refused []Refused
for _, s := range c.Shares {
no := func(why string) { refused = append(refused, Refused{Name: s.Name, Path: s.Path, Why: why}) }
if !path.IsAbs(s.Path) || strings.ContainsAny(s.Path, "\n\r\"\\") {
no(fmt.Sprintf("The share %s names %q, which is not an absolute path a configuration can carry.", s.Name, s.Path))
continue
}
real, err := m.Real(path.Clean(s.Path))
if err != nil {
no(fmt.Sprintf("The share %s's folder %s is not there (%v); the module never creates it.", s.Name, s.Path, err))
continue
}
if real == "/" {
no(fmt.Sprintf("The share %s is the machine's root, which is never shared.", s.Name))
continue
}
o, err := m.Stat(real)
if err != nil || !o.Dir {
no(fmt.Sprintf("The share %s's path %s is not a folder.", s.Name, real))
continue
}
if o.UID == 0 || o.GID == 0 {
no(fmt.Sprintf("The share %s's folder %s is owned by root (%d:%d), and every client would act as root there: give it an owner of its own.", s.Name, real, o.UID, o.GID))
continue
}
if p, complete := m.repositoryNear(real); p != "" {
no(fmt.Sprintf("The share %s's folder %s is at, inside or above the backup repository %s, and backups are never shared.", s.Name, real, p))
continue
} else if !complete {
no(fmt.Sprintf("The share %s's folder %s holds more folders than the module looks through (%d levels, at most %d), so it could not make sure no backup repository is under it.", s.Name, real, WalkDepth, WalkLimit))
continue
}
user, group := m.NameOf("/etc/passwd", o.UID), m.NameOf("/etc/group", o.GID)
if user == "" || group == "" {
no(fmt.Sprintf("The share %s's folder %s is owned by %d:%d, which this machine names no account or group for, so no client can be made its owner.", s.Name, real, o.UID, o.GID))
continue
}
if len(c.Users) == 0 {
no(fmt.Sprintf("The share %s is not served: the setting users names no login.", s.Name))
continue
}
out = append(out, Served{Name: s.Name, Path: real, Access: s.Access, UID: o.UID, GID: o.GID, User: user, Group: group, Users: c.Users})
}
return out, refused
}
// PasswordSecret is the secret holding a login's password.
func PasswordSecret(user string) string { return "smb-password-" + user }
// readSecret reads a login's password from its secret: one line; absent is not given.
func (m *Machine) readSecret(dir, user string) (string, bool, error) {
if dir == "" || !path.IsAbs(dir) {
return "", false, fmt.Errorf("the mesh gave no folder for the logins' secrets")
}
raw, err := m.Read(path.Join(dir, PasswordSecret(user)+".secret"))
if err != nil {
if !m.Exists(path.Join(dir, PasswordSecret(user)+".secret")) {
return "", false, nil
}
return "", false, err
}
pw := strings.TrimRight(raw, "\r\n")
if pw == "" || strings.ContainsAny(pw, "\r\n") {
return "", false, fmt.Errorf("its secret is not one line")
}
return pw, true, nil
}
func digest(user, pw string) string {
h := sha256.Sum256([]byte(user + "\x00" + pw))
return hex.EncodeToString(h[:])
}
// Logins judges every login of the setting and, when adopted, sets a password whose secret is new or changed
// (clause 5). A login in Samba's database with no secret given keeps its password; the module never removes
// or resets an entry by itself.
func (m *Machine) Logins(ctx context.Context, c Config, adopted bool) ([]Login, []health.Wait, []string) {
var logins []Login
var waits []health.Wait
var problems []string
set := map[string]string{}
if raw, err := m.Read(SetFile); err == nil {
_ = json.Unmarshal([]byte(raw), &set)
}
changed := false
for _, u := range c.Users {
l := Login{Name: u}
inDB := m.Run(ctx, "", "pdbedit", "-L", "-u", u).Code == 0
pw, given, err := m.readSecret(c.Passwords, u)
switch {
case err != nil:
l.Password, l.Why = "refused", "its password "+PasswordSecret(u)+" could not be read: "+err.Error()
problems = append(problems, fmt.Sprintf("the login %s: %s", u, l.Why))
case !given && inDB:
l.Password, l.Why = "kept as found", "no secret "+PasswordSecret(u)+" is given, so the password in Samba's database stands"
case !given:
l.Password, l.Why = "waits", "no secret "+PasswordSecret(u)+" is given and Samba's database has no such login"
waits = append(waits, health.Wait{Part: "the login " + u, Secret: PasswordSecret(u), What: "the password of the login " + u})
case !m.HasAccount(u):
l.Password, l.Why = "refused", "this machine has no account "+u+"; the module creates no account"
problems = append(problems, fmt.Sprintf("the login %s: %s", u, l.Why))
case !adopted:
l.Password, l.Why = "kept as found", "its secret is given and is set once the configuration is adopted"
case set[u] == digest(u, pw) && inDB:
l.Password = "set"
default:
// smbpasswd -s reads the new password twice from stdin; -a adds the login when it is not there.
args := []string{"-s", "-a", u}
if r := m.Run(ctx, pw+"\n"+pw+"\n", "smbpasswd", args...); r.Code != 0 {
l.Password, l.Why = "refused", "smbpasswd refused it: "+firstLine(r.Err)
problems = append(problems, fmt.Sprintf("the login %s: %s", u, l.Why))
break
}
set[u], changed, l.Password = digest(u, pw), true, "set"
}
logins = append(logins, l)
}
if changed {
raw, _ := json.MarshalIndent(set, "", " ")
if err := m.Write(SetFile, string(raw)+"\n", 0o600); err != nil {
problems = append(problems, "which passwords were set could not be kept: "+err.Error())
}
}
return logins, waits, problems
}
// Pass is one look; what it did is in the status it answers, which it has written.
func (m *Machine) Pass(ctx context.Context) Status {
s := Status{At: m.Now().UTC().Format(time.RFC3339), Mode: "found", Shares: []Served{}}
text, err := m.Read(ConfigFile)
if err != nil {
s.Problems = append(s.Problems, fmt.Sprintf("the assignment's shares (%s) could not be read: %v", ConfigFile, err))
}
c, problems := ParseConfig(text)
s.Problems = append(s.Problems, problems...)
served, refused := m.Judge(c)
s.Refused = refused
conf, _ := m.Read(SambaConf)
ours := strings.HasPrefix(conf, Marker)
if !ours {
s.Hand = HandShares(ParseSamba(conf), c.Shares)
}
var adoption *Adoption
if raw, err := m.Read(AdoptFile); err == nil {
var a Adoption
if json.Unmarshal([]byte(raw), &a) == nil && a.At != "" {
adoption = &a
}
}
if ours && adoption == nil {
// Written by the module before, and the word is gone: what it wrote stands, and is the module's.
adoption = &Adoption{At: "before", Why: "the configuration is already the module's"}
}
s.Adopted = adoption
if adoption != nil {
if !ours {
if missing := withoutCounterpart(s.Hand); len(missing) > 0 {
s.Problems = append(s.Problems, "the configuration is not taken over: "+strings.Join(missing, "; "))
adoption = nil
} else if served == nil && len(c.Shares) > 0 {
s.Problems = append(s.Problems, "the configuration is not taken over: no share the settings name can be served")
adoption = nil
}
}
}
if adoption != nil {
s.Mode = "adopted"
m.takeOver(ctx, &s, conf, ours, served, adoption)
} else {
for i := range served {
served[i].Live = false
}
}
logins, waits, lp := m.Logins(ctx, c, s.Mode == "adopted")
s.Logins, s.Waits = logins, waits
s.Problems = append(s.Problems, lp...)
if s.Mode == "adopted" {
s.Shares = m.markLive(ctx, served)
} else {
s.Shares = served
}
raw, _ := json.MarshalIndent(s, "", " ")
if err := m.Write(StatusFile, string(raw)+"\n", 0o644); err != nil {
s.Problems = append(s.Problems, "the status file could not be written: "+err.Error())
}
return s
}
// withoutCounterpart is each hand-written share the settings do not take the place of, in words.
func withoutCounterpart(hand []HandShare) []string {
var out []string
for _, h := range hand {
if h.Counterpart == "" {
out = append(out, fmt.Sprintf("the hand-written share %s (%s) has no counterpart: %s", h.Name, h.Path, h.Why))
}
}
return out
}
// takeOver writes the module's configuration when it differs from what is there: the first time, a dated
// copy of the hand-written one is kept; every time, testparm checks the candidate before it takes the
// configuration's place and Samba reads it. Then NetBIOS is stopped.
func (m *Machine) takeOver(ctx context.Context, s *Status, conf string, ours bool, served []Served, a *Adoption) {
want := Render(served)
if conf != want {
if err := m.Write(Candidate, want, 0o644); err != nil {
s.Problems = append(s.Problems, "the module's configuration could not be written: "+err.Error())
return
}
if r := m.Run(ctx, "", "testparm", "-s", m.at(Candidate)); r.Code != 0 {
s.Problems = append(s.Problems, "the module's configuration fails Samba's own check and is kept out of use: "+firstLine(r.Err))
return
}
if !ours {
copyAt := HandCopyPrefix + m.Now().UTC().Format("20060102T150405Z")
if err := m.Write(copyAt, conf, 0o644); err != nil {
s.Problems = append(s.Problems, "the hand-written configuration could not be kept, so it is not replaced: "+err.Error())
return
}
a.Copy = copyAt
if a.At != "before" {
raw, _ := json.MarshalIndent(a, "", " ")
_ = m.Write(AdoptFile, string(raw)+"\n", 0o644)
}
}
if err := m.Place(); err != nil {
s.Problems = append(s.Problems, "the module's configuration could not take its place: "+err.Error())
return
}
s.Written = m.Now().UTC().Format(time.RFC3339)
if r := m.Run(ctx, "", "smbcontrol", "smbd", "reload-config"); r.Code != 0 {
s.Problems = append(s.Problems, "Samba did not read its configuration again: "+firstLine(r.Err))
} else {
s.Reloaded = s.Written
}
}
if m.UnitState(NMBUnit) == "active" || m.UnitEnabled(NMBUnit) == "enabled" {
if r := m.Run(ctx, "", "systemctl", "disable", "--now", NMBUnit); r.Code != 0 {
s.Problems = append(s.Problems, "NetBIOS ("+NMBUnit+") could not be stopped: "+firstLine(r.Err))
}
}
s.NetBIOS = m.UnitState(NMBUnit)
}
// markLive marks each share Samba serves now, as testparm reads the configuration in use.
func (m *Machine) markLive(ctx context.Context, served []Served) []Served {
r := m.Run(ctx, "", "testparm", "-s", m.at(SambaConf))
live := map[string]bool{}
if r.Code == 0 {
for _, sec := range ParseSamba(r.Out) {
live[strings.ToLower(sec.Name)] = true
}
}
for i := range served {
served[i].Live = live[strings.ToLower(served[i].Name)] && m.UnitState(Unit) == "active"
}
return served
}
// RunConfig is the process: a pass now and every Every until the context ends.
func (m *Machine) RunConfig(ctx context.Context) error {
m.Pass(ctx)
t := time.NewTicker(Every)
defer t.Stop()
for {
select {
case <-ctx.Done():
return nil
case <-t.C:
m.Pass(ctx)
}
}
}
// LastStatus is the process's last look, or why there is none.
func (m *Machine) LastStatus() (*Status, error) {
text, err := m.Read(StatusFile)
if err != nil {
return nil, fmt.Errorf("the module's process has written no status (%s): %v", StatusFile, err)
}
var s Status
if err := json.Unmarshal([]byte(text), &s); err != nil {
return nil, fmt.Errorf("the module's status file is not one: %v", err)
}
return &s, nil
}
@@ -0,0 +1,301 @@
package main
import (
"context"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
// Defends novox/hq ADR 0341 clauses 2 to 7 and 11 to 13: an SMB share is served as its folder's owner, SMB
// 2.1 and later on 445 alone with NetBIOS off; before a person adopts it the hand-written configuration is
// kept as found and nothing is written; adoption keeps a dated copy, writes the module's configuration only
// once testparm passes it, and refuses while a hand-written share has no counterpart; a login without its
// secret keeps its password, and one with it is set once.
type fake struct {
t *testing.T
root string
answers map[string]Ran
ran []string
inputs map[string]string
now time.Time
}
func newFake(t *testing.T) *fake {
f := &fake{t: t, root: t.TempDir(), answers: map[string]Ran{}, inputs: map[string]string{}, now: time.Date(2026, 10, 13, 9, 30, 0, 0, time.UTC)}
f.answers["systemctl is-active smb.service"] = Ran{Out: "active\n"}
f.answers["systemctl is-active nmb.service"] = Ran{Out: "active\n"}
f.answers["systemctl is-enabled nmb.service"] = Ran{Out: "enabled\n"}
f.answers["pdbedit -L -u media"] = Ran{Out: "media:1001:\n"}
f.file("/etc/passwd", "root:x:0:0::/root:/bin/sh\nmedia:x:1001:2000::/home/media:/bin/sh\n")
f.file("/etc/group", "root:x:0:\nmedia:x:2000:\n")
return f
}
func (f *fake) machine() *Machine {
return &Machine{Root: f.root, UID: 0, Now: func() time.Time { return f.now },
ownerOf: func(info os.FileInfo) (int, int) {
if strings.HasPrefix(info.Name(), "rootowned") {
return 0, 0
}
return 1001, 2000
},
Run: func(_ context.Context, input, name string, args ...string) Ran {
line := strings.Join(append([]string{name}, args...), " ")
line = strings.ReplaceAll(line, f.root, "")
f.ran = append(f.ran, line)
if input != "" {
f.inputs[line] = input
}
if line == "systemctl disable --now nmb.service" {
f.answers["systemctl is-active nmb.service"] = Ran{Out: "inactive\n"}
f.answers["systemctl is-enabled nmb.service"] = Ran{Out: "disabled\n"}
}
if strings.HasPrefix(line, "testparm -s ") {
if r, ok := f.answers["testparm"]; ok {
return r
}
return Ran{Out: f.read(strings.TrimPrefix(line, "testparm -s "))}
}
return f.answers[line]
}}
}
func (f *fake) dir(p string) {
if err := os.MkdirAll(filepath.Join(f.root, p), 0o755); err != nil {
f.t.Fatal(err)
}
}
func (f *fake) file(p, content string) {
f.dir(filepath.Dir(p))
if err := os.WriteFile(filepath.Join(f.root, p), []byte(content), 0o644); err != nil {
f.t.Fatal(err)
}
}
func (f *fake) read(p string) string {
b, _ := os.ReadFile(filepath.Join(f.root, p))
return string(b)
}
func (f *fake) ran1(prefix string) []string {
var out []string
for _, r := range f.ran {
if strings.HasPrefix(r, prefix) {
out = append(out, r)
}
}
return out
}
// handWritten is the home server's configuration as issue 169's survey found it.
const handWritten = "[global]\n workgroup = WORKGROUP\n server string = home\n\n[media]\n path = /storage/media/\n valid users = media\n writable = yes\n"
func (f *fake) home(shares, users string) {
f.dir("/storage/media")
f.file(ConfigFile, "shares="+shares+"\nusers="+users+"\npasswords=/var/lib/mesh/smb-server/state\n")
f.file(SambaConf, handWritten)
}
func TestTheRenderingServesAsTheOwnerOnFourFortyFiveAloneWithoutNetBIOS(t *testing.T) {
text := Render([]Served{{Name: "media", Path: "/storage/media", Access: "rw", User: "media", Group: "media", Users: []string{"media"}}})
for _, want := range []string{Marker, "server min protocol = SMB2_10", "disable netbios = yes", "smb ports = 445", "map to guest = never",
"[media]", "path = /storage/media", "read only = no", "valid users = media", "force user = media", "force group = media"} {
if !strings.Contains(text, want) {
t.Errorf("the rendering lacks %q:\n%s", want, text)
}
}
if strings.Contains(text, "139") || strings.Contains(text, "1001") {
t.Errorf("the rendering names 139, or an owner as a number:\n%s", text)
}
}
func TestAShareIsRefusedAtRootRootOwnedMissingOrNearABackupAndNothingIsCreated(t *testing.T) {
f := newFake(t)
f.dir("/srv/rootowned")
f.dir("/srv/pool/library")
f.file("/srv/pool/library/deep/repo/config", "x")
for _, d := range []string{"keys", "snapshots", "data"} {
f.dir("/srv/pool/library/deep/repo/" + d)
}
f.dir("/srv/ok")
c, _ := ParseConfig("shares=root=/ owned=/srv/rootowned gone=/srv/gone lib=/srv/pool/library ok=/srv/ok:ro\nusers=media\n")
served, refused := f.machine().Judge(c)
why := map[string]string{}
for _, r := range refused {
why[r.Name] = r.Why
}
for name, says := range map[string]string{"root": "machine's root", "owned": "owned by root", "gone": "not there", "lib": "backup repository"} {
if !strings.Contains(why[name], says) {
t.Errorf("%s is not refused as %q: %q", name, says, why[name])
}
}
if len(served) != 1 || served[0].Name != "ok" || served[0].Access != "ro" || served[0].User != "media" {
t.Errorf("served: %+v", served)
}
if f.machine().Exists("/srv/gone") {
t.Error("the module created a missing folder")
}
}
func TestBeforeAdoptionTheHandWrittenConfigurationIsKeptAndNothingIsWritten(t *testing.T) {
f := newFake(t)
f.home("media=/storage/media", "media")
s := f.machine().Pass(context.Background())
if s.Mode != "found" || f.read(SambaConf) != handWritten {
t.Fatalf("mode %s; the hand-written configuration changed:\n%s", s.Mode, f.read(SambaConf))
}
if len(s.Hand) != 1 || s.Hand[0].Counterpart != "media" || s.Hand[0].ReadOnly {
t.Errorf("the hand-written share is not matched: %+v", s.Hand)
}
for _, r := range f.ran {
if strings.HasPrefix(r, "smbcontrol") || strings.HasPrefix(r, "systemctl disable") || strings.HasPrefix(r, "smbpasswd") {
t.Errorf("before adoption the process ran %q", r)
}
}
if s.Logins[0].Password != "kept as found" {
t.Errorf("the login: %+v", s.Logins)
}
if h := f.machine().Health(); !h.Healthy || !strings.Contains(h.Why, "adopt") {
t.Errorf("health before adoption: %+v", h)
}
}
func TestAdoptRefusesWhileAHandWrittenShareHasNoCounterpart(t *testing.T) {
f := newFake(t)
f.home("library=/storage/media", "media")
f.machine().Pass(context.Background())
if _, err := f.machine().Adopt(true, "take it over"); err == nil || !strings.Contains(err.Error(), "no share disappears") {
t.Fatalf("adopt with a hand-written share the settings do not name: %v", err)
}
if f.machine().Exists(AdoptFile) {
t.Error("a refused adopt left the word for the process")
}
f.file(ConfigFile, "shares=media=/storage/media\nusers=media\npasswords=/x\n")
f.machine().Pass(context.Background())
dry, err := f.machine().Adopt(false, "")
if err != nil || !dry.DryRun || len(dry.Beside) != 1 || !strings.Contains(dry.Beside[0], "every client as media:media") {
t.Fatalf("dry run: %+v %v", dry, err)
}
if f.machine().Exists(AdoptFile) {
t.Error("a dry run left the word for the process")
}
if _, err := f.machine().Adopt(true, ""); err == nil {
t.Error("adopt with confirm and no why was done")
}
}
func TestAdoptionKeepsADatedCopyChecksAndThenStopsNetBIOS(t *testing.T) {
f := newFake(t)
f.home("media=/storage/media", "media")
f.machine().Pass(context.Background())
f.file(AdoptFile, `{"at":"2026-10-13T09:31:00Z","why":"the operator keeps the share"}`)
f.ran = nil
s := f.machine().Pass(context.Background())
if s.Mode != "adopted" || len(s.Problems) != 0 {
t.Fatalf("status: %+v", s)
}
copyAt := HandCopyPrefix + "20261013T093000Z"
if f.read(copyAt) != handWritten {
t.Errorf("the dated copy is not the hand-written configuration: %q", f.read(copyAt))
}
if !strings.HasPrefix(f.read(SambaConf), Marker) || !strings.Contains(f.read(SambaConf), "force user = media") {
t.Errorf("the module's configuration is not in place:\n%s", f.read(SambaConf))
}
want := []string{"testparm -s " + Candidate, "smbcontrol smbd reload-config", "systemctl disable --now nmb.service"}
for _, w := range want {
if len(f.ran1(w)) != 1 {
t.Errorf("%q ran %d times: %v", w, len(f.ran1(w)), f.ran)
}
}
if !s.Shares[0].Live || s.NetBIOS != "inactive" {
t.Errorf("served: %+v netbios %s", s.Shares, s.NetBIOS)
}
// The next pass finds its own configuration and writes nothing again.
f.ran = nil
f.machine().Pass(context.Background())
if len(f.ran1("smbcontrol")) != 0 || len(f.ran1("systemctl disable")) != 0 {
t.Errorf("an unchanged configuration was written or reloaded again: %v", f.ran)
}
}
func TestAConfigurationFailingTestparmIsKeptOutOfUse(t *testing.T) {
f := newFake(t)
f.home("media=/storage/media", "media")
f.file(AdoptFile, `{"at":"2026-10-13T09:31:00Z","why":"take over"}`)
f.answers["testparm"] = Ran{Code: 1, Err: "Unknown parameter encountered"}
s := f.machine().Pass(context.Background())
if f.read(SambaConf) != handWritten {
t.Errorf("a configuration testparm refused took the hand-written one's place")
}
if len(f.ran1("smbcontrol")) != 0 || !strings.Contains(strings.Join(s.Problems, " "), "fails Samba's own check") {
t.Errorf("not said, or reloaded: %v %v", s.Problems, f.ran)
}
if h := f.machine().Health(); h.Healthy {
t.Errorf("healthy with a refused configuration: %+v", h)
}
}
func TestALoginKeepsItsPasswordWithoutASecretWaitsWhenNewAndIsSetOnce(t *testing.T) {
f := newFake(t)
f.home("media=/storage/media", "media guest2")
f.answers["pdbedit -L -u guest2"] = Ran{Code: 255}
f.file(AdoptFile, `{"at":"2026-10-13T09:31:00Z","why":"take over"}`)
s := f.machine().Pass(context.Background())
got := map[string]string{}
for _, l := range s.Logins {
got[l.Name] = l.Password
}
if got["media"] != "kept as found" || got["guest2"] != "waits" || len(f.ran1("smbpasswd")) != 0 {
t.Fatalf("logins %v; smbpasswd ran %v", got, f.ran1("smbpasswd"))
}
h := f.machine().Health()
if h.State != "waits" || len(h.Waits) != 1 || h.Waits[0].Secret != "smb-password-guest2" {
t.Errorf("health: %+v", h)
}
// The secret given: set once from stdin, never on the command line, and not again while unchanged.
f.file("/etc/passwd", f.read("/etc/passwd")+"guest2:x:1002:2000::/:/bin/sh\n")
f.file("/var/lib/mesh/smb-server/state/smb-password-guest2.secret", "s3cret\n")
f.machine().Pass(context.Background())
runs := f.ran1("smbpasswd")
if len(runs) != 1 || runs[0] != "smbpasswd -s -a guest2" || f.inputs[runs[0]] != "s3cret\ns3cret\n" {
t.Fatalf("smbpasswd: %v %q", runs, f.inputs)
}
f.answers["pdbedit -L -u guest2"] = Ran{Out: "guest2:1002:\n"}
f.machine().Pass(context.Background())
if len(f.ran1("smbpasswd")) != 1 {
t.Errorf("an unchanged password was set again: %v", f.ran1("smbpasswd"))
}
var set map[string]string
_ = json.Unmarshal([]byte(f.read(SetFile)), &set)
if strings.Contains(f.read(SetFile), "s3cret") || set["guest2"] == "" {
t.Errorf("the record of set passwords holds the password, or nothing: %s", f.read(SetFile))
}
if info, err := os.Stat(filepath.Join(f.root, SetFile)); err != nil || info.Mode().Perm() != 0o600 {
t.Errorf("the record of set passwords is not root's alone: %v", info.Mode())
}
}
func TestTheModuleWritesOnlyItsOwnFiles(t *testing.T) {
m := newFake(t).machine()
for _, p := range []string{"/etc/passwd", "/storage/media/x", "/etc/samba/other.conf", HandCopyPrefix + "x/../../evil"} {
if err := m.Write(p, "x", 0o644); err == nil {
t.Errorf("wrote %s", p)
}
}
}
func TestParseSambaReadsSectionsAndAccess(t *testing.T) {
secs := ParseSamba("; c\n[global]\nworkgroup = W\n[a]\n path = /x\n read only = no\n[b]\n path = /y\n[c]\n path = /z\n writeable = yes\n")
if len(secs) != 4 || secs[1].Params["path"] != "/x" {
t.Fatalf("%+v", secs)
}
if readOnly(secs[1].Params) || !readOnly(secs[2].Params) || readOnly(secs[3].Params) {
t.Error("read only is not read as Samba reads it")
}
}
+366
View File
@@ -0,0 +1,366 @@
package main
import (
"context"
"encoding/json"
"fmt"
"strings"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/health"
)
// Seat is the role this module holds on its machine (novox/hq ADR 0341).
const Seat = "node-smb-server"
// Stale is how old the process's last look may be before the tools say it stopped looking.
const Stale = 5 * time.Minute
func verb(name, description string, input map[string]any, run func(a map[string]any) (any, error)) stdio.Tool {
return stdio.Tool{Name: Seat + "." + name, Description: description, Input: input, Run: run}
}
func str(description string) map[string]any {
return map[string]any{"type": "string", "description": description}
}
func arg(a map[string]any, k string) string {
v, _ := a[k].(string)
return strings.TrimSpace(v)
}
// confirmed says whether an act was confirmed: the boolean true, or the string "true" the seat's schema passes.
func confirmed(a map[string]any) bool {
switch v := a["confirm"].(type) {
case bool:
return v
case string:
return strings.EqualFold(strings.TrimSpace(v), "true")
}
return false
}
// Tools are the seat's verbs and the module's health.
func Tools(m *Machine) []stdio.Tool {
return []stdio.Tool{
verb("shares", "Every SMB share this machine serves: its name, folder, rw or ro, the owner every client acts as "+
"(uid, gid, names), the logins allowed and whether each login's password was set from its secret, is kept "+
"as found or waits for it, and whether Samba serves it now; whether the configuration is the mesh's or still "+
"the hand-written one, and each hand-written share with whether the settings take its place; the shares "+
"refused, and why. A password is never shown. Replaces testparm -s, cat /etc/samba/smb.conf and pdbedit -L. (r)",
map[string]any{}, func(map[string]any) (any, error) { return m.Shares(), nil }),
verb("sessions", "Which devices are connected to which share now, with the login and protocol each uses, as "+
"Samba knows them (smbstatus, as root). Replaces smbstatus. (r)",
map[string]any{}, func(map[string]any) (any, error) { return m.Sessions(context.Background()) }),
verb("test", "Whether this machine serves one share now: asked for and not refused, served by Samba, Samba "+
"active, and 445 listening. Replaces smbclient -L. (r)",
map[string]any{"share": str("the share's name")},
func(a map[string]any) (any, error) { return m.Test(context.Background(), arg(a, "share")) }),
verb("reload", "Have Samba read its configuration again now (smbcontrol smbd reload-config, as root). The "+
"module's process does this itself whenever it writes the configuration; this is for after a change made "+
"outside it. Changes no shared folder. Replaces smbcontrol smbd reload-config. (a)",
map[string]any{}, func(map[string]any) (any, error) { return m.Reload(context.Background()) }),
verb("adopt", "Take over Samba's hand-written configuration: the module's process keeps a dated copy of it "+
"beside it and writes the mesh's own in its place, checked with testparm, then stops NetBIOS. Only when every "+
"hand-written share has its counterpart in the setting shares (same name, same folder), so no share "+
"disappears. Without confirm, a dry run: each hand-written share beside the module's rendering of it. A "+
"person's act, with why. Replaces edit /etc/samba/smb.conf. (a)",
map[string]any{
"confirm": str("\"true\" to do it; anything else, a dry run"),
"why": str("why, in a sentence (required with confirm)"),
},
func(a map[string]any) (any, error) { return m.Adopt(confirmed(a), arg(a, "why")) }),
{
Name: "smb_health",
Description: "Whether this machine's SMB shares are served, as the module's health check (novox/hq ADR 0240, " +
"ADR 0341): unhealthy when Samba is not active, the module's process has not looked for five minutes, a " +
"share is refused or not served, or the configuration fails Samba's check; waits when a login's password " +
"is its secret not yet given; with why. (r)",
Run: func(map[string]any) (any, error) { return m.Health(), nil },
},
}
}
// SharesAnswer is what `shares` answers.
type SharesAnswer struct {
Mode string `json:"configuration"`
Shares []Served `json:"shares"`
Refused []Refused `json:"refused,omitempty"`
Hand []HandShare `json:"hand_written,omitempty"`
Logins []Login `json:"logins,omitempty"`
Service string `json:"service"`
Looked string `json:"looked,omitempty"`
Said []string `json:"said"`
Notes []string `json:"notes,omitempty"`
}
// Shares is the machine's SMB shares now.
func (m *Machine) Shares() SharesAnswer {
a := SharesAnswer{Shares: []Served{}, Service: m.UnitState(Unit)}
st, err := m.LastStatus()
if err != nil {
a.Notes = append(a.Notes, err.Error())
a.Said = append(a.Said, "the module's process has not looked yet")
return a
}
a.Mode, a.Shares, a.Refused, a.Hand, a.Logins, a.Looked = st.Mode, st.Shares, st.Refused, st.Hand, st.Logins, st.At
a.Notes = st.Problems
if st.Mode == "found" {
a.Said = append(a.Said, "the configuration is the hand-written one, kept as found; the module takes it over by adopt, a person's act")
} else {
a.Said = append(a.Said, "the configuration is the module's")
}
for _, s := range st.Shares {
state := "served"
if !s.Live {
state = "NOT served"
}
a.Said = append(a.Said, fmt.Sprintf("%s: %s, %s, every client as %s:%s, for %s — %s", s.Name, s.Path, s.Access, s.User, s.Group, strings.Join(s.Users, " "), state))
}
for _, h := range st.Hand {
if h.Counterpart != "" {
a.Said = append(a.Said, fmt.Sprintf("hand-written %s (%s): the setting's share %s takes its place", h.Name, h.Path, h.Counterpart))
} else {
a.Said = append(a.Said, fmt.Sprintf("hand-written %s (%s): %s", h.Name, h.Path, h.Why))
}
}
for _, l := range st.Logins {
a.Said = append(a.Said, fmt.Sprintf("login %s: password %s", l.Name, l.Password))
}
for _, r := range st.Refused {
a.Said = append(a.Said, "refused: "+r.Why)
}
a.Said = append(a.Said, "Samba is "+a.Service)
return a
}
// Sessions is what smbstatus says of the connected devices.
func (m *Machine) Sessions(ctx context.Context) (any, error) {
r, err := m.AsRoot(ctx, "", "smbstatus", "--json")
if err != nil {
return nil, err
}
var v struct {
Sessions map[string]struct {
Machine string `json:"remote_machine"`
Username string `json:"username"`
Dialect string `json:"session_dialect"`
} `json:"sessions"`
Tcons map[string]struct {
Service string `json:"service"`
Machine string `json:"machine"`
} `json:"tcons"`
}
if err := json.Unmarshal([]byte(r.Out), &v); err != nil {
return nil, fmt.Errorf("smbstatus answered what this cannot read: %v", err)
}
type session struct {
Device string `json:"device"`
Login string `json:"login"`
Protocol string `json:"protocol"`
Shares []string `json:"shares"`
}
var out []session
for _, s := range v.Sessions {
se := session{Device: s.Machine, Login: s.Username, Protocol: s.Dialect}
for _, t := range v.Tcons {
if t.Machine == s.Machine {
se.Shares = append(se.Shares, t.Service)
}
}
out = append(out, se)
}
return map[string]any{"sessions": out}, nil
}
// TestAnswer is what `test` answers.
type TestAnswer struct {
Share string `json:"share"`
Served bool `json:"served"`
Service string `json:"service"`
Listening bool `json:"listening_445"`
Said string `json:"said"`
}
// Test says whether one share is served now.
func (m *Machine) Test(ctx context.Context, share string) (TestAnswer, error) {
if share == "" {
return TestAnswer{}, fmt.Errorf("name the share")
}
a := TestAnswer{Share: share, Service: m.UnitState(Unit)}
r := m.Run(ctx, "", "ss", "-ltnH", "sport", "=", ":445")
a.Listening = strings.TrimSpace(r.Out) != ""
st, err := m.LastStatus()
if err != nil {
return a, err
}
for _, s := range st.Shares {
if s.Name == share {
a.Served = s.Live && a.Service == "active" && a.Listening
}
}
for _, rf := range st.Refused {
if rf.Name == share {
a.Said = rf.Why
return a, nil
}
}
switch {
case a.Served:
a.Said = fmt.Sprintf("%s is served on 445", share)
case st.Mode == "found":
a.Said = fmt.Sprintf("%s is not served by the module: the configuration is still the hand-written one", share)
default:
a.Said = fmt.Sprintf("%s is not served now: Samba is %s, 445 listening: %v", share, a.Service, a.Listening)
}
return a, nil
}
// Reload has Samba read its configuration again.
func (m *Machine) Reload(ctx context.Context) (any, error) {
if _, err := m.AsRoot(ctx, "", "smbcontrol", "smbd", "reload-config"); err != nil {
return nil, err
}
return m.Shares(), nil
}
// AdoptAnswer is what `adopt` answers.
type AdoptAnswer struct {
DryRun bool `json:"dry_run"`
Done bool `json:"done"`
Copy string `json:"copy,omitempty"`
Beside []string `json:"beside"`
Said string `json:"said"`
}
// AdoptWait is how long a confirmed adopt waits for the process to take over.
var AdoptWait = 45 * time.Second
// Adopt leaves a person's word for the process to take over, after a dry run that shows each hand-written share
// beside its rendering. It refuses while a hand-written share has no counterpart.
func (m *Machine) Adopt(confirm bool, why string) (AdoptAnswer, error) {
a := AdoptAnswer{DryRun: !confirm}
st, err := m.LastStatus()
if err != nil {
return a, err
}
if st.Mode == "adopted" {
return a, fmt.Errorf("the configuration is already the module's; nothing to take over")
}
served := map[string]Served{}
for _, s := range st.Shares {
served[s.Name] = s
}
var missing []string
for _, h := range st.Hand {
if h.Counterpart == "" {
missing = append(missing, fmt.Sprintf("%s (%s): %s", h.Name, h.Path, h.Why))
a.Beside = append(a.Beside, fmt.Sprintf("hand-written %s, %s, valid users %s → no counterpart", h.Name, h.Path, h.ValidUsers))
continue
}
s, ok := served[h.Counterpart]
if !ok {
missing = append(missing, fmt.Sprintf("%s: the setting's share %s is refused", h.Name, h.Counterpart))
continue
}
a.Beside = append(a.Beside, fmt.Sprintf("hand-written %s, %s, valid users %s, read only %v → the mesh's %s, %s, %s, valid users %s, every client as %s:%s",
h.Name, h.Path, h.ValidUsers, h.ReadOnly, s.Name, s.Path, s.Access, strings.Join(s.Users, " "), s.User, s.Group))
}
if len(missing) > 0 {
return a, fmt.Errorf("not taken over, so no share disappears: %s", strings.Join(missing, "; "))
}
plan := "keep a dated copy of the hand-written configuration beside it, write the mesh's own in its place once testparm passes it, have Samba read it, and stop NetBIOS"
if !confirm {
a.Said = "Dry run: would " + plan + "."
return a, nil
}
if why == "" {
return a, fmt.Errorf("say why: adopt is a person's act, and the why is kept with it")
}
raw, _ := json.MarshalIndent(Adoption{At: m.Now().UTC().Format(time.RFC3339), Why: why}, "", " ")
if err := m.Write(AdoptFile, string(raw)+"\n", 0o644); err != nil {
return a, err
}
deadline := m.Now().Add(AdoptWait)
for {
if st, err := m.LastStatus(); err == nil && st.Mode == "adopted" && st.Written != "" {
a.Done = true
if st.Adopted != nil {
a.Copy = st.Adopted.Copy
}
a.Said = fmt.Sprintf("Did: %s. Why: %s", plan, why)
return a, nil
}
if !m.Now().Before(deadline) {
a.Said = "The word is left for the module's process, which has not taken over yet; read shares for why."
return a, nil
}
time.Sleep(2 * time.Second)
}
}
// HealthAnswer is the module's health: mesh-sdk's answer, with what was found.
type HealthAnswer struct {
health.Answer
Found []string `json:"found,omitempty"`
}
// Health judges the shares.
func (m *Machine) Health() HealthAnswer {
var found []string
if s := m.UnitState(Unit); s != "active" {
found = append(found, fmt.Sprintf("Samba (%s) is %s.", Unit, s))
}
st, err := m.LastStatus()
if err != nil {
found = append(found, err.Error()+".")
return HealthAnswer{Answer: health.Unhealthy(strings.Join(found, " ")), Found: found}
}
if at, err := time.Parse(time.RFC3339, st.At); err != nil || m.Now().Sub(at) > Stale {
found = append(found, "The module's process has not looked for five minutes.")
}
for _, p := range st.Problems {
found = append(found, sentence(p))
}
for _, r := range st.Refused {
found = append(found, r.Why)
}
if st.Mode == "adopted" {
for _, s := range st.Shares {
if !s.Live {
found = append(found, fmt.Sprintf("The share %s (%s) is not served by Samba.", s.Name, s.Path))
}
}
}
if len(found) > 0 {
return HealthAnswer{Answer: health.Unhealthy(strings.Join(found, " "), st.Waits...), Found: found}
}
if len(st.Waits) > 0 {
var why []string
for _, w := range st.Waits {
why = append(why, fmt.Sprintf("%s waits for %s.", sentence(w.Part), w.What))
}
return HealthAnswer{Answer: health.Waiting(strings.Join(why, " "), st.Waits...)}
}
if st.Mode == "found" {
return HealthAnswer{Answer: health.Healthy("Samba serves the hand-written configuration, kept as found; the module takes it over by node-smb-server.adopt, a person's act.")}
}
var parts []string
for _, s := range st.Shares {
parts = append(parts, s.Name)
}
return HealthAnswer{Answer: health.Healthy(fmt.Sprintf("Samba is active and serves %s on 445.", strings.Join(parts, ", ")))}
}
func sentence(s string) string {
s = strings.TrimSpace(s)
if s == "" {
return s
}
s = strings.ToUpper(s[:1]) + s[1:]
if !strings.HasSuffix(s, ".") {
s += "."
}
return s
}
+5
View File
@@ -0,0 +1,5 @@
module smbserver
go 1.22
require git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261010191001-33917f91ac3c
+8
View File
@@ -0,0 +1,8 @@
git.novox.be/novox/mesh-sdk/go v0.1.6 h1:9qzdYONYbJdWcu6sxQcq9v1LI0JxcfkiKYkMUzJSkVQ=
git.novox.be/novox/mesh-sdk/go v0.1.6/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
git.novox.be/novox/mesh-sdk/go v0.1.7 h1:C0sTQmtTiyYH7bnqZb7PusXnqA37gKuT7Nqjn9gG47w=
git.novox.be/novox/mesh-sdk/go v0.1.7/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261010190644-622b6b4f556d h1:BR8uwv4BqyLaBwITuLylrd9DRUnu0D5DjJw0RuQSXbk=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261010190644-622b6b4f556d/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261010191001-33917f91ac3c h1:l5onJwoIeH8yE/PjVlEeoPyXBsHp2NQiWLllz2fwX7s=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261010191001-33917f91ac3c/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
+144
View File
@@ -0,0 +1,144 @@
{
"module": "smb-server",
"version": "1",
"upgrade": {
"policy": "record",
"why": "the folders the devices on the home network open: a build that breaks Samba's configuration leaves every device's share refused, and the gate on this one machine does not see the devices (hq ADR 0236, ADR 0341)"
},
"capabilities": [
"package-manager",
"service-manager"
],
"claims": [
{
"name": "node-smb-server",
"scope": "node",
"serves": [
"shares",
"sessions",
"test",
"reload",
"adopt"
]
}
],
"tools": [
"smb_health"
],
"settings": {
"shares": {
"kind": "preference",
"default": "none",
"why": "a machine serves no SMB share until its assignment names one: space-separated name=folder, each optionally :ro, one share per folder, every client acting as the folder's owner (hq ADR 0341)"
},
"users": {
"kind": "preference",
"default": "none",
"why": "the logins allowed on every share, space-separated; each login's password is the secret smb-password-<login>, and a login already in Samba's database whose secret is not given keeps its password (hq ADR 0341)"
}
},
"own-secrets": {
"smb-password-*": {
"path": "${dir:state}/smb-password-*.secret",
"issued-by": "outside"
}
},
"listens": [
{
"name": "smb",
"port": 445,
"protocol": "tcp",
"from": "mesh",
"fixed": true,
"why": "SMB 2.1 and later, on 445 alone: NetBIOS is off, so 137, 138 and 139 are never opened. The home server's assignment binds it with the reach local-network (hq ADR 0340, ADR 0341), so the devices on its own network reach it and nothing beyond"
}
],
"resources": [
{
"id": "package",
"type": "package",
"package": "samba"
},
{
"id": "state",
"type": "directory",
"mode": "0700",
"place": "."
},
{
"id": "config-dir",
"type": "directory",
"path": "/etc/smb-server",
"mode": "0755"
},
{
"id": "run-dir",
"type": "directory",
"path": "/run/smb-server",
"mode": "0755"
},
{
"id": "lib-dir",
"type": "directory",
"path": "/var/lib/smb-server",
"mode": "0755"
},
{
"id": "adopt-dir",
"type": "directory",
"path": "/var/lib/smb-server/adopt",
"mode": "0755",
"owner": "${machine:account}"
},
{
"id": "config",
"type": "file",
"path": "/etc/smb-server/shares.conf",
"mode": "0644",
"content": "# Written by the mesh (module smb-server, novox/hq ADR 0341) from this machine's assignment.\n# Replaced on every push; change the settings, never this file.\n#\n# The shares: name=folder, or name=folder:ro, one share per folder. The users: the logins allowed. The\n# module's process serves each share with every client as the folder's owner, and writes Samba's\n# configuration only once a person adopted it (node-smb-server.adopt).\nshares=${setting:shares}\nusers=${setting:users}\npasswords=${dir:state}\n"
},
{
"id": "server",
"type": "service",
"unit": "smb.service",
"state": "running",
"boot": "enabled",
"health": {
"kind": "unit"
}
},
{
"id": "config-process",
"type": "process",
"name": "smb-server-config",
"artifact": "tools",
"run": [
"./smb-server",
"config"
],
"health": {
"kind": "tool",
"tool": "smb_health",
"interval": "60s",
"timeout": "10s",
"looks": 2,
"grace": "90s"
}
}
],
"build": {
"artifacts": [
{
"name": "tools",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/smb-server",
"binary": "smb-server",
"loads": [
"smb-server"
]
}
]
}
}
+2 -1
View File
@@ -131,7 +131,8 @@
"path": "${dir:mesh-state}/config.json",
"mode": "0600",
"content": "{\n \"site\": \"default\",\n \"password\": \"${secret:controller}\"\n}\n",
"merge": "json"
"merge": "json",
"protected": ["password"]
}
],
"requires": [