diff --git a/modules/betterbird/README.md b/modules/betterbird/README.md index 13bf2484..4e2a2acd 100644 --- a/modules/betterbird/README.md +++ b/modules/betterbird/README.md @@ -1,10 +1,12 @@ # betterbird Betterbird, the mail and calendar client of g14 and shanks (a Thunderbird fork, with the operator's -Exchange calendar through the Owl add-on), as a module (novox/hq issue 345, ADR 0208, ADR 0316, ADR 0317, -to-be 56). It requires `x11-display`, so it is assigned only where a display server is held on the same -machine. It installs Betterbird from a pinned upstream release, ships Owl with it through Betterbird's -enterprise policy, and declares the operator's mail accounts without their credentials. +Exchange calendar through the Owl add-on), as a module (novox/hq issue 345, issue 586, ADR 0208, ADR 0316, +ADR 0317, ADR 0329, to-be 56). It requires `x11-display`, so it is assigned only where a display server is held +on the same machine. It installs Betterbird from a pinned upstream release, ships Owl with it through +Betterbird's enterprise policy, and owns Betterbird's whole configuration as settings: the preferences, the +mail accounts without their credentials, the identities with their signatures, the CalDAV calendars, the +CardDAV address books, the filters and the tags. It exists because a symptom had no tools: on a workstation, a calendar reminder appeared as a transparent window with nothing in it (2026-10-09). Nothing in the mesh could say which window that @@ -18,13 +20,14 @@ issue 345. | Betterbird, the upstream release (`archive`, pinned by URL and sha256) | `/opt/betterbird` | owned, whole | | the enterprise policy (`install/distribution/policies.json`) | `/opt/betterbird/distribution` | owned, whole | | Owl's xpi (`owl`, pinned by the vendor's sha256) | `/opt/betterbird/owl` | owned, whole | -| the module's defaults (`install/defaults-pref/mesh-defaults.js`) and the spelling dictionaries' default (`vendor-prefs.js`) | `/opt/betterbird/defaults/pref` | owned: its two files | +| the module's defaults (`install/defaults-pref/mesh-defaults.js`: the preferences kept at Betterbird's own default, no value of the operator's) and the spelling dictionaries' default (`vendor-prefs.js`) | `/opt/betterbird/defaults/pref` | owned: its two files | | the tools' program, for the start's preparation | `/usr/local/lib/mesh-betterbird` | owned, whole | | the start | `/usr/local/bin/betterbird` | owned, whole | | the launcher entry | `/usr/local/share/applications/eu.betterbird.Betterbird.desktop` | owned, whole | -| the module's directory, with its `user.js` (setting `prefs`), `accounts.json` (setting `accounts`) and the record of adopted and created accounts | `~/.config/betterbird/` | owned, whole | +| the module's directory, with the settings as the node-engine writes them (`prefs.json`, `accounts.json`, `configuration.json`, `tags.json`), the records of what was adopted or created, and the filter files it replaced (`kept/`) | `~/.config/betterbird/` | owned, whole | | the profile's `user.js`, placed by the tools | the profile | the module's, or never touched | -| the account lists in the profile's `prefs.js` | the profile | written into: the module's keys only | +| the lists in the profile's `prefs.js` (accounts, outgoing servers, an adopted account's identities, the calendars' order), and the preferences of what the module created and the settings no longer hold | the profile | written into: the module's keys only, only while Betterbird is closed | +| each declared account's `msgFilterRules.dat` (setting `filters`) | the account's directory in the profile | owned, whole, only while Betterbird is closed | | the profile's own values of the preferences the module manages, in `prefs.js` | the profile | cleared only by `betterbird_config_apply` with apply, Betterbird quit | | the profile: mail, address books, calendars, logins | `~/.thunderbird/` | found (ADR 0182) | @@ -69,97 +72,85 @@ box out in the settings. `betterbird_check` says when Betterbird would ask (novo **No start of its own.** Betterbird is started by the operator from the launcher. The start (`/usr/local/bin/betterbird`) first runs `betterbird-tools prepare`, which, with no Betterbird of the -account running, places the user.js and writes the created accounts into the account lists; then it +account running, places the user.js and writes what Betterbird rewrites itself (the lists, the removals, the +filter files); then it starts Betterbird either way. `betterbird_restart` starts it as the transient user unit `mesh-betterbird`, so its output then goes to the journal. -## The configuration this module ships +## The preferences: locks, the setting prefs, and Betterbird's own defaults -Betterbird's configuration comes in three layers (novox/hq issue 345). The operator approved the set on -2026-10-12. It is the laptop's setup, with the start page locked off, reminders missed while asleep shown at the -next start, and chat off. +Betterbird's preferences come in three layers (novox/hq issue 345, issue 586). **The operator's own choices +live in the module's configuration, never in the catalogue**: the module ships only its policy's locks and the +names it keeps at Betterbird's own default. -| layer | file | who changes it | holds | +| layer | where | who changes it | holds | |---|---|---|---| | **locked** | `/opt/betterbird/distribution/policies.json` (`install/distribution/`) | only a reviewed catalogue change | what must never differ on a node; Betterbird's settings show it greyed out | -| **default** | `/opt/betterbird/defaults/pref/mesh-defaults.js` (`install/defaults-pref/`) | a catalogue change; or the operator in Betterbird's settings, where the change lands in the profile's `prefs.js` | the operator's setup: what a fresh node gets without anyone touching it | -| **node** | the setting `prefs`, written to `~/.config/betterbird/user.js` and placed in the profile's `user.js` | the operator, per node, through the settings layers | a node's exceptions, such as a font size for a larger screen | +| **declared** | the setting `prefs`, mesh-wide with a node's layer over it, written to `~/.config/betterbird/prefs.json` and placed in the profile's `user.js` | the operator, through the settings layers (trusted: the terminal or the operator's warrant) | the operator's setup, and a node's exceptions, such as a font size for a larger screen | +| **Betterbird's default** | Betterbird's own defaults; `mesh-defaults.js` names those the module manages there | a catalogue change | the preferences kept at Betterbird's default | -A locked value wins over everything. A node layer (`user.js`) wins over the profile's `prefs.js`, and -`prefs.js` wins over a default. +A locked value wins over everything; a declared one (in `user.js`) wins over the profile's `prefs.js` at every +start; `prefs.js` wins over a default. + +**The setting `prefs` is one entry per preference** (issue 586): `{"prefs": {"mail.uifontsize": 14, …}}`, the +value true, false, a whole number or a string. The node-engine merges the layers key by key, so a node's layer +overrides one preference, or takes one out with `null`. The operator's setup approved on 2026-10-12 (compact +density, 14-point text, the vertical view with multi-line cards and threads, no new-mail pop-up, sound or tray +icon, the calendar's day from 09:00 to 20:00, reminders missed while asleep shown, chat off, remote content +blocked, Do Not Track; 21 preferences) moved from the catalogue's defaults file into the mesh-wide layer of +`prefs`, set on the operator's Approve. + +**The order that leaves no node on Betterbird's own defaults** (issue 586): merge this build (the `record` +upgrade policy holds it back from the nodes); propose the mesh-wide layer through the controller's `settings` +verb with `propose`, carrying the current `accounts` verbatim and `prefs` with the 21 values; the operator +approves; only then push the desktop nodes. Caveat: between that approval and a node's push, any other send to a +desktop node writes the map's text into the old build's user.js, and `betterbird_check` there says the user.js is +refused until the push. **Locked** (the policy): Betterbird's updater off (`DisableAppUpdate`); telemetry off (`DisableTelemetry`); add-ons change only through the mesh's pins (`ExtensionUpdate` false); and three preferences locked off: `mail.shell.checkDefaultClient` (the ask to be the default client), `mail.inappnotifications.enabled` (the vendor's in-app notices), and `mailnews.start_page.enabled` (the start page, which otherwise loads from the -vendor at every start). The `Preferences` policy accepts only some prefixes. `privacy.` and `toolkit.` are not -among them, so such a preference can only be a default. +vendor at every start). -**Default** (`mesh-defaults.js`, each line commented with why): +**Kept at Betterbird's default** (`// keeps Betterbird's default: ` in `mesh-defaults.js`): the theme, +system colours, always-underlined links, manual spam marking, and the composer's attachment pane. A value the +profile saved for one of them is drift. -- display: compact density (`mail.uidensity` 0), 14-point interface text, and message bodies not darkened; -- layout: the vertical view with multi-line rows, the list as cards, threads grouped; -- no confirmation on Shift+Delete; -- new mail: no pop-up, no sound, no tray icon, and alerts through the desktop's notifier if a node turns them on; -- calendar: the day from 09:00 to 20:00, deleting without asking, the node's time zone, reminders shown, and - reminders missed while asleep shown at the next start; -- chat off; -- remote content blocked, and Do Not Track sent. +**Drift.** `betterbird_config` shows each managed preference (locked, declared, kept) through the layers, and +flags **drift**: a declared value the profile's `user.js` does not hold yet (remedy: `betterbird_user_js` with +apply, then `betterbird_restart`), or a value the profile saved where the module keeps Betterbird's default. +`betterbird_check` reports it. -A line `// keeps Betterbird's default: ` names a preference the module manages without a value of its own. -For these the operator's setup is Betterbird's own default: the theme, system colours, always-underlined -links, manual spam marking, and the composer's attachment pane. - -**Drift.** A default does not beat a value the profile saved in `prefs.js`. That value may be a choice made in -the settings window, or one the profile already held. `betterbird_config` shows each managed preference through -the layers, and flags **drift**: a saved value that beats the module's default without a node layer saying so. -`betterbird_check` reports it. There are two remedies: - -- `betterbird_config_apply` with `apply` takes the drifting values out of `prefs.js`, so the module's value - holds at the next start. It acts only while Betterbird is quit (no main process, and no `lock` in the profile), - because Betterbird rewrites `prefs.js` when it quits, and it says what it cleared. Without `apply` it lists - what it would clear. +- `betterbird_config_apply` with `apply` takes a kept preference's saved value out of `prefs.js`, so Betterbird's + default holds at the next start. It acts only while Betterbird is quit (no main process, and no `lock` in the + profile), and says what it cleared. - **The theme is the exception.** Betterbird keeps the active theme in its add-on database (`extensions.json`), - and `extensions.activeThemeID` only mirrors it. So the tools read the theme from the add-on database, and the - module's value is `default-theme@mozilla.org`. Another theme is drift whose remedy is to enable the default + and `extensions.activeThemeID` only mirrors it. Another theme is drift whose remedy is to enable the default theme in the add-on manager. `betterbird_config_apply` never clears it, and `keep` refuses it. +- `betterbird_config` with `keep` proposes this node's layer of `prefs` holding the profile's values of the named + preferences beside what the layer holds, through the controller's settings verb, for the operator's Approve. -**Never an account.** A name the tools withhold (a credential's, an account's, a server's), or an account's own -preference, is never managed, even when the defaults file lists it. `betterbird_config` leaves it out and names it -in `refused`, `betterbird_config_apply` never clears it, and `betterbird_check` says to take it out of the file. -- `betterbird_config` with `keep` proposes, through the controller's settings verb, a node layer of `prefs` that - holds the profile's value for this machine. The operator approves it on a channel that proves who answers. - The tool never writes `user.js` itself. The setting is one string, so the layer carries what `prefs` says on - this machine today, with the kept values added. +**One owner per value.** `prefs` refuses a preference whose name can hold a credential, an account's +(`mail.account.`, `mail.identity.`, `mail.server.`, `mail.smtpserver.` and the account lists, ADR 0316 §11), and +those the settings below own: `calendar.registry.`, `calendar.list.`, `ldap_2.servers.` and `mailnews.tags.` +(ADR 0329 §6). -## The user.js: how a preference is declared +## The user.js -Betterbird reads `user.js` in its profile when it starts, and a preference set there wins over the -one it saved in `prefs.js`. The profile's directory has a random name per machine -(`.default-default`), which a manifest cannot name. So: - -1. the node-engine writes `~/.config/betterbird/user.js` from the module's setting `prefs`, one line of - `user_pref("name", value);` statements (default: a comment, no preference); -2. the module's tools copy that file whole into the profile: `betterbird_user_js` with `apply`, or - `betterbird_restart`, which places it before it starts Betterbird again; -3. Betterbird reads it at that start. +Betterbird reads `user.js` in its profile when it starts, and a preference set there wins over the one it saved +in `prefs.js`. The profile's directory has a random name per machine (`.default-default`), which a +manifest cannot name. So the tools compose the user.js from the settings — `prefs`, then the declared accounts, +then the declared identities, calendars, address books and tags under the keys each resolved to — and place it +in the profile: at the start from the launcher, `betterbird_user_js` with `apply`, or `betterbird_restart`. **The profile is found** as the one the running Betterbird holds open; with none running, as the installation's default in `profiles.ini` (`[Install…] Default`), else the profile marked `Default=1`. -On the workstation measured these differ: `Profile1` is marked `Default=1`, and the installation's -default, the one in use, is the other. **A user.js the module did not write is never overwritten.** The module's begins with the line -`// Betterbird's user.js (module betterbird`; any other is the operator's, and `apply` refuses with -what to do. The tools refuse a declared file that holds anything but comments and `user_pref` -statements, or that sets a preference whose name can hold a credential. +`// Betterbird's user.js (module betterbird`; any other is the operator's, and `apply` refuses with what to do. -The file is **trusted** (`"trusted": true`): a preference can weaken how the mail client checks a server, -so the setting is changed only at the controller's terminal or proposed and set on the operator's warrant -(novox/hq issue 339, ADR 0277). An account's preferences (`mail.account.`, `mail.identity.`, `mail.server.`, -`mail.smtpserver.` and the account lists) are refused in `prefs`: an account is declared in `accounts`, so -it has one owner (ADR 0316 §11). ## The accounts: declared without credentials (ADR 0316 §10–§16, ADR 0317) The setting `accounts` reaches the machine as `~/.config/betterbird/accounts.json`, a trusted mergeable @@ -205,32 +196,80 @@ lines and its record go, the account stays the person's) and takes a created acc lists (its mail stays in the profile). **The value is derived, not retyped** (§9): `betterbird_accounts` with `propose` builds `accounts` from the -profile (every account but Owl's, no credential) and proposes it through the controller's settings verb; -the operator reads it whole at their desk (`settings proposals `) and approves on a channel that proves -who answers. Its answer to its caller holds counts and kinds only. +profile (every account but Owl's, no credential) and proposes it in the mesh-wide layer as it stands, through +the controller's settings verb; the operator approves on a channel that proves who answers. Its answer shows +the value it proposed (ADR 0329 §10). **A node with no profile yet.** The preparation needs the profile Betterbird makes at its first start: on a node set up from nothing, the declared accounts are written at the second start from the launcher (or at `betterbird_restart`). +## Identities, calendars, address books, filters and tags (ADR 0329) + +Five more settings, mesh-wide with a node's layer over each (an entry set to `null` in a node's layer is not on +that node). `identities`, `calendars`, `address-books` and `filters` reach the machine as +`~/.config/betterbird/configuration.json`, trusted as `accounts` is; `tags` as `tags.json`, not trusted (§5). +**No value holds a password, a token or a cookie**: a key shaped like one is refused, and so is an address whose +user part carries a password (§2). A setting holds no line break, so a signature is a list of its lines. + +| setting | an entry | matched in the profile by | +|---|---|---| +| `identities` | by a name: `account` (its name in `accounts`), and for another identity than the account's default `address` and `name`; `reply-to`, `organization`, `signature` (lines), `signature-html`, `signature-above-quote`; `key` in a node's layer | the account's default (no `address`), or an identity of that account with the address | +| `calendars` | by a name: `name`, `url` (CalDAV), `color`, `read-only`, `reminders`, `user`, `invitations` (an identities entry); `key` in a node's layer | its address | +| `address-books` | by a name: `name`, `url` (CardDAV), `user`; `key` in a node's layer | its address | +| `tags` | by its key: `name`, `color`, `order` | its key | +| `filters` | by an account's name: the ordered list, each with `name`, `enabled`, `when` (new-mail, manual, after-junk, sending, archiving, periodic), `match` (all, any, every-message), `conditions` (`field`, `op`, `value`, in Betterbird's words) and `actions` (`action` in Betterbird's words; `folder` as a path within the account, or `value`) | owned whole per account | + +**Adopted as accounts are** (§7): an entry recorded on this node is kept by its key; one under the module's own +key (`ident_mesh_`, `mesh_` for a calendar or an address book) is its creation; a `key` in the +node's layer is adopted when the profile holds it and it matches; else one match is adopted, none is created +under the module's key, and several are refused with the remedy. The declared fields go into the user.js under +that key. An entry leaving the setting releases what was adopted (its lines go) and removes the preferences of +what was created, never a calendar's events, an address book's contacts or a cache. An entry no setting declares +stays the person's. Owl's accounts and calendars, an ICS calendar (its address is often the secret), local +calendars and address books are never declared (§13). + +**Filters are owned whole per declared account** (§8): the module writes that account's `msgFilterRules.dat` +from the list, each folder's path turned into its address on this node, so a filter added in Betterbird is gone +at the next apply (drift shows it first). Before its first write it keeps the replaced file in +`~/.config/betterbird/kept//`; when the account leaves `filters`, the module stops writing and the file +stays the person's; `betterbird_filters` says where the kept copy is. A filter a setting cannot hold as it is (a +folder of another account, an add-on's action, a news filter) is named and not derived. + +**Written only while Betterbird is closed** (§9): the lists in `prefs.js` and the filter files, because +Betterbird rewrites them as it runs and quits; the user.js is placed as before. `betterbird_check` counts per +kind the changes that wait. + +**Derived, then proposed one layer at a time** (§10): `betterbird_derive` asks every desktop node holding +Betterbird for what its profile holds. An entry on every such node goes into the mesh-wide layer with the value of +the node the derive was called on, and the answer lists each field another node holds differently (which then +shows as drift there); one on some nodes only goes into those nodes' own layers. An account whose filter file +holds no filter is not put in `filters`. With `propose`, the next layer (the mesh-wide one, then each node's) is +proposed with the layer as it stands beside the derived kinds; never while a proposal of the module's waits, and +never while a desktop node does not answer. `prefs` on the call carries values of the setting `prefs` into the +mesh-wide layer. A `key` is never proposed for the mesh-wide layer. ## Tools -They are served by the node's runtime as the operator account (ADR 0175). **No answer carries mail, a -calendar entry, an address, a host, an account's name or a credential**: accounts are said by count, place -and key. +They are served by the node's runtime as the operator account (ADR 0175). **They show the operator's own +configuration with its values, and never a credential** (ADR 0329 §1, §2): every answer passes the module's +credential check, which withholds a field whose name says password, token, secret or cookie and the password +part of any address. No answer carries mail, a calendar entry or a contact. | tool | does | |---|---| -| `betterbird_status` (r) | whether it runs (pid, since, scope, children); what is installed and from where (the mesh's release, or the community package before the switch), its version and build beside the newest published build (Betterbird's own `versions.txt`); the policy; Owl's version against the pin and whether it is active; where its output goes; the profile and how it was found; the user.js (declared, placed, in effect); the declared accounts by count and, per place, adopted, created or refused with their keys; the add-ons in the profile (ids, versions); the preferences that decide how its windows and notifications look | -| `betterbird_prefs` (r) | named preferences or prefixes (ending in `.` or `*`, at most 200; an empty name, `*` or `.` is refused): value and source (the policy's lock, user.js, prefs.js or the default), and the default itself with the file that gives it. Betterbird's own defaults are read from `omni.ja` (`greprefs.js`, `defaults/pref/*.js`) and the installation's `defaults/pref`, the module's from its files there. Of several files, the one first by name holds, as Gecko reads them. A preference compiled into Betterbird is said as built in. Withheld without its value: a name that can hold a credential (`password`, `token`, `oauth`, `signon`, …); one that names an account, server, calendar, recent folder or person (`mail.server.`, `mail.identity.`, `mail.last_msg…`, `messenger.account.`, `mail.cloud_files.`, `calendar.registry.`, and of `mailnews.*` only a name that holds a server, account, identity or tag); and every `extensions.*` but the theme's id (`extensions.activeThemeID`). In every value answered, the home is `~`, and a mail address, a `user%40domain@host` URI and a host name are masked | -| `betterbird_config` (r; a with `keep`) | each managed preference: its lock, the module's default (or Betterbird's, where the module keeps that), the node layer, the profile's saved value, the value in effect, and drift; with `keep`, a node layer holding the profile's values proposed for this machine | -| `betterbird_config_apply` (r; a with `apply`) | the drifting values listed; with `apply`, while Betterbird is quit, taken out of `prefs.js`, each said with the value it had and the one that now holds | -| `betterbird_user_js` (r; a with `apply`) | the declared user.js against the profile's; `apply` places it | -| `betterbird_log` (r) | the last lines of the journal of `mesh-betterbird` (or of the reminder test), addresses and credentials masked (the value after `password=`, `token:` and the like, and the token after `Bearer`/`Basic`). A Betterbird the module did not start prints elsewhere, and the answer says where | -| `betterbird_restart` (a) | ends Betterbird (asked to quit, forced after 8 s), places the user.js where allowed, starts it as `mesh-betterbird`. **Refused while a composer window is open** (X instance `Msgcompose`, whatever its class), because the unsent message would be lost, unless `discard_unsent` is true, which only the operator's yes justifies. It answers within the runtime's 30 s: a start not yet seen after 4 s is said in `note`, and `betterbird_status` shows it | -| `betterbird_check` (r) | installed from the mesh's release and not behind the newest published build; the policy placed, and the module's defaults with it; no managed preference drifting from them; no ask at the start to be the default mail client; Owl at its pin and active; one running; profile found; the user.js on the machine, valid, placed, in effect; the declared accounts in the list, none refused, each signed in once; its output read | -| `betterbird_accounts` (r; a with `propose`) | the profile's accounts by count and kind (declarable, Owl's, others); with `propose`, the setting `accounts` built from them and proposed to the operator (ADR 0317 §9) | +| `betterbird_status` (r) | whether it runs (pid, since, scope, children); what is installed and from where, its version beside the newest published build; the policy; Owl's version against the pin and whether it is active; where its output goes; the profile and how it was found; the user.js (declared, placed, in effect, with every declared entry as it resolved); the declared accounts by count; the add-ons; the preferences that decide how its windows and notifications look | +| `betterbird_prefs` (r) | named preferences or prefixes: value and source, and the default with the file that gives it. Withheld without its value: a name that can hold a credential, one that names an account, server, calendar or person, and every `extensions.*` but the theme's id. Values masked (the home as `~`, addresses and hosts) | +| `betterbird_config` (r; a with `keep`) | each managed preference: its lock, the declared value, Betterbird's default where the module keeps that, the profile's saved value, the value in effect, and drift; with `keep`, this node's layer of `prefs` proposed with the profile's values | +| `betterbird_config_apply` (r; a with `apply`) | a kept preference's drifting values listed; with `apply`, while Betterbird is quit, taken out of `prefs.js` | +| `betterbird_user_js` (r; a with `apply`) | the user.js composed from the settings against the profile's; `apply` places it, writes what Betterbird rewrites while it is closed, and counts what waits | +| `betterbird_log` (r) | the last lines of the journal of `mesh-betterbird` (or of the reminder test), addresses and credentials masked | +| `betterbird_restart` (a) | ends Betterbird (asked to quit, forced after 8 s), places the user.js and writes what waited, starts it as `mesh-betterbird`. **Refused while a composer window is open**, unless `discard_unsent` is true, which only the operator's yes justifies | +| `betterbird_check` (r) | installed from the mesh's release; the policy and the module's defaults placed; no managed preference drifting; no ask to be the default mail client; Owl at its pin and active; one running; profile found; the user.js valid, placed, in effect; the declared accounts in the list, none refused, each signed in once; per kind, the declared entries drifting or refused and the changes waiting for Betterbird to close; its output read | +| `betterbird_accounts` (r; a with `propose`) | the profile's accounts with their values and the declared account each is, Owl's by name and address and marked; with `propose`, `accounts` built from them and proposed in the mesh-wide layer as it stands | +| `betterbird_identities`, `betterbird_calendars`, `betterbird_address_books`, `betterbird_filters`, `betterbird_tags` (r) | the profile's entries with their keys and values beside the declared ones and how each resolved (adopted, created or refused with why); an ICS calendar by its name only, Owl's marked; `betterbird_filters` also where the kept filter files are | +| `betterbird_derive` (r; a with `propose`) | the five kinds derived from the desktop nodes' profiles, per layer, with the fields another node holds differently and what was left out; with `propose`, the next layer proposed; `here` answers this node's profile, as another node asks it | +| `betterbird_drift` (r) | per kind and declared entry: holds, differs (each field with both values), lacks, or refused; and the profile's entries nothing declares | | `betterbird_reminder_test` (a) | a reminder on demand without the operator's calendar (below) | ### A reminder on demand @@ -293,9 +332,9 @@ outside the mesh. With this version, a push to a node (each waits for the operat | `/opt/betterbird` | the mesh's 153.4.0esr-bb10, in the same place, so the same profile opens | | the policy, Owl | placed; Owl installed into the profile by Betterbird at its next start where it is missing | | `/usr/local` | the start, the launcher entry, the tools' program | -| `~/.config/betterbird/` | `user.js`, `accounts.json` (no account until the setting `accounts` is approved) | +| `~/.config/betterbird/` | `prefs.json`, `accounts.json`, `configuration.json`, `tags.json` (each empty until its setting is approved) | | the profile | untouched until the next start from the launcher, `betterbird_user_js` with apply, or `betterbird_restart` | -| the configuration | the policy's locks at Betterbird's next start; the module's defaults wherever the profile saved no value of its own; a saved value is drift, said by `betterbird_check`, until `betterbird_config_apply` or a node layer | +| the configuration | the policy's locks at Betterbird's next start; the settings in the user.js from the next start. Pushed in the order above, the operator's 21 preferences are in `prefs` before the build arrives; pushed before that layer is approved, Betterbird runs with its own defaults for them until it is | ## Relies on @@ -305,4 +344,6 @@ outside the mesh. With this version, a push to a node (each waits for the operat - `systemd-run` and `journalctl` of the account's user manager. - The build seat's archive pinned by URL and sha256 (mesh-controller, ADR 0316 §2): a controller that does not know it refuses this manifest. -- The controller's `settings` verb, for `betterbird_accounts` with `propose` (`invokes`). +- The controller's `settings` verb, for every proposal and to read a layer as it stands; its `nodes` and `node` + verbs, for which machines hold the module; and the module's own `betterbird_derive` on those machines + (`invokes`). diff --git a/modules/betterbird/cmd/betterbird-tools/accounts.go b/modules/betterbird/cmd/betterbird-tools/accounts.go index cefe7ec7..227af178 100644 --- a/modules/betterbird/cmd/betterbird-tools/accounts.go +++ b/modules/betterbird/cmd/betterbird-tools/accounts.go @@ -398,10 +398,6 @@ func unquote(v string) string { // (§12): the declared accounts' keys added at the end, the module's keys of accounts no longer declared taken // out, every other key and line kept as it was. Only while Betterbird is not running. Says what changed. func (m *Machine) writeAccountLists(profile string, resolved []resolvedAccount) (string, error) { - if ps := m.mainProcs(); len(ps) > 0 { - return "", fmt.Errorf("Betterbird is running (pid %d), and it rewrites prefs.js when it quits: the account "+ - "list is written at its next start from the launcher, or by betterbird_restart", ps[0].PID) - } path := filepath.Join(profile, "prefs.js") raw, err := readBounded(path) if err != nil && !errors.Is(err, os.ErrNotExist) { @@ -418,16 +414,15 @@ func (m *Machine) writeAccountLists(profile string, resolved []resolvedAccount) if !changed { return "unchanged", nil } + // Only while Betterbird is closed: no main process and no lock on the profile (novox/hq ADR 0316 §12, ADR 0329 §9). + if why := m.closedFor(profile); why != "" { + return "", errors.New(why + ": the account list is written at its next start from the launcher, or by betterbird_restart") + } mode := os.FileMode(0o600) if st, err := os.Stat(path); err == nil { mode = st.Mode().Perm() } - tmp := path + ".mesh-new" - if err := os.WriteFile(tmp, []byte(out), mode); err != nil { - return "", err - } - if err := os.Rename(tmp, path); err != nil { - os.Remove(tmp) + if err := writeAtomic(path, []byte(out), mode); err != nil { return "", err } return fmt.Sprintf("written: %d account(s) the module created in the account list", len(wantAccounts)), nil @@ -608,11 +603,15 @@ func withoutSignIn(accounts map[string]Account, kept map[string]bool) []string { return out } -// AccountsAnswer is what betterbird_accounts answers its caller: counts and kinds, never an address, a host or -// a name (novox/hq ADR 0317 §9). The value itself goes only to the operator, in a proposal they read at -// their own desk. +// AccountsAnswer is what betterbird_accounts answers its caller: the profile's accounts with the operator's own +// values (novox/hq ADR 0329 §1, §12), Owl's marked and by name and address only (§13), and counts and kinds. +// Never a password, a token or anything of the login store (§2). type AccountsAnswer struct { Profile string `json:"profile"` + // Accounts are the profile's accounts, in its list's order. + Accounts []listedAccount `json:"accounts"` + // Derived is the value of the setting accounts the propose built, as it was proposed (ADR 0329 §10). + Derived map[string]Account `json:"derived,omitempty"` // Declarable are the profile's mail accounts a declaration can hold (IMAP and POP3), by kind. Declarable map[string]int `json:"declarable"` // KeptByOwl are the accounts Owl keeps (the work account through Exchange): never declared (ADR 0316 §13). @@ -626,6 +625,20 @@ type AccountsAnswer struct { Note string `json:"note"` } +// listedAccount is one account of the profile as betterbird_accounts lists it. +type listedAccount struct { + Key string `json:"key"` + Kind string `json:"kind"` + // DeclaredAs is the name the setting accounts declares it under, or empty: the person's own. + DeclaredAs string `json:"declared_as,omitempty"` + Owl bool `json:"owl,omitempty"` + // Name is what Betterbird shows it as, and Address its default identity's. + Name string `json:"name,omitempty"` + Address string `json:"address,omitempty"` + Value *Account `json:"value,omitempty"` + Note string `json:"note,omitempty"` +} + // profileDeclarable reads the profile's accounts: the value a declaration of them would be, by name, and // what is answered of them. func (m *Machine) profileDeclarable() (map[string]Account, AccountsAnswer, error) { @@ -634,13 +647,28 @@ func (m *Machine) profileDeclarable() (map[string]Account, AccountsAnswer, error return nil, AccountsAnswer{}, err } prefs := mergedPrefs(p.path) - a := AccountsAnswer{Profile: p.Dir, Declarable: map[string]int{}, NotDeclarable: map[string]int{}} + a := AccountsAnswer{Profile: p.Dir, Accounts: []listedAccount{}, Declarable: map[string]int{}, NotDeclarable: map[string]int{}} value := map[string]Account{} used := map[string]bool{} + declaredAs := map[string]string{} + if accounts, _, err := m.readAccounts(); err == nil { + for _, r := range resolveAccounts(accounts, prefs, m.readRecord(p.Dir)) { + if r.How != refused { + declaredAs[r.Keys.Account] = r.name + } + } + } for _, pa := range profileAccounts(prefs) { + listed := listedAccount{Key: pa.Key, Kind: pa.Kind, DeclaredAs: declaredAs[pa.Key], Owl: pa.Owl, + Name: unquote(prefs["mail.server."+pa.Server+".name"])} + if len(pa.Addresses) > 0 { + listed.Address = pa.Addresses[0] + } switch { case pa.Owl: a.KeptByOwl++ + listed.Note = "Owl's: never declared (novox/hq ADR 0316 §13)" + a.Accounts = append(a.Accounts, listed) continue case pa.Kind != "imap" && pa.Kind != "pop3": kind := pa.Kind @@ -648,6 +676,8 @@ func (m *Machine) profileDeclarable() (map[string]Account, AccountsAnswer, error kind = "local folders" } a.NotDeclarable[kind]++ + listed.Note = "not an account a declaration holds" + a.Accounts = append(a.Accounts, listed) continue } a.Declarable[pa.Kind]++ @@ -659,8 +689,14 @@ func (m *Machine) profileDeclarable() (map[string]Account, AccountsAnswer, error address = pa.Addresses[0] } name := nameFor(pa, address, used) + if declaredAs[pa.Key] != "" { + name = declaredAs[pa.Key] + } used[name] = true - value[name] = readAccount(prefs, pa) + acc := readAccount(prefs, pa) + value[name] = acc + listed.Value = &acc + a.Accounts = append(a.Accounts, listed) } if len(a.NotDeclarable) == 0 { a.NotDeclarable = nil @@ -681,9 +717,9 @@ func (m *Machine) ProfileAccounts(propose bool) (AccountsAnswer, error) { if err != nil { return a, err } - a.Note = "counts and kinds only: no address, host or name leaves through this answer. With propose, the " + - "setting accounts is built from these accounts (never Owl's, never a password or token) and proposed to " + - "the operator, who reads it whole at their desk before approving (novox/hq ADR 0317 §9)" + a.Note = "the profile's accounts with their values, never a password or token (novox/hq ADR 0329 §1, §2). With " + + "propose, the setting accounts is built from them (never Owl's) and proposed to the operator in the mesh-wide " + + "layer as it stands, who approves it on a channel that proves who answers (novox/hq ADR 0317 §9)" if !propose { return a, nil } @@ -701,21 +737,16 @@ func (m *Machine) ProfileAccounts(propose bool) (AccountsAnswer, error) { return a, fmt.Errorf("nothing was proposed: %s; betterbird_prefs shows the account's preferences by name", strings.Join(problems, "; ")) } - body, err := json.Marshal(map[string]any{"accounts": value}) + // A layer is replaced whole: the proposal carries the mesh-wide layer as it stands, with this value of accounts. + layer, err := readLayer("") if err != nil { - return a, err + return a, fmt.Errorf("nothing was proposed: %v", err) } - raw, err := asker("seat:mesh-controller.settings", map[string]any{"module": "betterbird", "values": string(body), - "propose": "true"}) - if err != nil { - return a, fmt.Errorf("the controller's settings verb did not answer: %v", err) - } - said, ok := verbSaid(raw) + layer["accounts"] = value + a.Derived = value + said, err := proposeLayer("", layer, false) a.Proposed = m.maskValue(said) - if !ok { - return a, fmt.Errorf("the controller did not take the proposal: %s", a.Proposed) - } - return a, nil + return a, err } // verbSaid is a verb's words and whether it went ahead, whatever wraps them: the controller's {ok, output}, diff --git a/modules/betterbird/cmd/betterbird-tools/accounts_test.go b/modules/betterbird/cmd/betterbird-tools/accounts_test.go index 87796008..0e96ab44 100644 --- a/modules/betterbird/cmd/betterbird-tools/accounts_test.go +++ b/modules/betterbird/cmd/betterbird-tools/accounts_test.go @@ -357,9 +357,12 @@ func TestDeclaredAccountsWithoutASignInAreCountedFromTheServerNamesAlone(t *test } } -func TestBetterbirdAccountsAnswersCountsAndProposesTheValueWithoutOwlOrACredential(t *testing.T) { +// ADR 0329 §1, §10, §12, §13: betterbird_accounts lists the accounts with their values, Owl's by name and address +// and marked; its propose carries the mesh-wide layer as it stands with accounts in it, and answers the value. +func TestBetterbirdAccountsListsTheValuesAndProposesThemInTheLayerAsItStands(t *testing.T) { f := newFake(t) f.profileWithAccounts() + f.write(profileDir+"/logins.json", `{"logins":[{"hostname":"imap://imap.example.org","encryptedUsername":"MDoEEPgAAAAA","encryptedPassword":"hunter2secret"}]}`) links(t, nil) a, err := f.ProfileAccounts(false) if err != nil { @@ -368,19 +371,33 @@ func TestBetterbirdAccountsAnswersCountsAndProposesTheValueWithoutOwlOrACredenti if a.Declarable["imap"] != 1 || a.KeptByOwl != 1 || a.NotDeclarable["local folders"] != 1 || a.Proposed != "" { t.Fatalf("%+v", a) } - raw, _ := json.Marshal(a) - for _, never := range []string{"example", "other", "Other", "work@", "smtp1"} { + raw, _ := json.Marshal(credentialChecked(a)) + for _, shown := range []string{"other@mail.example.org", "imap.example.org", "smtp.example.org", `"port":993`, `"work@corp.example.com"`} { + if !strings.Contains(string(raw), shown) { + t.Errorf("the answer shows the operator's own %s (ADR 0329 §1): %s", shown, raw) + } + } + for _, never := range []string{"hunter2", "MDoEEP"} { if strings.Contains(string(raw), never) { - t.Errorf("the answer to the caller names %q: %s", never, raw) + t.Errorf("the answer carries the login store's %q", never) + } + } + for _, l := range a.Accounts { + if l.Key == "account3" && (!l.Owl || l.Value != nil || l.Address != "work@corp.example.com") { + t.Errorf("Owl's account is listed by name and address, marked, never as a value to declare: %+v", l) } } - var asked map[string]any + var asked []map[string]any asker = func(key string, body any) (json.RawMessage, error) { if key != "seat:mesh-controller.settings" { t.Fatalf("asked %s", key) } - asked = body.(map[string]any) + args := body.(map[string]any) + asked = append(asked, args) + if args["propose"] == nil { + return json.RawMessage(`{"ok": true, "output": "{\n \"tags\": {\"$label1\": {\"name\": \"Important\"}}\n}\n approved by the operator\nbetterbird on the whole mesh, every value"}`), nil + } return json.RawMessage(`{"ok": true, "output": "proposed p-17 for betterbird on the whole mesh: accounts (other@mail.example.org); fingerprint ab12"}`), nil } t.Cleanup(func() { asker = nil }) @@ -388,10 +405,15 @@ func TestBetterbirdAccountsAnswersCountsAndProposesTheValueWithoutOwlOrACredenti if err != nil { t.Fatal(err) } - if asked["module"] != "betterbird" || asked["propose"] != "true" || asked["node"] != nil { - t.Fatalf("proposed for the whole mesh, never set: %v", asked) + if len(asked) != 2 || asked[1]["module"] != "betterbird" || asked[1]["propose"] != "true" || asked[1]["node"] != nil || + asked[1]["replace"] != nil { + t.Fatalf("the layer read, then proposed for the whole mesh, never set, never removing: %v", asked) } - value, err := parseAccounts([]byte(asked["values"].(string))) + var layer map[string]json.RawMessage + if err := json.Unmarshal([]byte(asked[1]["values"].(string)), &layer); err != nil || string(layer["tags"]) == "" { + t.Fatalf("the proposal carries the layer as it stands: %v %s", err, asked[1]["values"]) + } + value, err := parseAccounts([]byte(asked[1]["values"].(string))) if err != nil { t.Fatalf("the proposed value is one the module takes: %v", err) } @@ -400,6 +422,9 @@ func TestBetterbirdAccountsAnswersCountsAndProposesTheValueWithoutOwlOrACredenti other.Incoming.Port != 993 || other.Outgoing.Security != "tls" || other.Outgoing.Port != 465 { t.Fatalf("the value holds the person's account and not Owl's: %+v", value) } + if a.Derived["mail"].Address != "other@mail.example.org" { + t.Fatalf("the answer shows the value it proposed (ADR 0329 §10): %+v", a.Derived) + } if strings.Contains(a.Proposed, "other@") || !strings.Contains(a.Proposed, "p-17") { t.Fatalf("the controller's words come back with every address masked: %s", a.Proposed) } @@ -516,3 +541,27 @@ func TestAnAccountsOwnAliasesAndNonMailIdentitiesDoNotShareItsOutgoingServer(t * t.Fatalf("%v", err) } } + +// Review 121: the account lists wait for Betterbird to close by its profile's lock too, not only by its process. +func TestTheAccountListsWaitWhileTheProfileIsLocked(t *testing.T) { + f := newFake(t) + f.profileWithAccounts() + f.noPackage() + links(t, nil) + f.declare("// none") + f.declareAccounts(personal) + stopBetterbird(t, f) + f.write(profileDir+"/lock", "") + before := f.read(profileDir + "/prefs.js") + u, err := f.userJS(true) + if err != nil { + t.Fatal(err) + } + if !strings.HasPrefix(u.Lists, "not written") || !strings.Contains(u.Lists, "locked") || f.read(profileDir+"/prefs.js") != before { + t.Fatalf("with the profile locked and no Betterbird running, prefs.js is unchanged: %q", u.Lists) + } + os.Remove(filepath.Join(f.Root, profileDir, "lock")) + if u, err = f.userJS(true); err != nil || !strings.HasPrefix(u.Lists, "written") { + t.Fatalf("once unlocked the list is written: %q %v", u.Lists, err) + } +} diff --git a/modules/betterbird/cmd/betterbird-tools/adoption.go b/modules/betterbird/cmd/betterbird-tools/adoption.go index 8a65066e..02adc852 100644 --- a/modules/betterbird/cmd/betterbird-tools/adoption.go +++ b/modules/betterbird/cmd/betterbird-tools/adoption.go @@ -171,7 +171,8 @@ func refuseSharedOutgoing(resolved []resolvedAccount, prefs map[string]string, h users := map[string][]string{} seen := map[string]bool{} use := func(id string) { - if seen[id] || ours[id] { + // An identity the module created for a declared account sends as that account does (ADR 0329 §7). + if seen[id] || ours[id] || ownIdentity.MatchString(id) { return } seen[id] = true diff --git a/modules/betterbird/cmd/betterbird-tools/betterbird.go b/modules/betterbird/cmd/betterbird-tools/betterbird.go index 3cbc09e0..a76148e3 100644 --- a/modules/betterbird/cmd/betterbird-tools/betterbird.go +++ b/modules/betterbird/cmd/betterbird-tools/betterbird.go @@ -9,20 +9,20 @@ package main // preferences asked for are answered and never one that can hold a credential or names an account // (passwordSecret, accountPrefix); // - the profile's extensions.json, for the add-ons' ids and whether each is on; -// - the module's declared user.js (~/.config/betterbird/user.js), which the node-engine writes from -// the module's setting `prefs`, and its declared accounts (~/.config/betterbird/accounts.json, the -// setting `accounts`, accounts.go); +// - the module's settings as the node-engine writes them: prefs (~/.config/betterbird/prefs.json, prefs.go), +// accounts (~/.config/betterbird/accounts.json, accounts.go), and identities, calendars, address-books, +// filters and tags (configuration.json and tags.json, configuration.go); // - of logins.json, only the server name each kept sign-in is for, which Betterbird keeps in the clear, // to count the declared accounts that have not signed in yet (novox/hq ADR 0316 §16). // -// Never read: the mail store, the address books, a user name or password in logins.json, key4.db, -// cert9.db, cookies, the calendar's caches. Nothing of the operator's mail or calendar is answered, and of -// their accounts only counts and kinds. +// Never read: the mail store, the address books' contacts, a user name or password in logins.json, key4.db, +// cert9.db, cookies, the calendar's events and caches. The operator's own configuration is answered with its +// values (novox/hq ADR 0329 §1), never a credential (§2). // -// The files a tool writes in the profile are user.js, and only the module's own (the declared file and -// the declared accounts' preferences, betterbird_user_js with apply, betterbird_restart, and the start -// from the launcher), and the module's own keys in the account lists of prefs.js, only while Betterbird is -// not running (accounts.go). A user.js that is not the module's is never overwritten. +// The files a tool writes in the profile are user.js, and only the module's own (betterbird_user_js with apply, +// betterbird_restart, and the start from the launcher); and, only while Betterbird is not running, the module's +// own keys in the lists of prefs.js, the removals of what it created, and each declared account's +// msgFilterRules.dat (accounts.go, owned.go). A user.js that is not the module's is never overwritten. import ( "bufio" @@ -51,8 +51,6 @@ const ( testUnit = "mesh-betterbird-reminder-test" // profilesDir is where Betterbird keeps its profiles: Thunderbird's place, which Betterbird shares. profilesDir = ".thunderbird" - // declared is the module's user.js, written by the node-engine from the setting prefs. - declared = ".config/betterbird/user.js" // mark is the first line of the module's user.js: how a user.js is known to be the module's. mark = "// Betterbird's user.js (module betterbird" ) @@ -440,6 +438,12 @@ func validUserJS(text string) error { "accounts, never in prefs (novox/hq ADR 0316 §11)", i+1, name) } } + for _, p := range configurationOwned { + if strings.HasPrefix(name, p) { + return fmt.Errorf("line %d sets %s, which the setting calendars, address-books or tags owns, never "+ + "prefs (novox/hq ADR 0329 §6)", i+1, name) + } + } } } return nil @@ -489,6 +493,16 @@ type UserJS struct { Each []resolvedAccount `json:"accounts,omitempty"` // Lists says whether the keys of the accounts the module created were written into the account lists. Lists string `json:"account_lists,omitempty"` + // Configuration is every declared identity, calendar, address book and tag by its name: adopted, created or + // refused (with why), and its key (novox/hq ADR 0329 §7). + Configuration []resolvedEntry `json:"configuration,omitempty"` + ConfigurationProblem string `json:"configuration_problem,omitempty"` + // Filters are the declared accounts' filter files. + Filters []filterPlan `json:"filters,omitempty"` + // Closed says what was written of what Betterbird rewrites itself, or why it waits (§9). + Closed string `json:"written_while_closed,omitempty"` + // Waiting counts, per kind, the changes that wait for Betterbird to close. + Waiting map[string]int `json:"waiting_for_betterbird_to_close,omitempty"` } // refusedAccounts are the reasons of the declared accounts that were refused. @@ -508,9 +522,12 @@ func (u UserJS) refusedAccounts() []string { // it created into the profile's account lists when Betterbird is not running. A refused account gets nothing; // the others go ahead (novox/hq ADR 0317 §7). func (m *Machine) userJS(apply bool) (UserJS, error) { - a := UserJS{Declared: "~/" + declared} - declaredText, err := readBounded(m.home(declared)) - a.Present = err == nil + a := UserJS{Declared: "~/" + prefsFile} + declaredText, present, derr := m.declaredText() + a.Present = present + if derr != nil { + a.Present, a.Valid = true, derr.Error() + } p, perr := m.profile() if perr != nil { return a, perr @@ -518,19 +535,27 @@ func (m *Machine) userJS(apply bool) (UserJS, error) { target := filepath.Join(p.path, "user.js") a.Profile = m.tilde(strings.TrimPrefix(target, m.Root)) accounts, _, aerr := m.readAccounts() + merged := mergedPrefs(p.path) if aerr != nil { a.AccountsProblem = aerr.Error() accounts = nil } else { a.Accounts = len(accounts) - a.Each = resolveAccounts(accounts, mergedPrefs(p.path), m.readRecord(p.Dir)) + a.Each = resolveAccounts(accounts, merged, m.readRecord(p.Dir)) } + conf, cerr := m.readConfiguration() + if cerr != nil { + a.ConfigurationProblem = cerr.Error() + } + plan := m.planConfiguration(p, conf, accounts, a.Each, merged) + a.Configuration, a.Filters = plan.Entries, plan.filters + a.Waiting = m.waiting(p.path, plan, a.Each) var want []byte - if a.Present { - if verr := validUserJS(string(declaredText)); verr != nil { + if a.Present && a.Valid == "" { + if verr := validUserJS(declaredText); verr != nil { a.Valid = verr.Error() } - want = []byte(composeUserJS(string(declaredText), accounts, a.Each)) + want = []byte(composeUserJS(declaredText, accounts, a.Each) + plan.configBlock()) a.Content = m.answeredUserJS(string(want)) for _, line := range strings.Split(string(want), "\n") { for _, st := range splitStatements(strings.TrimSpace(line)) { @@ -554,11 +579,13 @@ func (m *Machine) userJS(apply bool) (UserJS, error) { if apply { switch { case !a.Present: - return a, fmt.Errorf("the module's user.js is not on this machine (~/%s): the module is not pushed here yet", declared) + return a, fmt.Errorf("the setting prefs is not on this machine (~/%s): the module is not pushed here yet", prefsFile) case a.Valid != "": return a, fmt.Errorf("the module's user.js is refused: %s; change the setting prefs", a.Valid) case a.AccountsProblem != "": return a, fmt.Errorf("nothing was placed: %s", a.AccountsProblem) + case a.ConfigurationProblem != "": + return a, fmt.Errorf("nothing was placed: %s", a.ConfigurationProblem) } saved, _ := readBounded(filepath.Join(p.path, "prefs.js")) savedPrefs := readPrefs(saved) @@ -590,6 +617,16 @@ func (m *Machine) userJS(apply bool) (UserJS, error) { } else { a.Lists = lists } + if closed, err := m.writeClosed(p, &plan); err != nil { + a.Closed = "not written: " + err.Error() + } else { + a.Closed = closed + } + a.Filters = plan.filters + if err := m.writeConfigRecord(p.Dir, plan.record); err != nil { + return a, fmt.Errorf("the user.js is placed, and what was adopted or created could not be recorded: %v", err) + } + a.Waiting = m.waiting(p.path, plan, a.Each) } if ps := m.mainProcs(); len(ps) > 0 { if st, err := os.Stat(target); err == nil { @@ -640,6 +677,20 @@ func (m *Machine) Prepare() error { problems = append(problems, "the account lists were "+u.Lists) } problems = append(problems, u.refusedAccounts()...) + if strings.HasPrefix(u.Closed, "not written") { + problems = append(problems, "what Betterbird rewrites itself was "+u.Closed) + } + // Said by kind, name and key only: the start's words reach the journal, which holds no address (ADR 0329 §1). + for _, e := range u.Configuration { + if e.How == refused { + problems = append(problems, fmt.Sprintf("%s %s is refused: %s", e.Kind, e.Name, e.Why)) + } + } + for _, fp := range u.Filters { + if fp.Refused != "" { + problems = append(problems, fmt.Sprintf("the filters of %s are refused: %s", fp.Account, fp.Refused)) + } + } if len(problems) > 0 { return fmt.Errorf("%s", strings.Join(problems, "; ")) } @@ -843,7 +894,7 @@ func (m *Machine) Restart(discard bool) (RestartAnswer, error) { a.UserJS = "not placed: " + err.Error() } else { a.UserJS = fmt.Sprintf("placed: %d preference(s) from ~/%s, %d declared account(s); account lists %s", - u.Statements, declared, u.Accounts, u.Lists) + u.Statements, prefsFile, u.Accounts, u.Lists) } // As detach does, with shorter bounds: the worst case of the whole call (3 s for the window tree, // 8 s to quit, 3 + 5 s to start, 4 s to see it) stays under the runtime's 30 s. @@ -962,6 +1013,23 @@ func (m *Machine) Check() (CheckAnswer, error) { add(fmt.Sprintf("%d declared account(s) have not signed in yet", c.WithoutSignIn), "a person signs in once, in Betterbird's own window, when it asks (novox/hq ADR 0316 §14)") } + // The five kinds of ADR 0329: drift and what waits for Betterbird to close, per kind (§9, §11, §12). + if _, err := m.readConfiguration(); err != nil { + add(err.Error(), "change the setting at the controller's terminal, or derive and propose it (betterbird_derive)") + } else if v, err := m.view(); err == nil { + drift := m.driftCounts(v) + waiting := m.waiting(v.p.path, v.plan, v.ra) + for _, k := range allKinds { + if n := drift[k]; n > 0 { + add(fmt.Sprintf("%d declared %s differ from the profile, are not in it yet or are refused", n, k), + "betterbird_drift with kind "+k+" says which and how; the module's user.js holds them from Betterbird's next start (betterbird_restart)") + } + if n := waiting[k]; n > 0 { + add(fmt.Sprintf("%d change(s) to %s are not written yet: they are written only while Betterbird is closed", n, k), + "quit Betterbird and start it from the launcher, or betterbird_restart (novox/hq ADR 0329 §9)") + } + } + } } ps := m.mainProcs() switch { @@ -975,7 +1043,7 @@ func (m *Machine) Check() (CheckAnswer, error) { case err != nil: add("no profile: "+err.Error(), "start Betterbird once") case !u.Present: - add("the module's user.js is not on this machine (~/"+declared+")", "the module is not pushed here: the controller's push") + add("the setting prefs is not on this machine (~/"+prefsFile+")", "the module is not pushed here: the controller's push") case u.Valid != "": add("the module's user.js is refused: "+u.Valid, "change the module's setting prefs") case u.AccountsProblem != "": diff --git a/modules/betterbird/cmd/betterbird-tools/betterbird_test.go b/modules/betterbird/cmd/betterbird-tools/betterbird_test.go index 5f017a90..4296f76d 100644 --- a/modules/betterbird/cmd/betterbird-tools/betterbird_test.go +++ b/modules/betterbird/cmd/betterbird-tools/betterbird_test.go @@ -1,6 +1,7 @@ package main import ( + "encoding/json" "os" "path/filepath" "strings" @@ -44,8 +45,20 @@ func (f *fake) betterbird() { f.write(profileDir+"/extensions.json", `{"addons":[{"id":"owl@beonex.com","active":true,"location":"app-profile"},{"id":"x@mozilla.org","active":true,"location":"app-builtin"}]}`) } +// declare writes the setting prefs as the node-engine does (prefs.json, novox/hq issue 586), from user_pref +// statements: each one entry of the map. func (f *fake) declare(prefs string) { - f.write(testHome+"/.config/betterbird/user.js", mark+", novox/hq issue 345). Owned by the mesh.\n//\n"+prefs+"\n") + f.t.Helper() + values := map[string]any{} + for name, lit := range statementsOf([]byte(prefs)) { + v, ok := prefValue(lit) + if !ok { + f.t.Fatalf("%s: %s is not a value the setting holds", name, lit) + } + values[name] = v + } + body, _ := json.Marshal(map[string]any{"prefs": values}) + f.write(testHome+"/"+prefsFile, string(body)) } func TestTheProfileIsTheOneTheRunningBetterbirdHoldsElseTheInstallationsDefault(t *testing.T) { @@ -106,9 +119,9 @@ func TestTheModulesUserJSIsPlacedOnlyWhereNoOperatorsFileIs(t *testing.T) { t.Fatalf("%+v %v", u, err) } placed, _ := os.ReadFile(filepath.Join(f.Root, profileDir, "user.js")) - declared, _ := os.ReadFile(filepath.Join(f.Root, testHome, ".config/betterbird/user.js")) - if string(placed) != string(declared) { - t.Fatal("placed whole, as declared") + declared, _, _ := f.declaredText() + if string(placed) != declared || !strings.Contains(declared, `user_pref("ui.key.x", "a;b");`) { + t.Fatalf("placed whole, as declared:\n%s", placed) } if !strings.HasPrefix(u.Effective, "yes") { t.Fatalf("the fake Betterbird started after the file's time: %q", u.Effective) diff --git a/modules/betterbird/cmd/betterbird-tools/config.go b/modules/betterbird/cmd/betterbird-tools/config.go index eb6a0651..25635e10 100644 --- a/modules/betterbird/cmd/betterbird-tools/config.go +++ b/modules/betterbird/cmd/betterbird-tools/config.go @@ -1,18 +1,20 @@ package main -// The configuration the module ships (novox/hq issue 345), in three layers: +// The configuration of Betterbird's preferences (novox/hq issue 345, issue 586), in three layers: // // - LOCKED: the enterprise policy (/opt/betterbird/distribution/policies.json). Its Preferences entries with // Status locked win over everything, and Betterbird's settings show them greyed out. -// - DEFAULT: the module's defaults file (/opt/betterbird/defaults/pref/mesh-defaults.js), the operator's -// setup. Betterbird reads it as a default, so a value the profile saved in prefs.js wins over it: that is -// drift, unless a node layer says so. -// - NODE: the setting prefs, which the node-engine writes to ~/.config/betterbird/user.js and the tools place -// in the profile's user.js, where it wins over prefs.js at every start. +// - DECLARED: the setting prefs, the operator's setup: mesh-wide, with a node's layer over it key by key. The +// node-engine writes it to ~/.config/betterbird/prefs.json and the tools place it in the profile's user.js, +// where it wins over prefs.js at every start. +// - BETTERBIRD'S DEFAULT: what Betterbird starts with. The module's defaults file +// (/opt/betterbird/defaults/pref/mesh-defaults.js) holds no value of the operator's any more (issue 586): it +// names the preferences the module keeps at Betterbird's own default, where a value the profile saved is drift. // // betterbird_config shows each managed preference through the layers and flags the drift; betterbird_config_apply -// clears a drifting value from prefs.js while Betterbird is not running; keep proposes a node layer that holds the -// profile's value for this node instead, through the controller's settings verb, never by writing user.js. +// clears a drifting value from prefs.js while Betterbird is not running; keep proposes a node layer of the setting +// prefs that holds the profile's value for this node instead, through the controller's settings verb, never by +// writing user.js. import ( "archive/zip" @@ -320,14 +322,14 @@ func statementsOf(raw []byte) map[string]string { // ConfigPref is one managed preference through the layers. type ConfigPref struct { Name string `json:"name"` - // Layer is locked (the policy), default (the module's defaults file) or "Betterbird's default" (a managed - // preference whose module value is Betterbird's own). + // Layer is locked (the policy), declared (the setting prefs), default (a value of the module's defaults file) or + // "Betterbird's default" (a managed preference whose module value is Betterbird's own). Layer string `json:"layer"` Locked string `json:"locked,omitempty"` ModuleDefault string `json:"module_default,omitempty"` BetterbirdDefault string `json:"betterbird_default,omitempty"` - // NodeLayer is what the setting prefs on this machine says (the module's user.js). - NodeLayer string `json:"node_layer,omitempty"` + // Declared is what the setting prefs says on this machine: mesh-wide, with the node's layer over it. + Declared string `json:"declared,omitempty"` // Profile is what the profile saved in its prefs.js. Profile string `json:"profile,omitempty"` Effective string `json:"effective"` @@ -349,6 +351,8 @@ type ConfigAnswer struct { Drifting []string `json:"drifting"` // Refused are names the module's defaults list that the tools never manage: an account's or a credential's. Refused []string `json:"refused,omitempty"` + // PrefsProblem says why the setting prefs on this machine is refused. + PrefsProblem string `json:"prefs_problem,omitempty"` // Proposed is the controller's answer to a node layer proposed with keep. Proposed string `json:"proposed,omitempty"` Note string `json:"note,omitempty"` @@ -372,8 +376,10 @@ func (m *Machine) config() (ConfigAnswer, Profile, error) { } locked := m.locks() own, _ := m.defaults(false) - declaredRaw, _ := readBounded(m.home(declared)) - node := statementsOf(declaredRaw) + node, _, nerr := m.readDeclaredPrefs() + if nerr != nil { + a.PrefsProblem = nerr.Error() + } prefsRaw, _ := readBounded(filepath.Join(p.path, "prefs.js")) saved := readPrefs(prefsRaw) userRaw, _ := readBounded(filepath.Join(p.path, "user.js")) @@ -384,8 +390,13 @@ func (m *Machine) config() (ConfigAnswer, Profile, error) { names = append(names, n) } sort.Strings(names) - for _, n := range sh.order { - if _, isLocked := locked[n]; !isLocked { + listed := map[string]bool{} + for _, n := range names { + listed[n] = true + } + for _, n := range append(append([]string{}, sh.order...), sortedKeys(node)...) { + if !listed[n] { + listed[n] = true names = append(names, n) } } @@ -395,7 +406,7 @@ func (m *Machine) config() (ConfigAnswer, Profile, error) { a.Refused = append(a.Refused, n+": "+why) continue } - c := ConfigPref{Name: n, NodeLayer: node[n], Profile: saved[n]} + c := ConfigPref{Name: n, Declared: node[n], Profile: saved[n]} if n == themePref && theme != "" { // The add-on database says which theme is active; the preference only mirrors it. c.Profile = quoted(theme) @@ -410,6 +421,8 @@ func (m *Machine) config() (ConfigAnswer, Profile, error) { switch v, isLocked := locked[n]; { case isLocked: c.Layer, c.Locked, c.module = "locked", v, v + case c.Declared != "": + c.Layer, c.module = "declared", c.Declared case sh.keeps[n]: c.Layer = "Betterbird's default" c.module = c.BetterbirdDefault @@ -430,23 +443,37 @@ func (m *Machine) config() (ConfigAnswer, Profile, error) { default: c.From = "Betterbird's built-in default (in no defaults file)" } - if why := drift(c); why != "" { + if why := drift(c, placed[n]); why != "" { c.Drift = why - if n == themePref { + switch { + case n == themePref: c.Remedy = themeRemedy + case c.Layer == "declared": + c.Remedy = "betterbird_user_js with apply places the setting in the profile's user.js; Betterbird reads it " + + "at its next start (betterbird_restart)" } a.Drifting = append(a.Drifting, n) } - c.NodeLayer, c.Profile, c.Effective = m.answer(n, c.NodeLayer), m.answer(n, c.Profile), m.answer(n, c.Effective) + c.Declared, c.Profile, c.Effective = m.answer(n, c.Declared), m.answer(n, c.Profile), m.answer(n, c.Effective) a.Prefs = append(a.Prefs, c) } return a, p, nil } -// drift says how the profile's saved value beats the module's, or "": a value in prefs.js that is not the -// module's, where no lock holds and no node layer names the preference. -func drift(c ConfigPref) string { - if c.Locked != "" || c.NodeLayer != "" || c.Profile == "" || c.Profile == c.module { +// drift says how the profile differs from the module's configuration, or "": for a declared preference, the +// profile's user.js not holding the declared value (so Betterbird does not start with it); for a preference kept at +// Betterbird's default, a value in prefs.js that is not it. A lock holds whatever the profile says. +func drift(c ConfigPref, placed string) string { + if c.Locked != "" { + return "" + } + if c.Declared != "" { + if placed == c.Declared { + return "" + } + return fmt.Sprintf("the setting prefs says %s, and the profile's user.js does not hold it yet", c.Declared) + } + if c.Profile == "" || c.Profile == c.module { return "" } want := c.module @@ -457,7 +484,7 @@ func drift(c ConfigPref) string { if c.Name == themePref { where = "the profile's active theme is" } - return fmt.Sprintf("%s %s, which beats the module's %s, and no node layer says so", where, c.Profile, want) + return fmt.Sprintf("%s %s, which beats the module's %s, and the setting prefs does not say so", where, c.Profile, want) } // Config is what betterbird_config answers; with keep it proposes a node layer of the setting prefs that holds @@ -468,8 +495,9 @@ func (m *Machine) Config(keep []string) (ConfigAnswer, error) { if err != nil { return a, err } - a.Note = "drift is a value the profile saved that beats the module's default without a node layer saying so: " + - "betterbird_config_apply clears it (Betterbird quit), or keep proposes a node layer that holds it for this machine" + a.Note = "drift is a declared value the profile's user.js does not hold yet (betterbird_user_js with apply), or a " + + "value the profile saved where the module keeps Betterbird's default: betterbird_config_apply clears that one " + + "(Betterbird quit), or keep proposes a node layer of the setting prefs that holds it for this machine" if len(keep) == 0 { return a, nil } @@ -494,50 +522,36 @@ func (m *Machine) Config(keep []string) (ConfigAnswer, error) { } kept[n] = saved[n] } - declaredRaw, err := readBounded(m.home(declared)) - if err != nil { - return a, fmt.Errorf("nothing was proposed: the module's user.js is not on this machine (~/%s)", declared) - } node := nodeName() if node == "" { return a, errors.New("nothing was proposed: this runtime does not say which machine it serves (MESH_NODE)") } - // The setting is one string, so a node layer replaces it whole: it carries what the setting says on this - // machine today, with the kept values in place of any the same names had. - var parts []string - for _, line := range strings.Split(string(declaredRaw), "\n") { - for _, st := range splitStatements(strings.TrimSpace(line)) { - if mm := prefLine.FindStringSubmatch(st); mm != nil { - if _, replaced := kept[mm[1]]; !replaced { - parts = append(parts, fmt.Sprintf("user_pref(%s, %s);", quoted(mm[1]), mm[2])) - } - } - } + // A layer is replaced whole, so the proposal carries this node's layer as it stands, with the kept values in + // the setting prefs beside what it held (novox/hq issue 586: one entry per preference). + layer, err := readLayer(node) + if err != nil { + return a, fmt.Errorf("nothing was proposed: %v", err) + } + prefs, _ := layer["prefs"].(map[string]any) + if prefs == nil { + prefs = map[string]any{} } for _, n := range keep { - if v, ok := kept[n]; ok { - parts = append(parts, fmt.Sprintf("user_pref(%s, %s);", quoted(n), v)) - delete(kept, n) + v, ok := prefValue(kept[n]) + if !ok { + return a, fmt.Errorf("nothing was proposed: the profile's value of %s is not one the setting can hold", n) } + prefs[n] = v } - value := strings.Join(parts, " ") - if err := validUserJS(mark + ")\n" + value + "\n"); err != nil { - return a, fmt.Errorf("nothing was proposed: the layer would be refused: %v", err) - } - body, _ := json.Marshal(map[string]string{"prefs": value}) - raw, err := asker("seat:mesh-controller.settings", map[string]any{"module": "betterbird", "node": node, - "values": string(body), "propose": "true"}) - if err != nil { - return a, fmt.Errorf("the controller's settings verb did not answer: %v", err) - } - said, ok := verbSaid(raw) + layer["prefs"] = prefs + said, err := proposeLayer(node, layer, false) a.Proposed = m.maskValue(said) - if !ok { - return a, fmt.Errorf("the controller did not take the proposal: %s", a.Proposed) + if err != nil { + return a, err } a.Note = "proposed for " + node + ": the operator approves it on a channel that proves who answers; the next " + "push writes it, and it is placed at Betterbird's next start from the launcher (or betterbird_restart). " + - "The layer carries the whole setting prefs as it stands on this machine, since a layer replaces it whole" + "The layer carries this node's layer as it stands, with the kept preferences in the setting prefs" return a, nil } diff --git a/modules/betterbird/cmd/betterbird-tools/config_test.go b/modules/betterbird/cmd/betterbird-tools/config_test.go index c4f8b669..c92a7288 100644 --- a/modules/betterbird/cmd/betterbird-tools/config_test.go +++ b/modules/betterbird/cmd/betterbird-tools/config_test.go @@ -13,8 +13,9 @@ import ( // The configuration the module ships (novox/hq issue 345): the policy's locks, the module's defaults file, and the // tools that show the layers, flag drift, clear it and propose a node layer. -// shippedDefaultsWant is the DEFAULT layer the operator approved on 2026-10-12: the laptop's setup, with -// reminders missed while asleep shown at the next start and chat off. +// shippedDefaultsWant is the setup the operator approved on 2026-10-12 (the laptop's, with reminders missed while +// asleep shown at the next start and chat off), which the module shipped as its defaults file until issue 586 moved +// it into the mesh-wide layer of the setting prefs: the catalogue holds none of it now. var shippedDefaultsWant = map[string]string{ "mail.uidensity": "0", "mail.uifontsize": "14", @@ -147,8 +148,13 @@ func TestTheModuleShipsTheApprovedLocksAndDefaults(t *testing.T) { text := string(repoFile(t, "install/defaults-pref/mesh-defaults.js")) sh := parseShipped([]byte(text)) - if !reflect.DeepEqual(sh.values, shippedDefaultsWant) { - t.Errorf("the defaults are the approved ones:\n got %v\nwant %v", sh.values, shippedDefaultsWant) + if len(sh.values) != 0 { + t.Errorf("the module's defaults hold no value of the operator's: they are the setting prefs (issue 586): %v", sh.values) + } + for n := range shippedDefaultsWant { + if strings.Contains(text, n) { + t.Errorf("%s is the operator's choice and lives in the setting prefs, never in the catalogue", n) + } } var keeps []string for _, n := range sh.order { @@ -198,8 +204,8 @@ func TestPrefsSayTheDefaultAndTheFileThatGivesIt(t *testing.T) { if p := by["offline.autoDetect"]; p.Value != "true" || p.Default != "true" || !strings.Contains(p.DefaultFrom, "all-thunderbird.js") { t.Errorf("of two files, the one first by name holds, as Gecko reads them: %+v", p) } - if p := by["mail.threadpane.listview"]; p.Value != "0" || !strings.Contains(p.From, "the module's default") { - t.Errorf("the module's default is over Betterbird's: %+v", p) + if p := by["mail.threadpane.listview"]; p.Value != "1" || strings.Contains(p.From, "module") || strings.Contains(p.DefaultFrom, "module") { + t.Errorf("the module ships no value of the operator's, so Betterbird's own default holds: %+v", p) } if p := by["gfx.webrender.software"]; p.Value != "false" || !strings.Contains(p.From, "greprefs.js") { t.Errorf("%+v", p) @@ -234,8 +240,8 @@ func TestMailnewsIsWithheldOnlyWhereItNamesAnAccount(t *testing.T) { } } -// T1: every managed preference through the layers, and drift where the profile's own value beats the module's -// default without a node layer saying so. +// T1: every managed preference through the layers, and drift: a declared value the profile's user.js does not hold +// yet, or a value the profile saved where the module keeps Betterbird's default. func TestTheConfigShowsTheLayersAndTheDrift(t *testing.T) { f := newFake(t) f.betterbird() @@ -243,13 +249,14 @@ func TestTheConfigShowsTheLayersAndTheDrift(t *testing.T) { f.withDefaults() f.betterbirdsOwn() links(t, nil) - f.declare(`user_pref("mail.uifontsize", 16);`) + f.declare(`user_pref("mail.uifontsize", 16); user_pref("mail.biff.show_alert", false); user_pref("calendar.alarms.showmissed", true);`) + f.write(profileDir+"/user.js", mark+")\n"+`user_pref("mail.uifontsize", 16);`+"\n"+`user_pref("calendar.alarms.showmissed", true);`+"\n") f.write(profileDir+"/prefs.js", "// Mozilla User Preferences\n"+ `user_pref("extensions.activeThemeID", "thunderbird-compact-dark@mozilla.org");`+"\n"+ - `user_pref("mail.uifontsize", 16);`+"\n"+ - `user_pref("mail.uidensity", 0);`+"\n"+ + `user_pref("mail.uifontsize", 12);`+"\n"+ `user_pref("mail.biff.show_alert", true);`+"\n"+ - `user_pref("mail.spam.manualMark", false);`+"\n"+ + `user_pref("mail.spam.manualMark", true);`+"\n"+ + `user_pref("browser.display.use_system_colors", false);`+"\n"+ `user_pref("mailnews.start_page.enabled", true);`+"\n") c, err := f.Config(nil) if err != nil { @@ -259,30 +266,34 @@ func TestTheConfigShowsTheLayersAndTheDrift(t *testing.T) { for _, p := range c.Prefs { by[p.Name] = p } - if len(c.Prefs) != 3+len(shippedDefaultsWant)+len(shippedKeepsWant) || !strings.HasPrefix(c.Defaults, "placed") { - t.Fatalf("every lock, default and kept default: %d, %s", len(c.Prefs), c.Defaults) + if len(c.Prefs) != 3+3+len(shippedKeepsWant) || !strings.HasPrefix(c.Defaults, "placed") { + t.Fatalf("every lock, declared preference and kept default: %d, %s", len(c.Prefs), c.Defaults) } - if !reflect.DeepEqual(c.Drifting, []string{"extensions.activeThemeID", "mail.biff.show_alert"}) { + if !reflect.DeepEqual(c.Drifting, []string{"extensions.activeThemeID", "mail.spam.manualMark", "mail.biff.show_alert"}) { t.Fatalf("drift: %v", c.Drifting) } - if p := by["mail.biff.show_alert"]; p.Layer != "default" || p.ModuleDefault != "false" || p.BetterbirdDefault != "true" || - p.Profile != "true" || p.Effective != "true" || p.From != "the profile's prefs.js" || !strings.Contains(p.Drift, "beats the module's false") { - t.Errorf("%+v", p) + if p := by["mail.biff.show_alert"]; p.Layer != "declared" || p.Declared != "false" || p.Profile != "true" || + p.Effective != "true" || !strings.Contains(p.Drift, "the setting prefs says false") || !strings.Contains(p.Remedy, "betterbird_user_js") { + t.Errorf("a declared value the profile's user.js does not hold yet: %+v", p) + } + if p := by["mail.spam.manualMark"]; p.Layer != "Betterbird's default" || p.BetterbirdDefault != "false" || + !strings.Contains(p.Drift, "beats the module's false") || p.Remedy != "" { + t.Errorf("a saved value where the module keeps Betterbird's default: %+v", p) } if p := by["extensions.activeThemeID"]; p.Layer != "Betterbird's default" || !strings.Contains(p.Drift, `"default-theme@mozilla.org"`) || !strings.Contains(p.Remedy, "add-on manager") || p.Profile != `"thunderbird-compact-dark@mozilla.org"` { t.Errorf("another theme is drift whose remedy is the add-on manager, its id answered whole: %+v", p) } - if p := by["mail.spam.manualMark"]; p.Drift != "" || p.BetterbirdDefault != "false" { + if p := by["browser.display.use_system_colors"]; p.Drift != "" { t.Errorf("a saved value equal to Betterbird's default is not drift: %+v", p) } - if p := by["mail.uifontsize"]; p.Drift != "" || p.NodeLayer != "16" { - t.Errorf("a node layer that says so is not drift: %+v", p) + if p := by["mail.uifontsize"]; p.Drift != "" || p.Declared != "16" || p.Effective != "16" || !strings.Contains(p.From, "user.js") { + t.Errorf("a declared value the user.js holds wins at the start, whatever prefs.js saved: %+v", p) } if p := by["mailnews.start_page.enabled"]; p.Layer != "locked" || p.Effective != "false" || p.Drift != "" { t.Errorf("a lock wins over the profile: %+v", p) } - if p := by["calendar.alarms.showmissed"]; p.Effective != "true" || p.From != "the module's default" { + if p := by["calendar.alarms.showmissed"]; p.Effective != "true" || p.Layer != "declared" { t.Errorf("%+v", p) } // T5: the check says the drift, with both remedies. @@ -292,7 +303,7 @@ func TestTheConfigShowsTheLayersAndTheDrift(t *testing.T) { } found := false for _, x := range ch.Findings { - found = found || (strings.Contains(x.What, "1 managed preference(s) drift") && strings.Contains(x.What, "mail.biff.show_alert") && + found = found || (strings.Contains(x.What, "1 managed preference(s) drift") && strings.Contains(x.What, "mail.spam.manualMark") && strings.Contains(x.Do, "betterbird_config_apply") && strings.Contains(x.Do, "keep")) } theme := false @@ -300,7 +311,11 @@ func TestTheConfigShowsTheLayersAndTheDrift(t *testing.T) { theme = theme || (strings.HasPrefix(x.What, "extensions.activeThemeID drifts") && strings.Contains(x.Do, "add-on manager") && !strings.Contains(x.Do, "betterbird_config_apply")) } - if !found || !theme { + declared := false + for _, x := range ch.Findings { + declared = declared || (strings.HasPrefix(x.What, "mail.biff.show_alert drifts") && strings.Contains(x.Do, "betterbird_user_js")) + } + if !found || !theme || !declared { t.Fatalf("the check reports the drift: %+v", ch.Findings) } os.Remove(filepath.Join(f.Root, meshDefaults)) @@ -326,7 +341,7 @@ func TestConfigApplyClearsTheDriftOnlyWhileBetterbirdIsQuit(t *testing.T) { prefs := "// Mozilla User Preferences\n" + `user_pref("extensions.activeThemeID", "thunderbird-compact-dark@mozilla.org");` + "\n" + `user_pref("mail.identity.id1.useremail", "someone@example.org");` + "\n" + - `user_pref("mail.biff.show_alert", true);` + "\n" + + `user_pref("mail.spam.manualMark", true);` + "\n" + `user_pref("mail.uidensity", 0);` + "\n" f.write(profileDir+"/prefs.js", prefs) dry, err := f.ConfigApply(false) @@ -365,7 +380,7 @@ func TestConfigApplyClearsTheDriftOnlyWhileBetterbirdIsQuit(t *testing.T) { for _, c := range a.Clears { said[c.Name] = c } - if c := said["mail.biff.show_alert"]; c.Was != "true" || c.Now != "false" { + if c := said["mail.spam.manualMark"]; c.Was != "true" || c.Now != "false" { t.Errorf("it says what it cleared and what holds now: %+v", c) } if _, cleared := said["extensions.activeThemeID"]; cleared || len(a.NotCleared) != 1 || !strings.Contains(a.NotCleared[0], "add-on manager") { @@ -384,46 +399,52 @@ func TestConfigApplyClearsTheDriftOnlyWhileBetterbirdIsQuit(t *testing.T) { } } -// T6: keeping a value on one node is a node layer proposed through the controller, never a write to user.js. +// T6: keeping a value on one node is that node's layer of the setting prefs proposed through the controller, one +// entry per preference beside what the layer holds (issue 586), never a write to user.js. func TestKeepProposesANodeLayerThroughTheController(t *testing.T) { f := newFake(t) f.betterbird() f.placedByTheMesh() f.withDefaults() + f.betterbirdsOwn() links(t, nil) f.declare(`user_pref("calendar.alarms.playsound", false); user_pref("mail.uifontsize", 12);`) - f.write(profileDir+"/prefs.js", `user_pref("mail.uifontsize", 16);`+"\n"+`user_pref("mail.biff.show_alert", true);`+"\n") + f.write(profileDir+"/prefs.js", `user_pref("mail.uifontsize", 16);`+"\n"+`user_pref("mail.spam.manualMark", true);`+"\n") oldNode, oldAsk := nodeName, asker t.Cleanup(func() { nodeName, asker = oldNode, oldAsk }) nodeName = func() string { return "shanks" } - var verb string - var args map[string]any + var asked []map[string]any asker = func(v string, a any) (json.RawMessage, error) { - verb, args = v, a.(map[string]any) + args := a.(map[string]any) + asked = append(asked, args) + if args["propose"] == nil { + return json.RawMessage(`{"ok": true, "output": "{\"prefs\": {\"calendar.alarms.playsound\": false}, \"identities\": {\"mail\": {\"account\": \"mail\"}}}\n set at the terminal\nbetterbird on shanks, every value"}`), nil + } return json.RawMessage(`{"ok": true, "output": "proposed: p-1"}`), nil } - before := f.read(testHome + "/.config/betterbird/user.js") - c, err := f.Config([]string{"mail.uifontsize", "mail.biff.show_alert"}) + before := f.read(testHome + "/" + prefsFile) + c, err := f.Config([]string{"mail.uifontsize", "mail.spam.manualMark"}) if err != nil { t.Fatal(err) } - if verb != "seat:mesh-controller.settings" || args["node"] != "shanks" || args["propose"] != "true" || args["module"] != "betterbird" { - t.Fatalf("%s %v", verb, args) + if len(asked) != 2 || asked[0]["node"] != "shanks" || asked[1]["node"] != "shanks" || asked[1]["propose"] != "true" || + asked[1]["module"] != "betterbird" { + t.Fatalf("the node's layer read, then proposed: %v", asked) } - var values map[string]string - if err := json.Unmarshal([]byte(args["values"].(string)), &values); err != nil { + var values map[string]any + if err := json.Unmarshal([]byte(asked[1]["values"].(string)), &values); err != nil { t.Fatal(err) } - want := `user_pref("calendar.alarms.playsound", false); user_pref("mail.uifontsize", 16); user_pref("mail.biff.show_alert", true);` - if values["prefs"] != want { - t.Fatalf("the layer keeps what the setting says and the kept values:\n got %s\nwant %s", values["prefs"], want) + want := map[string]any{"calendar.alarms.playsound": false, "mail.uifontsize": float64(16), "mail.spam.manualMark": true} + if !reflect.DeepEqual(values["prefs"], want) || values["identities"] == nil { + t.Fatalf("the layer as it stands with the kept values, one entry each:\n got %v\nwant %v", values, want) } - if c.Proposed != "proposed: p-1" || f.read(testHome+"/.config/betterbird/user.js") != before { - t.Fatalf("proposed, and user.js untouched: %+v", c) + if c.Proposed != "proposed: p-1" || f.read(testHome+"/"+prefsFile) != before { + t.Fatalf("proposed, and nothing written: %+v", c) } - for _, bad := range [][]string{{"mailnews.start_page.enabled"}, {"some.other.pref"}, {"mail.uidensity"}} { - verb = "" - if _, err := f.Config(bad); err == nil || verb != "" { + for _, bad := range [][]string{{"mailnews.start_page.enabled"}, {"some.other.pref"}, {"calendar.alarms.playsound"}} { + asked = nil + if _, err := f.Config(bad); err == nil || len(asked) != 0 { t.Errorf("%v: a lock, an unmanaged name or a value the profile did not save is refused: %v", bad, err) } } diff --git a/modules/betterbird/cmd/betterbird-tools/configuration.go b/modules/betterbird/cmd/betterbird-tools/configuration.go new file mode 100644 index 00000000..9a1620d3 --- /dev/null +++ b/modules/betterbird/cmd/betterbird-tools/configuration.go @@ -0,0 +1,471 @@ +package main + +// Betterbird's whole configuration, beside the accounts (novox/hq ADR 0329 §4–§6, issue 586). +// +// Five settings, each mesh-wide with a node's layer over it, reach the machine as two files the node-engine +// merges from the module's layers key by key: +// +// - ~/.config/betterbird/configuration.json: identities, calendars, address-books and filters, trusted as +// accounts is (§5), because each names where a password, a reply or forwarded mail goes; +// - ~/.config/betterbird/tags.json: tags, which names no server and sends no mail, and is not trusted. +// +// An entry is keyed by a name of the operator's (lower-case letters and digits, as an account's), a tag by its +// own key. A node's layer may set an entry to null to take it out on that node, and may give an identity, a +// calendar or an address book its `key`: the profile's key to adopt there (§4, §7). No value holds a password, +// a token or a cookie: a key shaped like one is refused whole, and so is an address whose user part carries a +// password (§2). A setting holds no line break (novox/hq issue 339), so a signature is a list of its lines. + +import ( + "bytes" + "encoding/json" + "errors" + "fmt" + "net/url" + "os" + "regexp" + "sort" + "strings" +) + +const ( + // configurationFile holds identities, calendars, address-books and filters; tagsFile holds tags. + configurationFile = ".config/betterbird/configuration.json" + tagsFile = ".config/betterbird/tags.json" +) + +// The five kinds, in the order the tools answer them. +const ( + kindIdentities = "identities" + kindCalendars = "calendars" + kindAddressBooks = "address-books" + kindFilters = "filters" + kindTags = "tags" +) + +var allKinds = []string{kindIdentities, kindCalendars, kindAddressBooks, kindFilters, kindTags} + +// Identity is one identity of a declared account. The account's default identity is the entry that gives no +// address: its name and address are the ones the setting accounts gives (§4). Any other entry is another +// identity of that account, matched by its address. +type Identity struct { + Account string `json:"account"` + Address string `json:"address,omitempty"` + Name string `json:"name,omitempty"` + ReplyTo string `json:"reply-to,omitempty"` + Organization string `json:"organization,omitempty"` + // Signature is the signature's text, a line to an element. + Signature []string `json:"signature,omitempty"` + SignatureHTML bool `json:"signature-html,omitempty"` + // SignatureAboveQuote puts the reply and its signature above the quoted message (Betterbird's "start my + // reply above the quote", signature below the reply); false is below the quote. + SignatureAboveQuote bool `json:"signature-above-quote,omitempty"` + Key string `json:"key,omitempty"` +} + +// Calendar is one CalDAV calendar. +type Calendar struct { + Name string `json:"name"` + URL string `json:"url"` + Color string `json:"color,omitempty"` + ReadOnly bool `json:"read-only,omitempty"` + // Reminders absent is on. + Reminders *bool `json:"reminders,omitempty"` + User string `json:"user,omitempty"` + // Invitations is the identities entry whose address answers invitations; absent, none. + Invitations string `json:"invitations,omitempty"` + Key string `json:"key,omitempty"` +} + +// AddressBook is one CardDAV address book. +type AddressBook struct { + Name string `json:"name"` + URL string `json:"url"` + User string `json:"user,omitempty"` + Key string `json:"key,omitempty"` +} + +// Tag is one tag, keyed in the setting by its own key. +type Tag struct { + Name string `json:"name"` + Color string `json:"color,omitempty"` + // Order is its place among the tags, from 1. + Order int `json:"order,omitempty"` +} + +// Filter is one message filter of an account, in its place in the account's list. +type Filter struct { + Name string `json:"name"` + Enabled bool `json:"enabled"` + // When it runs: new-mail, manual, after-junk, sending, archiving, periodic. + When []string `json:"when"` + // Match is all (every condition holds), any (one does) or every-message (no condition). + Match string `json:"match"` + Conditions []Condition `json:"conditions,omitempty"` + Actions []Action `json:"actions"` +} + +// Condition is one of a filter's conditions, in Betterbird's own words: a field, an operator, a value. +type Condition struct { + Field string `json:"field"` + Op string `json:"op"` + Value string `json:"value,omitempty"` +} + +// Action is one of a filter's actions, in Betterbird's own words. A folder is named by its path within the +// account (INBOX/Receipts), and becomes that folder's address on each node (§8). +type Action struct { + Action string `json:"action"` + Folder string `json:"folder,omitempty"` + Value string `json:"value,omitempty"` +} + +// Configuration is the five settings as the machine has them. +type Configuration struct { + Identities map[string]*Identity `json:"identities,omitempty"` + Calendars map[string]*Calendar `json:"calendars,omitempty"` + AddressBooks map[string]*AddressBook `json:"address-books,omitempty"` + Filters map[string][]Filter `json:"filters,omitempty"` + Tags map[string]*Tag `json:"tags,omitempty"` +} + +var ( + entryName = regexp.MustCompile(`^[a-z][a-z0-9]{0,30}$`) + tagKey = regexp.MustCompile(`^[$A-Za-z0-9_.\-]{1,64}$`) + colorWord = regexp.MustCompile(`^#[0-9A-Fa-f]{6}$`) + // secretName is a field's name that says it holds a credential: such a field is withheld from every answer (§2). + secretName = regexp.MustCompile(`(?i)(password|passwd|token|secret|cookie)`) + // userPassword is an address whose user part carries a password: scheme://user:password@host. + userPassword = regexp.MustCompile(`(?i)\b([a-z][a-z0-9+.\-]*://[^/:@\s]*):[^@/\s]+@`) +) + +// The words a filter is written in. +var ( + filterWhen = map[string]int{"new-mail": 0x1, "manual": 0x10, "after-junk": 0x20, "sending": 0x40, + "archiving": 0x80, "periodic": 0x100} + filterFields = map[string]bool{"subject": true, "from": true, "to": true, "cc": true, "to or cc": true, + "all addresses": true, "body": true, "date": true, "priority": true, "status": true, "age in days": true, + "size": true, "tag": true, "junk status": true, "junk percent": true, "junk score origin": true, + "has attachment status": true, "date received": true} + filterOps = map[string]bool{"contains": true, "doesn't contain": true, "is": true, "isn't": true, + "begins with": true, "ends with": true, "is empty": true, "isn't empty": true, "is before": true, + "is after": true, "is greater than": true, "is less than": true, "is higher than": true, + "is lower than": true, "is in ab": true, "isn't in ab": true, "matches": true, "doesn't match": true} + // filterActions are the actions a declared filter may take, and whether each names a folder or a value. + filterActions = map[string]string{"Move to folder": "folder", "Copy to folder": "folder", + "Change priority": "value", "Delete": "", "Mark read": "", "Mark unread": "", "Mark flagged": "", + "Ignore thread": "", "Ignore subthread": "", "Watch thread": "", "AddTag": "value", "JunkScore": "value", + "Forward": "value", "Stop execution": "", "Delete from Pop3 server": "", "Leave on Pop3 server": "", + "Fetch body from Pop3Server": ""} +) + +// readConfiguration reads the five settings. A missing file holds none of its kinds (the module is older +// here); a file that carries a key shaped like a credential, an address with a password, an unknown field or a +// value Betterbird cannot be given is refused whole, naming the entry and never a value. +func (m *Machine) readConfiguration() (Configuration, error) { + var c Configuration + var problems []string + for _, file := range []string{configurationFile, tagsFile} { + raw, err := readBounded(m.home(file)) + if errors.Is(err, os.ErrNotExist) { + continue + } + if err != nil { + return Configuration{}, err + } + if err := decodeInto(raw, file, &c); err != nil { + problems = append(problems, err.Error()) + } + } + if len(problems) > 0 { + return Configuration{}, errors.New(strings.Join(problems, "; ")) + } + if p := configurationProblems(c); len(p) > 0 { + return Configuration{}, fmt.Errorf("the configuration is refused: %s", strings.Join(p, "; ")) + } + c.dropNulls() + for _, cal := range c.Calendars { + cal.Color = strings.ToLower(cal.Color) + } + for _, t := range c.Tags { + t.Color = strings.ToUpper(t.Color) + } + return c, nil +} + +// decodeInto reads one settings file into c, refusing a credential-shaped key or an unknown field. +func decodeInto(raw []byte, file string, c *Configuration) error { + var tree any + if err := json.Unmarshal(raw, &tree); err != nil { + return fmt.Errorf("~/%s is not JSON: %v", file, err) + } + if where := secretKeyIn(tree, ""); where != "" { + return fmt.Errorf("~/%s carries %s, a key shaped like a credential: the configuration is declared without "+ + "any password, token or cookie (novox/hq ADR 0329 §2); take it out at the controller's terminal", file, where) + } + dec := json.NewDecoder(bytes.NewReader(raw)) + dec.DisallowUnknownFields() + var into Configuration + if err := dec.Decode(&into); err != nil { + return fmt.Errorf("~/%s does not hold the shape of the settings it takes: %v", file, err) + } + if file == tagsFile { + if into.Identities != nil || into.Calendars != nil || into.AddressBooks != nil || into.Filters != nil { + return fmt.Errorf("~/%s holds only tags", file) + } + c.Tags = into.Tags + return nil + } + if into.Tags != nil { + return fmt.Errorf("~/%s holds no tags: tags are their own setting", file) + } + c.Identities, c.Calendars, c.AddressBooks, c.Filters = into.Identities, into.Calendars, into.AddressBooks, into.Filters + return nil +} + +// dropNulls takes out the entries a node's layer set to null: not on this node. +func (c *Configuration) dropNulls() { + for k, v := range c.Identities { + if v == nil { + delete(c.Identities, k) + } + } + for k, v := range c.Calendars { + if v == nil { + delete(c.Calendars, k) + } + } + for k, v := range c.AddressBooks { + if v == nil { + delete(c.AddressBooks, k) + } + } + for k, v := range c.Filters { + if v == nil { + delete(c.Filters, k) + } + } + for k, v := range c.Tags { + if v == nil { + delete(c.Tags, k) + } + } +} + +// addressPassword says whether an address carries a password in its user part. +func addressPassword(s string) bool { + if userPassword.MatchString(s) { + return true + } + if u, err := url.Parse(s); err == nil && u.User != nil { + _, has := u.User.Password() + return has + } + return false +} + +// withoutPassword is an address with the password of its user part as ‹withheld› (§2). +func withoutPassword(s string) string { + return userPassword.ReplaceAllString(s, "${1}:‹withheld›@") +} + +// davURL says what is wrong with a calendar's or an address book's address, or "". +func davURL(s string) string { + if addressPassword(s) { + return "its address carries a password in its user part, which no setting holds (novox/hq ADR 0329 §2): " + + "Betterbird asks for the password at the first sign-in" + } + u, err := url.Parse(s) + if err != nil || (u.Scheme != "https" && u.Scheme != "http") || u.Host == "" { + return "its address is an https:// (or http://) address of the server" + } + if strings.ContainsAny(s, lineEnds+`"\`) { + return "its address holds a quote, a backslash or a line break" + } + return "" +} + +// configurationProblems is everything wrong with the five settings, each said by its kind and entry name. +func configurationProblems(c Configuration) []string { + var out []string + say := func(kind, name, format string, args ...any) { + out = append(out, fmt.Sprintf("%s %s: ", kind, name)+fmt.Sprintf(format, args...)) + } + text := func(kind, name, field, v string) { + if !plainText.MatchString(v) { + say(kind, name, "%s holds a quote, a backslash or a line break", field) + } + } + for _, name := range sortedKeys(c.Identities) { + id := c.Identities[name] + if id == nil { + continue + } + if !entryName.MatchString(name) { + say(kindIdentities, name, "a name is lower-case letters and digits, starting with a letter") + } + if !accountName.MatchString(id.Account) { + say(kindIdentities, name, "account names an entry of the setting accounts") + } + if id.Address == "" && id.Name != "" { + say(kindIdentities, name, "the account's default identity (no address) takes its name from the setting accounts") + } + if id.Address != "" && !addressShape.MatchString(id.Address) { + say(kindIdentities, name, "the address is not a mail address") + } + if id.ReplyTo != "" && !addressShape.MatchString(id.ReplyTo) { + say(kindIdentities, name, "reply-to is not a mail address") + } + for field, v := range map[string]string{"name": id.Name, "organization": id.Organization} { + text(kindIdentities, name, field, v) + } + for _, line := range id.Signature { + if strings.ContainsAny(line, lineEnds) { + say(kindIdentities, name, "a signature's line holds a line break: each line is its own element") + break + } + } + if id.Key != "" && !profileKey.MatchString(id.Key) { + say(kindIdentities, name, "key is an identity key of the profile, such as id3") + } + } + for _, name := range sortedKeys(c.Calendars) { + cal := c.Calendars[name] + if cal == nil { + continue + } + if !entryName.MatchString(name) { + say(kindCalendars, name, "a name is lower-case letters and digits, starting with a letter") + } + if why := davURL(cal.URL); why != "" { + say(kindCalendars, name, "%s", why) + } + if cal.Color != "" && !colorWord.MatchString(cal.Color) { + say(kindCalendars, name, "a colour is #rrggbb") + } + if cal.Invitations != "" && c.Identities[cal.Invitations] == nil { + say(kindCalendars, name, "invitations names %s, which the setting identities does not hold", cal.Invitations) + } + text(kindCalendars, name, "name", cal.Name) + text(kindCalendars, name, "user", cal.User) + if cal.Key != "" && !tagKey.MatchString(cal.Key) { + say(kindCalendars, name, "key is a calendar key of the profile") + } + } + for _, name := range sortedKeys(c.AddressBooks) { + ab := c.AddressBooks[name] + if ab == nil { + continue + } + if !entryName.MatchString(name) { + say(kindAddressBooks, name, "a name is lower-case letters and digits, starting with a letter") + } + if why := davURL(ab.URL); why != "" { + say(kindAddressBooks, name, "%s", why) + } + text(kindAddressBooks, name, "name", ab.Name) + text(kindAddressBooks, name, "user", ab.User) + if ab.Key != "" && !profileKey.MatchString(ab.Key) { + say(kindAddressBooks, name, "key is an address book key of the profile") + } + } + for _, key := range sortedKeys(c.Tags) { + t := c.Tags[key] + if t == nil { + continue + } + if !tagKey.MatchString(key) { + say(kindTags, key, "a tag's key is letters, digits, $, _, . or -") + } + if t.Color != "" && !colorWord.MatchString(t.Color) { + say(kindTags, key, "a colour is #rrggbb") + } + text(kindTags, key, "name", t.Name) + } + for _, account := range sortedKeys(c.Filters) { + if !accountName.MatchString(account) { + say(kindFilters, account, "filters are keyed by an account's name in the setting accounts") + } + for i, f := range c.Filters[account] { + for _, p := range filterProblems(f) { + say(kindFilters, account, "filter %d: %s", i+1, p) + } + } + } + return out +} + +// filterValue is what a filter's name, value or folder may hold: what its file can carry without quoting. +var filterValue = regexp.MustCompile(`^[^"\\()\n\r]*$`) + +func filterProblems(f Filter) []string { + var out []string + if f.Name == "" || !filterValue.MatchString(f.Name) { + out = append(out, "a name, without a quote, a backslash or a parenthesis") + } + if len(f.When) == 0 { + out = append(out, "when says when it runs: new-mail, manual, after-junk, sending, archiving or periodic") + } + for _, w := range f.When { + if _, ok := filterWhen[w]; !ok { + out = append(out, fmt.Sprintf("when %q is new-mail, manual, after-junk, sending, archiving or periodic", w)) + } + } + switch f.Match { + case "all", "any": + if len(f.Conditions) == 0 { + out = append(out, "match all or any needs a condition") + } + case "every-message": + if len(f.Conditions) > 0 { + out = append(out, "match every-message takes no condition") + } + default: + out = append(out, "match is all, any or every-message") + } + for i, c := range f.Conditions { + if !filterFields[c.Field] && !(strings.HasPrefix(c.Field, `"`) && strings.HasSuffix(c.Field, `"`) && len(c.Field) > 2) { + out = append(out, fmt.Sprintf("condition %d: the field %q is not one Betterbird filters on", i+1, c.Field)) + } + if !filterOps[c.Op] { + out = append(out, fmt.Sprintf("condition %d: the operator %q is not one of Betterbird's", i+1, c.Op)) + } + if !filterValue.MatchString(c.Value) { + out = append(out, fmt.Sprintf("condition %d: the value holds a quote, a backslash, a parenthesis or a line break", i+1)) + } + } + if len(f.Actions) == 0 { + out = append(out, "a filter takes an action") + } + for i, a := range f.Actions { + takes, ok := filterActions[a.Action] + switch { + case !ok: + out = append(out, fmt.Sprintf("action %d: %q is not an action a declared filter takes", i+1, a.Action)) + case takes == "folder" && (a.Folder == "" || a.Value != ""): + out = append(out, fmt.Sprintf("action %d: %s names a folder, by its path within the account", i+1, a.Action)) + case takes == "value" && (a.Value == "" || a.Folder != ""): + out = append(out, fmt.Sprintf("action %d: %s takes a value", i+1, a.Action)) + case takes == "" && (a.Value != "" || a.Folder != ""): + out = append(out, fmt.Sprintf("action %d: %s takes no value", i+1, a.Action)) + } + if !filterValue.MatchString(a.Value) || !filterValue.MatchString(a.Folder) || strings.HasPrefix(a.Folder, "/") || + strings.Contains(a.Folder, "://") { + out = append(out, fmt.Sprintf("action %d: a folder is a path within the account, and a value holds no quote, "+ + "backslash, parenthesis or line break", i+1)) + } + if a.Action == "Forward" && !addressShape.MatchString(a.Value) { + out = append(out, fmt.Sprintf("action %d: Forward names a mail address", i+1)) + } + } + return out +} + +// sortedFilterAccounts are the accounts filters are declared for. +func (c Configuration) sortedFilterAccounts() []string { + out := make([]string, 0, len(c.Filters)) + for a := range c.Filters { + out = append(out, a) + } + sort.Strings(out) + return out +} diff --git a/modules/betterbird/cmd/betterbird-tools/configuration_test.go b/modules/betterbird/cmd/betterbird-tools/configuration_test.go new file mode 100644 index 00000000..c9363119 --- /dev/null +++ b/modules/betterbird/cmd/betterbird-tools/configuration_test.go @@ -0,0 +1,518 @@ +package main + +import ( + "encoding/json" + "os" + "path/filepath" + "reflect" + "strings" + "testing" +) + +// Betterbird's whole configuration (novox/hq ADR 0329, issue 586): identities, calendars, address books, filters +// and tags declared mesh-wide, adopted or created as accounts are, filters owned whole per declared account, what +// Betterbird rewrites written only while it is closed, derived from the nodes' profiles one layer at a time, and +// shown with the operator's own values but never a credential. + +// profileWithConfiguration adds to profileWithAccounts' profile: account1's directory and signature, CalDAV +// calendars (one address twice), an ICS and an Owl calendar, a local and a CardDAV address book, two tags, and +// account1's filter file holding one filter of the person's. +func (f *fake) profileWithConfiguration() { + f.t.Helper() + prefs := f.read(profileDir+"/prefs.js") + + `user_pref("mail.server.server1.directory-rel", "[ProfD]ImapMail/imap.example.org");` + "\n" + + `user_pref("mail.server.server2.directory-rel", "[ProfD]Mail/Local Folders");` + "\n" + + `user_pref("mail.identity.id1.htmlSigText", "Regards");` + "\n" + + `user_pref("calendar.registry.c1.type", "caldav");` + "\n" + + `user_pref("calendar.registry.c1.uri", "https://dav.example.org/cal/home/");` + "\n" + + `user_pref("calendar.registry.c1.name", "Home");` + "\n" + + `user_pref("calendar.registry.c1.color", "#3366ff");` + "\n" + + `user_pref("calendar.registry.c2.type", "caldav");` + "\n" + + `user_pref("calendar.registry.c2.uri", "https://dav.example.org/cal/shared/");` + "\n" + + `user_pref("calendar.registry.c2.name", "Shared A");` + "\n" + + `user_pref("calendar.registry.c3.type", "caldav");` + "\n" + + `user_pref("calendar.registry.c3.uri", "https://dav.example.org/cal/shared/");` + "\n" + + `user_pref("calendar.registry.c3.name", "Shared B");` + "\n" + + `user_pref("calendar.registry.c4.type", "ics");` + "\n" + + `user_pref("calendar.registry.c4.uri", "https://feeds.example.org/secret-token-abc.ics");` + "\n" + + `user_pref("calendar.registry.c4.name", "Holidays");` + "\n" + + `user_pref("calendar.registry.c5.type", "owl");` + "\n" + + `user_pref("calendar.registry.c5.uri", "owl://work@corp.example.com/calendar");` + "\n" + + `user_pref("calendar.registry.c5.name", "Work");` + "\n" + + `user_pref("calendar.list.sortOrder", "c1 c2 c3 c4 c5");` + "\n" + + `user_pref("ldap_2.servers.pab.description", "Personal Address Book");` + "\n" + + `user_pref("ldap_2.servers.pab.dirType", 101);` + "\n" + + `user_pref("ldap_2.servers.ab1.description", "Contacts");` + "\n" + + `user_pref("ldap_2.servers.ab1.dirType", 102);` + "\n" + + `user_pref("ldap_2.servers.ab1.carddav.url", "https://dav.example.org/card/me/");` + "\n" + + `user_pref("ldap_2.servers.ab1.carddav.username", "me");` + "\n" + + `user_pref("mailnews.tags.$label1.tag", "Important");` + "\n" + + `user_pref("mailnews.tags.$label1.color", "#FF0000");` + "\n" + + `user_pref("mailnews.tags.personal.tag", "Personal");` + "\n" + + `user_pref("mailnews.tags.personal.color", "#00AA00");` + "\n" + f.write(profileDir+"/prefs.js", prefs) + f.write(profileDir+"/ImapMail/imap.example.org/"+filterFileName, "version=\"9\"\nlogging=\"no\"\nname=\"Old\"\n"+ + "enabled=\"yes\"\ntype=\"17\"\naction=\"Mark read\"\ncondition=\"AND (from,contains,news)\"\n") + f.write(profileDir+"/Mail/Local Folders/"+filterFileName, "version=\"9\"\nlogging=\"no\"\n") + f.write(profileDir+"/calendar-data/cache.sqlite", "events") +} + +const declaredConfiguration = `{ + "identities": { + "mail": {"account": "mail", "signature": ["-- ", "Other"], "signature-above-quote": true}, + "mailalias": {"account": "mail", "address": "alias@mail.example.org", "name": "Alias", "reply-to": "other@mail.example.org"} + }, + "calendars": { + "home": {"name": "Home", "url": "https://dav.example.org/cal/home", "color": "#3366CC", "user": "other", "invitations": "mail"}, + "work": {"name": "Team", "url": "https://dav.example.org/cal/team/"}, + "shared": {"name": "Shared", "url": "https://dav.example.org/cal/shared/"} + }, + "address-books": { + "contacts": {"name": "Contacts", "url": "https://dav.example.org/card/me/", "user": "other"}, + "team": {"name": "Team", "url": "https://dav.example.org/card/team/"} + }, + "filters": {} +}` + +const declaredTags = `{"tags": {"$label1": {"name": "Important", "color": "#ff0000", "order": 1}, "todo": {"name": "To do", "color": "#0000ff", "order": 2}}}` + +func (f *fake) declareConfiguration(body, tags string) { + f.write(testHome+"/"+configurationFile, body) + f.write(testHome+"/"+tagsFile, tags) +} + +func entriesBy(u UserJS) map[string]resolvedEntry { + out := map[string]resolvedEntry{} + for _, e := range u.Configuration { + out[e.Kind+"/"+e.Name] = e + } + return out +} + +// §2, §4, §6: the settings' shapes; a credential, an address with a password, a key on a tag and the four new +// prefixes in prefs are refused, said without the value. +func TestTheSettingsRefuseACredentialAPasswordedAddressATagsKeyAndTheirPrefixesInPrefs(t *testing.T) { + f := newFake(t) + f.declareConfiguration(declaredConfiguration, declaredTags) + c, err := f.readConfiguration() + if err != nil || len(c.Identities) != 2 || len(c.Calendars) != 3 || len(c.AddressBooks) != 2 || len(c.Tags) != 2 { + t.Fatalf("the declared shapes are taken: %+v %v", c, err) + } + if c.Calendars["home"].Color != "#3366cc" || c.Tags["todo"].Color != "#0000FF" { + t.Errorf("colours are said as Betterbird keeps them: %+v %+v", c.Calendars["home"], c.Tags["todo"]) + } + for _, bad := range []struct{ conf, tags, says string }{ + {`{"calendars": {"home": {"name": "H", "url": "https://dav.example.org/c/", "password": "hunter2"}}}`, `{}`, "shaped like a credential"}, + {`{"calendars": {"home": {"name": "H", "url": "https://me:hunter2@dav.example.org/c/"}}}`, `{}`, "carries a password"}, + {`{"address-books": {"b": {"name": "B", "url": "https://me:hunter2@dav.example.org/b/"}}}`, `{}`, "carries a password"}, + {`{}`, `{"tags": {"todo": {"name": "To do", "key": "x"}}}`, "does not hold the shape"}, + {`{"tags": {"todo": {"name": "To do"}}}`, `{}`, "holds no tags"}, + {`{"identities": {"x": {"account": "mail", "name": "Me"}}}`, `{}`, "default identity"}, + {`{"filters": {"mail": [{"name": "F", "enabled": true, "when": ["manual"], "match": "all", "actions": [{"action": "Delete"}]}]}}`, `{}`, "needs a condition"}, + {`{"filters": {"mail": [{"name": "F", "enabled": true, "when": ["manual"], "match": "every-message", "actions": [{"action": "Move to folder", "folder": "imap://x@y/INBOX"}]}]}}`, `{}`, "path within the account"}, + } { + f.declareConfiguration(bad.conf, bad.tags) + _, err := f.readConfiguration() + if err == nil || !strings.Contains(err.Error(), bad.says) || strings.Contains(err.Error(), "hunter2") { + t.Errorf("%s %s: %v", bad.conf, bad.tags, err) + } + } + // A key is an identity's, a calendar's or an address book's. + f.declareConfiguration(`{"identities": {"x": {"account": "mail", "address": "x@mail.example.org", "key": "id9"}}, + "calendars": {"c": {"name": "C", "url": "https://dav.example.org/c/", "key": "c9"}}, + "address-books": {"b": {"name": "B", "url": "https://dav.example.org/b/", "key": "ab9"}}}`, `{}`) + if _, err := f.readConfiguration(); err != nil { + t.Fatal(err) + } + // prefs refuses the four prefixes the new settings own, and the account's and a credential's as before. + for _, name := range []string{"calendar.registry.c1.color", "calendar.list.sortOrder", "ldap_2.servers.ab1.description", + "mailnews.tags.$label1.color", "mail.identity.id1.htmlSigText", "signon.rememberSignons"} { + body, _ := json.Marshal(map[string]any{"prefs": map[string]any{name: "x"}}) + if _, err := parseDeclaredPrefs(body); err == nil || !strings.Contains(err.Error(), name) { + t.Errorf("%s: %v", name, err) + } + } + got, err := parseDeclaredPrefs([]byte(`{"prefs": {"mail.uifontsize": 14, "mail.chat.enabled": false, "calendar.alarms.show": null, + "intl.date_time.pattern_override.date_short": "yyyy-MM-dd"}}`)) + if err != nil || !reflect.DeepEqual(got, map[string]string{"mail.uifontsize": "14", "mail.chat.enabled": "false", + "intl.date_time.pattern_override.date_short": `"yyyy-MM-dd"`}) { + t.Fatalf("one entry per preference, null taking one out: %v %v", got, err) + } + if _, err := parseDeclaredPrefs([]byte(`{"prefs": {"mail.uifontsize": 14.5}}`)); err == nil { + t.Fatal("a number Betterbird does not keep is refused") + } +} + +// §7, §9: each kind is adopted, created or refused as accounts are; the declared fields go into the user.js, and +// what Betterbird rewrites waits for it to close; leaving the setting releases or removes, never events. +func TestEachKindIsAdoptedOrCreatedAndWhatBetterbirdRewritesWaitsForItToClose(t *testing.T) { + f := newFake(t) + f.profileWithAccounts() + f.profileWithConfiguration() + f.noPackage() + links(t, nil) + f.declare("// none") + f.declareAccounts(adoptable) + f.declareConfiguration(declaredConfiguration, declaredTags) + before := f.read(profileDir + "/prefs.js") + + // Betterbird runs: the user.js is placed, prefs.js and the filter files wait (§9). + u, err := f.userJS(true) + if err != nil { + t.Fatal(err) + } + by := entriesBy(u) + for want, how := range map[string][2]string{ + "identities/mail": {adopted, "id1"}, "identities/mailalias": {created, "ident_mesh_mailalias"}, + "calendars/home": {adopted, "c1"}, "calendars/work": {created, "mesh_work"}, "calendars/shared": {refused, ""}, + "address-books/contacts": {adopted, "ab1"}, "address-books/team": {created, "mesh_team"}, + "tags/$label1": {adopted, "$label1"}, "tags/todo": {created, "todo"}, + } { + if e := by[want]; e.How != how[0] || e.Key != how[1] { + t.Errorf("%s: %+v, want %v", want, e, how) + } + } + if why := by["calendars/shared"].Why; !strings.Contains(why, "2 entries") || !strings.Contains(why, "c2, c3") || + !strings.Contains(why, "key in this node's layer") { + t.Errorf("an ambiguous match is refused with the keys and the remedy: %s", why) + } + placed := f.read(profileDir + "/user.js") + for _, want := range []string{ + `user_pref("mail.identity.id1.htmlSigText", "-- \nOther");`, + `user_pref("mail.identity.id1.reply_on_top", 1);`, + `user_pref("mail.identity.id1.sig_bottom", false);`, + `user_pref("mail.identity.ident_mesh_mailalias.useremail", "alias@mail.example.org");`, + `user_pref("mail.identity.ident_mesh_mailalias.smtpServer", "smtp1");`, + `user_pref("calendar.registry.c1.color", "#3366cc");`, + `user_pref("calendar.registry.c1.imip.identity.key", "id1");`, + `user_pref("calendar.registry.mesh_work.type", "caldav");`, + `user_pref("ldap_2.servers.ab1.carddav.username", "other");`, + `user_pref("ldap_2.servers.mesh_team.dirType", 102);`, + `user_pref("mailnews.tags.todo.color", "#0000FF");`, + `user_pref("mailnews.tags.todo.ordinal", "1002");`, + } { + if !strings.Contains(placed, want) { + t.Errorf("the user.js lacks %s", want) + } + } + for _, never := range []string{"calendar.registry.c2.", "calendar.registry.c3.", "calendar.registry.c4.", "calendar.registry.c5.", + "ldap_2.servers.pab.", "mailnews.tags.personal.", "calendar.list.sortOrder"} { + if strings.Contains(placed, never) { + t.Errorf("the user.js writes %s, which nothing declares or which is refused", never) + } + } + if !strings.HasPrefix(u.Closed, "not written") || !strings.Contains(u.Closed, "running") || + u.Waiting[kindIdentities] != 1 || u.Waiting[kindCalendars] != 1 { + t.Fatalf("while Betterbird runs prefs.js is not written, and the waiting changes are counted: %q %v", u.Closed, u.Waiting) + } + if f.read(profileDir+"/prefs.js") != before { + t.Fatal("prefs.js changed while Betterbird runs") + } + ch, _ := f.Check() + waits := false + for _, x := range ch.Findings { + waits = waits || (strings.Contains(x.What, "change(s) to calendars are not written yet") && strings.Contains(x.Do, "betterbird_restart")) + } + if !waits { + t.Fatalf("the check counts the waiting changes per kind: %+v", ch.Findings) + } + + // A key in the node's layer settles the ambiguous calendar; one the profile does not hold is refused. + f.declareConfiguration(strings.Replace(declaredConfiguration, `"url": "https://dav.example.org/cal/shared/"}`, + `"url": "https://dav.example.org/cal/shared/", "key": "c3"}`, 1), declaredTags) + if u, _ = f.userJS(false); entriesBy(u)["calendars/shared"].Key != "c3" || entriesBy(u)["calendars/shared"].How != adopted { + t.Fatalf("the named key is adopted: %+v", entriesBy(u)["calendars/shared"]) + } + f.declareConfiguration(strings.Replace(declaredConfiguration, `"url": "https://dav.example.org/cal/shared/"}`, + `"url": "https://dav.example.org/cal/shared/", "key": "c9"}`, 1), declaredTags) + if u, _ = f.userJS(false); entriesBy(u)["calendars/shared"].How != refused || !strings.Contains(entriesBy(u)["calendars/shared"].Why, "c9") { + t.Fatalf("a key the profile does not hold is refused: %+v", entriesBy(u)["calendars/shared"]) + } + f.declareConfiguration(strings.Replace(declaredConfiguration, `"url": "https://dav.example.org/cal/shared/"}`, + `"url": "https://dav.example.org/cal/shared/", "key": "c1"}`, 1), declaredTags) + if u, _ = f.userJS(false); entriesBy(u)["calendars/shared"].How != refused { + t.Fatalf("a key whose entry does not match is refused: %+v", entriesBy(u)["calendars/shared"]) + } + f.declareConfiguration(declaredConfiguration, declaredTags) + + // Closed: the lists gain the module's own keys at their end, every other line as it was. + stopBetterbird(t, f) + f.write(profileDir+"/lock", "") + if err := f.Prepare(); err == nil || !strings.Contains(err.Error(), "locked") { + t.Fatalf("the profile's lock holds the writes back: %v", err) + } + os.Remove(filepath.Join(f.Root, profileDir, "lock")) + if err := f.Prepare(); err != nil && !strings.Contains(err.Error(), "calendars shared is refused") { + t.Fatal(err) + } + after := f.read(profileDir + "/prefs.js") + for _, want := range []string{`user_pref("mail.account.account1.identities", "id1,ident_mesh_mailalias");`, + `user_pref("calendar.list.sortOrder", "c1 c2 c3 c4 c5 mesh_work");`} { + if !strings.Contains(after, want) { + t.Errorf("prefs.js lacks %s:\n%s", want, after) + } + } + bl, al := strings.Split(before, "\n"), strings.Split(after, "\n") + for i := range bl { + if bl[i] != al[i] && !strings.Contains(bl[i], ".identities") && !strings.Contains(bl[i], "sortOrder") { + t.Errorf("line %d changed: %q to %q", i+1, bl[i], al[i]) + } + } + + // Leaving: an adopted entry is released (its lines go, the profile keeps it), a created one's preferences are + // removed; the calendar's events stay. + f.write(profileDir+"/prefs.js", f.read(profileDir+"/prefs.js")+ + `user_pref("calendar.registry.mesh_work.calendarSyncState", "x");`+"\n"+`user_pref("mailnews.tags.todo.tag", "To do");`+"\n") + f.declareConfiguration(`{"identities": {"mail": {"account": "mail"}}}`, `{"tags": {}}`) + if err := f.Prepare(); err != nil { + t.Fatal(err) + } + final := f.read(profileDir + "/prefs.js") + for _, gone := range []string{"mesh_work", "ident_mesh_mailalias", "mailnews.tags.todo."} { + if strings.Contains(final, gone) { + t.Errorf("what the module created and the setting left is removed: %s is still there", gone) + } + } + for _, kept := range []string{`calendar.registry.c1.uri`, `ldap_2.servers.ab1.carddav.url`, `mailnews.tags.$label1.tag`, + `calendar.registry.c4.uri`} { + if !strings.Contains(final, kept) { + t.Errorf("released or undeclared, %s stays the person's", kept) + } + } + if f.read(profileDir+"/calendar-data/cache.sqlite") != "events" { + t.Error("a calendar's events are never touched") + } + if placed := f.read(profileDir + "/user.js"); strings.Contains(placed, "calendar.registry.c1.") || strings.Contains(placed, "ab1") { + t.Errorf("a released entry's lines leave the user.js:\n%s", placed) + } +} + +const declaredFilters = `{"filters": {"mail": [ + {"name": "Receipts", "enabled": true, "when": ["new-mail", "manual"], "match": "all", + "conditions": [{"field": "subject", "op": "contains", "value": "receipt"}], + "actions": [{"action": "Move to folder", "folder": "INBOX/Receipts"}, {"action": "Forward", "value": "archive@mail.example.org"}]}, + {"name": "Everything", "enabled": false, "when": ["manual"], "match": "every-message", "actions": [{"action": "Mark read"}]} +]}}` + +const renderedFilters = "version=\"9\"\nlogging=\"no\"\n" + + "name=\"Receipts\"\nenabled=\"yes\"\ntype=\"17\"\naction=\"Move to folder\"\n" + + "actionValue=\"imap://other2@imap.example.org/INBOX/Receipts\"\naction=\"Forward\"\nactionValue=\"archive@mail.example.org\"\n" + + "condition=\"AND (subject,contains,receipt)\"\n" + + "name=\"Everything\"\nenabled=\"no\"\ntype=\"16\"\naction=\"Mark read\"\ncondition=\"ALL\"\n" + +// §8: an account named in filters has its msgFilterRules.dat written whole from the list, the replaced file kept; +// an account never named keeps its file byte for byte; one taken out keeps the file it last had. +func TestFiltersAreOwnedWholePerDeclaredAccount(t *testing.T) { + f := newFake(t) + f.profileWithAccounts() + f.profileWithConfiguration() + f.noPackage() + links(t, nil) + f.declare("// none") + f.declareAccounts(adoptable) + f.declareConfiguration(declaredFilters, `{"tags": {}}`) + filters := profileDir + "/ImapMail/imap.example.org/" + filterFileName + local := profileDir + "/Mail/Local Folders/" + filterFileName + old, oldLocal := f.read(filters), f.read(local) + + u, err := f.userJS(true) + if err != nil { + t.Fatal(err) + } + if u.Waiting[kindFilters] != 1 || f.read(filters) != old { + t.Fatalf("while Betterbird runs the filter file waits: %v", u.Waiting) + } + stopBetterbird(t, f) + if err := f.Prepare(); err != nil { + t.Fatal(err) + } + if got := f.read(filters); got != renderedFilters { + t.Fatalf("the file is the declared list, its folder this node's address:\n%s", got) + } + keptPath := testHome + "/.config/betterbird/kept/60016jsp.default-default/mail." + filterFileName + if f.read(keptPath) != old { + t.Fatal("the replaced file is kept in the module's own directory") + } + if f.read(local) != oldLocal { + t.Fatal("an account never named keeps its file byte for byte") + } + if parsed := parseFilters(renderedFilters, "imap://other2@imap.example.org"); len(parsed.NotDeclarable) != 0 || + mustJSON(parsed.Filters) != mustJSON(mustConfig(t, declaredFilters).Filters["mail"]) { + t.Fatalf("the file reads back as the declared list: %+v", parsed) + } + + // A filter the person adds in Betterbird is drift first, and gone at the next apply; the kept file stays the first. + f.write(filters, renderedFilters+"name=\"Mine\"\nenabled=\"yes\"\ntype=\"16\"\naction=\"Delete\"\ncondition=\"ALL\"\n") + d, _ := f.Drift(kindFilters) + if len(d.Kinds) != 1 || len(d.Kinds[0].Declared) != 1 || d.Kinds[0].Declared[0].State != "differs" || + !strings.Contains(mustJSON(d.Kinds[0].Declared[0].Differs), "Mine") { + t.Fatalf("drift shows the added filter: %+v", d) + } + if err := f.Prepare(); err != nil { + t.Fatal(err) + } + if f.read(filters) != renderedFilters || f.read(keptPath) != old { + t.Fatal("the added filter is gone, and the kept file is still the one first replaced") + } + + // Taken out of filters: the module stops writing, the last file stays the person's, the kept copy is named. + f.declareConfiguration(`{"filters": {}}`, `{"tags": {}}`) + f.write(filters, renderedFilters+"name=\"Later\"\nenabled=\"yes\"\ntype=\"16\"\naction=\"Delete\"\ncondition=\"ALL\"\n") + lastHad := f.read(filters) + if err := f.Prepare(); err != nil { + t.Fatal(err) + } + if f.read(filters) != lastHad || f.read(keptPath) != old { + t.Fatal("an account taken out keeps its file byte for byte, and the kept copy stays") + } + l, _ := f.List(kindFilters) + if l.Kept["mail"] != "~/.config/betterbird/kept/60016jsp.default-default/mail."+filterFileName { + t.Fatalf("betterbird_filters says where the kept copy is: %+v", l.Kept) + } + d, _ = f.Drift(kindFilters) + if len(d.Kinds[0].Declared) != 0 || len(d.Kinds[0].Undeclared) != 0 { + t.Fatalf("drift no longer counts that account's filters: %+v", d.Kinds[0]) + } +} + +func mustConfig(t *testing.T, body string) Configuration { + t.Helper() + var c Configuration + if err := decodeInto([]byte(body), configurationFile, &c); err != nil { + t.Fatal(err) + } + return c +} + +// §11: per kind and entry, the field that differs with both values; an undeclared entry is listed; once the module's +// user.js is placed, what it declares holds. +func TestDriftNamesEachDifferingFieldAndTheUndeclaredEntries(t *testing.T) { + f := newFake(t) + f.profileWithAccounts() + f.profileWithConfiguration() + f.noPackage() + links(t, nil) + f.declare("// none") + f.declareAccounts(adoptable) + f.declareConfiguration(strings.Replace(declaredConfiguration, `"work": {"name": "Team", "url": "https://dav.example.org/cal/team/"},`, "", 1), declaredTags) + d, err := f.Drift("") + if err != nil { + t.Fatal(err) + } + by := map[string]KindDrift{} + for _, k := range d.Kinds { + by[k.Kind] = k + } + field := func(kind, name, f string) *fieldDiff { + for _, e := range by[kind].Declared { + if e.Name == name { + for _, x := range e.Differs { + if x.Field == f { + return &x + } + } + } + } + return nil + } + if x := field(kindIdentities, "mail", "signature"); x == nil || mustJSON(x.Profile) != `["Regards"]` { + t.Errorf("the signature differs, with both values: %+v", by[kindIdentities]) + } + if x := field(kindCalendars, "home", "color"); x == nil || x.Declared != "#3366cc" || x.Profile != "#3366ff" { + t.Errorf("the colour differs: %+v", by[kindCalendars]) + } + if x := field(kindAddressBooks, "contacts", "user"); x == nil || x.Profile != "me" { + t.Errorf("the user differs: %+v", by[kindAddressBooks]) + } + states := map[string]string{} + for _, k := range d.Kinds { + for _, e := range k.Declared { + states[k.Kind+"/"+e.Name] = e.State + } + } + if states["tags/todo"] != "lacks" || states["calendars/shared"] != refused || states["tags/$label1"] != "holds" || + states["address-books/team"] != "lacks" { + t.Errorf("lacks, refused and holds: %v", states) + } + undeclared := mustJSON(d) + for _, want := range []string{`"key":"personal"`, `"key":"c2"`, `"key":"c3"`} { + if !strings.Contains(undeclared, want) { + t.Errorf("the undeclared entries are listed: lacks %s", want) + } + } + if _, err := f.userJS(true); err != nil { + t.Fatal(err) + } + d, _ = f.Drift("") + for _, k := range d.Kinds { + for _, e := range k.Declared { + if e.State == "differs" { + t.Errorf("once the user.js is placed, %s %s holds: %+v", k.Kind, e.Name, e) + } + } + } +} + +// §1, §2, §13: the listing tools answer the operator's own values, Owl's marked, an ICS calendar by its name only, +// and never a credential: through the tools, a password in an address and a field named for one are withheld. +func TestTheListingsShowTheValuesOwlsMarkedAnICSByNameAndNoCredential(t *testing.T) { + f := newFake(t) + f.profileWithAccounts() + f.profileWithConfiguration() + f.write(profileDir+"/prefs.js", f.read(profileDir+"/prefs.js")+ + `user_pref("calendar.registry.c6.type", "caldav");`+"\n"+ + `user_pref("calendar.registry.c6.uri", "https://other:hunter2@dav.example.org/cal/x/");`+"\n"+ + `user_pref("calendar.registry.c6.name", "Old");`+"\n") + f.write(profileDir+"/logins.json", `{"logins":[{"hostname":"https://dav.example.org","encryptedPassword":"hunter3"}]}`) + links(t, nil) + f.declare("// none") + f.declareAccounts(adoptable) + f.declareConfiguration(declaredConfiguration, declaredTags) + old := machine + machine = f.Machine + t.Cleanup(func() { machine = old }) + answers := map[string]string{} + for _, tool := range tools() { + if strings.HasPrefix(tool.Name, "betterbird_") && contains([]string{"betterbird_identities", "betterbird_calendars", + "betterbird_address_books", "betterbird_filters", "betterbird_tags", "betterbird_accounts", "betterbird_drift"}, tool.Name) { + out, err := tool.Run(map[string]any{}) + if err != nil { + t.Fatalf("%s: %v", tool.Name, err) + } + answers[tool.Name] = mustJSON(out) + } + } + cals := answers["betterbird_calendars"] + for _, shown := range []string{"https://dav.example.org/cal/home/", `"name":"Work","url":"owl://work@corp.example.com/calendar"`, + `"owl":true`, `"name":"Holidays"`} { + if !strings.Contains(cals, shown) { + t.Errorf("betterbird_calendars shows %s: %s", shown, cals) + } + } + if strings.Contains(cals, "secret-token") { + t.Error("an ICS calendar is listed by its name only") + } + for name, a := range answers { + if strings.Contains(a, "hunter2") || strings.Contains(a, "hunter3") { + t.Errorf("%s carries a credential: %s", name, a) + } + } + if !strings.Contains(cals, "https://other:‹withheld›@dav.example.org/cal/x/") { + t.Errorf("an address's password is shown as ‹withheld›: %s", cals) + } + for _, shown := range []string{`"signature":["Regards"]`, "other@mail.example.org"} { + if !strings.Contains(answers["betterbird_identities"], shown) { + t.Errorf("betterbird_identities shows %s", shown) + } + } + if got := mustJSON(credentialChecked(map[string]any{"name": "x", "password": "p", "nested": []any{map[string]any{"oauthToken": "t"}}})); got != + `{"name":"x","nested":[{"oauthToken":"‹withheld›"}],"password":"‹withheld›"}` { + t.Errorf("a field named for a credential is withheld: %s", got) + } + // The start's words reach the journal: a refusal names kinds, names and keys, never an address. + stopBetterbird(t, f) + err := f.Prepare() + if err == nil || !strings.Contains(err.Error(), "calendars shared is refused") || strings.Contains(err.Error(), "example.org") { + t.Fatalf("%v", err) + } +} diff --git a/modules/betterbird/cmd/betterbird-tools/derive.go b/modules/betterbird/cmd/betterbird-tools/derive.go new file mode 100644 index 00000000..6d9b0484 --- /dev/null +++ b/modules/betterbird/cmd/betterbird-tools/derive.go @@ -0,0 +1,700 @@ +package main + +// The five settings derived from the desktop nodes' profiles and proposed for the operator's Approve, one layer +// at a time (novox/hq ADR 0329 §10, §12). +// +// betterbird_derive asks every desktop node that holds Betterbird for what its profile holds (its own derive with +// here), and lays the answers out per layer: an entry found on every such node goes into the mesh-wide layer, with +// the value of the node the derive was called on, and the answer lists every field another node holds differently, +// which then shows as drift there; an entry found on some nodes only goes into each of those nodes' layers, so +// nothing is added to another node until the operator moves it. With propose it proposes the next layer through +// the controller's settings verb: first the mesh-wide one, then each node's that needs one, each only once no +// proposal of the module's is open, against the layer as it then stands, which it carries whole beside the derived +// kinds (a layer is replaced whole). The ask shows the values in their shape (ADR 0277 §6); the operator reads them +// whole with `settings proposals ` at their desk. + +import ( + "encoding/json" + "errors" + "fmt" + "regexp" + "sort" + "strings" +) + +// derivedEntry is one entry a node's profile holds, in the setting's shape. Match is what makes two nodes' entries +// the same one: an identity's account and address, a calendar's or an address book's address, a tag's key, an +// account's name for its filters. +type derivedEntry struct { + Match string `json:"match"` + Name string `json:"name"` + Value json.RawMessage `json:"value"` +} + +// nodeDerivation is what one node's profile holds of the five kinds. +type nodeDerivation struct { + Node string `json:"node"` + Kinds map[string][]derivedEntry `json:"kinds"` + // NotDerived names what a setting does not hold, by name and why: Owl's, an ICS calendar, a local address + // book, an address carrying a password, a filter a setting cannot say. + NotDerived []string `json:"not_derived,omitempty"` +} + +var nameChars = regexp.MustCompile(`[^a-z0-9]`) + +// entryNameFor is a name of the operator's kind for an entry: lower-case letters and digits from its words, +// unique among those used. +func entryNameFor(words string, fallback string, used map[string]bool) string { + base := nameChars.ReplaceAllString(strings.ToLower(words), "") + if base == "" || base[0] < 'a' || base[0] > 'z' { + base = fallback + base + } + if len(base) > 28 { + base = base[:28] + } + name := base + for i := 2; used[name]; i++ { + name = fmt.Sprintf("%s%d", base, i) + } + used[name] = true + return name +} + +func raw(v any) json.RawMessage { + b, _ := json.Marshal(v) + return b +} + +// deriveHere reads this node's profile into the five kinds. +func (m *Machine) deriveHere(kinds []string) (nodeDerivation, error) { + d := nodeDerivation{Node: nodeName(), Kinds: map[string][]derivedEntry{}} + p, err := m.profile() + if err != nil { + return d, err + } + prefs := mergedPrefs(p.path) + accounts, _, err := m.readAccounts() + if err != nil { + return d, err + } + nameOf := map[string]string{} + serverOf := map[string]string{} + for _, r := range resolveAccounts(accounts, prefs, m.readRecord(p.Dir)) { + if r.How != refused { + nameOf[r.Keys.Account], serverOf[r.name] = r.name, r.Keys.Server + } + } + want := map[string]bool{} + for _, k := range kinds { + want[k] = true + } + // Identities, by their account's name; the default carries no name or address (the setting accounts' own). + identityName := map[string]string{} + used := map[string]bool{} + byAccount := map[string]bool{} + for _, pi := range m.profileIdentities(prefs) { + account, declared := nameOf[pi.AccountKey] + switch { + case pi.Owl: + if pi.Default { + d.NotDerived = append(d.NotDerived, fmt.Sprintf("identities: %s <%s> is Owl's (novox/hq ADR 0316 §13)", + pi.Value.Name, pi.Value.Address)) + } + continue + case !declared: + if !byAccount[pi.AccountKey] { + byAccount[pi.AccountKey] = true + d.NotDerived = append(d.NotDerived, fmt.Sprintf("identities: those of the profile's account %s, which the "+ + "setting accounts does not hold", pi.AccountKey)) + } + continue + } + v := pi.Value + v.Account = account + e := derivedEntry{} + if pi.Default { + v.Address, v.Name = "", "" + e.Match, e.Name = account+"|", entryNameFor(account, "id", used) + } else { + local, _, _ := strings.Cut(v.Address, "@") + e.Match, e.Name = account+"|"+strings.ToLower(v.Address), entryNameFor(account+local, "id", used) + } + identityName[pi.Key] = e.Name + e.Value = raw(v) + if want[kindIdentities] { + d.Kinds[kindIdentities] = append(d.Kinds[kindIdentities], e) + } + } + // Calendars: CalDAV only; Owl's are listed, an ICS one by its name only, a local one by its name (§13). + used = map[string]bool{} + for _, pc := range profileCalendars(prefs) { + if !want[kindCalendars] { + break + } + switch { + case pc.Kind == "owl" || strings.HasPrefix(pc.Kind, "owl"): + d.NotDerived = append(d.NotDerived, fmt.Sprintf("calendars: %s (%s) is Owl's", pc.Value.Name, pc.Value.URL)) + continue + case pc.Kind == "ics": + d.NotDerived = append(d.NotDerived, fmt.Sprintf("calendars: %s is subscribed by an ICS address, which is "+ + "often itself the secret: listed by its name only, never declared (novox/hq ADR 0329 §13)", pc.Value.Name)) + continue + case pc.Kind != "caldav": + d.NotDerived = append(d.NotDerived, fmt.Sprintf("calendars: %s is %s, kept in the profile: its events are never declared", + pc.Value.Name, pc.Kind)) + continue + case addressPassword(pc.Value.URL): + d.NotDerived = append(d.NotDerived, fmt.Sprintf("calendars: %s at %s carries a password in its address: refused "+ + "(novox/hq ADR 0329 §2)", pc.Value.Name, withoutPassword(pc.Value.URL))) + continue + } + v := pc.Value + v.Invitations = identityName[pc.IdentityKey] + d.Kinds[kindCalendars] = append(d.Kinds[kindCalendars], derivedEntry{Match: strings.ToLower(strings.TrimRight(v.URL, "/")), + Name: entryNameFor(v.Name, "calendar", used), Value: raw(v)}) + } + // Address books: CardDAV only. + used = map[string]bool{} + for _, b := range profileAddressBooks(prefs) { + if !want[kindAddressBooks] { + break + } + switch { + case !b.CardDAV: + d.NotDerived = append(d.NotDerived, fmt.Sprintf("address-books: %s is %s: its contacts are never declared", b.Value.Name, b.Kind)) + continue + case addressPassword(b.Value.URL): + d.NotDerived = append(d.NotDerived, fmt.Sprintf("address-books: %s at %s carries a password in its address: "+ + "refused (novox/hq ADR 0329 §2)", b.Value.Name, withoutPassword(b.Value.URL))) + continue + } + d.Kinds[kindAddressBooks] = append(d.Kinds[kindAddressBooks], derivedEntry{ + Match: strings.ToLower(strings.TrimRight(b.Value.URL, "/")), Name: entryNameFor(b.Value.Name, "book", used), Value: raw(b.Value)}) + } + // Tags, by their own key. + if want[kindTags] { + for _, t := range profileTags(prefs) { + d.Kinds[kindTags] = append(d.Kinds[kindTags], derivedEntry{Match: t.Key, Name: t.Key, Value: raw(t.Value)}) + } + } + // Filters of each declared account whose file holds one or more: an account with none keeps its file as the + // person's until a setting names it. + if want[kindFilters] { + for _, name := range sortedKeys(serverOf) { + dir := m.accountDirectory(p.path, prefs, serverOf[name]) + if dir == "" { + continue + } + text, err := readBounded(dir + "/" + filterFileName) + if err != nil { + continue + } + parsed := parseFilters(string(text), folderBase(prefs, serverOf[name])) + for _, why := range parsed.NotDeclarable { + d.NotDerived = append(d.NotDerived, "filters of "+name+": "+why) + } + if len(parsed.Filters) > 0 { + d.Kinds[kindFilters] = append(d.Kinds[kindFilters], derivedEntry{Match: name, Name: name, Value: raw(parsed.Filters)}) + } + } + } + return d, nil +} + +// difference is one field another node holds differently from the value the mesh-wide layer gets. +type difference struct { + Kind string `json:"kind"` + Entry string `json:"entry"` + Node string `json:"node"` + Field string `json:"field"` + MeshWide any `json:"mesh_wide"` + There any `json:"there"` +} + +// DeriveAnswer is what betterbird_derive answers its caller: the values it derived, which ADR 0329 §10 lets it show. +type DeriveAnswer struct { + // Here is this node's own derivation, when asked with here. + Here *nodeDerivation `json:"here,omitempty"` + // From are the desktop nodes derived from, and NotHeard those holding Betterbird that did not answer. + From []string `json:"derived_from,omitempty"` + NotHeard []string `json:"not_heard,omitempty"` + // Layers are the derived values per layer: "mesh-wide", then each node's own. + Layers map[string]map[string]any `json:"layers,omitempty"` + Differences []difference `json:"differences,omitempty"` + NotDerived map[string][]string `json:"not_derived,omitempty"` + // Next is the layer a propose proposes now, or why none. + Next string `json:"next,omitempty"` + Proposed string `json:"proposed,omitempty"` + Note string `json:"note,omitempty"` +} + +const meshWide = "mesh-wide" + +// Derive derives the kinds asked for (all five when none) from the desktop nodes, and with propose proposes the +// next layer. prefs are values for the setting prefs to carry in the mesh-wide layer (issue 586: the operator's +// setup moved out of the catalogue); layer names the layer to propose instead of the next in order. +func (m *Machine) Derive(kinds []string, here, propose bool, prefs map[string]any, layer string) (DeriveAnswer, error) { + var a DeriveAnswer + if len(kinds) == 0 { + kinds = allKinds + } + for _, k := range kinds { + if !contains(allKinds, k) { + return a, fmt.Errorf("kinds are %s; %q is not one", strings.Join(allKinds, ", "), k) + } + } + if here { + d, err := m.deriveHere(kinds) + if err != nil { + return a, err + } + a.Here = &d + return a, nil + } + self := nodeName() + if self == "" { + return a, errors.New("this runtime does not say which machine it serves (MESH_NODE), so the node derived on is unknown") + } + holders, err := holdersOfBetterbird() + if err != nil { + return a, err + } + derived := map[string]nodeDerivation{} + for _, n := range holders { + if n == self { + d, err := m.deriveHere(kinds) + if err != nil { + return a, fmt.Errorf("this node's profile: %v", err) + } + derived[n] = d + continue + } + d, err := deriveOn(n, kinds) + if err != nil { + a.NotHeard = append(a.NotHeard, n+": "+err.Error()) + continue + } + derived[n] = d + } + if _, ok := derived[self]; !ok { + return a, fmt.Errorf("%s does not hold Betterbird as the controller says: derive on a desktop node", self) + } + a.From = sortedKeys(derived) + a.Layers, a.Differences = layOut(self, derived, kinds) + a.NotDerived = map[string][]string{} + for n, d := range derived { + if len(d.NotDerived) > 0 { + a.NotDerived[n] = d.NotDerived + } + } + if len(a.NotDerived) == 0 { + a.NotDerived = nil + } + if len(prefs) > 0 { + a.Layers[meshWide]["prefs"] = prefs + } + a.Note = "the values each layer would get; an entry on every desktop node is mesh-wide with this node's value, one " + + "on some nodes only is in those nodes' own layers. With propose, the next layer is proposed for the operator's " + + "Approve, carrying the layer as it stands beside these kinds; nothing changes until then (novox/hq ADR 0329 §10)" + if !propose { + return a, nil + } + if len(a.NotHeard) > 0 { + return a, fmt.Errorf("nothing was proposed: %d desktop node(s) holding Betterbird did not answer (%s), and a "+ + "layer derived without them would put on every node what only some hold; derive again when they are awake", + len(a.NotHeard), strings.Join(a.NotHeard, "; ")) + } + if open, err := openProposals(); err != nil { + return a, fmt.Errorf("nothing was proposed: whether a proposal of the module's is open cannot be read: %v", err) + } else if len(open) > 0 { + return a, fmt.Errorf("nothing was proposed: the module's proposal %s waits for the operator; the next layer is "+ + "proposed once it is set or declined (novox/hq ADR 0329 §10, ADR 0277 §10)", strings.Join(open, ", ")) + } + order := []string{meshWide} + for _, n := range sortedKeys(a.Layers) { + if n != meshWide { + order = append(order, n) + } + } + if layer != "" { + if _, ok := a.Layers[layer]; !ok { + return a, fmt.Errorf("nothing was proposed: layer is %s", strings.Join(order, " or ")) + } + order = []string{layer} + } + for _, l := range order { + node := l + if l == meshWide { + node = "" + } + current, err := readLayer(node) + if err != nil { + return a, fmt.Errorf("nothing was proposed: %v", err) + } + next := withKinds(current, a.Layers[l]) + if l == meshWide { + if prefs, ok := next["prefs"]; ok { + body, _ := json.Marshal(map[string]any{"prefs": prefs}) + if _, err := parseDeclaredPrefs(body); err != nil { + return a, fmt.Errorf("nothing was proposed: %v", err) + } + } + } + if err := proposable(next, l == meshWide); err != nil { + return a, fmt.Errorf("nothing was proposed for %s: %v", l, err) + } + if mustJSON(next) == mustJSON(current) { + continue + } + a.Next = l + said, err := proposeLayer(node, next, true) + a.Proposed = m.maskValue(said) + if err != nil { + return a, err + } + a.Note = "proposed the " + l + " layer for the operator's Approve; the next layer is proposed by calling again " + + "once this one is set or declined" + return a, nil + } + a.Next = "none: every layer already holds what was derived" + return a, nil +} + +// proposable checks a layer's derived kinds as the module would read them, so the operator is never asked to set +// a value the module then refuses. A key to adopt names one node's profile, so the mesh-wide layer holds none (§4). +func proposable(layer map[string]any, mesh bool) error { + conf := map[string]any{} + for _, k := range []string{kindIdentities, kindCalendars, kindAddressBooks, kindFilters} { + if v, ok := layer[k]; ok { + conf[k] = v + } + } + var c Configuration + if err := decodeInto([]byte(mustJSON(conf)), configurationFile, &c); err != nil { + return err + } + if v, ok := layer[kindTags]; ok { + if err := decodeInto([]byte(mustJSON(map[string]any{kindTags: v})), tagsFile, &c); err != nil { + return err + } + } + // A node's layer may name an identity the mesh-wide layer holds for a calendar's invitations: that is judged + // where the layers meet, on the node. + var problems []string + if mesh { + for name, id := range c.Identities { + if id != nil && id.Key != "" { + problems = append(problems, "identities "+name+": a key is given in a node's own layer only") + } + } + for name, cal := range c.Calendars { + if cal != nil && cal.Key != "" { + problems = append(problems, "calendars "+name+": a key is given in a node's own layer only") + } + } + for name, ab := range c.AddressBooks { + if ab != nil && ab.Key != "" { + problems = append(problems, "address-books "+name+": a key is given in a node's own layer only") + } + } + } + for _, p := range configurationProblems(c) { + if !strings.Contains(p, "invitations names") { + problems = append(problems, p) + } + } + if len(problems) > 0 { + return errors.New(strings.Join(problems, "; ")) + } + return nil +} + +// layOut puts each kind's entries into the layers (§10). +func layOut(self string, derived map[string]nodeDerivation, kinds []string) (map[string]map[string]any, []difference) { + layers := map[string]map[string]any{meshWide: {}} + var diffs []difference + nodes := sortedKeys(derived) + for _, kind := range kinds { + type seen struct { + by map[string]derivedEntry + } + groups := map[string]*seen{} + var order []string + for _, n := range append([]string{self}, nodes...) { + for _, e := range derived[n].Kinds[kind] { + g := groups[e.Match] + if g == nil { + g = &seen{by: map[string]derivedEntry{}} + groups[e.Match] = g + order = append(order, e.Match) + } + if _, already := g.by[n]; !already { + g.by[n] = e + } + } + } + mesh := map[string]any{} + used := map[string]bool{} + for _, match := range order { + g := groups[match] + if len(g.by) == len(nodes) { + mine := g.by[self] + name := uniqueName(mine.Name, used) + var v any + _ = json.Unmarshal(mine.Value, &v) + mesh[name] = v + for _, n := range nodes { + if n == self { + continue + } + diffs = append(diffs, fieldDifferences(kind, name, n, mine.Value, g.by[n].Value)...) + } + } + } + for _, match := range order { + g := groups[match] + if len(g.by) == len(nodes) { + continue + } + for _, n := range sortedKeys(g.by) { + e := g.by[n] + if layers[n] == nil { + layers[n] = map[string]any{} + } + nodeKind, _ := layers[n][kind].(map[string]any) + if nodeKind == nil { + nodeKind = map[string]any{} + layers[n][kind] = nodeKind + } + var v any + _ = json.Unmarshal(e.Value, &v) + nodeKind[uniqueName(e.Name, used)] = v + } + } + if len(mesh) > 0 { + layers[meshWide][kind] = mesh + } + } + return layers, diffs +} + +func uniqueName(name string, used map[string]bool) string { + n := name + for i := 2; used[n]; i++ { + n = fmt.Sprintf("%s%d", name, i) + } + used[n] = true + return n +} + +// fieldDifferences are the fields a node holds differently from the mesh-wide value of one entry. +func fieldDifferences(kind, entry, node string, mesh, there json.RawMessage) []difference { + var a, b map[string]any + if json.Unmarshal(mesh, &a) != nil || json.Unmarshal(there, &b) != nil { + if mustJSON(json.RawMessage(mesh)) != mustJSON(json.RawMessage(there)) { + var x, y any + _ = json.Unmarshal(mesh, &x) + _ = json.Unmarshal(there, &y) + return []difference{{Kind: kind, Entry: entry, Node: node, Field: "the whole list", MeshWide: x, There: y}} + } + return nil + } + var out []difference + for _, f := range sortedKeys(mergeKeys(a, b)) { + if mustJSON(a[f]) != mustJSON(b[f]) { + out = append(out, difference{Kind: kind, Entry: entry, Node: node, Field: f, MeshWide: a[f], There: b[f]}) + } + } + return out +} + +func mergeKeys(a, b map[string]any) map[string]bool { + out := map[string]bool{} + for k := range a { + out[k] = true + } + for k := range b { + out[k] = true + } + return out +} + +// withKinds is a layer with the derived kinds in place of its own, every other key kept. +func withKinds(layer map[string]any, kinds map[string]any) map[string]any { + out := map[string]any{} + for k, v := range layer { + out[k] = v + } + for k, v := range kinds { + if k == "prefs" { + merged := map[string]any{} + if was, ok := out["prefs"].(map[string]any); ok { + for n, x := range was { + merged[n] = x + } + } + for n, x := range v.(map[string]any) { + merged[n] = x + } + out[k] = merged + continue + } + out[k] = v + } + return out +} + +func contains(list []string, s string) bool { + for _, x := range list { + if x == s { + return true + } + } + return false +} + +// readLayer is the module's layer as the controller holds it: the mesh-wide one for node "", else that node's. +func readLayer(node string) (map[string]any, error) { + args := map[string]any{"module": "betterbird"} + if node != "" { + args["node"] = node + } + raw, err := asker("seat:mesh-controller.settings", args) + if err != nil { + return nil, fmt.Errorf("the controller's settings verb did not answer: %v", err) + } + said, ok := verbSaid(raw) + if !ok { + return nil, fmt.Errorf("the controller did not answer the layer: %s", said) + } + // A missing layer is said in exactly these words; anything else must read as the layer's JSON. A proposal + // replaces the layer whole, so an answer read wrongly as empty would drop what the layer holds: refused, loudly. + text := strings.TrimSpace(said) + where := "the whole mesh" + if node != "" { + where = node + } + if strings.HasPrefix(text, "betterbird on "+where+": no layer") { + return map[string]any{}, nil + } + out := map[string]any{} + if err := json.NewDecoder(strings.NewReader(text)).Decode(&out); err != nil || !strings.HasPrefix(text, "{") { + first, _, _ := strings.Cut(text, "\n") + return nil, fmt.Errorf("the controller's answer for the layer of betterbird on %s cannot be read as the layer "+ + "(it begins %q), so nothing is proposed: a layer is replaced whole and would lose what it holds", where, first) + } + return out, nil +} + +// proposeLayer proposes a layer whole for the operator's Approve (ADR 0277); replace says a key the layer had +// and the proposal does not hold is meant to go, which a re-derive needs when an entry left every profile. +func proposeLayer(node string, values map[string]any, replace bool) (string, error) { + body, err := json.Marshal(values) + if err != nil { + return "", err + } + args := map[string]any{"module": "betterbird", "values": string(body), "propose": "true"} + if replace { + args["replace"] = "true" + } + if node != "" { + args["node"] = node + } + raw, err := asker("seat:mesh-controller.settings", args) + if err != nil { + return "", fmt.Errorf("the controller's settings verb did not answer: %v", err) + } + said, ok := verbSaid(raw) + if !ok { + return said, fmt.Errorf("the controller did not take the proposal: %s", said) + } + return said, nil +} + +var ( + proposalHead = regexp.MustCompile(`^(\S+)\s+(?:set|clear) betterbird on (.+?)\s+\(`) + waitingLine = regexp.MustCompile(`^\s+waiting for the operator until`) +) + +// openProposals are the module's proposals that wait for the operator, by id and layer. +func openProposals() ([]string, error) { + raw, err := asker("seat:mesh-controller.settings", map[string]any{"proposals": "true"}) + if err != nil { + return nil, err + } + said, ok := verbSaid(raw) + if !ok { + return nil, errors.New(said) + } + var out []string + current := "" + for _, line := range strings.Split(said, "\n") { + if mm := proposalHead.FindStringSubmatch(line); mm != nil { + current = mm[1] + " (" + mm[2] + ")" + continue + } + if !strings.HasPrefix(line, " ") { + current = "" + } + if current != "" && waitingLine.MatchString(line) { + out = append(out, current) + current = "" + } + } + return out, nil +} + +// holdersOfBetterbird are the nodes the controller has assigned the module to: the desktop nodes. +func holdersOfBetterbird() ([]string, error) { + raw, err := asker("seat:mesh-controller.nodes", map[string]any{}) + if err != nil { + return nil, fmt.Errorf("the controller's nodes verb did not answer: %v", err) + } + said, _ := verbSaid(raw) + var nodes []struct { + Name string `json:"name"` + } + if err := json.Unmarshal([]byte(said), &nodes); err != nil { + return nil, fmt.Errorf("the controller's nodes cannot be read: %v", err) + } + var out []string + for _, n := range nodes { + raw, err := asker("seat:mesh-controller.node", map[string]any{"node": n.Name}) + if err != nil { + return nil, fmt.Errorf("the controller's node verb did not answer for %s: %v", n.Name, err) + } + said, _ := verbSaid(raw) + for _, line := range strings.Split(said, "\n") { + if rest, ok := strings.CutPrefix(strings.TrimSpace(line), "assigned:"); ok { + for _, mod := range strings.Split(rest, ",") { + if strings.TrimSpace(mod) == "betterbird" { + out = append(out, n.Name) + } + } + } + } + } + sort.Strings(out) + if len(out) == 0 { + return nil, errors.New("the controller names no node the module is assigned to") + } + return out, nil +} + +// deriveOn asks another node's betterbird for what its profile holds. +func deriveOn(node string, kinds []string) (nodeDerivation, error) { + raw, err := asker("betterbird.betterbird_derive@"+node, map[string]any{"here": true, "kinds": kinds}) + if err != nil { + return nodeDerivation{}, err + } + said, ok := verbSaid(raw) + if !ok { + return nodeDerivation{}, errors.New(said) + } + var a DeriveAnswer + if err := json.Unmarshal([]byte(said), &a); err != nil || a.Here == nil { + return nodeDerivation{}, fmt.Errorf("its answer cannot be read") + } + a.Here.Node = node + return *a.Here, nil +} diff --git a/modules/betterbird/cmd/betterbird-tools/derive_test.go b/modules/betterbird/cmd/betterbird-tools/derive_test.go new file mode 100644 index 00000000..549e98d9 --- /dev/null +++ b/modules/betterbird/cmd/betterbird-tools/derive_test.go @@ -0,0 +1,263 @@ +package main + +import ( + "encoding/json" + "errors" + "fmt" + "reflect" + "strings" + "testing" +) + +// ADR 0329 §10, §12: derived from every desktop node's profile, laid out per layer, and proposed one layer at a +// time, carrying the layer as it stands. + +// derivingMesh is two desktop nodes, g14 (this one, f) and shanks (f2), and a server that holds no Betterbird, as the +// controller answers them; layers are the controller's layers, open the proposals that wait. +type derivingMesh struct { + t *testing.T + shanks *fake + layers map[string]map[string]any + open bool + proposed []map[string]any + asked []string + asleep bool + // garbled answers the layer in words that are not its JSON. + garbled bool +} + +func (d *derivingMesh) ask(key string, body any) (json.RawMessage, error) { + d.asked = append(d.asked, key) + args, _ := body.(map[string]any) + answer := func(ok bool, text string) (json.RawMessage, error) { + b, _ := json.Marshal(map[string]any{"ok": ok, "output": text}) + return b, nil + } + switch { + case key == "seat:mesh-controller.nodes": + return answer(true, `[{"name": "g14"}, {"name": "novox"}, {"name": "shanks"}]`) + case key == "seat:mesh-controller.node": + if args["node"] == "novox" { + return answer(true, "novox\n assigned: postgres, nats\n") + } + return answer(true, fmt.Sprintf("%s\n last heard from here\n assigned: adwaita, betterbird, dunst\n", args["node"])) + case key == "betterbird.betterbird_derive@shanks": + if d.asleep { + return nil, errors.New("nats: no responders available for request") + } + here, err := d.shanks.deriveHere(stringsOf(args["kinds"])) + if err != nil { + d.t.Fatal(err) + } + text, _ := json.Marshal(credentialChecked(DeriveAnswer{Here: &here})) + return json.Marshal(map[string]any{"content": []any{map[string]any{"type": "text", "text": string(text)}}}) + case key == "seat:mesh-controller.settings" && args["proposals"] == "true": + if d.open { + return answer(true, "sab12 set betterbird on the whole mesh (accounts, identities)\n by g14.node-tools at 2026-10-13 01:00; fingerprint ab12\n waiting for the operator until 2026-10-14 01:00\n"+ + "scd34 set mounts on shanks (sources)\n by g14.node-tools at 2026-10-11 00:05; fingerprint dd0b\n chosen: done\n") + } + return answer(true, "scd34 set mounts on shanks (sources)\n by g14.node-tools; fingerprint dd0b\n chosen: done\n") + case key == "seat:mesh-controller.settings" && args["propose"] == "true": + d.proposed = append(d.proposed, args) + return answer(true, "proposal sab12: Set the settings of betterbird?\n fingerprint ab12") + case key == "seat:mesh-controller.settings": + node, _ := args["node"].(string) + if d.garbled { + return answer(true, "Settings of betterbird on "+node+":\n accounts = {…}\n") + } + layer := d.layers[node] + if layer == nil { + where := node + if where == "" { + where = "the whole mesh" + } + return answer(true, "betterbird on "+where+": no layer — the module's definition says\nbetterbird on "+where+", every value") + } + b, _ := json.MarshalIndent(layer, "", " ") + return answer(true, string(b)+"\n approved by the operator\nbetterbird on the whole mesh, every value and where it comes from:\n") + } + d.t.Fatalf("asked %s %v", key, body) + return nil, nil +} + +func stringsOf(v any) []string { + switch x := v.(type) { + case []string: + return x + case []any: + var out []string + for _, s := range x { + out = append(out, s.(string)) + } + return out + } + return nil +} + +func TestDeriveLaysEachEntryOutPerLayerAndProposesOneLayerAtATime(t *testing.T) { + f := newFake(t) + f.profileWithAccounts() + f.profileWithConfiguration() + f.write(profileDir+"/prefs.js", f.read(profileDir+"/prefs.js")+ + `user_pref("calendar.registry.c6.type", "caldav");`+"\n"+ + `user_pref("calendar.registry.c6.uri", "https://other:hunter2@dav.example.org/cal/x/");`+"\n"+ + `user_pref("calendar.registry.c6.name", "Old");`+"\n") + f.declareAccounts(adoptable) + links(t, nil) + f2 := newFake(t) + f2.profileWithAccounts() + f2.profileWithConfiguration() + // shanks holds the same accounts and calendars, another signature, another tag colour and a calendar of its own. + f2.write(profileDir+"/prefs.js", strings.NewReplacer(`"Regards"`, `"Cheers"`, `"#FF0000"`, `"#CC0000"`).Replace(f2.read(profileDir+"/prefs.js"))+ + `user_pref("calendar.registry.s1.type", "caldav");`+"\n"+ + `user_pref("calendar.registry.s1.uri", "https://dav.example.org/cal/standup/");`+"\n"+ + `user_pref("calendar.registry.s1.name", "Standup");`+"\n") + f2.declareAccounts(adoptable) + mesh := &derivingMesh{t: t, shanks: f2, layers: map[string]map[string]any{"": {"accounts": map[string]any{"mail": "as approved"}}}} + oldNode, oldAsk := nodeName, asker + t.Cleanup(func() { nodeName, asker = oldNode, oldAsk }) + nodeName = func() string { return "g14" } + asker = mesh.ask + + a, err := f.Derive(nil, false, false, nil, "") + if err != nil { + t.Fatal(err) + } + if !reflect.DeepEqual(a.From, []string{"g14", "shanks"}) || len(a.NotHeard) != 0 { + t.Fatalf("derived from the nodes holding Betterbird: %v %v", a.From, a.NotHeard) + } + whole := mustJSON(a) + mw := a.Layers[meshWide] + ids, _ := mw[kindIdentities].(map[string]any) + if mail, _ := ids["mail"].(map[string]any); mail == nil || mustJSON(mail["signature"]) != `["Regards"]` || mail["address"] != nil { + t.Errorf("an identity on both nodes is mesh-wide with this node's value, the default without its address: %v", ids) + } + cals, _ := mw[kindCalendars].(map[string]any) + if cals["home"] == nil || cals["standup"] != nil { + t.Errorf("a calendar both hold is mesh-wide, one only shanks holds is not: %v", cals) + } + if sh := mustJSON(a.Layers["shanks"]); !strings.Contains(sh, "https://dav.example.org/cal/standup/") { + t.Errorf("an entry on one node only goes into that node's layer: %s", sh) + } + tags, _ := mw[kindTags].(map[string]any) + if l1, _ := tags["$label1"].(map[string]any); l1 == nil || l1["color"] != "#FF0000" { + t.Errorf("the tag is mesh-wide with this node's colour: %v", tags) + } + diffs := map[string]difference{} + for _, d := range a.Differences { + diffs[d.Kind+"/"+d.Entry+"/"+d.Field] = d + } + if d := diffs["identities/mail/signature"]; d.Node != "shanks" || mustJSON(d.There) != `["Cheers"]` { + t.Errorf("the answer lists the field shanks holds differently: %+v", a.Differences) + } + if d := diffs["tags/$label1/color"]; d.There != "#CC0000" || d.MeshWide != "#FF0000" { + t.Errorf("%+v", a.Differences) + } + for _, never := range []string{"secret-token", "hunter2", "owl://", `"Work"`} { + if strings.Contains(mustJSON(a.Layers), never) { + t.Errorf("no ICS address, credential or Owl calendar is derived: %s in %s", never, mustJSON(a.Layers)) + } + } + if !strings.Contains(whole, "Holidays") || !strings.Contains(whole, "‹withheld›") || strings.Contains(mustJSON(credentialChecked(a)), "hunter2") { + t.Errorf("what is left out is named, an ICS calendar by its name and a password as ‹withheld›: %v", a.NotDerived) + } + if len(mesh.proposed) != 0 { + t.Fatal("without propose nothing is proposed") + } + + // Propose: the mesh-wide layer first, as it stands with the derived kinds, and the operator's setup of prefs. + prefs := map[string]any{} + for n, v := range shippedDefaultsWant { + prefs[n], _ = prefValue(v) + } + a, err = f.Derive(nil, false, true, prefs, "") + if err != nil { + t.Fatal(err) + } + if len(mesh.proposed) != 1 || mesh.proposed[0]["node"] != nil || mesh.proposed[0]["replace"] != "true" || a.Next != meshWide { + t.Fatalf("the mesh-wide layer is proposed first: %v %s", mesh.proposed, a.Next) + } + var values map[string]any + _ = json.Unmarshal([]byte(mesh.proposed[0]["values"].(string)), &values) + if values["accounts"] == nil || values[kindIdentities] == nil || values[kindTags] == nil || values["prefs"] == nil { + t.Fatalf("the layer as it stands, with every derived kind and prefs: %v", values) + } + body, _ := json.Marshal(map[string]any{"prefs": values["prefs"]}) + if got, err := parseDeclaredPrefs(body); err != nil || !reflect.DeepEqual(got, shippedDefaultsWant) { + t.Fatalf("the operator's 21 preferences, one entry each: %v %v", got, err) + } + if !strings.Contains(a.Proposed, "sab12") { + t.Fatalf("%s", a.Proposed) + } + + // While it waits nothing more is proposed. + mesh.open = true + if _, err := f.Derive(nil, false, true, nil, ""); err == nil || !strings.Contains(err.Error(), "sab12 (the whole mesh)") || len(mesh.proposed) != 1 { + t.Fatalf("one open proposal of the module's at a time: %v", err) + } + // Approved: the mesh-wide layer holds it, so the next layer is shanks's own. + mesh.open = false + mesh.layers[""] = values + a, err = f.Derive(nil, false, true, prefs, "") + if err != nil { + t.Fatal(err) + } + if len(mesh.proposed) != 2 || mesh.proposed[1]["node"] != "shanks" || a.Next != "shanks" || + !strings.Contains(mesh.proposed[1]["values"].(string), "standup") { + t.Fatalf("then the node's own layer: %v", mesh.proposed) + } + var shanks map[string]any + if err := json.Unmarshal([]byte(mesh.proposed[1]["values"].(string)), &shanks); err != nil { + t.Fatal(err) + } + mesh.layers["shanks"] = shanks + if a, err = f.Derive(nil, false, true, prefs, ""); err != nil || len(mesh.proposed) != 2 || !strings.HasPrefix(a.Next, "none") { + t.Fatalf("nothing left to propose: %v %s", err, a.Next) + } + + // A desktop node that does not answer: derived without it, never proposed. + mesh.asleep = true + a, err = f.Derive(nil, false, true, nil, meshWide) + if err == nil || len(a.NotHeard) != 1 || !strings.Contains(err.Error(), "did not answer") || len(mesh.proposed) != 2 { + t.Fatalf("a layer is never proposed without a desktop node: %v %v", err, a.NotHeard) + } + + // A key to adopt is a node's: the mesh-wide layer's proposal refuses one. + if err := proposable(map[string]any{kindCalendars: map[string]any{"c": map[string]any{"name": "C", + "url": "https://dav.example.org/c/", "key": "c3"}}}, true); err == nil || !strings.Contains(err.Error(), "node's own layer") { + t.Fatalf("%v", err) + } +} + +// Review 121: a layer the controller answers in any other words than its JSON or "no layer" refuses the proposal, +// because a proposal replaces the layer whole and would drop accounts. +func TestALayerThatCannotBeReadRefusesTheProposal(t *testing.T) { + f := newFake(t) + f.profileWithAccounts() + f.profileWithConfiguration() + f.declareAccounts(adoptable) + links(t, nil) + f2 := newFake(t) + f2.profileWithAccounts() + f2.declareAccounts(adoptable) + mesh := &derivingMesh{t: t, shanks: f2, garbled: true} + oldNode, oldAsk := nodeName, asker + t.Cleanup(func() { nodeName, asker = oldNode, oldAsk }) + nodeName = func() string { return "g14" } + asker = mesh.ask + if _, err := f.Derive(nil, false, true, nil, ""); err == nil || !strings.Contains(err.Error(), "cannot be read as the layer") || + len(mesh.proposed) != 0 { + t.Fatalf("the derive refuses: %v %v", err, mesh.proposed) + } + if _, err := f.ProfileAccounts(true); err == nil || !strings.Contains(err.Error(), "cannot be read as the layer") || + len(mesh.proposed) != 0 { + t.Fatalf("betterbird_accounts refuses: %v %v", err, mesh.proposed) + } + if _, err := readLayer(""); err == nil { + t.Fatal("an unreadable layer is an error, never an empty layer") + } + mesh.garbled = false + if got, err := readLayer("shanks"); err != nil || len(got) != 0 { + t.Fatalf("no layer, in the controller's words, is an empty layer: %v %v", got, err) + } +} diff --git a/modules/betterbird/cmd/betterbird-tools/filters.go b/modules/betterbird/cmd/betterbird-tools/filters.go new file mode 100644 index 00000000..493ae3db --- /dev/null +++ b/modules/betterbird/cmd/betterbird-tools/filters.go @@ -0,0 +1,271 @@ +package main + +// An account's filters, owned whole per declared account (novox/hq ADR 0329 §8). +// +// Betterbird keeps an account's filters in msgFilterRules.dat in the account's own directory: a header, then per +// filter its name, whether it is on, when it runs (a bit mask), its actions and one condition line. For an account +// the setting filters names, the module writes that file from the declared list, naming each folder by its path +// within the account and turning it into the folder's address on this node; a filter in the file the setting does +// not hold is gone after the apply. Before its first write on a node the module keeps the file it replaces, in its +// own directory in the home (ADR 0182). It writes only while Betterbird does not run, because Betterbird rewrites +// the file itself (§9). + +import ( + "bufio" + "fmt" + "net/url" + "path/filepath" + "regexp" + "sort" + "strconv" + "strings" +) + +// filterFileName is the file an account's filters are in. +const filterFileName = "msgFilterRules.dat" + +// accountDirectory is an account's directory in the profile, as its server's preferences name it, or "". +func accountDirectory(profile string, prefs map[string]string, server string) string { + p := "mail.server." + server + "." + if rel := unquote(prefs[p+"directory-rel"]); strings.HasPrefix(rel, "[ProfD]") { + dir := filepath.Join(profile, filepath.FromSlash(strings.TrimPrefix(rel, "[ProfD]"))) + if strings.HasPrefix(filepath.Clean(dir), filepath.Clean(profile)+string(filepath.Separator)) { + return dir + } + return "" + } + return "" +} + +// folderBase is the address of an account's root folder on this node: imap://user@host for IMAP, mailbox:// +// for POP3 and local folders, the user's @ written %40 as Betterbird writes it. +func folderBase(prefs map[string]string, server string) string { + p := "mail.server." + server + "." + kind := unquote(prefs[p+"type"]) + host := unquote(prefs[p+"hostname"]) + user := unquote(prefs[p+"userName"]) + scheme := "mailbox" + switch kind { + case "imap": + scheme = "imap" + case "none": + host, user = "Local Folders", "nobody" + } + return scheme + "://" + strings.ReplaceAll(url.PathEscape(user), "@", "%40") + "@" + url.PathEscape(host) +} + +// folderAddress is a folder's address on this node from its path within the account. +func folderAddress(base, path string) string { + parts := strings.Split(strings.Trim(path, "/"), "/") + for i, p := range parts { + parts[i] = url.PathEscape(p) + } + return base + "/" + strings.Join(parts, "/") +} + +// folderPath is a folder's path within the account from its address, when the address is the account's. +func folderPath(base, address string) (string, bool) { + rest, ok := strings.CutPrefix(address, base+"/") + if !ok { + return "", false + } + parts := strings.Split(rest, "/") + for i, p := range parts { + u, err := url.PathUnescape(p) + if err != nil { + return "", false + } + parts[i] = u + } + return strings.Join(parts, "/"), true +} + +// whenOf is a filter's type mask from its words; whenWords the reverse, or false when the mask holds a bit no +// word says (a news filter's). +func whenOf(words []string) int { + n := 0 + for _, w := range words { + n |= filterWhen[w] + } + return n +} + +func whenWords(mask int) ([]string, bool) { + var out []string + left := mask + for _, w := range []string{"new-mail", "manual", "after-junk", "sending", "archiving", "periodic"} { + if mask&filterWhen[w] != 0 { + out = append(out, w) + left &^= filterWhen[w] + } + } + return out, left == 0 +} + +// renderFilters is the file Betterbird reads for a declared list of filters on this node. +func renderFilters(filters []Filter, base string) string { + var b strings.Builder + b.WriteString("version=\"9\"\nlogging=\"no\"\n") + for _, f := range filters { + fmt.Fprintf(&b, "name=\"%s\"\n", f.Name) + fmt.Fprintf(&b, "enabled=\"%s\"\n", map[bool]string{true: "yes", false: "no"}[f.Enabled]) + fmt.Fprintf(&b, "type=\"%d\"\n", whenOf(f.When)) + for _, a := range f.Actions { + fmt.Fprintf(&b, "action=\"%s\"\n", a.Action) + switch { + case a.Folder != "": + fmt.Fprintf(&b, "actionValue=\"%s\"\n", folderAddress(base, a.Folder)) + case a.Value != "": + fmt.Fprintf(&b, "actionValue=\"%s\"\n", a.Value) + } + } + fmt.Fprintf(&b, "condition=\"%s\"\n", conditionText(f)) + } + return b.String() +} + +func conditionText(f Filter) string { + if f.Match == "every-message" { + return "ALL" + } + join := "AND" + if f.Match == "any" { + join = "OR" + } + var parts []string + for _, c := range f.Conditions { + parts = append(parts, fmt.Sprintf("%s (%s,%s,%s)", join, c.Field, c.Op, c.Value)) + } + return strings.Join(parts, " ") +} + +var ( + filterLine = regexp.MustCompile(`^([A-Za-z]+)="((?:[^"\\]|\\.)*)"$`) + clause = regexp.MustCompile(`(AND|OR) \(([^,()]+),([^,()]+),([^()]*)\)`) +) + +// parsedFilters is a filter file read into the setting's shape. NotDeclarable names, by the filter's name, the +// filters a setting cannot hold as they are (a folder of another account, an action of an add-on, a news filter). +type parsedFilters struct { + Filters []Filter + NotDeclarable []string +} + +// parseFilters reads a filter file for an account whose folders are under base. +func parseFilters(text, base string) parsedFilters { + var out parsedFilters + var cur *Filter + bad := "" + finish := func() { + if cur == nil { + return + } + if bad != "" { + out.NotDeclarable = append(out.NotDeclarable, cur.Name+": "+bad) + } else if p := filterProblems(*cur); len(p) > 0 { + out.NotDeclarable = append(out.NotDeclarable, cur.Name+": "+p[0]) + } else { + out.Filters = append(out.Filters, *cur) + } + cur, bad = nil, "" + } + s := bufio.NewScanner(strings.NewReader(text)) + s.Buffer(make([]byte, 64<<10), 4<<20) + for s.Scan() { + mm := filterLine.FindStringSubmatch(strings.TrimSpace(s.Text())) + if mm == nil { + continue + } + field, value := mm[1], strings.NewReplacer(`\"`, `"`, `\\`, `\`).Replace(mm[2]) + if field == "name" { + finish() + cur = &Filter{Name: value} + continue + } + if cur == nil { + continue + } + switch field { + case "enabled": + cur.Enabled = value == "yes" + case "type": + n, _ := strconv.Atoi(value) + words, ok := whenWords(n) + if !ok && bad == "" { + bad = "it runs on news or at a time no setting word says" + } + cur.When = words + case "action": + cur.Actions = append(cur.Actions, Action{Action: value}) + case "actionValue": + if len(cur.Actions) == 0 { + continue + } + a := &cur.Actions[len(cur.Actions)-1] + if filterActions[a.Action] == "folder" { + path, ok := folderPath(base, value) + if !ok && bad == "" { + bad = "it moves or copies to a folder of another account" + } + a.Folder = path + } else { + a.Value = value + } + case "condition": + switch { + case value == "ALL": + cur.Match = "every-message" + default: + all := clause.FindAllStringSubmatch(value, -1) + if len(all) == 0 && bad == "" { + bad = "its condition cannot be read" + } + for _, c := range all { + if c[1] == "OR" { + cur.Match = "any" + } else if cur.Match == "" { + cur.Match = "all" + } + cur.Conditions = append(cur.Conditions, Condition{Field: c[2], Op: c[3], Value: c[4]}) + } + } + } + } + finish() + return out +} + +// filterDifferences says how a file's filters differ from the declared list, by place and name: missing, +// not declared, or differing. +func filterDifferences(declared, have []Filter) []string { + var out []string + for i := 0; i < len(declared) || i < len(have); i++ { + switch { + case i >= len(have): + out = append(out, fmt.Sprintf("filter %d (%s) is declared and not in the file", i+1, declared[i].Name)) + case i >= len(declared): + out = append(out, fmt.Sprintf("filter %d (%s) is in the file and not declared: removed at the next apply", i+1, have[i].Name)) + default: + d, h := mustJSON(declared[i]), mustJSON(have[i]) + if d != h { + out = append(out, fmt.Sprintf("filter %d: declared %s, the file holds %s", i+1, d, h)) + } + } + } + return out +} + +// keptFilterFile is where the module keeps the file it replaced at its first write of an account's filters. +func (m *Machine) keptFilterFile(profileDir, account string) string { + base := filepath.Base(profileDir) + return m.home(".config/betterbird/kept", base, account+"."+filterFileName) +} + +func sortedAccountsOf(fs map[string][]Filter) []string { + out := make([]string, 0, len(fs)) + for k := range fs { + out = append(out, k) + } + sort.Strings(out) + return out +} diff --git a/modules/betterbird/cmd/betterbird-tools/listing.go b/modules/betterbird/cmd/betterbird-tools/listing.go new file mode 100644 index 00000000..d65bd9a5 --- /dev/null +++ b/modules/betterbird/cmd/betterbird-tools/listing.go @@ -0,0 +1,480 @@ +package main + +// What the tools show of the five kinds (novox/hq ADR 0329 §1, §11–§13): the profile's entries beside the declared +// ones, with the operator's own values, and how a node differs from what is declared. Never a credential (§2): +// every answer passes credentialChecked, which withholds a field whose name says password, token, secret or cookie +// and the password part of any address. + +import ( + "encoding/json" + "fmt" + "strings" +) + +// configView is the profile and the settings, read once for an answer. +type configView struct { + p Profile + prefs map[string]string + accounts map[string]Account + ra []resolvedAccount + conf Configuration + problem string + plan configPlan + rec configRecord +} + +func (m *Machine) view() (configView, error) { + var v configView + p, err := m.profile() + if err != nil { + return v, err + } + v.p, v.prefs = p, mergedPrefs(p.path) + accounts, _, aerr := m.readAccounts() + if aerr != nil { + v.problem = aerr.Error() + } + v.accounts = accounts + v.ra = resolveAccounts(accounts, v.prefs, m.readRecord(p.Dir)) + conf, cerr := m.readConfiguration() + if cerr != nil { + v.problem = strings.TrimPrefix(v.problem+"; "+cerr.Error(), "; ") + } + v.conf = conf + v.rec = m.readConfigRecord(p.Dir) + v.plan = m.planConfiguration(p, conf, accounts, v.ra, v.prefs) + return v, nil +} + +// listedEntry is one entry of the profile, with its values: Owl's marked, an ICS calendar by its name only (§13). +type listedEntry struct { + Key string `json:"key"` + Kind string `json:"kind,omitempty"` + // Account is an identity's account key in the profile, or the account name filters are listed under. + Account string `json:"account,omitempty"` + Owl bool `json:"owl,omitempty"` + // DeclaredAs is the declared entry this one is, or empty: the person's own. + DeclaredAs string `json:"declared_as,omitempty"` + Value any `json:"value,omitempty"` + Note string `json:"note,omitempty"` +} + +// declaredEntry is one declared entry, its value and how it resolved on this node. +type declaredEntry struct { + Name string `json:"name"` + Value any `json:"value"` + How string `json:"how,omitempty"` + Key string `json:"key,omitempty"` + Why string `json:"why,omitempty"` +} + +// KindListing is what a listing tool answers. +type KindListing struct { + Profile string `json:"profile"` + Kind string `json:"kind"` + InProfile []listedEntry `json:"in_the_profile"` + Declared []declaredEntry `json:"declared"` + // Kept says where the module keeps the filter files it replaced, per account (§8). + Kept map[string]string `json:"kept_filter_files,omitempty"` + Problem string `json:"problem,omitempty"` +} + +func (v configView) resolved(kind string) map[string]resolvedEntry { + out := map[string]resolvedEntry{} + for _, e := range v.plan.Entries { + if e.Kind == kind { + out[e.Name] = e + } + } + return out +} + +// declaredBy is, per profile key, the declared entry of a kind adopted or created under it. +func (v configView) declaredBy(kind string) map[string]string { + out := map[string]string{} + for _, e := range v.plan.Entries { + if e.Kind == kind && e.How != refused { + out[e.Key] = e.Name + } + } + return out +} + +// List answers one kind: the profile's entries beside the declared ones. +func (m *Machine) List(kind string) (KindListing, error) { + v, err := m.view() + a := KindListing{Kind: kind, InProfile: []listedEntry{}, Declared: []declaredEntry{}} + if err != nil { + return a, err + } + a.Profile, a.Problem = v.p.Dir, v.problem + res := v.resolved(kind) + by := v.declaredBy(kind) + declare := func(name string, value any) { + r := res[name] + a.Declared = append(a.Declared, declaredEntry{Name: name, Value: value, How: r.How, Key: r.Key, Why: r.Why}) + } + switch kind { + case kindIdentities: + for _, pi := range m.profileIdentities(v.prefs) { + a.InProfile = append(a.InProfile, listedEntry{Key: pi.Key, Account: pi.AccountKey, Owl: pi.Owl, + DeclaredAs: by[pi.Key], Value: pi.Value}) + } + for _, name := range sortedKeys(v.conf.Identities) { + declare(name, v.conf.Identities[name]) + } + case kindCalendars: + for _, pc := range profileCalendars(v.prefs) { + e := listedEntry{Key: pc.Key, Kind: pc.Kind, DeclaredAs: by[pc.Key], Value: pc.Value} + switch { + case strings.HasPrefix(pc.Kind, "owl"): + e.Owl, e.Value = true, map[string]string{"name": pc.Value.Name, "url": pc.Value.URL} + e.Note = "Owl's: never declared (novox/hq ADR 0316 §13)" + case pc.Kind == "ics": + e.Value = map[string]string{"name": pc.Value.Name} + e.Note = "subscribed by an ICS address, which is often itself the secret: its name only, never declared (novox/hq ADR 0329 §13)" + case pc.Kind != "caldav": + e.Note = "kept in the profile: its events are never declared" + } + a.InProfile = append(a.InProfile, e) + } + for _, name := range sortedKeys(v.conf.Calendars) { + declare(name, v.conf.Calendars[name]) + } + case kindAddressBooks: + for _, b := range profileAddressBooks(v.prefs) { + e := listedEntry{Key: b.Key, Kind: b.Kind, DeclaredAs: by[b.Key], Value: b.Value} + if !b.CardDAV { + e.Value, e.Note = map[string]string{"name": b.Value.Name}, "its contacts are kept in the profile and never declared" + } + a.InProfile = append(a.InProfile, e) + } + for _, name := range sortedKeys(v.conf.AddressBooks) { + declare(name, v.conf.AddressBooks[name]) + } + case kindTags: + for _, t := range profileTags(v.prefs) { + a.InProfile = append(a.InProfile, listedEntry{Key: t.Key, DeclaredAs: by[t.Key], Value: t.Value}) + } + for _, key := range sortedKeys(v.conf.Tags) { + declare(key, v.conf.Tags[key]) + } + case kindFilters: + for _, f := range m.profileFilters(v) { + a.InProfile = append(a.InProfile, f) + } + for _, account := range v.conf.sortedFilterAccounts() { + d := declaredEntry{Name: account, Value: v.conf.Filters[account], How: "owned whole"} + for _, fp := range v.plan.filters { + if fp.Account == account { + d.Key, d.Why = fp.File, strings.TrimSpace(fp.Refused+" "+fp.Waiting) + if fp.Refused != "" { + d.How = refused + } + } + } + a.Declared = append(a.Declared, d) + } + a.Kept = map[string]string{} + for _, account := range sortedKeys(v.rec.Filters) { + if r := v.rec.Filters[account]; r.Kept != "" { + a.Kept[account] = r.Kept + } + } + if len(a.Kept) == 0 { + a.Kept = nil + } + default: + return a, fmt.Errorf("a kind is one of %s", strings.Join(allKinds, ", ")) + } + return a, nil +} + +// profileFilters are the filters of each declared account in the profile, by the account's name. +func (m *Machine) profileFilters(v configView) []listedEntry { + var out []listedEntry + for _, r := range v.ra { + if r.How == refused { + continue + } + dir := m.accountDirectory(v.p.path, v.prefs, r.Keys.Server) + if dir == "" { + continue + } + text, err := readBounded(dir + "/" + filterFileName) + if err != nil { + continue + } + parsed := parseFilters(string(text), folderBase(v.prefs, r.Keys.Server)) + e := listedEntry{Key: m.tilde(strings.TrimPrefix(dir+"/"+filterFileName, m.Root)), Account: r.name, Value: parsed.Filters} + if _, declared := v.conf.Filters[r.name]; declared { + e.DeclaredAs = r.name + } + if len(parsed.NotDeclarable) > 0 { + e.Note = "not declarable as they are: " + strings.Join(parsed.NotDeclarable, "; ") + } + out = append(out, e) + } + return out +} + +// fieldDiff is one field a profile holds differently from the declared value. +type fieldDiff struct { + Field string `json:"field"` + Declared any `json:"declared"` + Profile any `json:"profile"` +} + +// DriftEntry is one declared entry against the profile: holds, differs, lacks or refused (§11). +type DriftEntry struct { + Name string `json:"name"` + State string `json:"state"` + Key string `json:"key,omitempty"` + Differs []fieldDiff `json:"differs,omitempty"` + Why string `json:"why,omitempty"` +} + +// KindDrift is one kind's drift, and the profile's entries nothing declares. +type KindDrift struct { + Kind string `json:"kind"` + Declared []DriftEntry `json:"declared"` + Undeclared []listedEntry `json:"undeclared,omitempty"` +} + +// DriftAnswer is what betterbird_drift answers. +type DriftAnswer struct { + Profile string `json:"profile"` + Kinds []KindDrift `json:"kinds"` + Note string `json:"note"` + Problem string `json:"problem,omitempty"` +} + +// Drift answers, per kind, how the profile differs from what is declared: the profile as Betterbird reads it at +// its next start (prefs.js with user.js over it), so a declared value the module placed and Betterbird has not +// started with yet holds; the drift then is only what the module did not place. +func (m *Machine) Drift(kind string) (DriftAnswer, error) { + v, err := m.view() + a := DriftAnswer{Kinds: []KindDrift{}} + if err != nil { + return a, err + } + a.Profile, a.Problem = v.p.Dir, v.problem + a.Note = "the profile as Betterbird reads it at its next start (prefs.js, the module's user.js over it): differs " + + "names each field with the declared value and the profile's; lacks is an entry the profile does not hold yet " + + "(betterbird_user_js with apply, betterbird_restart); undeclared entries stay the person's" + kinds := allKinds + if kind != "" { + if !contains(allKinds, kind) { + return a, fmt.Errorf("a kind is one of %s", strings.Join(allKinds, ", ")) + } + kinds = []string{kind} + } + for _, k := range kinds { + a.Kinds = append(a.Kinds, m.kindDrift(v, k)) + } + return a, nil +} + +func (m *Machine) kindDrift(v configView, kind string) KindDrift { + d := KindDrift{Kind: kind, Declared: []DriftEntry{}} + res := v.resolved(kind) + by := v.declaredBy(kind) + entry := func(name string, declared any, profile func(key string) (any, bool)) { + r := res[name] + e := DriftEntry{Name: name, Key: r.Key} + switch { + case r.How == refused: + e.State, e.Why = refused, r.Why + default: + have, ok := profile(r.Key) + if !ok { + e.State = "lacks" + break + } + e.Differs = differing(declared, have) + e.State = map[bool]string{true: "holds", false: "differs"}[len(e.Differs) == 0] + } + d.Declared = append(d.Declared, e) + } + switch kind { + case kindIdentities: + ids := map[string]profileIdentity{} + for _, pi := range m.profileIdentities(v.prefs) { + ids[pi.Key] = pi + if by[pi.Key] == "" && !pi.Owl { + d.Undeclared = append(d.Undeclared, listedEntry{Key: pi.Key, Account: pi.AccountKey, Value: pi.Value}) + } + } + for _, name := range sortedKeys(v.conf.Identities) { + want := *v.conf.Identities[name] + want.Key = "" + entry(name, want, func(key string) (any, bool) { + pi, ok := ids[key] + if !ok { + return nil, false + } + have := pi.Value + have.Account = want.Account + if want.Address == "" { + have.Address, have.Name = "", "" + } + return have, true + }) + } + case kindCalendars: + cals := map[string]profileCalendar{} + for _, pc := range profileCalendars(v.prefs) { + cals[pc.Key] = pc + if by[pc.Key] == "" && pc.Kind == "caldav" { + d.Undeclared = append(d.Undeclared, listedEntry{Key: pc.Key, Kind: pc.Kind, Value: pc.Value}) + } + } + idName := v.declaredBy(kindIdentities) + for _, name := range sortedKeys(v.conf.Calendars) { + want := *v.conf.Calendars[name] + want.Key = "" + entry(name, want, func(key string) (any, bool) { + pc, ok := cals[key] + if !ok { + return nil, false + } + have := pc.Value + have.Invitations = idName[pc.IdentityKey] + if have.Reminders == nil && want.Reminders != nil && *want.Reminders { + on := true + have.Reminders = &on + } + return have, true + }) + } + case kindAddressBooks: + books := map[string]profileAddressBook{} + for _, b := range profileAddressBooks(v.prefs) { + books[b.Key] = b + if by[b.Key] == "" && b.CardDAV { + d.Undeclared = append(d.Undeclared, listedEntry{Key: b.Key, Kind: b.Kind, Value: b.Value}) + } + } + for _, name := range sortedKeys(v.conf.AddressBooks) { + want := *v.conf.AddressBooks[name] + want.Key = "" + entry(name, want, func(key string) (any, bool) { + b, ok := books[key] + return b.Value, ok + }) + } + case kindTags: + tags := map[string]profileTag{} + for _, t := range profileTags(v.prefs) { + tags[t.Key] = t + if by[t.Key] == "" { + d.Undeclared = append(d.Undeclared, listedEntry{Key: t.Key, Value: t.Value}) + } + } + for _, key := range sortedKeys(v.conf.Tags) { + want := *v.conf.Tags[key] + entry(key, want, func(k string) (any, bool) { + t, ok := tags[k] + have := t.Value + if want.Order == 0 { + have.Order = 0 + } + return have, ok + }) + } + case kindFilters: + files := map[string]listedEntry{} + for _, f := range m.profileFilters(v) { + files[f.Account] = f + released := v.rec.Filters[f.Account].Released + if f.DeclaredAs == "" && !released { + if list, _ := f.Value.([]Filter); len(list) > 0 { + d.Undeclared = append(d.Undeclared, f) + } + } + } + for _, fp := range v.plan.filters { + e := DriftEntry{Name: fp.Account, Key: fp.File} + switch { + case fp.Refused != "": + e.State, e.Why = refused, fp.Refused + case fp.path == "" || !fp.exists: + e.State, e.Why = "lacks", strings.TrimSpace(fp.Waiting) + default: + have, _ := files[fp.Account].Value.([]Filter) + for _, line := range filterDifferences(v.conf.Filters[fp.Account], have) { + e.Differs = append(e.Differs, fieldDiff{Field: line}) + } + if fp.have != fp.want && len(e.Differs) == 0 { + e.Differs = append(e.Differs, fieldDiff{Field: "the file's text, beyond the filters' own words"}) + } + e.State = map[bool]string{true: "holds", false: "differs"}[len(e.Differs) == 0] + } + d.Declared = append(d.Declared, e) + } + } + return d +} + +// differing are the fields of two values of the same shape that differ. +func differing(declared, have any) []fieldDiff { + var a, b map[string]any + _ = json.Unmarshal([]byte(mustJSON(declared)), &a) + _ = json.Unmarshal([]byte(mustJSON(have)), &b) + var out []fieldDiff + for _, f := range sortedKeys(mergeKeys(a, b)) { + if mustJSON(a[f]) != mustJSON(b[f]) { + out = append(out, fieldDiff{Field: f, Declared: a[f], Profile: b[f]}) + } + } + return out +} + +// driftCounts are, per kind, how many declared entries do not hold. +func (m *Machine) driftCounts(v configView) map[string]int { + out := map[string]int{} + for _, k := range allKinds { + for _, e := range m.kindDrift(v, k).Declared { + if e.State != "holds" { + out[k]++ + } + } + } + return out +} + +// credentialChecked is an answer as it may leave the tool (§2): a field whose name says password, token, secret +// or cookie withheld, and the password part of any address as ‹withheld›. +func credentialChecked(v any) any { + b, err := json.Marshal(v) + if err != nil { + return v + } + var tree any + if json.Unmarshal(b, &tree) != nil { + return v + } + return scrubTree(tree) +} + +func scrubTree(n any) any { + switch x := n.(type) { + case map[string]any: + for k, v := range x { + if secretName.MatchString(k) { + x[k] = "‹withheld›" + continue + } + x[k] = scrubTree(v) + } + return x + case []any: + for i, v := range x { + x[i] = scrubTree(v) + } + return x + case string: + return withoutPassword(x) + } + return n +} diff --git a/modules/betterbird/cmd/betterbird-tools/main.go b/modules/betterbird/cmd/betterbird-tools/main.go index ff2331a1..5c00de48 100644 --- a/modules/betterbird/cmd/betterbird-tools/main.go +++ b/modules/betterbird/cmd/betterbird-tools/main.go @@ -2,14 +2,17 @@ // mail and calendar client, in the operator's session, served by the node's runtime as the operator // account. The module holds no seat, so every tool is its own. // -// The operator's mail, calendars, accounts and credentials are never read: a tool reads processes, -// profiles.ini, and of the profile only prefs.js, user.js and extensions.json, and answers no -// preference that can hold a credential or names an account. +// The operator's mail, calendar events, contacts and credentials are never read: a tool reads processes, +// profiles.ini, and of the profile its prefs.js, user.js, extensions.json, each account's msgFilterRules.dat and +// of logins.json only the server each kept sign-in is for. The tools show the operator's own configuration, an +// account's, an identity's, a calendar's, an address book's, a filter's and a tag's values, and never a credential +// (novox/hq ADR 0329 §1, §2): every answer passes credentialChecked. package main import ( "fmt" "os" + "strings" "time" stdio "git.novox.be/novox/mesh-sdk/go" @@ -34,6 +37,31 @@ func main() { var machine = NewMachine() func tools() []stdio.Tool { + list := toolList() + for i := range list { + run := list[i].Run + list[i].Run = func(args map[string]any) (any, error) { + out, err := run(args) + return credentialChecked(out), err + } + } + return list +} + +// listTool is the listing tool of one of the five kinds (novox/hq ADR 0329 §12). +func listTool(kind, what string) stdio.Tool { + return stdio.Tool{ + Name: "betterbird_" + strings.ReplaceAll(kind, "-", "_"), + Description: "The " + what + " of the profile Betterbird uses beside the ones the module's setting " + kind + + " declares (novox/hq ADR 0329): each of the profile's with its key, its values and the declared entry it is " + + "(none: the person's own), Owl's marked; each declared one with its value and how it resolved on this node " + + "(adopted, created or refused with why, and its key). The operator's own values, never a password, token " + + "or cookie. betterbird_drift says how they differ. (r)", + Run: func(map[string]any) (any, error) { return machine.List(kind) }, + } +} + +func toolList() []stdio.Tool { return []stdio.Tool{ { Name: "betterbird_status", @@ -42,9 +70,9 @@ func tools() []stdio.Tool { "or the community package before the switch), its version and build beside the newest published build; " + "whether the enterprise policy is placed; Owl's version against the mesh's pin and whether it is active; " + "where its output goes; the profile it uses and how that was found; the module's user.js (declared, " + - "placed, in effect); the declared accounts by count (in the profile's list, other accounts, not signed " + - "in yet), never an address, host or name; the add-ons in the profile; and the preferences that decide " + - "how its windows and notifications look. (r)", + "placed, in effect, with every declared identity, calendar, address book and tag as it resolved); the " + + "declared accounts by count (in the profile's list, other accounts, not signed in yet); the add-ons in the " + + "profile; and the preferences that decide how its windows and notifications look. (r)", Run: func(map[string]any) (any, error) { return machine.Status() }, }, { @@ -69,13 +97,13 @@ func tools() []stdio.Tool { }, { Name: "betterbird_config", - Description: "The configuration the module ships, preference by preference: the value its policy locks, " + - "its default (mesh-defaults.js, or Betterbird's own where the module keeps that), the node layer (the " + - "setting prefs on this machine), the profile's own saved value and the value in effect. It flags drift: " + - "a value the profile saved that beats the module's default without a node layer saying so. With keep " + - "(names), it proposes a node layer of the setting prefs for this machine holding the profile's values of " + - "those preferences, through the controller's settings verb, for the operator's Approve; it never writes " + - "user.js. (r; a with keep)", + Description: "Betterbird's preferences through the module's layers, preference by preference: the value its " + + "policy locks, the setting prefs (the operator's setup, mesh-wide with this node's layer over it, one entry " + + "per preference), Betterbird's own default where the module keeps that, the profile's own saved value and " + + "the value in effect. It flags drift: a declared value the profile's user.js does not hold yet, or a value " + + "the profile saved where the module keeps Betterbird's default. With keep (names), it proposes this node's " + + "layer of the setting prefs holding the profile's values of those preferences beside what the layer holds, " + + "through the controller's settings verb, for the operator's Approve; it never writes user.js. (r; a with keep)", Input: map[string]any{ "keep": map[string]any{"type": "array", "items": map[string]any{"type": "string"}, "description": "managed preferences whose profile value this machine keeps: proposed as a node layer, e.g. [\"mail.uifontsize\"]"}, @@ -108,15 +136,16 @@ func tools() []stdio.Tool { }, { Name: "betterbird_user_js", - Description: "The module's user.js — the preferences the module declares (setting prefs), then the declared " + - "accounts' preferences under the module's own keys (setting accounts) — against the profile's user.js: " + - "whether it is placed, and whether the running Betterbird started after it was. With apply, place it in " + - "the profile (only where the profile has no user.js or the module's own; an operator's file is never " + - "overwritten). A declared account the profile already holds at its address is adopted under its own keys; " + - "one it does not hold is created under the module's keys, which are added to the account lists when " + - "Betterbird is not running; an ambiguous one is refused alone, with the remedy (novox/hq ADR 0317). " + - "Accounts' values are withheld in the answer. Betterbird reads it at its next start: betterbird_restart. " + - "(r; a with apply)", + Description: "The module's user.js — the setting prefs, then the declared accounts' preferences (setting " + + "accounts), then the declared identities, calendars, address books and tags — against the profile's " + + "user.js: whether it is placed, and whether the running Betterbird started after it was. With apply, place " + + "it in the profile (only where the profile has no user.js or the module's own; an operator's file is never " + + "overwritten). A declared entry the profile already holds is adopted under its own keys; one it does not " + + "hold is created under the module's keys; an ambiguous one is refused alone, with the remedy (novox/hq ADR " + + "0317, ADR 0329 §7). What Betterbird rewrites itself — the account and identity lists, the calendars' " + + "order, the removals and each declared account's msgFilterRules.dat — is written only while Betterbird is " + + "closed, and the answer counts what waits (§8, §9). Accounts' values are withheld in the user.js shown. " + + "Betterbird reads it at its next start: betterbird_restart. (r; a with apply)", Input: map[string]any{ "apply": map[string]any{"type": "boolean", "description": "place the declared user.js in the profile (default false)"}, }, @@ -179,19 +208,20 @@ func tools() []stdio.Tool { "behind the newest published build; the enterprise policy placed and the module's defaults with it; no " + "managed preference drifting from them; Owl at its pin and active; one " + "Betterbird runs; its profile is found; the module's user.js on the machine, valid, placed and in " + - "effect; the declared accounts in the profile's list, none refused, each signed in once; its output " + - "read. Answers ok and each finding with what to do, never an address, host or name. (r)", + "effect; the declared accounts in the profile's list, none refused, each signed in once; per kind, no " + + "declared identity, calendar, address book, filter or tag drifting or refused, and no change waiting for " + + "Betterbird to close; its output read. Answers ok and each finding with what to do. (r)", Run: func(map[string]any) (any, error) { return machine.Check() }, }, { Name: "betterbird_accounts", - Description: "The accounts of the profile Betterbird uses, by count and kind: the IMAP and POP3 accounts a " + - "declaration can hold, the accounts Owl keeps (never declared), and the rest. Never an address, a host " + - "or a name, and never anything of the login store. With propose, it builds the module's setting " + - "accounts from them (every account but Owl's, without any password or token) and proposes it through " + - "the controller's settings verb for the operator's Approve on a channel that proves who answers " + - "(novox/hq ADR 0317, ADR 0277): nothing changes until then, and the operator reads the value whole at " + - "their desk. (r; a with propose)", + Description: "The accounts of the profile Betterbird uses with their values (name, address, user, servers' " + + "hosts and ports) and the declared account each is, Owl's marked by name and address (never declared), " + + "with counts by kind. Never a password, a token or anything of the login store (novox/hq ADR 0329 §1, §2). " + + "With propose, it builds the module's setting accounts from them (every account but Owl's) and proposes " + + "it in the mesh-wide layer as it stands, through the controller's settings verb, for the operator's " + + "Approve on a channel that proves who answers (novox/hq ADR 0317, ADR 0277); the answer shows the value it " + + "proposed. (r; a with propose)", Input: map[string]any{ "propose": map[string]any{"type": "boolean", "description": "build the setting accounts from the profile and propose it to the operator (default false)"}, }, @@ -203,6 +233,76 @@ func tools() []stdio.Tool { return machine.ProfileAccounts(propose) }, }, + listTool(kindIdentities, "identities (each account's default and its other identities, with name, address, reply-to, organisation and signature)"), + listTool(kindCalendars, "calendars (CalDAV with name, address, colour and options; an ICS one by its name only)"), + listTool(kindAddressBooks, "address books (CardDAV with name, address and user; a local one by its name)"), + listTool(kindFilters, "filters (per declared account: its msgFilterRules.dat as filters, folders by their path within the account; where the module keeps a file it replaced)"), + listTool(kindTags, "tags (key, name, colour, place)"), + { + Name: "betterbird_derive", + Description: "Derive the module's settings identities, calendars, address-books, filters and tags (or the kinds " + + "given) from the profiles of every desktop node holding Betterbird, laid out per layer: an entry on every " + + "such node goes into the mesh-wide layer with this node's value, and the answer lists each field another " + + "node holds differently; one on some nodes only goes into those nodes' own layers. Owl's, ICS calendars, " + + "local address books and any address carrying a password are left out and named. prefs puts values of the " + + "setting prefs into the mesh-wide layer. With propose, the next layer (the mesh-wide first, then each " + + "node's) is proposed for the operator's Approve through the controller's settings verb, carrying the layer " + + "as it stands beside the derived kinds; refused while a proposal of the module's waits, or while a desktop " + + "node does not answer (novox/hq ADR 0329 §10, ADR 0277). here answers this node's profile only, as the " + + "other nodes are asked. The values are the operator's own; never a credential. (r; a with propose)", + Input: map[string]any{ + "kinds": map[string]any{"type": "array", "items": map[string]any{"type": "string", "enum": allKinds}, + "description": "the kinds to derive (default all five)"}, + "here": map[string]any{"type": "boolean", "description": "this node's profile only (default false)"}, + "propose": map[string]any{"type": "boolean", "description": "propose the next layer to the operator (default false)"}, + "prefs": map[string]any{"type": "object", "description": "values of the setting prefs for the mesh-wide " + + "layer, by preference name, e.g. {\"mail.uifontsize\": 14}"}, + "layer": map[string]any{"type": "string", "description": "the layer to propose instead of the next: mesh-wide or a node's name"}, + }, + Run: func(args map[string]any) (any, error) { + kinds, err := texts(args, "kinds") + if err != nil { + return nil, err + } + here, err := flag(args, "here", false) + if err != nil { + return nil, err + } + propose, err := flag(args, "propose", false) + if err != nil { + return nil, err + } + layer, err := text(args, "layer", false) + if err != nil { + return nil, err + } + var prefs map[string]any + if v, ok := args["prefs"]; ok && v != nil { + if prefs, ok = v.(map[string]any); !ok { + return nil, fmt.Errorf("prefs is an object of preference names and values, not %T", v) + } + } + return machine.Derive(kinds, here, propose, prefs, layer) + }, + }, + { + Name: "betterbird_drift", + Description: "How this node's profile differs from the declared identities, calendars, address books, " + + "filters and tags (novox/hq ADR 0329 §11): per declared entry whether the profile holds it, holds another " + + "value (each differing field with the declared value and the profile's), lacks it, or it is refused; and " + + "the profile's entries nothing declares, which stay the person's. The profile as Betterbird reads it at " + + "its next start. (r)", + Input: map[string]any{ + "kind": map[string]any{"type": "string", "enum": allKinds, "description": "one kind (default every kind)"}, + }, + Run: func(args map[string]any) (any, error) { + kind, err := text(args, "kind", false) + if err != nil { + return nil, err + } + return machine.Drift(kind) + }, + }, { Name: "betterbird_reminder_test", Description: "OPENS WINDOWS ON THE OPERATOR'S SCREEN: tell the operator before calling it. Shows a calendar " + diff --git a/modules/betterbird/cmd/betterbird-tools/manifest_test.go b/modules/betterbird/cmd/betterbird-tools/manifest_test.go index af081953..1a7846d0 100644 --- a/modules/betterbird/cmd/betterbird-tools/manifest_test.go +++ b/modules/betterbird/cmd/betterbird-tools/manifest_test.go @@ -13,8 +13,8 @@ import ( ) // betterbird's shape (novox/hq issue 345, ADR 0208, ADR 0316): the X display on its own machine; its directory -// in the home with the user.js the setting prefs fills and the accounts file the setting accounts fills, both -// trusted; the community package declared absent before anything is placed in /opt/betterbird; Betterbird from +// in the home with the mergeable files the settings fill, each naming the keys it takes (ADR 0326): prefs, accounts +// and configuration (identities, calendars, address-books, filters) trusted, tags not (ADR 0329 §5); the community package declared absent before anything is placed in /opt/betterbird; Betterbird from // its pinned upstream release, Owl pinned by its vendor's digest, the policy and the dictionaries' default // unpacked into the installation; the tools' program for the start's preparation, the start and the launcher; // every new build waiting for the operator's push; and the Go bundle serving exactly the listed tools. @@ -84,7 +84,7 @@ func TestItPlacesBetterbirdWhereTheCommunityPackageDidAndOnlyAfterItIsGone(t *te for _, r := range m.Resources { ids = append(ids, r["id"].(string)) } - want := "configuration-dir,user-js,accounts,community-package,installation,owl,policies,spelling,preparation,start,launcher" + want := "configuration-dir,prefs,accounts,configuration,tags,community-package,installation,owl,policies,spelling,preparation,start,launcher" if strings.Join(ids, ",") != want { t.Fatalf("the resources, in the order the node-engine applies them: %v, want %s", ids, want) } @@ -102,7 +102,7 @@ func TestItPlacesBetterbirdWhereTheCommunityPackageDidAndOnlyAfterItIsGone(t *te } } for _, never := range []string{".thunderbird", "prefs.js", "logins.json", "key4.db"} { - if strings.Contains(raw, never) { + if regexp.MustCompile(regexp.QuoteMeta(never) + `\b`).MatchString(raw) { t.Errorf("module.json names %q: the profile is Betterbird's", never) } } @@ -114,29 +114,58 @@ func TestItPlacesBetterbirdWhereTheCommunityPackageDidAndOnlyAfterItIsGone(t *te } } -func TestTheUserJSAndTheAccountsAreTrustedFilesOfTheModulesOwn(t *testing.T) { +// ADR 0329 §4–§6, ADR 0326, issue 586: every setting reaches the machine as a mergeable file of the account's that +// names the keys it takes; four kinds and accounts and prefs are trusted, tags is not; no setting has a default in the +// definition, since each is the operator's own value; and each file's default declares nothing. +func TestTheSettingsAreMergeableFilesThatNameTheirKeysAndTagsAloneIsUntrusted(t *testing.T) { m, _ := readManifest(t) - file := resource(t, m, "user-js") - content := file["content"].(string) - if !strings.HasPrefix(content, mark) || !strings.Contains(content, "${setting:prefs}") || file["trusted"] != true { - t.Fatalf("the user.js begins with the mark, holds the setting, and is trusted (only the terminal or the operator's warrant sets it): %v", file) + if len(m.Settings) != 0 { + t.Fatalf("no setting has a default in the definition: the operator's setup is the settings' mesh-wide layer (issue 586, ADR 0262): %v", m.Settings) } - // The default is a comment, so a machine without the setting places a valid, empty user.js. - def := m.Settings["prefs"].(map[string]any)["default"].(string) - if err := validUserJS(strings.ReplaceAll(content, "${setting:prefs}", def)); err != nil { - t.Fatal(err) + files := map[string]struct { + path, mode string + takes []any + trusted bool + }{ + "prefs": {prefsFile, "0644", []any{"prefs"}, true}, + "accounts": {accountsFile, "0600", []any{"accounts"}, true}, + "configuration": {configurationFile, "0600", []any{"identities", "calendars", "address-books", "filters"}, true}, + "tags": {tagsFile, "0644", []any{"tags"}, false}, } - acc := resource(t, m, "accounts") - if acc["path"] != "${machine:account-home}/"+accountsFile || acc["merge"] != "json" || acc["trusted"] != true || - acc["mode"] != "0600" || acc["owner"] != "${machine:account}" { - t.Fatalf("the setting accounts reaches the machine as a trusted mergeable file of the account's (ADR 0316 §10, §15): %v", acc) + merging := 0 + for _, r := range m.Resources { + if r["merge"] != nil { + merging++ + } } - got, err := parseAccounts([]byte(acc["content"].(string))) + if merging != len(files) { + t.Fatalf("%d mergeable files, want %d", merging, len(files)) + } + for id, want := range files { + r := resource(t, m, id) + if r["path"] != "${machine:account-home}/"+want.path || r["merge"] != "json" || r["trusted"] != want.trusted || + r["mode"] != want.mode || r["owner"] != "${machine:account}" || !reflect.DeepEqual(r["takes-settings"], want.takes) { + t.Errorf("%s: %v", id, r) + } + var content map[string]any + if err := json.Unmarshal([]byte(r["content"].(string)), &content); err != nil || len(content) != len(want.takes) { + t.Errorf("%s's default holds each key it takes, empty: %v %v", id, content, err) + } + } + got, err := parseAccounts([]byte(resource(t, m, "accounts")["content"].(string))) if err != nil || len(got) != 0 { t.Fatalf("its default declares no account: %v %v", got, err) } - if _, declared := m.Settings["accounts"]; declared { - t.Fatal("accounts is the operator's value and has no default in the definition (ADR 0262)") + prefs, err := parseDeclaredPrefs([]byte(resource(t, m, "prefs")["content"].(string))) + if err != nil || len(prefs) != 0 { + t.Fatalf("its default declares no preference: %v %v", prefs, err) + } + var c Configuration + if err := decodeInto([]byte(resource(t, m, "configuration")["content"].(string)), configurationFile, &c); err != nil { + t.Fatal(err) + } + if err := decodeInto([]byte(resource(t, m, "tags")["content"].(string)), tagsFile, &c); err != nil { + t.Fatal(err) } } @@ -278,8 +307,10 @@ func TestTheToolsAgreeWithTheManifest(t *testing.T) { if b["kind"] != "bundle" || b["language"] != "go" || b["from"] != "cmd/betterbird-tools" || b["binary"] != "betterbird-tools" { t.Errorf("the Go tools bundle: %v", b) } - // betterbird_accounts proposes the setting accounts through the controller's settings verb (ADR 0317 §9). - if !reflect.DeepEqual(m.Invokes, []string{"seat:mesh-controller.settings"}) { - t.Errorf("the module invokes the controller's settings verb and nothing else: %v", m.Invokes) + // The tools propose through the controller's settings verb (ADR 0317 §9), and betterbird_derive asks the controller + // which nodes hold the module and each of them for its profile (ADR 0329 §10). + if !reflect.DeepEqual(m.Invokes, []string{"seat:mesh-controller.settings", "seat:mesh-controller.nodes", + "seat:mesh-controller.node", "betterbird.betterbird_derive"}) { + t.Errorf("the module invokes the controller's settings, nodes and node verbs and its own derive: %v", m.Invokes) } } diff --git a/modules/betterbird/cmd/betterbird-tools/owned.go b/modules/betterbird/cmd/betterbird-tools/owned.go new file mode 100644 index 00000000..bba5df4e --- /dev/null +++ b/modules/betterbird/cmd/betterbird-tools/owned.go @@ -0,0 +1,749 @@ +package main + +// How the five settings of ADR 0329 reach the profile (§7–§9). +// +// Identities, calendars, address books and tags are matched as accounts are (ADR 0317): an entry the module +// recorded on this node is kept by its recorded key; else an entry under the module's own key is its creation; +// else a key a node's layer names is adopted when the profile holds it and it matches; else the profile's +// entries that match (an identity by its address within its account, a calendar and an address book by their +// address, a tag by its key): one is adopted, none means the entry is created under the module's own key, more +// than one is refused with the remedy. Their declared fields are written into the user.js the module places, +// where they win at every start. An entry leaving the setting releases what was adopted and removes the +// preferences of what was created; never a calendar's events, an address book's contacts or a cache. +// +// What Betterbird rewrites itself is written only while it does not run (§9): the lists in prefs.js (an adopted +// account's identities, the calendars' order) and each account's msgFilterRules.dat. Until then the module's +// check counts, per kind, the changes that wait. + +import ( + "bufio" + "bytes" + "encoding/json" + "errors" + "fmt" + "os" + "path/filepath" + "regexp" + "strconv" + "strings" +) + +// configRecordFile is where the module records, per profile, what it adopted or created of the five kinds and +// which filter files it wrote (ADR 0329 §7, §8, ADR 0317 §3). +const configRecordFile = ".config/betterbird/configuration-record.json" + +// resolvedEntry is one declared entry as it resolved on this node. +type resolvedEntry struct { + Kind string `json:"kind"` + Name string `json:"name"` + How string `json:"how"` + Key string `json:"key,omitempty"` + Why string `json:"why,omitempty"` +} + +type recordedEntry struct { + How string `json:"how"` + Key string `json:"key"` +} + +// filterRecord is what the module did with one account's filter file on a profile. +type filterRecord struct { + // File is the account's filter file as the module last wrote it, and Kept the file it replaced at its first write. + File string `json:"file"` + Kept string `json:"kept,omitempty"` + // Released says the account left the setting filters: the module no longer writes its file (§8). + Released bool `json:"released,omitempty"` +} + +type configRecord struct { + Entries map[string]map[string]recordedEntry `json:"entries,omitempty"` + Filters map[string]filterRecord `json:"filters,omitempty"` +} + +func (m *Machine) readConfigRecord(profileDir string) configRecord { + all := map[string]configRecord{} + if raw, err := readBounded(m.home(configRecordFile)); err == nil { + _ = json.Unmarshal(raw, &all) + } + r := all[profileDir] + if r.Entries == nil { + r.Entries = map[string]map[string]recordedEntry{} + } + if r.Filters == nil { + r.Filters = map[string]filterRecord{} + } + return r +} + +func (m *Machine) writeConfigRecord(profileDir string, r configRecord) error { + all := map[string]configRecord{} + if raw, err := readBounded(m.home(configRecordFile)); err == nil { + _ = json.Unmarshal(raw, &all) + } + all[profileDir] = r + body, err := json.MarshalIndent(all, "", " ") + if err != nil { + return err + } + return writeAtomic(m.home(configRecordFile), append(body, '\n'), 0o600) +} + +// writeAtomic writes a file beside itself and renames it into place. +func writeAtomic(path string, data []byte, mode os.FileMode) error { + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + return err + } + tmp := path + ".mesh-new" + if err := writeSynced(tmp, data, mode); err != nil { + os.Remove(tmp) + return err + } + if err := os.Rename(tmp, path); err != nil { + os.Remove(tmp) + return err + } + return nil +} + +// The module's own keys for what it creates: each names the entry. +func ownIdentityKey(name string) string { return "ident_mesh_" + name } +func ownCalendarKey(name string) string { return "mesh_" + name } +func ownBookKey(name string) string { return "mesh_" + name } + +var ( + ownIdentity = regexp.MustCompile(`^ident_mesh_[a-z][a-z0-9]{0,30}$`) + ownMeshKey = regexp.MustCompile(`^mesh_[a-z][a-z0-9]{0,30}$`) +) + +// configPlan is what an apply does for the five kinds on this profile. +type configPlan struct { + Entries []resolvedEntry + // lines are the user.js statements, a comment before each entry's. + lines []string + // identityLists are, per adopted account key, the identity keys of the module's own it wants in its list. + identityLists map[string][]string + // calendarOrder are the calendar keys the module created, for calendar.list.sortOrder. + calendarOrder []string + // removals are the preference prefixes of what the module created and the setting no longer holds. + removals []string + filters []filterPlan + // released are the accounts that left the setting filters since the module last wrote their file. + released []string + record configRecord + problem string +} + +// filterPlan is one declared account's filter file on this node. +type filterPlan struct { + Account string `json:"account"` + File string `json:"file,omitempty"` + // Waiting says why the file cannot be written yet, when it cannot. + Waiting string `json:"waiting,omitempty"` + Refused string `json:"refused,omitempty"` + want string + have string + exists bool + path string +} + +func (p filterPlan) changes() bool { return p.Refused == "" && p.path != "" && p.want != p.have } + +// planConfiguration resolves every declared entry against the profile and says what an apply writes. +func (m *Machine) planConfiguration(profile Profile, c Configuration, accounts map[string]Account, + ra []resolvedAccount, prefs map[string]string) configPlan { + rec := m.readConfigRecord(profile.Dir) + plan := configPlan{identityLists: map[string][]string{}, + record: configRecord{Entries: map[string]map[string]recordedEntry{}, Filters: map[string]filterRecord{}}} + byName := map[string]resolvedAccount{} + for _, r := range ra { + byName[r.name] = r + } + have := profileAccounts(prefs) + listOfAccount := map[string][]string{} + for _, a := range have { + listOfAccount[a.Key] = a.Identities + } + identityKeyOf := map[string]string{} + + // Identities. + taken := map[string]bool{} + for _, name := range sortedKeys(c.Identities) { + id := c.Identities[name] + e := resolvedEntry{Kind: kindIdentities, Name: name} + acct, declared := byName[id.Account] + switch { + case !declared: + e.How, e.Why = refused, fmt.Sprintf("its account %s is not in the setting accounts", id.Account) + case acct.How == refused: + e.How, e.Why = refused, fmt.Sprintf("its account %s is refused: betterbird_check says why", id.Account) + case id.Address == "": + e.How, e.Key = acct.How, acct.Keys.Identity + if taken[e.Key] { + e.How, e.Key, e.Why = refused, "", fmt.Sprintf("account %s has its default identity declared twice", id.Account) + } + default: + list := listOfAccount[acct.Keys.Account] + var others []string + if len(list) > 1 { + others = list[1:] + } + present := func(k string) bool { + for _, x := range others { + if x == k { + return true + } + } + return false + } + matches := func(k string) bool { + return strings.EqualFold(unquote(prefs["mail.identity."+k+".useremail"]), id.Address) + } + e = resolveEntry(kindIdentities, name, id.Key, ownIdentityKey(name), rec.Entries[kindIdentities][name], + others, present, matches, taken, "an identity of its account with its address", + "give the one to adopt as key in this node's layer of the setting identities") + if e.How == created { + plan.identityLists[acct.Keys.Account] = append(plan.identityLists[acct.Keys.Account], e.Key) + } + } + if e.How != refused { + taken[e.Key] = true + identityKeyOf[name] = e.Key + plan.lines = append(plan.lines, fmt.Sprintf("// identity %s: %s under %s", name, e.How, e.Key)) + plan.lines = append(plan.lines, identityLines(e, id, acct)...) + } + plan.Entries = append(plan.Entries, e) + } + // A created account's identity list is the module's whole: its default and the identities it created. + for _, name := range sortedKeys(byName) { + r := byName[name] + if r.How == created && len(plan.identityLists[r.Keys.Account]) > 0 { + list := append([]string{r.Keys.Identity}, plan.identityLists[r.Keys.Account]...) + plan.lines = append(plan.lines, fmt.Sprintf("user_pref(%s, %s);", quoted("mail.account."+r.Keys.Account+".identities"), + quoted(strings.Join(list, ",")))) + delete(plan.identityLists, r.Keys.Account) + } + } + // Adopted accounts keep their list in prefs.js: the module's own keys are written into it while Betterbird is + // closed. An adopted account whose list holds none of them and wants none is left out. + for _, a := range have { + if _, wanted := plan.identityLists[a.Key]; wanted { + continue + } + for _, k := range a.Identities { + if ownIdentity.MatchString(k) { + plan.identityLists[a.Key] = []string{} + break + } + } + } + + // Calendars. + cals := profileCalendars(prefs) + calKeys := map[string]profileCalendar{} + var caldav []string + for _, pc := range cals { + calKeys[pc.Key] = pc + if pc.Kind == "caldav" { + caldav = append(caldav, pc.Key) + } + } + taken = map[string]bool{} + for _, name := range sortedKeys(c.Calendars) { + cal := c.Calendars[name] + present := func(k string) bool { pc, ok := calKeys[k]; return ok && pc.Kind == "caldav" } + matches := func(k string) bool { return sameURL(calKeys[k].Value.URL, cal.URL) } + e := resolveEntry(kindCalendars, name, cal.Key, ownCalendarKey(name), rec.Entries[kindCalendars][name], + caldav, present, matches, taken, "a CalDAV calendar at its address", + "remove the duplicate in Betterbird, or give the one to adopt as key in this node's layer of the setting calendars") + if e.How != refused { + taken[e.Key] = true + if e.How == created { + plan.calendarOrder = append(plan.calendarOrder, e.Key) + } + plan.lines = append(plan.lines, fmt.Sprintf("// calendar %s: %s under %s", name, e.How, e.Key)) + plan.lines = append(plan.lines, calendarLines(e, cal, identityKeyOf[cal.Invitations])...) + } + plan.Entries = append(plan.Entries, e) + } + + // Address books. + books := profileAddressBooks(prefs) + bookKeys := map[string]profileAddressBook{} + var carddav []string + for _, b := range books { + bookKeys[b.Key] = b + if b.CardDAV { + carddav = append(carddav, b.Key) + } + } + taken = map[string]bool{} + for _, name := range sortedKeys(c.AddressBooks) { + ab := c.AddressBooks[name] + present := func(k string) bool { b, ok := bookKeys[k]; return ok && b.CardDAV } + matches := func(k string) bool { return sameURL(bookKeys[k].Value.URL, ab.URL) } + e := resolveEntry(kindAddressBooks, name, ab.Key, ownBookKey(name), rec.Entries[kindAddressBooks][name], + carddav, present, matches, taken, "a CardDAV address book at its address", + "remove the duplicate in Betterbird, or give the one to adopt as key in this node's layer of the setting address-books") + if e.How != refused { + taken[e.Key] = true + plan.lines = append(plan.lines, fmt.Sprintf("// address book %s: %s under %s", name, e.How, e.Key)) + plan.lines = append(plan.lines, addressBookLines(e, ab, name)...) + } + plan.Entries = append(plan.Entries, e) + } + + // Tags: a tag's key is its own, unique in a profile, so a match is never ambiguous. + tagKeys := map[string]bool{} + for _, t := range profileTags(prefs) { + tagKeys[t.Key] = true + } + for _, key := range sortedKeys(c.Tags) { + t := c.Tags[key] + e := resolvedEntry{Kind: kindTags, Name: key, Key: key, How: adopted} + if was, ok := rec.Entries[kindTags][key]; ok && was.How == created { + e.How = created + } else if !tagKeys[key] { + e.How = created + } + plan.lines = append(plan.lines, fmt.Sprintf("// tag %s: %s", key, e.How)) + plan.lines = append(plan.lines, tagLines(key, t)...) + plan.Entries = append(plan.Entries, e) + } + + // What the setting no longer holds: an adopted entry is released (its lines go), a created one's preferences + // are removed from prefs.js while Betterbird is closed. + declaredNow := map[string]map[string]bool{} + for _, e := range plan.Entries { + if declaredNow[e.Kind] == nil { + declaredNow[e.Kind] = map[string]bool{} + } + declaredNow[e.Kind][e.Name] = true + if e.How != refused { + if plan.record.Entries[e.Kind] == nil { + plan.record.Entries[e.Kind] = map[string]recordedEntry{} + } + plan.record.Entries[e.Kind][e.Name] = recordedEntry{How: e.How, Key: e.Key} + } + } + for _, kind := range []string{kindIdentities, kindCalendars, kindAddressBooks, kindTags} { + for _, name := range sortedKeys(rec.Entries[kind]) { + was := rec.Entries[kind][name] + if declaredNow[kind][name] || was.How != created { + continue + } + switch kind { + case kindIdentities: + plan.removals = append(plan.removals, "mail.identity."+was.Key+".") + case kindCalendars: + plan.removals = append(plan.removals, "calendar.registry."+was.Key+".") + case kindAddressBooks: + plan.removals = append(plan.removals, "ldap_2.servers."+was.Key+".") + case kindTags: + plan.removals = append(plan.removals, "mailnews.tags."+was.Key+".") + } + } + } + + // Filters, owned whole per declared account (§8). + for _, account := range sortedAccountsOf(c.Filters) { + fp := filterPlan{Account: account} + r, ok := byName[account] + switch { + case !ok: + fp.Refused = fmt.Sprintf("its account %s is not in the setting accounts", account) + case r.How == refused: + fp.Refused = fmt.Sprintf("its account %s is refused: betterbird_check says why", account) + default: + dir := m.accountDirectory(profile.path, prefs, r.Keys.Server) + if dir == "" { + fp.Waiting = "Betterbird has not made the account's directory yet: it does at its first start with the account" + break + } + fp.path = filepath.Join(dir, filterFileName) + fp.File = m.tilde(strings.TrimPrefix(fp.path, m.Root)) + fp.want = renderFilters(c.Filters[account], folderBase(prefs, r.Keys.Server)) + raw, err := readBounded(fp.path) + fp.exists = err == nil + fp.have = string(raw) + } + was := rec.Filters[account] + was.Released = false + if fp.path != "" { + was.File = fp.File + } + if was.File != "" || was.Kept != "" { + plan.record.Filters[account] = was + } + plan.filters = append(plan.filters, fp) + } + for _, account := range sortedKeys(rec.Filters) { + if _, still := c.Filters[account]; still { + continue + } + was := rec.Filters[account] + if !was.Released { + plan.released = append(plan.released, account) + } + was.Released = true + plan.record.Filters[account] = was + } + return plan +} + +// resolveEntry resolves an identity, a calendar or an address book as ADR 0317 resolves an account (§7). +func resolveEntry(kind, name, layerKey, own string, was recordedEntry, candidates []string, present, matches func(string) bool, + taken map[string]bool, what, remedy string) resolvedEntry { + e := resolvedEntry{Kind: kind, Name: name} + refuse := func(format string, args ...any) resolvedEntry { + e.How, e.Key, e.Why = refused, "", fmt.Sprintf(format, args...) + return e + } + // 1. Recorded, while the profile still holds it. + if was.Key != "" && !taken[was.Key] && (present(was.Key) || was.How == created) && (layerKey == "" || layerKey == was.Key) { + e.How, e.Key = was.How, was.Key + return e + } + // 2. Under the module's own key: its creation. + if present(own) { + e.How, e.Key = created, own + return e + } + // 3. The key a node's layer names. + if layerKey != "" { + switch { + case !present(layerKey): + return refuse("the key %s this node's layer names is not in the profile", layerKey) + case taken[layerKey]: + return refuse("the key %s is already another declared entry's", layerKey) + case !matches(layerKey): + return refuse("the key %s this node's layer names is not %s", layerKey, what) + } + e.How, e.Key = adopted, layerKey + return e + } + // 4. By the match. + var found []string + for _, k := range candidates { + if !taken[k] && !ownMeshKey.MatchString(k) && !ownIdentity.MatchString(k) && matches(k) { + found = append(found, k) + } + } + switch len(found) { + case 0: + e.How, e.Key = created, own + case 1: + e.How, e.Key = adopted, found[0] + default: + return refuse("%d entries in the profile are %s (%s): %s (novox/hq ADR 0329 §7)", len(found), what, + strings.Join(found, ", "), remedy) + } + return e +} + +// sameURL compares two addresses without a trailing slash's difference. +func sameURL(a, b string) bool { + return a != "" && strings.EqualFold(strings.TrimRight(a, "/"), strings.TrimRight(b, "/")) +} + +func pref(name, value string) string { return fmt.Sprintf("user_pref(%s, %s);", quoted(name), value) } + +func boolWord(b bool) string { return strconv.FormatBool(b) } + +// identityLines are an identity's declared fields under its key. The default identity's name and address are +// the setting accounts' (§4), so only another identity carries them here. +func identityLines(e resolvedEntry, id *Identity, acct resolvedAccount) []string { + p := "mail.identity." + e.Key + "." + var out []string + if id.Address != "" { + out = append(out, pref(p+"fullName", jsQuoted(id.Name)), pref(p+"useremail", jsQuoted(id.Address))) + if e.How == created { + out = append(out, pref(p+"valid", "true"), pref(p+"smtpServer", quoted(acct.Keys.SMTP))) + } + } + above := 0 + if id.SignatureAboveQuote { + above = 1 + } + out = append(out, + pref(p+"reply_to", jsQuoted(id.ReplyTo)), + pref(p+"organization", jsQuoted(id.Organization)), + pref(p+"htmlSigText", jsQuoted(strings.Join(id.Signature, "\n"))), + pref(p+"htmlSigFormat", boolWord(id.SignatureHTML)), + pref(p+"attach_signature", "false"), + pref(p+"reply_on_top", strconv.Itoa(above)), + pref(p+"sig_bottom", boolWord(!id.SignatureAboveQuote))) + return out +} + +func calendarLines(e resolvedEntry, cal *Calendar, identityKey string) []string { + p := "calendar.registry." + e.Key + "." + var out []string + if e.How == created { + out = append(out, pref(p+"type", quoted("caldav")), pref(p+"cache.enabled", "true"), + pref(p+"calendar-main-in-composite", "true")) + } + reminders := cal.Reminders == nil || *cal.Reminders + out = append(out, pref(p+"uri", jsQuoted(cal.URL)), pref(p+"name", jsQuoted(cal.Name))) + if cal.Color != "" { + out = append(out, pref(p+"color", jsQuoted(strings.ToLower(cal.Color)))) + } + out = append(out, pref(p+"readOnly", boolWord(cal.ReadOnly)), pref(p+"suppressAlarms", boolWord(!reminders)), + pref(p+"username", jsQuoted(cal.User)), pref(p+"imip.identity.key", jsQuoted(identityKey))) + return out +} + +func addressBookLines(e resolvedEntry, ab *AddressBook, name string) []string { + p := "ldap_2.servers." + e.Key + "." + var out []string + if e.How == created { + out = append(out, pref(p+"dirType", "102"), pref(p+"filename", quoted("abook-mesh_"+name+".sqlite"))) + } + return append(out, pref(p+"description", jsQuoted(ab.Name)), pref(p+"carddav.url", jsQuoted(ab.URL)), + pref(p+"carddav.username", jsQuoted(ab.User))) +} + +func tagLines(key string, t *Tag) []string { + p := "mailnews.tags." + key + "." + out := []string{pref(p+"tag", jsQuoted(t.Name))} + if t.Color != "" { + out = append(out, pref(p+"color", jsQuoted(strings.ToUpper(t.Color)))) + } + if t.Order > 0 { + out = append(out, pref(p+"ordinal", quoted(tagOrdinal(t.Order)))) + } + return out +} + +// configBlock is the user.js block of the five kinds. +func (p configPlan) configBlock() string { + if len(p.lines) == 0 { + return "" + } + return "\n// The identities, calendars, address books and tags the settings declare (novox/hq ADR 0329), under the\n" + + "// keys each was adopted or created under. No password, token or cookie.\n" + strings.Join(p.lines, "\n") + "\n" +} + +// closedFor says why the profile's own files may not be written now, or "": Betterbird runs, or the profile's +// lock says one holds it (§9). +func (m *Machine) closedFor(profile string) string { + if ps := m.mainProcs(); len(ps) > 0 { + return fmt.Sprintf("Betterbird is running (pid %d), and it rewrites these files when it quits", ps[0].PID) + } + if _, err := os.Lstat(filepath.Join(profile, "lock")); err == nil { + return "the profile is locked (its lock file is there): a Betterbird holds it, or one ended without quitting" + } + return "" +} + +// prefsChanges is prefs.js as the plan wants it, and how many changes each kind has in it. +func (p configPlan) prefsChanges(text string) (string, map[string]int) { + counts := map[string]int{} + var lines []string + done := map[string]bool{} + s := bufio.NewScanner(strings.NewReader(text)) + s.Buffer(make([]byte, 64<<10), 4<<20) + for s.Scan() { + line := s.Text() + mm := prefLine.FindStringSubmatch(line) + if mm == nil { + lines = append(lines, line) + continue + } + name := mm[1] + removed := false + for _, prefix := range p.removals { + if strings.HasPrefix(name, prefix) { + removed = true + counts[kindOfPrefix(prefix)]++ + break + } + } + if removed { + continue + } + if acct, ok := strings.CutPrefix(name, "mail.account."); ok && strings.HasSuffix(acct, ".identities") { + key := strings.TrimSuffix(acct, ".identities") + if want, ok := p.identityLists[key]; ok { + done[name] = true + next := listedWithOwn(listOf(mm[2]), want, ownIdentity.MatchString, ",") + if next != strings.Join(listOf(mm[2]), ",") { + line = pref(name, quoted(next)) + counts[kindIdentities]++ + } + } + } + if name == "calendar.list.sortOrder" { + done[name] = true + have := strings.Fields(unquote(mm[2])) + next := listedWithOwn(have, p.calendarOrder, ownMeshKey.MatchString, " ") + if next != strings.Join(have, " ") { + line = pref(name, quoted(next)) + counts[kindCalendars]++ + } + } + lines = append(lines, line) + } + if !done["calendar.list.sortOrder"] && len(p.calendarOrder) > 0 { + lines = append(lines, pref("calendar.list.sortOrder", quoted(strings.Join(p.calendarOrder, " ")))) + counts[kindCalendars]++ + } + out := strings.Join(lines, "\n") + if text == "" || strings.HasSuffix(text, "\n") { + out += "\n" + } + return out, counts +} + +func kindOfPrefix(prefix string) string { + switch { + case strings.HasPrefix(prefix, "mail.identity."): + return kindIdentities + case strings.HasPrefix(prefix, "calendar."): + return kindCalendars + case strings.HasPrefix(prefix, "ldap_2."): + return kindAddressBooks + } + return kindTags +} + +// listedWithOwn is a list with the module's own keys made the wanted ones, every other key in its place. +func listedWithOwn(have, wanted []string, own func(string) bool, sep string) string { + want := map[string]bool{} + for _, k := range wanted { + want[k] = true + } + var out []string + there := map[string]bool{} + for _, k := range have { + if own(k) && !want[k] { + continue + } + out = append(out, k) + there[k] = true + } + for _, k := range wanted { + if !there[k] { + out = append(out, k) + } + } + return strings.Join(out, sep) +} + +// Waiting counts, per kind, the changes of what Betterbird rewrites that wait for it to close (§9). +func (m *Machine) waiting(profile string, p configPlan, ra []resolvedAccount) map[string]int { + raw, _ := readBounded(filepath.Join(profile, "prefs.js")) + _, counts := p.prefsChanges(string(raw)) + var wantAccounts, wantSMTP []string + for _, r := range ra { + if r.How == created { + wantAccounts, wantSMTP = append(wantAccounts, r.Keys.Account), append(wantSMTP, r.Keys.SMTP) + } + } + if _, changed := withListed(string(raw), map[string][]string{ + "mail.accountmanager.accounts": wantAccounts, "mail.smtpservers": wantSMTP}); changed { + counts["accounts"]++ + } + for _, f := range p.filters { + if f.changes() { + counts[kindFilters]++ + } + } + for k, v := range counts { + if v == 0 { + delete(counts, k) + } + } + return counts +} + +// writeClosed writes what Betterbird rewrites itself, only while it is closed: prefs.js's lists and the +// removals, and each declared account's filter file, keeping the file it replaces at its first write. It +// answers what it wrote, or why it waits. +func (m *Machine) writeClosed(profile Profile, p *configPlan) (string, error) { + var said []string + path := filepath.Join(profile.path, "prefs.js") + raw, err := readBounded(path) + if err != nil && !errors.Is(err, os.ErrNotExist) { + return "", err + } + next, counts := p.prefsChanges(string(raw)) + pending := next != string(raw) + for _, f := range p.filters { + pending = pending || f.changes() + } + if !pending { + return "unchanged", nil + } + if why := m.closedFor(profile.path); why != "" { + return "", errors.New(why + ": the identities' and calendars' lists, the removals and the filter files are " + + "written at its next start from the launcher, or by betterbird_restart") + } + if next != string(raw) { + mode := os.FileMode(0o600) + if st, err := os.Stat(path); err == nil { + mode = st.Mode().Perm() + } + if err := writeAtomic(path, []byte(next), mode); err != nil { + return "", err + } + said = append(said, fmt.Sprintf("prefs.js: %d change(s) to the lists and removals", sum(counts))) + } + for i, f := range p.filters { + if !f.changes() { + continue + } + was := p.record.Filters[f.Account] + if was.Kept == "" && f.exists { + kept := m.keptFilterFile(profile.Dir, f.Account) + if err := writeAtomic(kept, []byte(f.have), 0o600); err != nil { + return strings.Join(said, "; "), fmt.Errorf("the filter file of %s was not written: the file it replaces could not be kept: %v", f.Account, err) + } + was.Kept = m.tilde(strings.TrimPrefix(kept, m.Root)) + } + mode := os.FileMode(0o644) + if st, err := os.Stat(f.path); err == nil { + mode = st.Mode().Perm() + } + if err := writeAtomic(f.path, []byte(f.want), mode); err != nil { + return strings.Join(said, "; "), err + } + was.File = f.File + p.record.Filters[f.Account] = was + p.filters[i].have, p.filters[i].exists = f.want, true + said = append(said, "the filters of "+f.Account) + } + if len(said) == 0 { + return "unchanged", nil + } + return "written: " + strings.Join(said, "; "), nil +} + +func sum(m map[string]int) int { + n := 0 + for _, v := range m { + n += v + } + return n +} + +// accountDirectory is an account's directory in the profile, as its server's preferences name it, or "" when +// they name none within the profile. +func (m *Machine) accountDirectory(profile string, prefs map[string]string, server string) string { + if dir := accountDirectory(profile, prefs, server); dir != "" { + return dir + } + abs := unquote(prefs["mail.server."+server+".directory"]) + if abs == "" { + return "" + } + dir := m.path(abs) + if strings.HasPrefix(filepath.Clean(dir), filepath.Clean(profile)+string(filepath.Separator)) { + return dir + } + return "" +} + +func mustJSON(v any) string { + b, _ := json.Marshal(v) + return string(bytes.TrimSpace(b)) +} diff --git a/modules/betterbird/cmd/betterbird-tools/prefs.go b/modules/betterbird/cmd/betterbird-tools/prefs.go new file mode 100644 index 00000000..1d4b9dca --- /dev/null +++ b/modules/betterbird/cmd/betterbird-tools/prefs.go @@ -0,0 +1,228 @@ +package main + +// The setting prefs, one entry per preference (novox/hq issue 586, ADR 0329; ADR 0316 §11 for the user.js). +// +// Until issue 586 the setting was one line of user_pref(…) statements, so a node's layer replaced it whole and +// the operator's own choices were shipped in the catalogue as the module's defaults file. Now the setting is a +// map from a preference's name to its value, and reaches the machine as ~/.config/betterbird/prefs.json, a +// file the node-engine merges from the module's layers key by key: the mesh-wide layer holds the operator's +// setup, and a node's layer overrides one preference, or takes one out with null. The module itself ships only +// its policy's locks and Betterbird's own defaults (mesh-defaults.js names the preferences it manages there). +// +// The tools turn the setting into the head of the user.js they place in the profile, where Betterbird reads it +// at its start and lets it win over what it saved itself. A preference that can hold a credential, or that +// another setting owns (an account's, an identity's, a calendar's, an address book's, a tag's), is refused, so +// each value has one owner (ADR 0316 §11, ADR 0329 §6). + +import ( + "bytes" + "encoding/json" + "errors" + "fmt" + "math" + "os" + "regexp" + "sort" + "strconv" + "strings" +) + +// prefsFile is the setting prefs on the machine, written by the node-engine. +const prefsFile = ".config/betterbird/prefs.json" + +// configurationOwned are the preferences the settings of ADR 0329 own: the setting prefs refuses them (§6). +var configurationOwned = []string{"calendar.registry.", "calendar.list.", "ldap_2.servers.", "mailnews.tags."} + +// prefName is a preference's name as the setting may give it. +var prefName = regexp.MustCompile(`^[A-Za-z0-9_][A-Za-z0-9_.\-\[\]@{}]{0,199}$`) + +// prefOwner says why the setting prefs may not set a preference, or "" when it may. +func prefOwner(name string) string { + if passwordSecret.MatchString(name) { + return "a preference that can hold a credential: not through the mesh" + } + for _, p := range accountOwned { + if strings.HasPrefix(name, p) { + return "an account's preference: an account is declared in the setting accounts, never in prefs (novox/hq ADR 0316 §11)" + } + } + for _, p := range configurationOwned { + if strings.HasPrefix(name, p) { + return "owned by the setting calendars, address-books or tags, never by prefs (novox/hq ADR 0329 §6)" + } + } + return "" +} + +// readDeclaredPrefs reads the setting prefs on the machine: each preference's value as user.js writes it. A +// missing file says the module is not pushed here in this form; a file that names a preference another setting +// owns, or holds a value user.js cannot carry, is refused whole, naming the preference. +func (m *Machine) readDeclaredPrefs() (map[string]string, bool, error) { + raw, err := readBounded(m.home(prefsFile)) + if errors.Is(err, os.ErrNotExist) { + return map[string]string{}, false, nil + } + if err != nil { + return nil, false, err + } + prefs, err := parseDeclaredPrefs(raw) + return prefs, true, err +} + +func parseDeclaredPrefs(raw []byte) (map[string]string, error) { + var top map[string]json.RawMessage + if err := json.Unmarshal(raw, &top); err != nil { + return nil, fmt.Errorf("~/%s is not a JSON object: %v", prefsFile, err) + } + out := map[string]string{} + body, ok := top["prefs"] + if !ok || string(body) == "null" { + return out, nil + } + var values map[string]any + dec := json.NewDecoder(bytes.NewReader(body)) + dec.UseNumber() + if err := dec.Decode(&values); err != nil { + return nil, fmt.Errorf("the setting prefs is a map from a preference's name to its value: %v", err) + } + var problems []string + for _, name := range sortedKeys(values) { + if !prefName.MatchString(name) { + problems = append(problems, fmt.Sprintf("%q is not a preference's name", name)) + continue + } + if why := prefOwner(name); why != "" { + problems = append(problems, name+": "+why) + continue + } + lit, err := prefLiteral(values[name]) + if err != nil { + problems = append(problems, name+": "+err.Error()) + continue + } + if lit != "" { + out[name] = lit + } + } + if len(problems) > 0 { + return nil, fmt.Errorf("the setting prefs is refused: %s", strings.Join(problems, "; ")) + } + return out, nil +} + +// prefLiteral is a setting's value as user.js writes it: true or false, a whole number, or a string. null is +// no value: a node's layer takes the mesh-wide one out with it, and Betterbird's own default holds there. +func prefLiteral(v any) (string, error) { + switch x := v.(type) { + case nil: + return "", nil + case bool: + return strconv.FormatBool(x), nil + case json.Number: + n, err := x.Int64() + if err != nil || n > math.MaxInt32 || n < math.MinInt32 { + return "", fmt.Errorf("%s is not a whole number Betterbird keeps", x) + } + return strconv.FormatInt(n, 10), nil + case float64: + if x != math.Trunc(x) || x > math.MaxInt32 || x < math.MinInt32 { + return "", fmt.Errorf("%v is not a whole number Betterbird keeps", x) + } + return strconv.FormatInt(int64(x), 10), nil + case string: + if strings.ContainsAny(x, lineEnds) { + return "", errors.New("a string preference is one line") + } + return jsQuoted(x), nil + } + return "", fmt.Errorf("a value is true, false, a whole number or a string, not %T", v) +} + +// prefValue is a preference's literal as the setting holds it: the reverse of prefLiteral. +func prefValue(lit string) (any, bool) { + switch { + case lit == "true": + return true, true + case lit == "false": + return false, true + } + if n, err := strconv.ParseInt(lit, 10, 32); err == nil { + return n, true + } + if strings.HasPrefix(lit, `"`) { + if s, err := strconv.Unquote(lit); err == nil { + return s, true + } + } + return nil, false +} + +// lineEnds are the characters a reader may take as a line's end: a setting holds none (novox/hq issue 339). +const lineEnds = "\n\r\v\f\x00\u0085

" + +// jsQuoted is a string as a preference file writes it: quote, backslash and the control characters escaped the +// way Gecko's preference parser reads them. +func jsQuoted(s string) string { + var b strings.Builder + b.WriteByte('"') + for _, r := range s { + switch { + case r == '"': + b.WriteString(`\"`) + case r == '\\': + b.WriteString(`\\`) + case r == '\n': + b.WriteString(`\n`) + case r == '\r': + b.WriteString(`\r`) + case r == '\t': + b.WriteString(`\t`) + case r < 0x20 || r == 0x7f || r == 0x2028 || r == 0x2029 || r == 0x85: + fmt.Fprintf(&b, `\u%04x`, r) + default: + b.WriteRune(r) + } + } + b.WriteByte('"') + return b.String() +} + +// declaredText is the head of the user.js the module places: its mark and the setting prefs, one statement a +// line, by name. present says the setting is on this machine; err says why it is refused. +func (m *Machine) declaredText() (string, bool, error) { + prefs, present, err := m.readDeclaredPrefs() + if err != nil || !present { + return "", present, err + } + return composeDeclared(prefs), true, nil +} + +// composeDeclared is the user.js head for a set of declared preferences. +func composeDeclared(prefs map[string]string) string { + var b strings.Builder + b.WriteString(mark + ", novox/hq issue 345). Owned by the mesh: the tools write it into the profile from the\n" + + "// module's settings at Betterbird's start (betterbird_user_js with apply, betterbird_restart, the launcher), and\n" + + "// Betterbird reads it at its start, where it wins over what Betterbird saved. Change the settings, never this file.\n" + + "//\n// The setting prefs (novox/hq issue 586): the operator's setup, mesh-wide with a node's own layer over it.\n") + names := make([]string, 0, len(prefs)) + for n := range prefs { + names = append(names, n) + } + sort.Strings(names) + for _, n := range names { + fmt.Fprintf(&b, "user_pref(%s, %s);\n", quoted(n), prefs[n]) + } + if len(names) == 0 { + b.WriteString("// none\n") + } + return b.String() +} + +func sortedKeys[V any](m map[string]V) []string { + out := make([]string, 0, len(m)) + for k := range m { + out = append(out, k) + } + sort.Strings(out) + return out +} diff --git a/modules/betterbird/cmd/betterbird-tools/profilekinds.go b/modules/betterbird/cmd/betterbird-tools/profilekinds.go new file mode 100644 index 00000000..8c57a67c --- /dev/null +++ b/modules/betterbird/cmd/betterbird-tools/profilekinds.go @@ -0,0 +1,178 @@ +package main + +// The profile's identities, calendars, address books and tags, read from its preferences into the setting's +// shape (novox/hq ADR 0329). The profile is read as Betterbird starts with it: prefs.js, with user.js over it. +// Nothing here reads the login store, a calendar's events or an address book's contacts (§13). + +import ( + "path/filepath" + "sort" + "strconv" + "strings" +) + +// profileIdentity is one identity the profile holds. +type profileIdentity struct { + Key string + AccountKey string + // Default says it is its account's first identity. + Default bool + Owl bool + Value Identity +} + +// profileIdentities reads every identity of every account in the profile's list, the default first. +func (m *Machine) profileIdentities(prefs map[string]string) []profileIdentity { + var out []profileIdentity + for _, a := range profileAccounts(prefs) { + for i, id := range a.Identities { + out = append(out, profileIdentity{Key: id, AccountKey: a.Key, Default: i == 0, Owl: a.Owl, + Value: m.readIdentity(prefs, id)}) + } + } + return out +} + +// readIdentity reads one identity's fields. A signature the profile reads from a file is read as its text, since +// a file's path differs per node (§10). +func (m *Machine) readIdentity(prefs map[string]string, key string) Identity { + p := "mail.identity." + key + "." + text := func(name string) string { return unquote(prefs[p+name]) } + id := Identity{Address: text("useremail"), Name: text("fullName"), ReplyTo: text("reply_to"), + Organization: text("organization"), SignatureHTML: prefs[p+"htmlSigFormat"] == "true", + SignatureAboveQuote: prefs[p+"reply_on_top"] == "1" && prefs[p+"sig_bottom"] == "false"} + sig := text("htmlSigText") + if prefs[p+"attach_signature"] == "true" { + if file := text("sig_file"); file != "" { + sig = m.signatureFile(file) + } + } + if sig = strings.TrimRight(strings.ReplaceAll(sig, "\r\n", "\n"), "\n"); sig != "" { + id.Signature = strings.Split(sig, "\n") + } + return id +} + +// signatureFile is a signature file's text, read only within the home and bounded. +func (m *Machine) signatureFile(file string) string { + clean := filepath.Clean(file) + if !strings.HasPrefix(clean, strings.TrimSuffix(m.Home, "/")+"/") { + return "" + } + raw, err := readBounded(m.path(clean)) + if err != nil || len(raw) > 64<<10 { + return "" + } + return string(raw) +} + +// keysUnder are the distinct second-level keys of the preferences under a prefix: calendar.registry..…. +func keysUnder(prefs map[string]string, prefix string) []string { + seen := map[string]bool{} + for name := range prefs { + if rest, ok := strings.CutPrefix(name, prefix); ok { + if key, _, found := strings.Cut(rest, "."); found && key != "" { + seen[key] = true + } + } + } + return sortedKeys(seen) +} + +// profileCalendar is one calendar of the profile: Kind is Betterbird's (caldav, ics, storage, owl …). +type profileCalendar struct { + Key string + Kind string + // IdentityKey is the identity that answers its invitations. + IdentityKey string + Value Calendar +} + +func profileCalendars(prefs map[string]string) []profileCalendar { + var out []profileCalendar + for _, key := range keysUnder(prefs, "calendar.registry.") { + p := "calendar.registry." + key + "." + text := func(name string) string { return unquote(prefs[p+name]) } + c := profileCalendar{Key: key, Kind: text("type"), IdentityKey: text("imip.identity.key"), + Value: Calendar{Name: text("name"), URL: text("uri"), Color: strings.ToLower(text("color")), + ReadOnly: prefs[p+"readOnly"] == "true", User: text("username")}} + if prefs[p+"suppressAlarms"] == "true" { + off := false + c.Value.Reminders = &off + } + if c.Kind == "" && c.Value.URL == "" { + continue + } + out = append(out, c) + } + return out +} + +// profileAddressBook is one address book of the profile: CardDAV is the kind a setting holds. +type profileAddressBook struct { + Key string + CardDAV bool + Kind string + Value AddressBook +} + +// Betterbird's kinds of address book (dirType). +var addressBookKinds = map[string]string{"0": "LDAP", "101": "local", "102": "CardDAV"} + +func profileAddressBooks(prefs map[string]string) []profileAddressBook { + var out []profileAddressBook + for _, key := range keysUnder(prefs, "ldap_2.servers.") { + p := "ldap_2.servers." + key + "." + if _, has := prefs[p+"dirType"]; !has && prefs[p+"description"] == "" { + continue + } + kind := addressBookKinds[prefs[p+"dirType"]] + if kind == "" { + kind = "other" + } + out = append(out, profileAddressBook{Key: key, CardDAV: kind == "CardDAV", Kind: kind, + Value: AddressBook{Name: unquote(prefs[p+"description"]), URL: unquote(prefs[p+"carddav.url"]), + User: unquote(prefs[p+"carddav.username"])}}) + } + return out +} + +// profileTag is one tag of the profile. +type profileTag struct { + Key string + ordinal string + Value Tag +} + +// profileTags reads the tags, each with its place as Betterbird orders them. +func profileTags(prefs map[string]string) []profileTag { + var out []profileTag + for _, key := range keysUnder(prefs, "mailnews.tags.") { + p := "mailnews.tags." + key + "." + if _, has := prefs[p+"tag"]; !has { + continue + } + out = append(out, profileTag{Key: key, ordinal: unquote(prefs[p+"ordinal"]), + Value: Tag{Name: unquote(prefs[p+"tag"]), Color: strings.ToUpper(unquote(prefs[p+"color"]))}}) + } + // Betterbird orders the tags by their ordinal, a tag without one by its key in its place. + at := func(t profileTag) string { + if t.ordinal != "" { + return t.ordinal + } + return t.Key + } + sort.SliceStable(out, func(i, j int) bool { + if at(out[i]) != at(out[j]) { + return at(out[i]) < at(out[j]) + } + return out[i].Key < out[j].Key + }) + for i := range out { + out[i].Value.Order = i + 1 + } + return out +} + +// tagOrdinal is the ordinal a tag's place is written as, so Betterbird's order by text is the declared one. +func tagOrdinal(order int) string { return strconv.Itoa(1000 + order) } diff --git a/modules/betterbird/cmd/betterbird-tools/reminder.go b/modules/betterbird/cmd/betterbird-tools/reminder.go index d8d52604..d60cc684 100644 --- a/modules/betterbird/cmd/betterbird-tools/reminder.go +++ b/modules/betterbird/cmd/betterbird-tools/reminder.go @@ -165,8 +165,8 @@ func (m *Machine) ReminderTest(after time.Duration, carry []string, stop bool) ( // The module's user.js first, then what is carried, then the test's own preferences: the last // statement wins, so neither can take the test's calendar or its quiet start away. user := "" - if mine, err := readBounded(m.home(declared)); err == nil && validUserJS(string(mine)) == nil { - user = string(mine) + "\n" + if mine, present, err := m.declaredText(); err == nil && present && validUserJS(mine) == nil { + user = mine + "\n" } user += strings.Join(carried, "\n") + "\n" + testPrefs(strings.TrimPrefix(calendar, m.Root)) if err := os.WriteFile(filepath.Join(dir, "profile", "user.js"), []byte(user), 0o600); err != nil { diff --git a/modules/betterbird/install/defaults-pref/mesh-defaults.js b/modules/betterbird/install/defaults-pref/mesh-defaults.js index 6134173c..3cd76a27 100644 --- a/modules/betterbird/install/defaults-pref/mesh-defaults.js +++ b/modules/betterbird/install/defaults-pref/mesh-defaults.js @@ -1,85 +1,20 @@ -// Written by the mesh (module betterbird, novox/hq issue 345): the operator's setup of Betterbird, taken from -// the laptop's profile and approved by the operator on 2026-10-12. Betterbird reads this file at its start as -// a default: a value the profile saved in its prefs.js (a choice made in Betterbird's settings window) wins -// over it, a node's layer of the setting prefs (user.js) wins over both, and the policy's locks -// (distribution/policies.json) win over everything. betterbird_config shows the layers and the drift; -// betterbird_config_apply brings a profile back to these values. Change it in the catalogue, never here. +// Written by the mesh (module betterbird, novox/hq issue 345, issue 586): the preferences the module manages at +// Betterbird's own default. It holds no value of the operator's: the operator's setup is the module's setting +// prefs, mesh-wide with a node's own layer over it, which the tools place in the profile's user.js, where it wins +// at every start; the policy's locks (distribution/policies.json) win over everything. betterbird_config shows the +// layers and the drift. Change it in the catalogue, never here. // -// A line "keeps Betterbird's default" names a preference the module manages without writing a value: the -// operator's setup is Betterbird's own default there, and a value the profile saved is drift. - -// --- Theme and display --- +// A line "keeps Betterbird's default" names a preference the module manages without writing a value: Betterbird's +// own default holds there, and a value the profile saved is drift, which betterbird_config_apply clears. // keeps Betterbird's default: extensions.activeThemeID -// The default theme (default-theme@mozilla.org) follows the desktop's dark mode; compactness comes from -// mail.uidensity, not a theme. Betterbird keeps the active theme in its add-on database, so another theme is -// changed back in the add-on manager, never by clearing prefs.js. +// The default theme (default-theme@mozilla.org) follows the desktop's dark mode. Betterbird keeps the active theme +// in its add-on database, so another theme is changed back in the add-on manager, never by clearing prefs.js. // keeps Betterbird's default: browser.display.use_system_colors // Betterbird's own colours, which the default theme is drawn for. // keeps Betterbird's default: layout.css.always_underline_links // Links are underlined only where the message's own style says so. - -// Compact density: more of the message list fits on a laptop screen (Betterbird's default is 1, normal). -pref("mail.uidensity", 0); -// 14-point interface text, the operator's choice; a larger desk screen sets its own size in a node layer. -pref("mail.uifontsize", 14); -// Message bodies are shown as their sender wrote them, not darkened. -pref("mail.dark-reader.enabled", false); - -// --- Layout --- - -// Vertical view: folders, the message list and the message side by side (Betterbird's default is 0, classic). -pref("mail.pane_config.dynamic", 2); -// Each row of the message list on several lines, so sender, subject and date all show in the narrow column. -pref("mail.pane_config.multiline_all", true); -// The message list as cards (Betterbird's default is 1, the table). -pref("mail.threadpane.listview", 0); -// Threads grouped in every folder by default (Betterbird's default is 0, unthreaded). -pref("mailnews.default_view_flags", 1); - -// --- Mail handling and compose --- - -// No confirmation before Shift+Delete removes a message for good. -pref("mail.warn_on_shift_delete", false); // keeps Betterbird's default: mail.spam.manualMark // Marking a message as junk by hand does nothing more than mark it. // keeps Betterbird's default: mail.compose.show_attachment_pane // The composer shows its attachment pane only when a message has attachments. - -// --- Notifications --- - -// No pop-up for new mail: the operator reads mail when they choose to. -pref("mail.biff.show_alert", false); -// No sound for new mail. -pref("mail.biff.play_sound", false); -// No tray icon for new mail. -pref("mail.biff.show_tray_icon", false); -// If a node's layer turns alerts back on, they go through the desktop's notification daemon. -pref("mail.biff.use_system_alert", true); - -// --- Calendar --- - -// The calendar's working day starts at 09:00 (Betterbird's default is 08:00). -pref("calendar.view.daystarthour", 9); -// and ends at 20:00 (Betterbird's default is 17:00). -pref("calendar.view.dayendhour", 20); -// A calendar item is deleted without asking. -pref("calendar.item.promptDelete", false); -// The calendar follows the node's time zone, because the laptop travels. -pref("calendar.timezone.useSystemTimezone", true); -// Reminders show in their window (Betterbird's default, made explicit: the reminder is why this module exists). -pref("calendar.alarms.show", true); -// Reminders missed while the laptop slept show at the next start (Betterbird's default is false). -pref("calendar.alarms.showmissed", true); - -// --- Chat --- - -// No chat button: the operator uses no chat account in Betterbird (Betterbird's default is true). -pref("mail.chat.enabled", false); - -// --- Privacy --- - -// No remote content in a message unless its sender is allowed (Betterbird's default, made explicit). -pref("mailnews.message_display.disable_remote_image", true); -// Ask sites not to track. A default only: the policy cannot lock a privacy. preference. -pref("privacy.donottrackheader.enabled", true); diff --git a/modules/betterbird/module.json b/modules/betterbird/module.json index 3681ea52..45383ab4 100644 --- a/modules/betterbird/module.json +++ b/modules/betterbird/module.json @@ -9,15 +9,11 @@ "x11-display" ], "invokes": [ - "seat:mesh-controller.settings" + "seat:mesh-controller.settings", + "seat:mesh-controller.nodes", + "seat:mesh-controller.node", + "betterbird.betterbird_derive" ], - "settings": { - "prefs": { - "kind": "preference", - "default": "// none", - "why": "Betterbird runs with its own defaults until a machine's assignment names a preference: one line of user_pref(\"name\", value); statements, which the module's user.js holds and Betterbird reads at its start (novox/hq issue 345). An account's preferences (mail.account., mail.identity., mail.server., mail.smtpserver. and the account lists) are refused here: an account is declared in the setting accounts, so it has one owner (novox/hq ADR 0316 §11)" - } - }, "tools": [ "betterbird_status", "betterbird_prefs", @@ -28,6 +24,13 @@ "betterbird_restart", "betterbird_check", "betterbird_accounts", + "betterbird_identities", + "betterbird_calendars", + "betterbird_address_books", + "betterbird_filters", + "betterbird_tags", + "betterbird_derive", + "betterbird_drift", "betterbird_reminder_test" ], "resources": [ @@ -39,12 +42,16 @@ "mode": "0755" }, { - "id": "user-js", + "id": "prefs", "type": "file", - "path": "${machine:account-home}/.config/betterbird/user.js", + "path": "${machine:account-home}/.config/betterbird/prefs.json", "owner": "${machine:account}", "mode": "0644", - "content": "// Betterbird's user.js (module betterbird, novox/hq issue 345). Owned by the mesh: the node-engine\n// writes it here at every push, and the module's tools copy it whole into Betterbird's profile\n// (betterbird_user_js with apply, betterbird_restart), where Betterbird reads it at its start and lets it\n// win over the preferences Betterbird saves itself. Change the module's setting prefs, never this file or the profile's copy.\n//\n// Only comments and user_pref(\"name\", value); statements; the tools refuse anything else, and any\n// preference whose name can hold a credential.\n${setting:prefs}\n", + "merge": "json", + "takes-settings": [ + "prefs" + ], + "content": "{\n \"prefs\": {}\n}\n", "trusted": true }, { @@ -60,6 +67,35 @@ "content": "{\n \"accounts\": {}\n}\n", "trusted": true }, + { + "id": "configuration", + "type": "file", + "path": "${machine:account-home}/.config/betterbird/configuration.json", + "owner": "${machine:account}", + "mode": "0600", + "merge": "json", + "takes-settings": [ + "identities", + "calendars", + "address-books", + "filters" + ], + "content": "{\n \"identities\": {},\n \"calendars\": {},\n \"address-books\": {},\n \"filters\": {}\n}\n", + "trusted": true + }, + { + "id": "tags", + "type": "file", + "path": "${machine:account-home}/.config/betterbird/tags.json", + "owner": "${machine:account}", + "mode": "0644", + "merge": "json", + "takes-settings": [ + "tags" + ], + "content": "{\n \"tags\": {}\n}\n", + "trusted": false + }, { "id": "community-package", "type": "package", @@ -101,7 +137,7 @@ "type": "file", "path": "/usr/local/bin/betterbird", "mode": "0755", - "content": "#!/bin/sh\n# Betterbird's start, written by the mesh (module betterbird, novox/hq ADR 0316). Replaced at every push.\n#\n# Before Betterbird starts, the module's tools place its user.js in the profile (the setting prefs, and the\n# accounts the setting accounts declares) and add the declared accounts to the profile's account list. They do\n# it only while no Betterbird of this account runs, because Betterbird rewrites its saved preferences when it quits; a\n# second start while one runs only hands over to it. What the preparation could not do it says here, and\n# betterbird_check says it too. Betterbird starts either way: its configuration never holds the mail back.\n/usr/local/lib/mesh-betterbird/betterbird-tools prepare || echo \"betterbird: the mesh's preparation did not complete; betterbird_check says why\" >&2\nexec /opt/betterbird/betterbird \"$@\"\n" + "content": "#!/bin/sh\n# Betterbird's start, written by the mesh (module betterbird, novox/hq ADR 0316). Replaced at every push.\n#\n# Before Betterbird starts, the module's tools place its user.js in the profile (the settings prefs, accounts, identities, calendars,\n# address-books and tags) and write what Betterbird rewrites itself: the account lists and each declared account's filters. They do\n# it only while no Betterbird of this account runs, because Betterbird rewrites its saved preferences when it quits; a\n# second start while one runs only hands over to it. What the preparation could not do it says here, and\n# betterbird_check says it too. Betterbird starts either way: its configuration never holds the mail back.\n/usr/local/lib/mesh-betterbird/betterbird-tools prepare || echo \"betterbird: the mesh's preparation did not complete; betterbird_check says why\" >&2\nexec /opt/betterbird/betterbird \"$@\"\n" }, { "id": "launcher",