Compare commits

..
Author SHA1 Message Date
jschoubben 0cf7b2344f Write a moved contributor's declared defaults into the provisioner client tests' stand-in mesh-config (issue 1780) 2026-10-21 21:48:59 +02:00
mesh-tools 3793175894 Merge pull request 'Make minio's link-key secrets 40 characters and keep every mc argument's secret out of its errors (issue 1766)' (#422) from fix/minio-link-key-length into main 2026-10-21 19:00:13 +00:00
jschoubben b70f246622 Merge remote-tracking branch 'origin/main' into fix/minio-link-key-length
mesh/merge-gate head b70f24662298 merged into trunk cdbdb4747474 — pass: builds minio, sent nowhere; no bus step; 1 wait(s) for a person; every machine com…
mesh/repo-check head b70f24662298 merged into trunk cdbdb4747474 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-…
mesh/delivery delivered
2026-10-21 20:52:18 +02:00
mesh-tools cdbdb47474 Merge pull request 'xterm: every resource a config value, today's values as defaults (issue 1656)' (#416) from fix/1656-xterm-config into main 2026-10-21 18:36:28 +00:00
jschoubben 689e61081d Merge remote-tracking branch 'origin/main' into fix/minio-link-key-length
mesh/merge-gate head 689e61081df9 merged into trunk cdbdb4747474 — pass: builds minio, sent nowhere; no bus step; 1 wait(s) for a person; every machine com…
mesh/repo-check head 689e61081df9 merged into trunk cdbdb4747474 — fail: its merge-check.sh failed: mesh-controller: 1 problem(s) in 139 manifest(s): modul…
mesh/delivery superseded: a newer head of the same pull request
2026-10-21 20:22:24 +02:00
jschoubben c83f48b821 Make link-key secrets 40 characters and keep every mc argument's secret out of its errors (issue 1766) 2026-10-21 20:21:56 +02:00
mesh-tools db5e6c5b48 Merge pull request 'Name the wait in the Stop hook's suggested calls, route adopt's counter reset through fsys, and stop writing empty counter files' (#421) from fix/1749-1757-wait-list into main 2026-10-21 18:19:19 +00:00
jschoubben b9dcf19c3f Merge remote-tracking branch 'origin/main' into fix/1656-xterm-config
mesh/repo-check head b9dcf19c3f91 merged into trunk db5e6c5b48f4 beside mesh-controller#420@9d260412 — pass: its merge-check.sh passed, each part in its to…
mesh/merge-gate head b9dcf19c3f91 merged into trunk db5e6c5b48f4 — pass: group fix/1656-xterm-config composed: every machine composes with the change as it…
mesh/delivery failed: its walk failed, carrying its own commit: a gate on a first machine (what it carried put back), a build, a machine
mesh/delivery-group group fix/1656-xterm-config failed: a member failed
2026-10-21 19:43:29 +02:00
jochen e3a714d5f0 Stop writing an empty refusal-counter file on every stop of every session (issue 1749)
mesh/merge-gate head e3a714d5f0f0 merged into trunk c3564c1accf3 — pass: builds claude-code → ace, g14, novox, shanks; no bus step; every machine composes …
mesh/repo-check head e3a714d5f0f0 merged into trunk c3564c1accf3 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-…
mesh/delivery delivered
writeRefusalCounters wrote "{}" and, since it now routes through fsys
(this branch's earlier commit), created the waits folder on every single
stop, even for a session that never had a wait: one counter file per
session id, piling up on a node that uses the wait list at all. It now
removes the file instead of writing empty counters, a no-op when there
was never one to begin with, so a stop with nothing to record leaves
neither the file nor the folder; a refusal still writes it, and the
stop right after the wait is done (or let through) removes it again.
2026-10-21 19:41:05 +02:00
jochen 08cec8cb1c Route adopt's refusal-counter reset through fsys, and name the wait in the Stop hook's suggestions (issues 1749, 1757)
issue 1749: AdoptName reset both session ids' refusal-counter files with
os.WriteFile, bypassing fsys; on a node whose agent has its own account
(ADR 0266) the module runs as the operator's account and cannot write
into the agent's home directly, so the reset silently failed. writeRefusalCounters
now goes through fsys (MkdirAll then WriteFile), as every other write into
the agent's home does; it is also called from the Stop hook itself, which
already runs as the agent's own account, where fsys is its zero value and
behaves exactly as the plain write did before.

issue 1757: the Stop hook's suggested watched and said calls left out `by`,
which claude_code_wait requires on every op, so a session following the
suggestion word for word was refused. Both now carry "by" from the wait's
own By field (already kept per wait since issue 1629). The SessionStart
hook's suggested adopt call already carried by; added a test for it too,
alongside the Stop hook's two, each extracting the exact JSON the hook
prints and feeding it back to the tool's own dispatch to prove it is
accepted as printed.
2026-10-21 19:35:33 +02:00
jschoubben e4cbffbaa9 Merge remote-tracking branch 'origin/main' into fix/1656-xterm-config
mesh/merge-gate out of date: judged against trunk 7a7a4034852a, the trunk is now c3564c1accf3; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk 7a7a4034852a, the trunk is now c3564c1accf3; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
2026-10-21 19:16:12 +02:00
jschoubben f178aab641 Make every xterm resource a config value with today's value as default; tools read the rendered resources (issue 1656)
mesh/delivery-group group fix/1656-xterm-config rejected: a member's own check failed
mesh/merge-gate waits for its group's composed check
mesh/repo-check head f178aab641dc merged into trunk 3259765d46bc beside mesh-controller#420@69a67876 — fail: its merge-check.sh failed: files asking for a …
mesh/delivery superseded: a newer head of the same pull request
2026-10-21 12:51:40 +02:00
40 changed files with 1441 additions and 952 deletions
+1 -1
View File
@@ -1,4 +1,4 @@
58
59
The long-running resources of this catalogue that do not say how they are ready (`health`, novox/hq ADR 0240
rule 8), as `module check` counts them. It may only go down: merge-check.sh fails a change that raises it, and
one that lowers it writes the new number on the first line. From 2026-11-18, or once it is 0, a long-running
@@ -0,0 +1,139 @@
package main
// Every call a hook prints for a session to copy must be accepted by claude_code_wait's own argument check
// (issue 1757): the Stop hook's suggested watched and said calls, and the SessionStart hook's suggested
// adopt call.
import (
"encoding/json"
"strings"
"testing"
"time"
)
// suggestedCall extracts the one JSON object embedded in a hook's message, right after "claude_code_wait "
// — what a session is expected to copy verbatim — and parses it into the arguments a tool call takes.
func suggestedCall(t *testing.T, msg string) map[string]any {
t.Helper()
i := strings.Index(msg, "claude_code_wait ")
if i < 0 {
t.Fatalf("no suggested call in: %s", msg)
}
rest := msg[i+len("claude_code_wait "):]
start := strings.Index(rest, "{")
if start < 0 {
t.Fatalf("no JSON object in: %s", msg)
}
depth, end := 0, -1
for j, c := range rest[start:] {
switch c {
case '{':
depth++
case '}':
depth--
if depth == 0 {
end = start + j + 1
}
}
if end >= 0 {
break
}
}
if end < 0 {
t.Fatalf("unbalanced JSON object in: %s", msg)
}
var args map[string]any
if err := json.Unmarshal([]byte(rest[start:end]), &args); err != nil {
t.Fatalf("the suggested call is not valid JSON: %v (%s)", err, rest[start:end])
}
return args
}
// Issue 1757: the Stop hook's suggested watched call carries `by`, and the tool accepts it as printed.
func TestTheStopHooksSuggestedWatchedCallIsAcceptedByTheTool(t *testing.T) {
p, _ := node(t, "g14")
store := newMemWaits()
waiting = &Waiting{Store: store, Now: func() time.Time { return t0 }}
t.Cleanup(func() { waiting = nil })
s := serve(t, p, &bus{kept: map[string]map[string]any{}}, memConfig{})
text, isErr := s.call(t, "claude_code_wait", map[string]any{"op": "add", "session": "verify-0489-test", "by": "fix-1757@g14",
"what": "the operator's Approve", "read": map[string]any{"address": "mesh-delivery.recheck"}, "bound_minutes": 10})
if isErr {
t.Fatalf("add: %s", text)
}
f, err := readWaitsFile(waitsFilePath(p, "verify-0489-test"))
if err != nil || len(f.Waits) != 1 {
t.Fatalf("%+v %v", f, err)
}
_, reason := waitBlocksStop("verify-0489-test", f.Waits[0], t0.Add(time.Minute))
if !strings.Contains(reason, "\"by\":\"fix-1757\"") {
t.Fatalf("the suggested call does not carry by: %s", reason)
}
args := suggestedCall(t, reason)
text, isErr = s.call(t, "claude_code_wait", args)
if isErr {
t.Fatalf("the suggested watched call, as printed, was refused: %s", text)
}
}
// Issue 1757: the Stop hook's suggested said call (past its bound) carries `by` too.
func TestTheStopHooksSuggestedSaidCallIsAcceptedByTheTool(t *testing.T) {
p, _ := node(t, "g14")
store := newMemWaits()
clock := t0
waiting = &Waiting{Store: store, Now: func() time.Time { return clock }}
t.Cleanup(func() { waiting = nil })
s := serve(t, p, &bus{kept: map[string]map[string]any{}}, memConfig{})
text, isErr := s.call(t, "claude_code_wait", map[string]any{"op": "add", "session": "verify-0489-test", "by": "fix-1757@g14",
"what": "the operator's Approve", "read": map[string]any{"address": "mesh-delivery.recheck"}, "bound_minutes": 1})
if isErr {
t.Fatalf("add: %s", text)
}
clock = t0.Add(2 * time.Minute) // past its 1-minute bound, for the live tool call too
f, err := readWaitsFile(waitsFilePath(p, "verify-0489-test"))
if err != nil || len(f.Waits) != 1 {
t.Fatalf("%+v %v", f, err)
}
_, reason := waitBlocksStop("verify-0489-test", f.Waits[0], clock)
if !strings.Contains(reason, "\"by\":\"fix-1757\"") {
t.Fatalf("the suggested call does not carry by: %s", reason)
}
args := suggestedCall(t, reason)
text, isErr = s.call(t, "claude_code_wait", args)
if isErr {
t.Fatalf("the suggested said call, as printed, was refused: %s", text)
}
}
// Issue 1757: the SessionStart hook's suggested adopt call already carries `by` — checked, as asked, so a
// regression here is caught the same way as the Stop hook's.
func TestTheSessionStartHooksSuggestedAdoptCallIsAcceptedByTheTool(t *testing.T) {
p, _ := node(t, "g14")
store := newMemWaits()
waiting = &Waiting{Store: store, Now: func() time.Time { return t0 }}
t.Cleanup(func() { waiting = nil })
s := serve(t, p, &bus{kept: map[string]map[string]any{}}, memConfig{})
text, isErr := s.call(t, "claude_code_wait", map[string]any{"op": "add", "session": "sess-old", "by": "fix-1757@g14",
"what": "the operator's Approve", "read": map[string]any{"address": "mesh-delivery.recheck"}, "bound_minutes": 10})
if isErr {
t.Fatalf("add: %s", text)
}
msg := SessionStartMessage("sess-new", WaitsDirPath(p), t0.Add(time.Minute))
if !strings.Contains(msg, "\"by\":\"fix-1757@g14\"") {
t.Fatalf("the suggested adopt call does not carry by: %s", msg)
}
args := suggestedCall(t, msg)
text, isErr = s.call(t, "claude_code_wait", args)
if isErr {
t.Fatalf("the suggested adopt call, as printed, was refused: %s", text)
}
}
+26 -7
View File
@@ -52,12 +52,12 @@ func waitBlocksStop(session string, w Wait, now time.Time) (bool, string) {
}
return true, fmt.Sprintf("wait %s (%s) is open and nothing follows it yet. Read where it stands with %s, then "+
"%s, and mark it watched: claude_code_wait "+
"{\"op\":\"watched\",\"session\":%q,\"id\":%q,\"watcher\":\"...\"}.", w.ID, w.What, w.Read.Address, watcherAdvice(w),
session, w.ID)
"{\"op\":\"watched\",\"session\":%q,\"by\":%q,\"id\":%q,\"watcher\":\"...\"}.", w.ID, w.What, w.Read.Address,
watcherAdvice(w), session, w.By, w.ID)
}
return true, fmt.Sprintf("wait %s (%s) is past its bound and the operator has not been told. Tell the operator what "+
"waits, since when, and what you are doing about it, then mark it: claude_code_wait "+
"{\"op\":\"said\",\"session\":%q,\"id\":%q}.", w.ID, w.What, session, w.ID)
"{\"op\":\"said\",\"session\":%q,\"by\":%q,\"id\":%q}.", w.ID, w.What, session, w.By, w.ID)
}
// watcherAdvice is what a session is told to do to follow a wait (ADR 0489 §3, review round 1 item 3): a
@@ -113,15 +113,34 @@ func readRefusalCounters(waitsDir, session string) map[string]int {
return m
}
// writeRefusalCounters writes the counter file beside the waits file. Its caller decides what an error here
// means: harmless when the stop is already allowed, but when the stop was about to be refused, a cap that
// cannot be written is a cap that cannot take effect (ADR 0489 §5 "never loops", review round 1 item 2).
// writeRefusalCounters keeps the counter file beside the waits file in step with m, through fsys like every
// other write into the agent's home (novox/hq ADR 0266, issue 1749): called from the Stop hook, which
// already runs as the agent's own account (fsys is then its zero value, a plain write or remove, as
// before), and from the tool's AdoptName, which runs as the operator's account and must reach a confined
// node's agent home through the sudo helper fsys carries such a write through. Its caller decides what an
// error here means: harmless when the stop is already allowed, but when the stop was about to be refused, a
// cap that cannot be written is a cap that cannot take effect (ADR 0489 §5 "never loops", review round 1
// item 2).
//
// Empty, m is removed rather than written as "{}" (issue 1749, follow-up): a stop with nothing to record —
// every stop of every session with no waits, the ordinary case — must leave neither the file nor, the first
// time, the waits folder itself; removing a file that was never there is not an error.
func writeRefusalCounters(waitsDir, session string, m map[string]int) error {
path := refusalCountersPath(waitsDir, session)
if len(m) == 0 {
if err := fsys.Remove(path); err != nil && !errors.Is(err, fs.ErrNotExist) {
return err
}
return nil
}
raw, err := json.Marshal(m)
if err != nil {
return err
}
return os.WriteFile(refusalCountersPath(waitsDir, session), raw, 0o600)
if err := fsys.MkdirAll(waitsDir, 0o700); err != nil {
return err
}
return fsys.WriteFile(path, raw, 0o600)
}
// notice says a fail-open or let-through reason on both the log (stderr) and the screen: Claude Code shows a
@@ -170,13 +170,72 @@ func TestRunStopKeepsItsCounterBesideTheWaitsFile(t *testing.T) {
if !strings.Contains(stdout.String(), "systemMessage") || !strings.Contains(stdout.String(), "unwatched") {
t.Fatalf("the let-through notice did not reach stdout: %s", stdout.String())
}
raw, err := os.ReadFile(refusalCountersPath(waitsDir, "sess-6"))
// Once let through, the counters are empty again: the file is removed, not left holding "{}" (issue
// 1749, follow-up).
if _, err := os.Stat(refusalCountersPath(waitsDir, "sess-6")); !os.IsNotExist(err) {
t.Fatalf("the counter file was not removed once its counters emptied: %v", err)
}
}
// Issue 1749, follow-up: a stop for a session with no waits at all leaves nothing behind — not the counter
// file, not even the waits folder a node that never added a wait would otherwise gain on every stop of
// every session.
func TestStopWithNoWaitsLeavesNoFolderAndNoFile(t *testing.T) {
waitsDir := filepath.Join(t.TempDir(), "waits") // not created: this is what the hook is handed cold
var stdout, stderr bytes.Buffer
stdin := strings.NewReader(`{"session_id":"sess-none","stop_hook_active":false}`)
if code := runStop(waitsDir, stdin, &stdout, &stderr); code != 0 {
t.Fatalf("code %d, stderr %s", code, stderr.String())
}
if _, err := os.Stat(waitsDir); !os.IsNotExist(err) {
t.Fatalf("a stop with no waits created the waits folder: %v", err)
}
}
// Issue 1749, follow-up: a refusal writes the counter file (there is something to record).
func TestStopWithARefusalWritesTheCounterFile(t *testing.T) {
waitsDir := t.TempDir()
f := WaitsFile{Session: "sess-7", At: stopT0, Waits: []Wait{openWait("w-7", 10)}}
writeWaitsFixture(t, filepath.Join(waitsDir, "sess-7.json"), f)
var stdout, stderr bytes.Buffer
stdin := strings.NewReader(`{"session_id":"sess-7","stop_hook_active":false}`)
if code := runStop(waitsDir, stdin, &stdout, &stderr); code != 2 {
t.Fatalf("code %d, stderr %s", code, stderr.String())
}
raw, err := os.ReadFile(refusalCountersPath(waitsDir, "sess-7"))
if err != nil {
t.Fatal(err)
}
var counters map[string]int
if json.Unmarshal(raw, &counters) != nil || len(counters) != 0 {
t.Fatalf("the counter file was not reset: %s", raw)
if json.Unmarshal(raw, &counters) != nil || counters["w-7"] != 1 {
t.Fatalf("the counter file: %s", raw)
}
}
// Issue 1749, follow-up: once the wait is done (no longer in the waits file), the next stop removes the
// counter file rather than leaving a stale "{}" behind.
func TestStopAfterTheWaitIsDoneRemovesTheCounterFile(t *testing.T) {
waitsDir := t.TempDir()
path := filepath.Join(waitsDir, "sess-8.json")
writeWaitsFixture(t, path, WaitsFile{Session: "sess-8", At: stopT0, Waits: []Wait{openWait("w-8", 10)}})
stdin := strings.NewReader(`{"session_id":"sess-8","stop_hook_active":false}`)
var stdout, stderr bytes.Buffer
if code := runStop(waitsDir, stdin, &stdout, &stderr); code != 2 {
t.Fatalf("refusal: code %d, stderr %s", code, stderr.String())
}
if _, err := os.Stat(refusalCountersPath(waitsDir, "sess-8")); err != nil {
t.Fatalf("the counter file was not written: %v", err)
}
// The wait is done: it is no longer in the waits file (DoneWait removes it, waits.go).
writeWaitsFixture(t, path, WaitsFile{Session: "sess-8", At: stopT0, Waits: []Wait{}})
var stdout2, stderr2 bytes.Buffer
stdin2 := strings.NewReader(`{"session_id":"sess-8","stop_hook_active":false}`)
if code := runStop(waitsDir, stdin2, &stdout2, &stderr2); code != 0 {
t.Fatalf("after done: code %d, stderr %s", code, stderr2.String())
}
if _, err := os.Stat(refusalCountersPath(waitsDir, "sess-8")); !os.IsNotExist(err) {
t.Fatalf("the counter file was not removed after the wait was done: %v", err)
}
}
@@ -273,3 +332,31 @@ func TestHookRegisterStillRefusesEveryHook(t *testing.T) {
t.Fatalf("a Stop hook was not refused: %v %s", isError, text)
}
}
// Issue 1749: adopt's refusal-counter reset writes into the agent's home (beside the waits file) and must
// go through fsys like every other such write (novox/hq ADR 0266) — on a confined node the module runs as
// the operator's account, which cannot write there directly; only the helper, as the agent's own account,
// can. The confined-node fake (agentFiles, agent_account_test.go) proves it: a write that reaches the real
// filesystem only through the helper's request/response protocol bumps its count.
func TestWriteRefusalCountersGoesThroughTheHelper(t *testing.T) {
f, home, counted := agentFiles(t)
was := fsys
fsys = f
t.Cleanup(func() { fsys = was })
waitsDir := filepath.Join(home, ".claude", "nox-mesh", "waits")
if err := writeRefusalCounters(waitsDir, "sess-1", map[string]int{"w-1": 2}); err != nil {
t.Fatal(err)
}
if counted.asked.Load() == 0 {
t.Fatal("the counter file was written directly, never through the helper")
}
raw, err := os.ReadFile(filepath.Join(waitsDir, "sess-1.refusals.json"))
if err != nil {
t.Fatal(err)
}
var m map[string]int
if json.Unmarshal(raw, &m) != nil || m["w-1"] != 2 {
t.Fatalf("the written counters: %s", raw)
}
}
+1 -13
View File
@@ -16,18 +16,6 @@ installation's are settings on the assignment, `settings set mailu <file>`:
header from — the address the proxy forwards with. The front's own hostname is the name of its
`web` route, told to it by the mesh.
## Mail client autoconfiguration
`mailu-automx` answers what mail clients ask for their settings (Thunderbird, Outlook, Apple) for the domains in
`AMX_MAIL_DOMAINS`, naming the IMAP and SMTP servers and ports of mailu's environment (`AMX_IMAP_ADDRESS`,
`AMX_IMAP_PORT`, `AMX_SMTP_ADDRESS`, `AMX_SMTP_PORT`). It is `cmd/automx`, a Go program of the standard library
only in this module's `go.mod` (ADR 0480 clauses 1, 2, 9). It runs in the mirrored alpine image the route proxy's
`trust` step runs, from the `program-automx` archive mounted read-only at `/opt/mesh`; the container is restarted
with the archive and with `mailu.env`, which the program reads at each start. It keeps no database. The file the
Python automx kept in `data-automx` is left where it is on the machine, not deleted by the mesh (ADR 0480
clause 10). The answers it gave before are the fixtures in `cmd/automx/testdata/captured`, with how they were
taken (`CAPTURED`) and what the program answers differently (`DIFFERENCES`).
## Files it does not know
Each directory or archive this module declares says how it treats a file it did not write (ADR 0422, ADR 0416).
@@ -35,7 +23,7 @@ Each directory or archive this module declares says how it treats a file it did
- `${dir:mesh-state}`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
- `${dir:state}`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
- `${dir:grants}`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
- the archive `program-automx` (`${dir:state}/automx`): everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
- `${dir:data-automx}`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
- `${dir:data-certs}`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
- `${dir:data-data}`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
- `${dir:data-dkim}`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
+87
View File
@@ -0,0 +1,87 @@
// What the autoconfiguration module (automx2) holds, as stored: its provider, server, domain and server_domain
// rows, read from its SQLite file inside its container with the sqlite3 program the image carries, opened
// read-only. Nothing is written and nothing is interpreted: the rows are the evidence for what automx answers
// to mail clients (novox/hq issue 1630, ADR 0480), and the reason the answer may be empty.
import type { Exec } from "./health.js";
export const AUTOMX_DATABASE = "/data/db.sqlite";
/** The tables read, each with its columns and its ordering. Fixed text: the tool takes no query from its caller. */
export const AUTOMX_TABLES: Record<string, { columns: string[]; order: string }> = {
provider: { columns: ["id", "name", "short_name"], order: "id" },
server: { columns: ["id", "prio", "name", "port", "type", "socket_type", "user_name", "authentication"], order: "id" },
domain: { columns: ["id", "name", "provider_id", "ldapserver_id"], order: "id" },
server_domain: { columns: ["server_id", "domain_id"], order: "domain_id, server_id" },
};
/**
* The one statement that reads a table. The sqlite3 in the mailu-automx image does not know `-json` (it answers
* "sqlite3: Error: unknown option: -json", issue 1630), so each column is printed as `<type>:<hex of its bytes>`, which
* the oldest sqlite3 prints in its default list mode: no quoting, no separator or newline inside a cell, and a NULL
* (`null:`) apart from an empty text (`text:`).
*/
export function selectOf(table: string): string {
const t = AUTOMX_TABLES[table];
return `SELECT ${t.columns.map((c) => `typeof(${c}) || ':' || hex(${c})`).join(", ")} FROM ${table} ORDER BY ${t.order}`;
}
export type Row = Record<string, string | number | null>;
export type AutomxRows = {
database: string;
tables: Record<string, Row[]>;
/** servers whose name is empty: the client is told an empty host name for them */
empty_names: { id: number; type: string }[];
/** tables that could not be read, each with what sqlite3 or docker said on one line */
unreadable: Record<string, string>;
};
/** One cell as selectOf prints it: `<type>:<hex>`; text is the bytes read as UTF-8, a number is read from its text. */
function cellOf(cell: string): string | number | null {
const at = cell.indexOf(":");
if (at < 0 || cell.length - at - 1 > 0 && !/^([0-9A-Fa-f]{2})+$/.test(cell.slice(at + 1))) throw new Error(`sqlite3 printed ${JSON.stringify(cell.slice(0, 40))}, not a cell`);
const type = cell.slice(0, at);
const bytes = Buffer.from(cell.slice(at + 1), "hex").toString("utf8");
switch (type) {
case "null": return null;
case "integer": case "real": return Number(bytes);
default: return bytes;
}
}
/** sqlite3 prints one line per row, cells apart by `|`, and nothing at all for a table without rows. */
export function parseRows(table: string, output: string): Row[] {
const columns = AUTOMX_TABLES[table].columns;
const rows: Row[] = [];
for (const line of output.split("\n")) {
if (line.trim() === "") continue;
const cells = line.replace(/\r$/, "").split("|");
if (cells.length !== columns.length) throw new Error(`sqlite3 printed ${cells.length} cells for ${columns.length} columns`);
rows.push(Object.fromEntries(columns.map((c, i) => [c, cellOf(cells[i])])));
}
return rows;
}
/** The command that reads one table: read-only, one fixed statement. */
export function readCommand(table: string): string[] {
return ["sqlite3", "-readonly", AUTOMX_DATABASE, selectOf(table)];
}
/** Read every table; one that fails is said in `unreadable` and does not hide the others. */
export async function readAutomx(exec: Exec, container: string): Promise<AutomxRows> {
const answer: AutomxRows = { database: AUTOMX_DATABASE, tables: {}, empty_names: [], unreadable: {} };
for (const table of Object.keys(AUTOMX_TABLES)) {
try {
answer.tables[table] = parseRows(table, await exec(container, readCommand(table)));
} catch (e) {
const err = e as { stderr?: string; message?: string };
const text = (err?.stderr ?? "").trim() || (err?.message ?? String(e)).trim();
answer.unreadable[table] = text.split("\n").map((l) => l.trim()).filter(Boolean)[0] ?? "it failed and said nothing";
}
}
for (const s of answer.tables.server ?? []) {
if (String(s.name ?? "").trim() === "") answer.empty_names.push({ id: Number(s.id), type: String(s.type) });
}
return answer;
}
+44
View File
@@ -0,0 +1,44 @@
# automx2 — the autoconfig/autodiscover responder, carried by the mailu module as its own
# artifact: it is a config-baked sidecar of this mail server, not a standalone application
# (novox/hq ADR 0015 draws that line at applications).
#
# The base is named rather than pinned (novox/hq issue 044): declared in module.json's
# `build.on`. The build context is the module's own directory; every ADD says so.
ARG PYTHON_BASE
FROM ${PYTHON_BASE}
RUN apk add --no-cache bash sqlite
WORKDIR /automx2
ADD automx/files/setupvenv.sh /automx2/setupvenv.sh
ADD automx/files/start /automx2/start
ADD automx/files/setup /automx2/setup
ADD automx/files/setup-db /automx2/setup-db
ADD automx/files/add-domains /automx2/add-domains
RUN chmod u+x setupvenv.sh start add-domains setup setup-db
RUN ./setupvenv.sh \
&& . .venv/bin/activate \
&& pip install automx2==2021.6
# The launcher `start` expects. In the predecessor's image this wrapper appeared during a build
# step that never made it into the files this module carries — the image worked and the recipe
# could not reproduce it. Written here explicitly, verbatim from the proven image, so the build
# is the whole truth about the image again.
RUN mkdir -p .venv/scripts && printf '%s\n' \
'#!/usr/bin/env bash' \
'set -euo pipefail' \
'. .venv/bin/activate' \
"export FLASK_ENV='production'" \
"export FLASK_APP='automx2.server:app'" \
'flask "$@"' > .venv/scripts/flask.sh && chmod +x .venv/scripts/flask.sh
ENV AUTOMX2_CONF=/etc/automx2.conf
ADD automx/files/automx2.conf /etc/automx2.conf
# VOLUME deliberately absent: the anonymous /data volume is exactly what lost db.sqlite on
# every recreate (measured on novox 2026-08-10). The manifest binds a real directory instead.
ENTRYPOINT ["/bin/sh"]
CMD ["./start"]
EXPOSE 4243
+49
View File
@@ -0,0 +1,49 @@
#!/usr/bin/env bash
set -e
echo "${MAIL_DOMAINS}"
# Split domains into array
IFS=', ' read -r -a array <<< "${AMX_MAIL_DOMAINS}"
# User configurable section -- START
PROVIDER_ID=001
SQL_CMD="";
# Iterate domains resulting from split on second arg
for element in "${array[@]}"
do
# Set vars
DOMAIN=$element
PROVIDER_NAME=$DOMAIN
PROVIDER_SHORTNAME=$DOMAIN
# Optional LDAP server
#LDAP_SERVER="ldap.${DOMAIN}"
# User configurable section -- END
s1_id=$((PROVIDER_ID + 1))
s2_id=$((PROVIDER_ID + 2))
s3_id=$((PROVIDER_ID + 3))
dom_id=$((PROVIDER_ID + 4))
s3_id='NULL'
SQL_CMD=$(cat <<EOT
$SQL_CMD
INSERT INTO provider(id, name, short_name) VALUES(${PROVIDER_ID}, '${PROVIDER_NAME}', '${PROVIDER_SHORTNAME}');
INSERT INTO server(id, port, type, name, socket_type, user_name, authentication)
VALUES(${s1_id}, ${AMX_IMAP_PORT}, 'imap', '${AMX_IMAP_SERVER}', 'STARTTLS', '%EMAILADDRESS%', 'password-cleartext');
INSERT INTO server(id, port, type, name, socket_type, user_name, authentication)
VALUES(${s2_id}, ${AMX_SMTP_PORT}, 'smtp', '${AMX_SMTP_ADDRESS}', 'STARTTLS', '%EMAILADDRESS%', 'password-cleartext');
INSERT INTO domain(id, name, provider_id, ldapserver_id) VALUES(${dom_id}, '${DOMAIN}', ${PROVIDER_ID}, ${s3_id});
INSERT INTO server_domain(server_id, domain_id) VALUES(${s1_id}, ${dom_id});
INSERT INTO server_domain(server_id, domain_id) VALUES(${s2_id}, ${dom_id});
EOT
)
PROVIDER_ID=$((PROVIDER_ID+10))
done
echo -e ${SQL_CMD}
echo -e ${SQL_CMD} | sqlite3 /data/db.sqlite
+21
View File
@@ -0,0 +1,21 @@
[automx2]
# A typical production setup would use loglevel = WARNING
loglevel = WARNING
# Echo SQL commands into log? Used for debugging.
db_echo = false
# In-memory SQLite database
# db_uri = sqlite:///:memory:
# SQLite database in a UNIX-like file system
db_uri = sqlite:////data/db.sqlite
# MySQL database on a remote server. This example does not use an encrypted
# connection and is therefore *not* recommended for production use.
#db_uri = mysql://username:password@server.example.com/db
# Number of proxy servers between automx2 and the client (default: 0).
# If your logs only show 127.0.0.1 or ::1 as the source IP for incoming
# connections, proxy_count probably needs to be changed.
proxy_count = 1
+12
View File
@@ -0,0 +1,12 @@
#!/usr/bin/env bash
set -e
if [ ! -e /data/db.sqlite ]; then
# DB SETUP
echo "SETTING UP DB"
./setup-db
echo "ADDING DOMAINS"
# Add the domains
./add-domains
fi
+84
View File
@@ -0,0 +1,84 @@
#!/usr/bin/env bash
set -e
# LDAP-Server
LDAP=$(cat <<EOT
CREATE TABLE ldapserver(
id INT PRIMARY KEY NOT NULL,
name TEXT NOT NULL,
port INT NOT NULL,
use_ssl INT NOT NULL,
search_base TEXT NOT NULL,
search_filter TEXT NOT NULL,
attr_uid TEXT NOT NULL,
attr_cn TEXT NOT NULL,
bind_password TEXT NOT NULL,
bind_user TEXT NOT NULL
);
EOT
)
# Provider
PROVIDER=$(cat <<EOT
CREATE TABLE provider(
id INT PRIMARY KEY NOT NULL,
name TEXT NOT NULL,
short_name TEXT NOT NULL
);
EOT
)
# Server
SERVER=$(cat <<EOT
CREATE TABLE server(
id INT PRIMARY KEY NOT NULL,
prio INT NOT NULL DEFAULT 10,
name TEXT NOT NULL,
port INT NOT NULL,
type TEXT NOT NULL,
socket_type TEXT NOT NULL,
user_name TEXT NOT NULL,
authentication TEXT NOT NULL
);
EOT
)
# Domain
DOMAIN=$(cat <<EOT
CREATE TABLE domain(
id INT PRIMARY KEY NOT NULL,
name TEXT NOT NULL,
provider_id INT NOT NULL,
ldapserver_id INT NULL,
FOREIGN KEY(ldapserver_id) REFERENCES ldapserver(id),
FOREIGN KEY(provider_id) REFERENCES provider(id)
);
CREATE UNIQUE INDEX domain_name ON domain(name);
EOT
)
# Server-Domain
SERVER_DOMAIN=$(cat <<EOT
CREATE TABLE server_domain(
server_id INT NOT NULL,
domain_id INT NOT NULL,
FOREIGN KEY(server_id) REFERENCES server(id),
FOREIGN KEY(domain_id) REFERENCES domain(id)
);
EOT
)
## TODO Foreign keys
SQL_CMD=$(cat <<EOT
$LDAP
$PROVIDER
$SERVER
$DOMAIN
$SERVER_DOMAIN
EOT
)
echo -e ${SQL_CMD}
echo -e ${SQL_CMD} | sqlite3 /data/db.sqlite
+38
View File
@@ -0,0 +1,38 @@
#!/usr/bin/env bash
# vim:ts=4:sw=4:noet
#
# Creates a Python 3 virtual environment. The target directory can be passed
# as a parameter. The default path is '.venv' in the current directory.
dir="${1:-.venv}"
echo "Setup dir $dir"
set -e
if [ -d "${dir}" ]; then
echo >&2 "Directory '${dir}' already exists, exiting."
exit 1
fi
python3 -m venv "${dir}"
source "${dir}/bin/activate"
set +e
pip install -U pip setuptools wheel || true
#set -e
## vim:tabstop=4:noexpandtab
##
## Creates a Python 3 virtual environment. The target directory can be passed
## as a parameter. The default path is 'venv' in the current directory.
#
#dir="${1:-venv}"
#
#set -e
#if [ -d "${dir}" ]; then
# echo "Directory '${dir}' already exists, exiting." >&2
# exit 1
#fi
#python3 -m venv "${dir}"
#. "${dir}/bin/activate"
#
#set +e
#pip install -U pip setuptools || true
+8
View File
@@ -0,0 +1,8 @@
#!/usr/bin/env bash
set -e
# Setup
./setup
# Start
./.venv/scripts/flask.sh run --host=0.0.0.0 --port=4243
-260
View File
@@ -1,260 +0,0 @@
package main
import (
"bytes"
"io"
"net/http"
"net/http/httptest"
"net/url"
"os"
"path/filepath"
"sort"
"strconv"
"strings"
"testing"
)
// The environment mailu gives automx on novox when the fixtures were captured (files/mailu.env): the mail domain,
// the route name for both servers, the ports.
var capturedEnvironment = map[string]string{
"AMX_MAIL_DOMAINS": "novox.be",
"AMX_IMAP_ADDRESS": "mail.novox.be",
"AMX_IMAP_PORT": "143",
"AMX_SMTP_ADDRESS": "mail.novox.be",
"AMX_SMTP_PORT": "587",
"AUTOMX_PORT": "4243",
}
func getenvOf(m map[string]string) func(string) string {
return func(k string) string { return m[k] }
}
func testServer(t *testing.T) *httptest.Server {
t.Helper()
cfg, err := configFrom(getenvOf(capturedEnvironment))
if err != nil {
t.Fatal(err)
}
s := httptest.NewServer(newHandler(cfg))
t.Cleanup(s.Close)
return s
}
const captured = "testdata/captured"
// A captured request: its first line is the method and the URL the client used, then the headers sent, then after
// a blank line the body sent (CAPTURED, how the files were written).
type request struct {
method, url string
header [][2]string
body string
}
func readRequest(t *testing.T, stem string) request {
t.Helper()
raw, err := os.ReadFile(stem + ".request")
if err != nil {
t.Fatal(err)
}
head, body, hasBody := strings.Cut(string(raw), "\n\n")
if !hasBody {
head = strings.TrimSuffix(head, "\n")
}
lines := strings.Split(head, "\n")
method, target, ok := strings.Cut(lines[0], " ")
if !ok {
t.Fatalf("%s: first line %q is not METHOD URL", stem, lines[0])
}
r := request{method: method, url: target, body: body}
for _, l := range lines[1:] {
k, v, ok := strings.Cut(l, ": ")
if !ok {
t.Fatalf("%s: header line %q", stem, l)
}
r.header = append(r.header, [2]string{k, v})
}
return r
}
func readFile(t *testing.T, name string) string {
t.Helper()
b, err := os.ReadFile(name)
if err != nil {
t.Fatal(err)
}
return string(b)
}
// A fixture asks for its body to differ only where DIFFERENCES says so: a stem with a .corrected.body is answered
// with that, and its Content-Length is the corrected body's.
func expectedBody(t *testing.T, stem string) (body string, corrected bool) {
if _, err := os.Stat(stem + ".corrected.body"); err == nil {
return readFile(t, stem+".corrected.body"), true
}
return readFile(t, stem+".body"), false
}
func TestEveryCapturedRequestIsAnsweredAsAutomxAnswered(t *testing.T) {
stems, _ := filepath.Glob(filepath.Join(captured, "*.request"))
if len(stems) < 17 {
t.Fatalf("%d captured requests found, want the 17 of issue 1630", len(stems))
}
srv := testServer(t)
client := &http.Client{CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }}
for _, f := range stems {
stem := strings.TrimSuffix(f, ".request")
t.Run(filepath.Base(stem), func(t *testing.T) {
rq := readRequest(t, stem)
u, err := url.Parse(rq.url)
if err != nil {
t.Fatal(err)
}
// The mesh's route reaches the program with the client's name and scheme; the program is told them in
// the forwarding headers (the Location of fixture 14 holds them).
req, err := http.NewRequest(rq.method, srv.URL+u.RequestURI(), strings.NewReader(rq.body))
if err != nil {
t.Fatal(err)
}
for _, h := range rq.header {
req.Header.Set(h[0], h[1])
}
req.Host = u.Host
req.Header.Set("X-Forwarded-Proto", u.Scheme)
resp, err := client.Do(req)
if err != nil {
t.Fatal(err)
}
defer resp.Body.Close()
got, _ := io.ReadAll(resp.Body)
if want := strings.TrimRight(readFile(t, stem+".status"), "\n"); resp.Status != want {
t.Errorf("status %q, captured %q", resp.Status, want)
}
wantBody, corrected := expectedBody(t, stem)
if !bytes.Equal(got, []byte(wantBody)) {
t.Errorf("body\n%q\nwant\n%q", got, wantBody)
}
want := map[string]string{}
for _, l := range strings.Split(strings.TrimRight(readFile(t, stem+".headers"), "\n"), "\n") {
if k, v, ok := strings.Cut(l, ": "); ok {
want[k] = v
}
}
if corrected {
want["Content-Length"] = strconv.Itoa(len(wantBody))
}
// Server names the Python program that answered (DIFFERENCES); Date changes on every request.
delete(want, "Server")
var extra []string
for k := range resp.Header {
if k == "Date" || k == "Server" {
continue
}
if _, ok := want[k]; !ok {
extra = append(extra, k)
}
}
sort.Strings(extra)
if len(extra) > 0 {
t.Errorf("headers the capture does not have: %v", extra)
}
for k, v := range want {
if g := resp.Header.Get(k); g != v {
t.Errorf("header %s: %q, captured %q", k, g, v)
}
}
})
}
}
// DIFFERENCES: the one change clause 9 of ADR 0480 allows to an answer is the IMAP server's name, written from
// AMX_IMAP_ADDRESS where automx wrote an empty one.
func TestTheOnlyCorrectedAnswerIsTheImapServersName(t *testing.T) {
corrected, _ := filepath.Glob(filepath.Join(captured, "*.corrected.body"))
if len(corrected) != 3 {
t.Fatalf("%d corrected answers, want the three Thunderbird answers (01, 02, 03): %v", len(corrected), corrected)
}
for _, c := range corrected {
orig := readFile(t, strings.TrimSuffix(c, ".corrected.body")+".body")
want := strings.Replace(orig, "<hostname />", "<hostname>mail.novox.be</hostname>", 1)
if strings.Contains(want, "<hostname />") || want == orig {
t.Fatalf("%s: the capture has no single empty hostname to correct", c)
}
if got := readFile(t, c); got != want {
t.Errorf("%s differs from the capture by more than the IMAP server's name", c)
}
}
}
func TestTheImapServerIsNamedFromTheImapAddress(t *testing.T) {
env := map[string]string{}
for k, v := range capturedEnvironment {
env[k] = v
}
env["AMX_IMAP_ADDRESS"] = "imap.example.test"
env["AMX_IMAP_PORT"] = "993"
cfg, err := configFrom(getenvOf(env))
if err != nil {
t.Fatal(err)
}
rec := httptest.NewRecorder()
newHandler(cfg).ServeHTTP(rec, httptest.NewRequest("GET", "/mail/config-v1.1.xml?emailaddress=a@novox.be", nil))
body := rec.Body.String()
if !strings.Contains(body, `<incomingServer type="imap"><hostname>imap.example.test</hostname><port>993</port>`) {
t.Errorf("incoming server not named from AMX_IMAP_ADDRESS and AMX_IMAP_PORT: %s", body)
}
if !strings.Contains(body, `<outgoingServer type="smtp"><hostname>mail.novox.be</hostname><port>587</port>`) {
t.Errorf("outgoing server changed: %s", body)
}
}
func TestEveryDomainOfTheEnvironmentIsServed(t *testing.T) {
env := map[string]string{}
for k, v := range capturedEnvironment {
env[k] = v
}
env["AMX_MAIL_DOMAINS"] = "novox.be, example.test"
cfg, err := configFrom(getenvOf(env))
if err != nil {
t.Fatal(err)
}
h := newHandler(cfg)
for _, d := range []string{"novox.be", "example.test"} {
rec := httptest.NewRecorder()
h.ServeHTTP(rec, httptest.NewRequest("GET", "/mail/config-v1.1.xml?emailaddress=a@"+d, nil))
if rec.Code != 200 || !strings.Contains(rec.Body.String(), "<domain>"+d+"</domain>") {
t.Errorf("%s: %d %s", d, rec.Code, rec.Body.String())
}
}
}
// Failure is loud (issue 1631 AC2): a missing variable stops the program and its name is in the message.
func TestAMissingVariableStopsTheProgramNamingIt(t *testing.T) {
for name := range capturedEnvironment {
t.Run(name, func(t *testing.T) {
env := map[string]string{}
for k, v := range capturedEnvironment {
env[k] = v
}
delete(env, name)
if _, err := configFrom(getenvOf(env)); err == nil || !strings.Contains(err.Error(), name) {
t.Fatalf("configFrom without %s: %v", name, err)
}
var out bytes.Buffer
if code := run(getenvOf(env), &out, func(*http.Server) error { t.Fatal("served without its configuration"); return nil }); code == 0 || !strings.Contains(out.String(), name) {
t.Errorf("run without %s: exit %d, said %q", name, code, out.String())
}
})
}
}
func TestAPortThatIsNotANumberIsRefusedNamingIt(t *testing.T) {
env := map[string]string{}
for k, v := range capturedEnvironment {
env[k] = v
}
env["AMX_SMTP_PORT"] = "submission"
if _, err := configFrom(getenvOf(env)); err == nil || !strings.Contains(err.Error(), "AMX_SMTP_PORT") {
t.Fatalf("got %v", err)
}
}
-290
View File
@@ -1,290 +0,0 @@
// automx answers mail clients' autoconfiguration requests for the domains mailu serves: Thunderbird's
// /mail/config-v1.1.xml, Outlook's /autodiscover/autodiscover.xml and Apple's /mobileconfig. It replaces the
// Python automx2 image (novox/hq ADR 0480 clauses 1, 9, 10): standard library only, no database, every answer
// made from mailu's environment at the start, so a change of the domains, servers or ports reaches clients when
// the program is started again.
//
// The answers are those automx2 2021.6 gave, captured in testdata/captured (CAPTURED, issue 1630), byte for byte,
// but for the one thing testdata/captured/DIFFERENCES lists: the IMAP server is named, from AMX_IMAP_ADDRESS.
package main
import (
"encoding/xml"
"errors"
"fmt"
"html"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
// The environment, all required: a program that cannot say which servers clients use must not run (failure is
// loud). The names are those of mailu's files/mailu.env.
const (
envDomains = "AMX_MAIL_DOMAINS"
envImapHost = "AMX_IMAP_ADDRESS"
envImapPort = "AMX_IMAP_PORT"
envSmtpHost = "AMX_SMTP_ADDRESS"
envSmtpPort = "AMX_SMTP_PORT"
envListen = "AUTOMX_PORT"
)
type config struct {
domains []string
imapHost, smtpHost string
imapPort, smtpPort int
listen int
}
// configFrom reads the environment; it names every variable that is missing or wrong.
func configFrom(getenv func(string) string) (config, error) {
var c config
var problems []string
text := func(name string) string {
v := strings.TrimSpace(getenv(name))
if v == "" {
problems = append(problems, name+" is not set")
}
return v
}
port := func(name string) int {
v := text(name)
if v == "" {
return 0
}
n, err := strconv.Atoi(v)
if err != nil || n < 1 || n > 65535 {
problems = append(problems, fmt.Sprintf("%s is %q, not a port number", name, v))
return 0
}
return n
}
// automx2's add-domains split the list at commas and spaces.
for _, d := range strings.FieldsFunc(text(envDomains), func(r rune) bool { return r == ',' || r == ' ' }) {
c.domains = append(c.domains, strings.ToLower(d))
}
c.imapHost, c.imapPort = text(envImapHost), port(envImapPort)
c.smtpHost, c.smtpPort = text(envSmtpHost), port(envSmtpPort)
c.listen = port(envListen)
if len(problems) > 0 {
return config{}, errors.New(strings.Join(problems, "; "))
}
return c, nil
}
// What automx2 sent (Werkzeug), copied from the capture.
const (
htmlType = "text/html; charset=utf-8"
xmlType = "application/xml"
notFound = `<!doctype html>
<html lang=en>
<title>404 Not Found</title>
<h1>Not Found</h1>
<p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>
`
notAllowed = `<!doctype html>
<html lang=en>
<title>405 Method Not Allowed</title>
<h1>Method Not Allowed</h1>
<p>The method is not allowed for the requested URL.</p>
`
internalError = `<!doctype html>
<html lang=en>
<title>500 Internal Server Error</title>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.</p>
`
badRequest = `<!doctype html>
<html lang=en>
<title>400 Bad Request</title>
<h1>Bad Request</h1>
<p>The browser (or proxy) sent a request that this server could not understand.</p>
`
index = `<html><body>Show Thunderbird-style <a href="/mail/config-v1.1.xml?emailaddress=abc@example.com">XML configuration</a> for abc@example.com.</body></html>`
)
func reply(w http.ResponseWriter, status int, contentType, body string) {
w.Header().Set("Content-Type", contentType)
if status != http.StatusNoContent {
w.Header().Set("Content-Length", strconv.Itoa(len(body)))
}
w.WriteHeader(status)
if status != http.StatusNoContent {
io.WriteString(w, body)
}
}
func missing(w http.ResponseWriter, argument string) {
reply(w, http.StatusBadRequest, htmlType, fmt.Sprintf("Missing request argument %q", argument))
}
type handler struct{ config }
func newHandler(c config) http.Handler { return handler{c} }
func (h handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
var allow string
switch r.URL.Path {
case "/", "/mail/config-v1.1.xml", "/mobileconfig/":
allow = "GET, HEAD, OPTIONS"
case "/autodiscover/autodiscover.xml":
allow = "POST, OPTIONS"
case "/mobileconfig":
h.redirectToSlash(w, r)
return
default:
reply(w, http.StatusNotFound, htmlType, notFound)
return
}
if r.Method == http.MethodOptions {
w.Header().Set("Allow", allow)
reply(w, http.StatusOK, htmlType, "")
return
}
if !strings.Contains(", "+allow+",", ", "+r.Method+",") {
w.Header().Set("Allow", allow)
reply(w, http.StatusMethodNotAllowed, htmlType, notAllowed)
return
}
switch r.URL.Path {
case "/":
reply(w, http.StatusOK, htmlType, index)
case "/mail/config-v1.1.xml":
h.byAddress(w, r.URL.Query().Get("emailaddress"), r.URL.Query().Has("emailaddress"), "emailaddress", h.thunderbird)
case "/mobileconfig/":
h.byAddress(w, r.URL.Query().Get("emailaddress"), r.URL.Query().Has("emailaddress"), "emailaddress", h.apple)
case "/autodiscover/autodiscover.xml":
h.outlook(w, r)
}
}
// byAddress is the part the three protocols share: the address is required, and a domain that is not served is
// answered with no content.
func (h handler) byAddress(w http.ResponseWriter, address string, present bool, argument string, answer func(http.ResponseWriter, int, string)) {
if !present {
missing(w, argument)
return
}
domain := strings.ToLower(address[strings.LastIndex(address, "@")+1:])
for i, d := range h.domains {
if strings.Contains(address, "@") && d == domain {
answer(w, i, d)
return
}
}
reply(w, http.StatusNoContent, htmlType, "")
}
// thunderbird is Mozilla's autoconfiguration, config-v1.1. The provider id is automx2's: the provider rows were
// numbered 1, 11, 21 ... by add-domains.
func (h handler) thunderbird(w http.ResponseWriter, i int, domain string) {
d := html.EscapeString(domain)
server := func(kind, host string, port int) string {
tag := map[string]string{"imap": "incomingServer", "smtp": "outgoingServer"}[kind]
return fmt.Sprintf(`<%s type="%s"><hostname>%s</hostname><port>%d</port><socketType>STARTTLS</socketType>`+
`<username>%%EMAILADDRESS%%</username><authentication>password-cleartext</authentication></%s>`,
tag, kind, html.EscapeString(host), port, tag)
}
body := fmt.Sprintf(`<clientConfig version="1.1"><emailProvider id="automx2-%d"><identity /><domain>%s</domain>`+
`<displayName>%s</displayName><displayShortName>%s</displayShortName>%s%s</emailProvider></clientConfig>`,
1+10*i, d, d, d, server("imap", h.imapHost, h.imapPort), server("smtp", h.smtpHost, h.smtpPort))
reply(w, http.StatusOK, xmlType, body)
}
// apple and outlook answer a served domain as automx2 did: with an error (DIFFERENCES, issue 1691). They are
// kept as captured here, so the move to a Go program changes the answer only where clause 9 says.
func (h handler) apple(w http.ResponseWriter, _ int, _ string) {
reply(w, http.StatusBadRequest, htmlType, badRequest)
}
// redirectToSlash is Werkzeug's strict-slash redirect, to the name the client used.
func (h handler) redirectToSlash(w http.ResponseWriter, r *http.Request) {
scheme := "http"
if p := strings.TrimSpace(strings.Split(r.Header.Get("X-Forwarded-Proto"), ",")[0]); p != "" {
scheme = p
}
host := r.Host
if f := strings.TrimSpace(strings.Split(r.Header.Get("X-Forwarded-Host"), ",")[0]); f != "" {
host = f
}
target := scheme + "://" + host + "/mobileconfig/"
if r.URL.RawQuery != "" {
target += "?" + r.URL.RawQuery
}
w.Header().Set("Location", target)
e := html.EscapeString(target)
reply(w, http.StatusPermanentRedirect, htmlType, fmt.Sprintf(`<!doctype html>
<html lang=en>
<title>Redirecting...</title>
<h1>Redirecting...</h1>
<p>You should be redirected automatically to the target URL: <a href="%s">%s</a>. If not, click the link.
`, e, e))
}
// outlook reads the request's EMailAddress anywhere in the document, whatever the namespace. A body that is not
// one XML document is automx2's 500.
func (h handler) outlook(w http.ResponseWriter, r *http.Request) {
dec := xml.NewDecoder(io.LimitReader(r.Body, 1<<20))
var address string
var found, elements, inAddress = false, 0, false
for {
tok, err := dec.Token()
if err == io.EOF {
break
}
if err != nil {
reply(w, http.StatusInternalServerError, htmlType, internalError)
return
}
switch t := tok.(type) {
case xml.StartElement:
elements++
if t.Name.Local == "EMailAddress" && !found {
inAddress = true
}
case xml.CharData:
if inAddress {
address += string(t)
}
case xml.EndElement:
if inAddress && t.Name.Local == "EMailAddress" {
inAddress, found = false, true
}
}
}
if elements == 0 {
reply(w, http.StatusInternalServerError, htmlType, internalError)
return
}
h.byAddress(w, strings.TrimSpace(address), found, "EMailAddress", func(w http.ResponseWriter, _ int, _ string) {
reply(w, http.StatusInternalServerError, htmlType, internalError)
})
}
// run is the program: configuration from the environment, then serving. A configuration that cannot be read is
// said, with the variables' names, and the exit code is not zero.
func run(getenv func(string) string, out io.Writer, serve func(*http.Server) error) int {
c, err := configFrom(getenv)
if err != nil {
fmt.Fprintf(out, "automx: %v\n", err)
return 1
}
srv := &http.Server{
Addr: ":" + strconv.Itoa(c.listen),
Handler: newHandler(c),
ReadHeaderTimeout: 10 * time.Second,
}
if err := serve(srv); err != nil {
fmt.Fprintf(out, "automx: %v\n", err)
return 1
}
return 0
}
func main() {
os.Exit(run(os.Getenv, os.Stderr, func(s *http.Server) error { return s.ListenAndServe() }))
}
-202
View File
@@ -1,202 +0,0 @@
package main
import (
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
)
// The digest route-proxy's `trust` artifact runs: one alpine image is held for both (ADR 0480 clause 3). It was
// read from mesh-controller's modules/route-proxy/module.json and is the one hello-web runs.
const alpineDigest = "alpine@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b"
type manifest struct {
Tools []string `json:"tools"`
Resources []map[string]any `json:"resources"`
Listens []struct {
Name string `json:"name"`
Port int `json:"port"`
} `json:"listens"`
Data struct {
Own []struct {
ID string `json:"id"`
Path string `json:"path"`
} `json:"own"`
} `json:"data"`
Build map[string]any `json:"build"`
}
func readManifest(t *testing.T) manifest {
t.Helper()
var m manifest
if err := json.Unmarshal([]byte(readFile(t, "../../module.json")), &m); err != nil {
t.Fatal(err)
}
return m
}
func (m manifest) resource(id string) map[string]any {
for _, r := range m.Resources {
if r["id"] == id {
return r
}
}
return nil
}
func (m manifest) artifact(name string) map[string]any {
arts, _ := m.Build["artifacts"].([]any)
for _, a := range arts {
if a, _ := a.(map[string]any); a["name"] == name {
return a
}
}
return nil
}
func strs(v any) []string {
var out []string
l, _ := v.([]any)
for _, e := range l {
out = append(out, e.(string))
}
return out
}
// Clause 2: a Go bundle from a package of the module's own go.mod, stamped for alpine; an archive resource unpacks
// it; the container mounts that read-only at /opt/mesh, calls the program there and names the archive in restart-on.
func TestAutomxIsAGoBundleForAlpineMountedAtOptMeshAndRestartedWithIt(t *testing.T) {
m := readManifest(t)
b := m.artifact("automx")
if b == nil || b["kind"] != "bundle" || b["language"] != "go" || b["system"] != "alpine" || b["from"] != "cmd/automx" || b["binary"] != "automx" {
t.Fatalf("artifact automx is %v, want a Go bundle, system alpine, from cmd/automx, binary automx", b)
}
if _, loads := b["loads"]; loads {
t.Errorf("a program run in a container is not loaded by the tool runner: %v", b["loads"])
}
goMod := readFile(t, "../../go.mod")
if !strings.HasPrefix(goMod, "module mailu\n") {
t.Errorf("the module's go.mod is %q; the bundle is a package of it", goMod)
}
if _, err := os.Stat(filepath.Join("..", "..", filepath.FromSlash(b["from"].(string)), "main.go")); err != nil {
t.Errorf("the bundle's package is not in the module's directory: %v", err)
}
unpack := m.resource("program-automx")
if unpack == nil || unpack["type"] != "archive" || unpack["artifact"] != "automx" || unpack["unknown"] != "program" {
t.Fatalf("resource program-automx is %v, want an archive of the bundle", unpack)
}
where, _ := unpack["path"].(string)
c := m.resource("automx")
if c == nil || c["type"] != "container" {
t.Fatalf("no container automx: %v", c)
}
if got := strs(c["volumes"]); len(got) != 1 || got[0] != where+":/opt/mesh:ro" {
t.Errorf("volumes %v, want the archive's place %q mounted read-only at /opt/mesh", got, where)
}
if got := strs(c["args"]); len(got) != 1 || got[0] != "/opt/mesh/automx" {
t.Errorf("args %v, want the program at /opt/mesh/automx", got)
}
if !contains(strs(c["restart-on"]), "program-automx") {
t.Errorf("restart-on %v does not name the archive: a new program would leave the old one running", c["restart-on"])
}
if !contains(strs(c["restart-on"]), "config-env") {
t.Errorf("restart-on %v does not name the environment file the program reads at its start", c["restart-on"])
}
}
// Clause 3: an upstream artifact pinned by digest, the one route-proxy's trust artifact runs; never an image
// built from a Dockerfile (clause 11), and no Dockerfile or base to build one on.
func TestAutomxRunsTheMirroredAlpineImageAndBuildsNoImage(t *testing.T) {
m := readManifest(t)
c := m.resource("automx")
base := m.artifact(c["artifact"].(string))
if base == nil || base["kind"] != "upstream" || base["from"] != alpineDigest {
t.Errorf("the container runs artifact %v, want an upstream artifact %s", base, alpineDigest)
}
if _, has := c["image"]; has {
t.Errorf("the container names an image %v, fetched from a registry on the machine", c["image"])
}
for _, a := range m.Build["artifacts"].([]any) {
if a.(map[string]any)["kind"] == "image" {
t.Errorf("artifact %v is an image built from a Dockerfile (ADR 0452)", a)
}
}
if _, has := m.Build["on"]; has {
t.Errorf("build.on %v: nothing is built on a base image any more", m.Build["on"])
}
for _, gone := range []string{"../../automx/Dockerfile", "../../automx"} {
if _, err := os.Stat(gone); err == nil {
t.Errorf("%s is still there", gone)
}
}
}
// Clause 1: the container stays as it was for what watches it.
func TestAutomxKeepsItsContainerNameNetworkPortAndEnvironment(t *testing.T) {
m := readManifest(t)
c := m.resource("automx")
if c["name"] != "mailu-automx" || c["network"] != "mailu" {
t.Errorf("container %v on network %v, want mailu-automx on mailu", c["name"], c["network"])
}
if got := strs(c["ports"]); len(got) != 1 || got[0] != "4243" {
t.Errorf("ports %v, want 4243", got)
}
if got := strs(c["env-file"]); len(got) != 1 || got[0] != "${dir:state}/mailu.env" {
t.Errorf("env-file %v, want mailu's environment only", got)
}
if !strings.Contains("\n"+readFile(t, "../../files/mailu.env"), "\nAUTOMX_PORT=4243\n") {
t.Errorf("mailu.env does not say AUTOMX_PORT=4243")
}
for _, l := range m.Listens {
if l.Name == "autoconfig" && l.Port == 4243 {
return
}
}
t.Errorf("the autoconfig endpoint is not on 4243: %v", m.Listens)
}
// Clause 10: a database nobody reads is gone from the declaration (the file stays on disk), and the container says
// whether it answers with one captured request and its captured answer.
func TestAutomxKeepsNoDatabaseAndSaysWhetherItAnswers(t *testing.T) {
m := readManifest(t)
if m.resource("data-automx") != nil {
t.Errorf("the data-automx directory is still declared")
}
for _, it := range m.Data.Own {
if strings.Contains(it.ID, "automx") || strings.Contains(it.Path, "data-automx") {
t.Errorf("data item %v is still declared", it)
}
}
for _, tool := range m.Tools {
if tool == "automx_rows" {
t.Errorf("the tool automx_rows reads a database that no longer exists (issue 1700)")
}
}
h, _ := m.resource("automx")["health"].(map[string]any)
if h["kind"] != "http" || h["endpoint"] != "autoconfig" {
t.Fatalf("health %v, want an http check of the autoconfig endpoint", h)
}
// The request is captured fixture 06 and the answer its status and body.
rq := readRequest(t, filepath.Join(captured, "06-autoconfig-root"))
if u := strings.SplitN(rq.url, "/", 4); "/"+u[len(u)-1] != h["path"] || rq.method != "GET" {
t.Errorf("health path %v is not captured request %s %s", h["path"], rq.method, rq.url)
}
if st := readFile(t, filepath.Join(captured, "06-autoconfig-root.status")); !strings.HasPrefix(st, "200 ") || h["status"] != float64(200) {
t.Errorf("health status %v, captured %q", h["status"], st)
}
if body, _ := h["body"].(string); body == "" || !strings.Contains(readFile(t, filepath.Join(captured, "06-autoconfig-root.body")), body) {
t.Errorf("health body %q is not in the captured answer", h["body"])
}
}
func contains(l []string, s string) bool {
for _, e := range l {
if e == s {
return true
}
}
return false
}
@@ -1 +0,0 @@
<clientConfig version="1.1"><emailProvider id="automx2-1"><identity /><domain>novox.be</domain><displayName>novox.be</displayName><displayShortName>novox.be</displayShortName><incomingServer type="imap"><hostname>mail.novox.be</hostname><port>143</port><socketType>STARTTLS</socketType><username>%EMAILADDRESS%</username><authentication>password-cleartext</authentication></incomingServer><outgoingServer type="smtp"><hostname>mail.novox.be</hostname><port>587</port><socketType>STARTTLS</socketType><username>%EMAILADDRESS%</username><authentication>password-cleartext</authentication></outgoingServer></emailProvider></clientConfig>
@@ -1 +0,0 @@
<clientConfig version="1.1"><emailProvider id="automx2-1"><identity /><domain>novox.be</domain><displayName>novox.be</displayName><displayShortName>novox.be</displayShortName><incomingServer type="imap"><hostname>mail.novox.be</hostname><port>143</port><socketType>STARTTLS</socketType><username>%EMAILADDRESS%</username><authentication>password-cleartext</authentication></incomingServer><outgoingServer type="smtp"><hostname>mail.novox.be</hostname><port>587</port><socketType>STARTTLS</socketType><username>%EMAILADDRESS%</username><authentication>password-cleartext</authentication></outgoingServer></emailProvider></clientConfig>
@@ -1 +0,0 @@
<clientConfig version="1.1"><emailProvider id="automx2-1"><identity /><domain>novox.be</domain><displayName>novox.be</displayName><displayShortName>novox.be</displayShortName><incomingServer type="imap"><hostname>mail.novox.be</hostname><port>143</port><socketType>STARTTLS</socketType><username>%EMAILADDRESS%</username><authentication>password-cleartext</authentication></incomingServer><outgoingServer type="smtp"><hostname>mail.novox.be</hostname><port>587</port><socketType>STARTTLS</socketType><username>%EMAILADDRESS%</username><authentication>password-cleartext</authentication></outgoingServer></emailProvider></clientConfig>
+1 -12
View File
@@ -14,13 +14,6 @@ Each capture is four files with the same stem:
The files were written from the tool's answers by a script (not by retyping each by hand); the tool returns JSON, so
a later capture that wants exact bytes needs a tool that writes the answer to a file.
A fixture is not the bytes on the wire in four ways (issue 1701):
- Date is left out of .headers (it changes on every request).
- .headers are sorted by name.
- .status holds the tool's status line without the protocol (the tool reported "HTTP/1.1" apart).
- Header names are as the tool's Go HTTP client gives them (canonical case, after the mesh's route), not as
automx wrote them.
What the answers show (read, not guessed):
- GET /mail/config-v1.1.xml?emailaddress=... answers 200 application/xml on all three names; same body on each.
- The incoming (IMAP) server has an EMPTY hostname (<hostname />), port 143, STARTTLS. The outgoing server is
@@ -34,8 +27,4 @@ What the answers show (read, not guessed):
- GET /mobileconfig?emailaddress=... redirects (308) to /mobileconfig/?emailaddress=...; for the served domain that
answers 400 with no profile; unknown domain 204; no emailaddress 400.
- GET / on autoconfig answers 200 with a small HTML page linking to an example address.
- The database rows behind the answers were read once with the tool automx_rows (mailu, issue 1630 AC4), which
ran sqlite3 read-only inside mailu-automx. The tool is retired with the database (ADR 0480 clause 10, issue
1700), so nothing reads the rows any more; the answers above are what they gave.
- The Go program that replaced automx2 answers these requests as they are captured here, but for what
DIFFERENCES lists.
- Not captured: the database rows (no mesh tool reads /data/db.sqlite; issue 1630 AC4).
-25
View File
@@ -1,25 +0,0 @@
How the Go program (modules/mailu/cmd/automx) answers differently from the automx2 2021.6 answers captured here
(issue 1631, ADR 0480 clause 9). automx_test.go answers every .request and compares with the capture; this is the
whole list of where the two differ.
1. The IMAP server's name (the one correction clause 9 allows). automx2 answered <hostname /> for the incoming
server because its setup script read AMX_IMAP_SERVER, which mailu's environment does not set (it sets
AMX_IMAP_ADDRESS). The program names it from AMX_IMAP_ADDRESS (and the port from AMX_IMAP_PORT).
Fixtures 01, 02 and 03 each have a <stem>.corrected.body: the capture with <hostname /> replaced by
<hostname>mail.novox.be</hostname> and nothing else (a test checks that). Their Content-Length is that file's
size, 634, not the captured 612.
2. The Server header. The capture has "Server: Werkzeug/3.1.9 Python/3.9.5". The program sends no Server header, and
Date, which the captured .headers leave out, is sent as net/http sends it.
Not changed, though issue 1691 calls them faults; they are kept as captured because clause 9 allows only the first
correction, and their cause is not known from the capture:
- Outlook's POST for a served domain (fixture 08) answers 500.
- Apple's /mobileconfig/ for a served domain (fixture 15) answers 400 with no profile.
Whether automx2 failed there because of the empty IMAP name is a guess; it can be tested only against the
running program after the delivery. Issue 1691 stays open for them.
Assumptions the fixtures do not settle (answers to requests nobody captured): a domain is compared in lower case;
an address without "@" is an unknown domain (204); HEAD is answered as GET and OPTIONS with the Allow header and
no body; the redirect of /mobileconfig takes its scheme and host from X-Forwarded-Proto and X-Forwarded-Host when
present (the route sends the client's, as fixture 14's Location shows) and from the request otherwise.
-3
View File
@@ -1,3 +0,0 @@
module mailu
go 1.22
+24 -34
View File
@@ -71,7 +71,8 @@
"mailu_list_domains",
"mailu_read_mail",
"mailu_search_mail",
"mailu_health"
"mailu_health",
"automx_rows"
],
"emits": [
"user.created",
@@ -215,6 +216,12 @@
"class": "rebuildable",
"why": "certificates, issued again"
},
{
"id": "automx",
"path": "${dir:data-automx}",
"class": "rebuildable",
"why": "client autoconfiguration, written again"
},
{
"id": "fetchmail",
"path": "${dir:data-fetchmail}",
@@ -301,6 +308,12 @@
"mode": "0700",
"unknown": "program"
},
{
"id": "data-automx",
"type": "directory",
"mode": "0700",
"unknown": "program"
},
{
"id": "config-env",
"type": "file",
@@ -668,25 +681,11 @@
"takes-config": ["domain", "proxy-address", "sitename", "website", "url", "container"],
"merge": "json"
},
{
"id": "program-automx",
"type": "archive",
"path": "${dir:state}/automx",
"artifact": "automx",
"unknown": "program"
},
{
"id": "automx",
"type": "container",
"name": "mailu-automx",
"artifact": "alpine",
"health": {
"kind": "http",
"endpoint": "autoconfig",
"path": "/",
"status": 200,
"body": "Show Thunderbird-style"
},
"artifact": "automx",
"network": "mailu",
"env-file": [
"${dir:state}/mailu.env"
@@ -695,18 +694,17 @@
"4243"
],
"volumes": [
"${dir:state}/automx:/opt/mesh:ro"
],
"args": [
"/opt/mesh/automx"
],
"restart-on": [
"program-automx",
"config-env"
"${dir:data-automx}:/data"
]
}
],
"build": {
"on": [
{
"arg": "PYTHON_BASE",
"image": "python@sha256:25f3cfeaceca14921366af4d1240b56457ef46273bdb508c7b0e8f469f6fd228"
}
],
"artifacts": [
{
"name": "code",
@@ -733,16 +731,8 @@
},
{
"name": "automx",
"kind": "bundle",
"language": "go",
"system": "alpine",
"from": "cmd/automx",
"binary": "automx"
},
{
"name": "alpine",
"kind": "upstream",
"from": "alpine@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b"
"kind": "image",
"from": "automx/Dockerfile"
},
{
"name": "files",
+103
View File
@@ -0,0 +1,103 @@
// What automx_rows reads (novox/hq issue 1630): the rows of automx2's database as stored, through a read-only
// sqlite3 inside its container. The database here is a real SQLite file made by the test with the tables
// automx/files/setup-db creates and the rows add-domains writes — with the IMAP server's name empty, the case
// the live instance is in. The fake container runs each command's statement against that database and prints
// it the way the image's sqlite3 does in list mode (one line per row, cells apart by "|", nothing for no rows); that sqlite3
// refuses `-json` ("sqlite3: Error: unknown option: -json", captured from novox/mailu.automx_rows, issue 1630). Run against the compiled module.
import { test } from "node:test";
import assert from "node:assert/strict";
import { DatabaseSync } from "node:sqlite";
import { readAutomx, readCommand, selectOf, parseRows, AUTOMX_DATABASE } from "../dist/automx.js";
function database(imapName: string): DatabaseSync {
const db = new DatabaseSync(":memory:");
db.exec(`
CREATE TABLE provider(id INT PRIMARY KEY NOT NULL, name TEXT NOT NULL, short_name TEXT NOT NULL);
CREATE TABLE server(id INT PRIMARY KEY NOT NULL, prio INT NOT NULL DEFAULT 10, name TEXT NOT NULL, port INT NOT NULL,
type TEXT NOT NULL, socket_type TEXT NOT NULL, user_name TEXT NOT NULL, authentication TEXT NOT NULL);
CREATE TABLE domain(id INT PRIMARY KEY NOT NULL, name TEXT NOT NULL, provider_id INT NOT NULL, ldapserver_id INT NULL);
CREATE TABLE server_domain(server_id INT NOT NULL, domain_id INT NOT NULL);
INSERT INTO provider VALUES (1, 'example.org', 'example.org');
INSERT INTO server(id, port, type, name, socket_type, user_name, authentication)
VALUES (2, 143, 'imap', '${imapName}', 'STARTTLS', '%EMAILADDRESS%', 'password-cleartext');
INSERT INTO server(id, port, type, name, socket_type, user_name, authentication)
VALUES (3, 587, 'smtp', 'mail.example.org', 'STARTTLS', '%EMAILADDRESS%', 'password-cleartext');
INSERT INTO domain VALUES (5, 'example.org', 1, NULL);
INSERT INTO server_domain VALUES (2, 5), (3, 5);
`);
return db;
}
function container(db: DatabaseSync, asked: string[][] = []) {
return async (_name: string, command: string[]) => {
asked.push(command);
if (command.includes("-json")) throw Object.assign(new Error("Command failed: docker exec"), { stderr: "sqlite3: Error: unknown option: -json\n" });
const rows = db.prepare(command[command.length - 1]).all() as unknown as Record<string, string>[];
return rows.length === 0 ? "" : rows.map((r) => Object.values(r).join("|")).join("\n") + "\n";
};
}
test("the rows come back as stored, and an empty server name is said", async () => {
const answer = await readAutomx(container(database("")), "mailu-automx");
assert.equal(answer.database, AUTOMX_DATABASE);
assert.deepEqual(answer.unreadable, {});
assert.deepEqual(answer.tables.provider, [{ id: 1, name: "example.org", short_name: "example.org" }]);
assert.deepEqual(answer.tables.domain, [{ id: 5, name: "example.org", provider_id: 1, ldapserver_id: null }]);
assert.deepEqual(answer.tables.server_domain, [{ server_id: 2, domain_id: 5 }, { server_id: 3, domain_id: 5 }]);
assert.deepEqual(
answer.tables.server.map((s) => [s.id, s.type, s.name, s.port]),
[[2, "imap", "", 143], [3, "smtp", "mail.example.org", 587]],
);
assert.deepEqual(answer.empty_names, [{ id: 2, type: "imap" }]);
});
test("a named IMAP server is not reported empty", async () => {
const answer = await readAutomx(container(database("mail.example.org")), "mailu-automx");
assert.deepEqual(answer.empty_names, []);
});
test("every command opens the file read-only and runs one fixed SELECT", async () => {
const asked: string[][] = [];
await readAutomx(container(database("x"), asked), "mailu-automx");
assert.equal(asked.length, 4);
for (const c of asked) {
assert.deepEqual(c.slice(0, 3), ["sqlite3", "-readonly", AUTOMX_DATABASE]);
assert.match(c[3], /^SELECT /);
assert.equal(c.length, 4);
assert.ok(!c.includes("-json"));
}
assert.deepEqual(readCommand("provider"), ["sqlite3", "-readonly", AUTOMX_DATABASE, selectOf("provider")]);
});
test("a table that cannot be read is said and does not hide the others", async () => {
const db = database("x");
const exec = async (name: string, command: string[]) => {
if (command[3].includes("FROM domain ")) throw Object.assign(new Error("Command failed: docker exec"), { stderr: "Error: unable to open database file\nmore" });
return container(db)(name, command);
};
const answer = await readAutomx(exec, "mailu-automx");
assert.deepEqual(answer.unreadable, { domain: "Error: unable to open database file" });
assert.equal(answer.tables.provider.length, 1);
assert.equal(answer.tables.domain, undefined);
});
test("a NULL, an empty text, a number and a text holding a bar or a newline are read apart", () => {
const hex = (t: string) => Buffer.from(t).toString("hex").toUpperCase();
const line = [`integer:${hex("5")}`, `text:${hex("a|b\nc")}`, "text:", "null:"].join("|");
const rows = parseRows("domain", line + "\n");
assert.deepEqual(rows, [{ id: 5, name: "a|b\nc", provider_id: "", ldapserver_id: null }]);
});
test("output that is not cells is an unreadable table, not a guess", async () => {
const exec = async () => "Error: something else\n";
const answer = await readAutomx(exec, "mailu-automx");
assert.equal(Object.keys(answer.unreadable).length, 4);
});
test("the image's refusal of -json is said as the table's reason", async () => {
const exec = async (_n: string, command: string[]) => container(database("x"))("", ["sqlite3", "-json", ...command.slice(1)]);
const answer = await readAutomx(exec, "mailu-automx");
assert.deepEqual(answer.unreadable.domain, "sqlite3: Error: unknown option: -json");
});
+2 -1
View File
@@ -135,6 +135,7 @@ test("the health tool is served without the admin API, on the smtp container it
await import("../dist/tools/index.js");
const mailu = collectTools({ MESH_MAILU_SMTP_CONTAINER: "mailu-smtp" }).find((m) => m.module === "mailu");
const names = (mailu?.tools ?? []).map((t) => t.name);
assert.deepEqual(names, ["mailu_health"]);
assert.deepEqual(names, ["mailu_health", "automx_rows"]);
assert.match(mailu!.tools[0].description, /inside mailu-smtp/);
assert.match(mailu!.tools[1].description, /read-only, inside mailu-automx/);
});
+19 -1
View File
@@ -4,6 +4,7 @@
import { registerModuleTools, type ToolDefinition } from "@novox/mesh-sdk/tools";
import { MailuClient } from "../client.js";
import { readAutomx } from "../automx.js";
import { dockerExec, mailFlows, DEFERRED_BOUND_SECONDS, PUBLIC_NAME, SPREAD_DOMAINS } from "../health.js";
/**
@@ -26,6 +27,23 @@ export function getHealthTool(env: NodeJS.ProcessEnv): ToolDefinition {
};
}
/**
* What automx holds, as stored (novox/hq issue 1630): the rows behind the answers it gives mail clients. It needs
* no admin API, only the automx container, so like the health tool it is served without the API.
*/
export function getAutomxTool(env: NodeJS.ProcessEnv): ToolDefinition {
const container = env.MESH_MAILU_AUTOMX_CONTAINER || "mailu-automx";
return {
name: "automx_rows",
description:
`Read the autoconfiguration module's database, as stored, read-only, inside ${container}: its provider, ` +
"server, domain and server_domain rows, and the servers whose name is empty (a mail client is then told " +
"an empty host name). Writes nothing.",
input: {},
run: async () => readAutomx(dockerExec(), container),
};
}
export function getMailuTools(mailu: MailuClient): ToolDefinition[] {
return [
{
@@ -173,5 +191,5 @@ registerModuleTools("mailu", (env) => {
} catch {
tools = [];
}
return [...tools, getHealthTool(env)];
return [...tools, getHealthTool(env), getAutomxTool(env)];
});
+1 -1
View File
@@ -8,5 +8,5 @@
"skipLibCheck": true,
"noEmit": true
},
"include": ["client.ts", "health.ts", "index.ts", "tools/index.ts"]
"include": ["client.ts", "health.ts", "automx.ts", "index.ts", "tools/index.ts"]
}
@@ -703,3 +703,23 @@ func TestNatsResticsFilesFillFromTheBackupPlacement(t *testing.T) {
b.assign(t, "restic", plan("# a\n/srv/a\n", "# a\nstate\n"))
eventually(t, "a module taken away refills backups.conf", func() bool { return e.read(t, backups) == head+"# a\n/srv/a\n" })
}
// defaults writes the defaults a module's manifest declares as mesh-config resolves them (source default), for a
// moved module whose templates read config values (issue 1780).
func (b *aBus) defaults(t *testing.T, module string, manifest []byte) {
t.Helper()
var m struct {
Config map[string]struct {
Default json.RawMessage `json:"default"`
} `json:"config"`
}
if err := json.Unmarshal(manifest, &m); err != nil {
t.Fatal(err)
}
for key, d := range m.Config {
if len(d.Default) > 0 && string(d.Default) != "null" {
b.put(t, ConfigBucket, node+"."+module+"."+key, fmt.Sprintf(`{"node":%q,"module":%q,"key":%q,"value":%s,`+
`"source":"default"}`, node, module, key, d.Default))
}
}
}
@@ -158,6 +158,7 @@ func TestNatsTheEnvironmentOfTheCataloguesModulesIsWhatTheOldFormComposed(t *tes
if !manifest.Moved || len(manifest.Items) == 0 {
t.Fatalf("%s is not a moved contributor of items", m)
}
b.defaults(t, m, read(m+"/module.json"))
texts := map[string]string{}
var planned []PlannedItem
for _, it := range manifest.Items {
@@ -63,6 +63,7 @@ func TestNatsTheSessionFilesFilledFromTheirContributorsItemsAreWhatTheOldFormCom
if !c.Moved || len(c.Shell) != 0 {
t.Fatalf("%s is not moved or still holds shell entries", m)
}
b.defaults(t, m, []byte(read(m+"/module.json")))
texts := map[string]string{}
var planned []PlannedItem
for _, it := range c.Items {
+36 -5
View File
@@ -256,11 +256,22 @@ export class MinioClient {
if (argv.some((a) => a.includes(this.rootPassword))) {
throw new Error("refused to run mc: its command line would carry the root password (novox/hq issue 282)");
}
const { stdout } = await execFileAsync(this.mcBin, argv, {
timeout: 30_000,
env: { ...process.env, MC_HOST_mesh: mcHost(this.baseUrl, this.rootUser, this.rootPassword) },
});
return stdout.trim();
const mcHostValue = mcHost(this.baseUrl, this.rootUser, this.rootPassword);
try {
const { stdout } = await execFileAsync(this.mcBin, argv, {
timeout: 30_000,
env: { ...process.env, MC_HOST_mesh: mcHostValue },
});
return stdout.trim();
} catch (err) {
// A fresh error with only the redacted message: execFile's own carries the whole command line, and
// its cmd, stdout and stderr properties would carry it on (issue 1766).
const known = [mcHostValue, this.rootPassword, encodeURIComponent(this.rootPassword), ...secretArgs(argv)];
const e = err as { stderr?: unknown; message?: string };
const reason = typeof e.stderr === "string" && e.stderr.trim() !== "" ? e.stderr.trim() : (e.message ?? String(err));
const shown = argv.filter((a) => a !== this.mcConfigDir).join(" ");
throw new Error(redactSecrets(`mc ${shown} failed: ${reason}`, known));
}
}
// --- SigV4 request plumbing ---------------------------------------------
@@ -360,6 +371,26 @@ function uriEncode(str: string, encodeSlash: boolean): string {
return out;
}
/** The values that follow --secret-key and --access-key in an argument list. */
function secretArgs(argv: string[]): string[] {
const out: string[] = [];
argv.forEach((a, i) => { if ((a === "--secret-key" || a === "--access-key") && i + 1 < argv.length) out.push(argv[i + 1]); });
return out;
}
/**
* The one redaction every `mc` failure goes through (issue 1766): the token after --secret-key or
* --access-key (as a separate word or after "="), any MC_HOST_ value, and each known secret, wherever they
* appear in the text, become "[redacted]".
*/
export function redactSecrets(text: string, known: string[] = []): string {
let out = text
.replace(/(--(?:secret|access)-key(?:=|\s+))('[^']*'|"[^"]*"|\S+)/g, "$1[redacted]")
.replace(/(MC_HOST_\w+=)\S+/g, "$1[redacted]");
for (const k of known) if (k !== "") out = out.split(k).join("[redacted]");
return out;
}
/**
* The MC_HOST_<alias> URL `mc` reads an alias from: the endpoint with the credentials in its userinfo,
* each percent-encoded so a reserved character in either cannot end it early.
+7 -1
View File
@@ -36,6 +36,9 @@ export function linkPolicy(bucket: string): string {
});
}
/** A link key's secret: 40 hex characters, minio's limit for a secret key (issue 1766). */
const LINK_SECRET = /^[0-9a-f]{40}$/;
export function linkSecretPath(dir: string, bucket: string): string {
return join(dir, `${bucket}.secret`);
}
@@ -47,9 +50,12 @@ export async function ensureLinkKey(admin: LinkAdmin, dir: string, bucket: strin
const path = linkSecretPath(dir, bucket);
mkdirSync(dir, { recursive: true, mode: 0o700 });
let secret = existsSync(path) ? readFileSync(path, "utf8").trim() : "";
// A stored secret of the wrong shape (an earlier version made 64 characters; minio takes at most 40) is
// replaced, and its key made again (issue 1766).
if (!LINK_SECRET.test(secret)) secret = "";
if (secret !== "" && (await admin.accessKeyExists(id))) return;
if (secret === "") {
secret = randomBytes(32).toString("hex");
secret = randomBytes(20).toString("hex");
writeFileSync(path, secret + "\n", { mode: 0o600 });
chmodSync(path, 0o600);
}
+55
View File
@@ -115,3 +115,58 @@ test("the link key id is the one the object-store tool derives (cmd/object-store
// 1516/AC1
assert.equal(linkKeyId("brother"), "link0b4bd77cec705cf5");
});
// Issue 1766: a link key's secret is minio's length, and no mc failure carries an argument's secret.
import { chmodSync, writeFileSync, mkdirSync } from "node:fs";
import { MinioClient } from "../client.ts";
test("a link key's secret is 40 characters, and a stored 64-character secret is replaced", async () => {
// 1766/AC1
const { minio, dir } = setup();
await ensureLinkKey(minio, dir, "fresh");
assert.equal(minio.keys.get(linkKeyId("fresh"))!.secret.length, 40);
assert.match(minio.keys.get(linkKeyId("fresh"))!.secret, /^[0-9a-f]{40}$/);
mkdirSync(dir, { recursive: true });
const old = "ab".repeat(32);
writeFileSync(linkSecretPath(dir, "old"), old + "\n");
await ensureLinkKey(minio, dir, "old");
const made = minio.keys.get(linkKeyId("old"))!.secret;
assert.equal(made.length, 40);
assert.notEqual(made, old);
assert.equal(readFileSync(linkSecretPath(dir, "old"), "utf8").trim(), made);
});
function failingMc(): { client: MinioClient; root: string } {
const d = mkdtempSync(join(process.env.TMPDIR ?? tmpdir(), "minio-mc-"));
const bin = join(d, "mc");
// Says its arguments and its alias on stderr, then fails, the way a refused command does.
writeFileSync(bin, '#!/bin/sh\necho "args: $*" >&2\necho "env: MC_HOST_mesh=$MC_HOST_mesh" >&2\nexit 1\n');
chmodSync(bin, 0o755);
const root = "RootPass-s3cret/ok";
const client = new MinioClient({ endpoint: "http://127.0.0.1:9000", rootUser: "root", rootPassword: root, region: "us-east-1", mcBin: bin, mcConfigDir: d });
return { client, root };
}
test("an mc failure's message holds no --secret-key value and no MC_HOST value", async () => {
// 1766/AC2
const { client, root } = failingMc();
const secret = "SuperSecretValue0123456789";
const err: any = await client.createAccessKey("b", "ACCESSKEYID1234567", secret).then(() => null, (e) => e);
assert.ok(err, "the failure is raised");
for (const text of [String(err.message), String(err.stack ?? ""), JSON.stringify(Object.getOwnPropertyNames(err).map((k) => err[k]))]) {
assert.ok(!text.includes(secret), "no secret key");
assert.ok(!text.includes("ACCESSKEYID1234567"), "no access key");
assert.ok(!text.includes(root) && !text.includes(encodeURIComponent(root)), "no root password");
assert.ok(!text.includes("root:"), "no MC_HOST value");
}
assert.match(err.message, /\[redacted\]/);
});
test("a consumer key's create passes the minted secret unchanged", async () => {
// 1766/AC3
const s = setup();
await s.h.create(provision);
assert.equal(s.minio.keys.get("brother-key")!.secret, "consumer-secret");
});
+12 -5
View File
@@ -21,17 +21,24 @@ They are those both workstations ran, with three changes:
- **scrollback:** `saveLines` 10000 instead of 2000.
The palette (background, foreground and the sixteen colours) is today's, unchanged. So are the
clipboard keys: Ctrl+Shift+C copies to the clipboard and Ctrl+Shift+V pastes from it. The bundle's
`Resources()` is the one source of these values. The manifest test holds the manifest to it, and
`xterm_colours` compares against it.
clipboard keys: Ctrl+Shift+C copies to the clipboard and Ctrl+Shift+V pastes from it.
Every one of these values is a **config value** (ADR 0348; issue 1656) with today's value as its default:
`face`, `size` (the predefined `font-size`, 6 to 72), `save-lines`, `term-name`, `render-font`, `bold-mode`,
`always-bold-mode`, `allow-bold-fonts`, `background`, `foreground`, `color0` to `color15` (each `#rrggbb`),
and the two clipboard bindings `copy-binding` and `paste-binding`. `files/xresources-normal` holds only
`${config:<key>}` placeholders, so a machine changes its font size through mesh-config. The manifest test holds
the file to the keys, and the file filled with the defaults to the text every machine had before.
The tools keep no copy of the values: `xterm_font` and `xterm_colours` read the `XTerm*` lines of the rendered
resources file `xorg` writes (`~/.config/xorg/xresources`) as the module's value, and say so when it is not there.
## Tools
| tool | | what |
|---|---|---|
| `node-terminal-emulator.open` | d | a terminal in the session: the login shell, or a command run through it (`$SHELL -lc`), in a directory (the home by default), with an optional title, and `hold` to keep it open. It starts as a transient unit of the account's service manager (`systemd-run --user`), so it outlives the call and every runtime restart |
| `xterm_font` | r/a | the face and size in force and the installed font they resolve to (`fc-match`); set face or size for terminals opened from now on. Only installed faces are accepted |
| `xterm_colours` | r | background, foreground, cursor and the sixteen colours as `#rrggbb`, each beside the module's default |
| `xterm_font` | r/a | the face and size in force beside the module's value (from the rendered resources), and the installed font they resolve to (`fc-match`); set face or size for terminals opened from now on. Only installed faces are accepted |
| `xterm_colours` | r | background, foreground, cursor and the sixteen colours as `#rrggbb`, each beside the module's value from the rendered resources |
## What it leaves found
@@ -0,0 +1,274 @@
package main
import (
"context"
"os"
"path/filepath"
"regexp"
"strconv"
"strings"
"testing"
"xterm/internal/desktop"
)
// todaysResources is the file every machine had before the module took config values (novox/hq issue 1656).
var todaysResources = strings.Join([]string{
"! xterm: the face, the palette and the clipboard keys. Scoped to the XTerm class, so they",
"! outrank any `*name` resource of yours and touch no other X program.",
"XTerm*termName: xterm-256color",
"XTerm*saveLines: 10000",
"XTerm*renderFont: true",
"XTerm*faceName: JetBrainsMono Nerd Font",
"XTerm*faceSize: 12",
"XTerm*boldMode: false",
"XTerm*alwaysBoldMode: false",
"XTerm*allowBoldFonts: true",
"XTerm*background: #000000",
"XTerm*foreground: #a8a8a8",
"XTerm*color0: #000000",
"XTerm*color8: #666666",
"XTerm*color1: #9e1828",
"XTerm*color9: #bc5766",
"XTerm*color2: #008800",
"XTerm*color10: #61a171",
"XTerm*color3: #d2bb4b",
"XTerm*color11: #e7db52",
"XTerm*color4: #414171",
"XTerm*color12: #5085af",
"XTerm*color5: #963c59",
"XTerm*color13: #a97a99",
"XTerm*color6: #418179",
"XTerm*color14: #6ba4a4",
"XTerm*color7: #bebebe",
"XTerm*color15: #ffffff",
"XTerm*VT100.Translations: #override \\n\\",
" Ctrl Shift ~Alt <Key> C: copy-selection(CLIPBOARD) \\n\\",
" Ctrl Shift ~Alt <Key> V: insert-selection(CLIPBOARD)",
"",
}, "\n")
var placeholder = regexp.MustCompile(`\$\{config:([a-z0-9][a-z0-9_.-]*)\}`)
// asText is a default as the controller writes a value into a file: a string as it is, a number without a
// needless fraction, a boolean as true or false.
func asText(t *testing.T, v any) string {
t.Helper()
switch v := v.(type) {
case string:
return v
case float64:
return strconv.FormatFloat(v, 'f', -1, 64)
case bool:
return strconv.FormatBool(v)
}
t.Fatalf("a default of no type: %v", v)
return ""
}
func template(t *testing.T) string {
t.Helper()
m := readManifest(t)
if len(m.Shell) != 1 {
t.Fatalf("%+v", m.Shell)
}
return m.Shell[0].Code
}
// 1656/AC1: every value the resources set is a config value with today's value as its default.
func TestEveryResourceValueIsAConfigValueWithTodaysDefault(t *testing.T) {
m := readManifest(t)
want := map[string]any{
"face": "JetBrainsMono Nerd Font", "size": 12.0, "save-lines": 10000.0, "term-name": "xterm-256color",
"render-font": true, "bold-mode": false, "always-bold-mode": false, "allow-bold-fonts": true,
"background": "#000000", "foreground": "#a8a8a8",
"color0": "#000000", "color1": "#9e1828", "color2": "#008800", "color3": "#d2bb4b",
"color4": "#414171", "color5": "#963c59", "color6": "#418179", "color7": "#bebebe",
"color8": "#666666", "color9": "#bc5766", "color10": "#61a171", "color11": "#e7db52",
"color12": "#5085af", "color13": "#a97a99", "color14": "#6ba4a4", "color15": "#ffffff",
"copy-binding": "Ctrl Shift ~Alt <Key> C", "paste-binding": "Ctrl Shift ~Alt <Key> V",
}
if len(m.Config) != len(want) {
t.Errorf("declares %d config values, want %d", len(m.Config), len(want))
}
for key, def := range want {
c, ok := m.Config[key]
if !ok {
t.Errorf("%s: not declared", key)
continue
}
if c.Default != def || c.Required || strings.TrimSpace(c.Description) == "" {
t.Errorf("%s: %+v, want default %v and a description", key, c, def)
}
wantType := map[bool]string{true: "string"}[true]
switch def.(type) {
case float64:
wantType = "number"
case bool:
wantType = "boolean"
}
if c.Type != wantType {
t.Errorf("%s: type %q, want %q", key, c.Type, wantType)
}
}
size := m.Config["size"]
if size.Predefined != "font-size" || size.Minimum == nil || *size.Minimum != 6 || size.Maximum == nil || *size.Maximum != 72 {
t.Errorf("size: %+v, want the predefined font-size from 6 to 72", size)
}
names := []string{"background", "foreground"}
for i := 0; i < 16; i++ {
names = append(names, "color"+strconv.Itoa(i))
}
for _, key := range names {
c := m.Config[key]
re, err := regexp.Compile("^(?:" + c.Pattern + ")$")
if err != nil || c.Pattern == "" {
t.Errorf("%s: pattern %q", key, c.Pattern)
continue
}
if !re.MatchString("#0a1B2c") || re.MatchString("red") || re.MatchString("#12345") || re.MatchString("#1234567") {
t.Errorf("%s: pattern %q is not #rrggbb", key, c.Pattern)
}
}
for _, key := range []string{"copy-binding", "paste-binding"} {
re := regexp.MustCompile("^(?:" + m.Config[key].Pattern + ")$")
if !re.MatchString("Ctrl Shift ~Alt <Key> C") || re.MatchString("a: b") || re.MatchString("a\nb") || re.MatchString(`a\`) {
t.Errorf("%s: pattern %q lets a value end the resource", key, m.Config[key].Pattern)
}
}
}
// 1656/AC2: no literal value is left in the resources, and every declared key is used.
func TestEveryResourceValueIsAPlaceholderOfADeclaredKeyAndEveryKeyIsUsed(t *testing.T) {
m := readManifest(t)
text := template(t)
used := map[string]bool{}
for _, l := range strings.Split(text, "\n") {
switch {
case l == "", strings.HasPrefix(l, "!"):
continue
case strings.HasPrefix(l, "XTerm*VT100.Translations: #override"):
continue
}
_, value, ok := strings.Cut(l, ": ")
if strings.HasPrefix(l, " ") {
// a translation line: `<binding>: <action> \n\`
value, _, _ = strings.Cut(strings.TrimSpace(l), ": ")
ok = true
}
if !ok {
t.Errorf("not a resource: %q", l)
continue
}
f := placeholder.FindStringSubmatch(value)
if f == nil || f[0] != value {
t.Errorf("a literal value, not a ${config:<key>}: %q", l)
continue
}
if _, declared := m.Config[f[1]]; !declared {
t.Errorf("%q names a key the module does not declare", l)
}
used[f[1]] = true
}
for key := range m.Config {
if !used[key] {
t.Errorf("%s is declared and no resource uses it", key)
}
}
}
// 1656/AC3: filled with the defaults, the resources are the file every machine had.
func TestFilledWithTheDefaultsTheResourcesAreTodaysFileByteForByte(t *testing.T) {
m := readManifest(t)
got := placeholder.ReplaceAllStringFunc(template(t), func(ref string) string {
c, ok := m.Config[placeholder.FindStringSubmatch(ref)[1]]
if !ok {
t.Fatalf("%s: no such key", ref)
}
return asText(t, c.Default)
})
if got != todaysResources {
t.Fatalf("filled with the defaults:\n%s\nwant:\n%s", got, todaysResources)
}
}
// renderedWith is the resources file xorg writes with this module's piece filled from the defaults, and
// the given values changed.
func renderedWith(t *testing.T, changed map[string]string) string {
t.Helper()
m := readManifest(t)
text := placeholder.ReplaceAllStringFunc(template(t), func(ref string) string {
key := placeholder.FindStringSubmatch(ref)[1]
if v, ok := changed[key]; ok {
return v
}
return asText(t, m.Config[key].Default)
})
return "! The mesh's X resources (module xorg)\nXft.dpi: 96\n# xterm\n" + text + "Xft.rgba: rgb\n"
}
func deskWithResources(t *testing.T, rendered string, f *fake) xterm {
t.Helper()
path := filepath.Join(t.TempDir(), "xresources")
if rendered != "" {
if err := os.WriteFile(path, []byte(rendered), 0o644); err != nil {
t.Fatal(err)
}
}
return xterm{d: f.desk(true), resources: path}
}
// 1656/AC4: the tools' defaults are what the module's resources were rendered with, not constants.
func TestFontReportsTheDefaultTheModuleRenderedNotAConstant(t *testing.T) {
f := &fake{query: "XTerm*faceName:\tInter\nXTerm*faceSize:\t13\n"}
x := deskWithResources(t, renderedWith(t, map[string]string{"size": "10", "face": "Fira Code"}), f)
got, err := x.font(context.Background(), desktop.Args{})
if err != nil {
t.Fatal(err)
}
if j := asJSON(got); !strings.Contains(j, `"default":{"face":"Fira Code","size":10}`) {
t.Fatalf("%s", j)
}
}
// 1656/AC4
func TestColoursReportTheDefaultTheModuleRenderedNotAConstant(t *testing.T) {
f := &fake{query: "XTerm*color1:\t#aa0000\nXTerm*foreground:\t#112233\nXTerm*color2:\trgb:00/88/01\n"}
x := deskWithResources(t, renderedWith(t, map[string]string{"color1": "#aa0000", "foreground": "#112233"}), f)
got, err := x.colours(context.Background(), desktop.Args{})
if err != nil {
t.Fatal(err)
}
j := asJSON(got)
for _, want := range []string{`{"name":"color1","value":"#aa0000","default":"#aa0000"}`,
`{"name":"foreground","value":"#112233","default":"#112233"}`,
`{"name":"color2","value":"#008801","default":"#008800","differs":true}`} {
if !strings.Contains(j, want) {
t.Errorf("lacks %s in %s", want, j)
}
}
}
// 1656/AC4: without the rendered file there is no default to report, said loudly.
func TestWithoutTheRenderedResourcesTheToolsSayWhy(t *testing.T) {
x := deskWithResources(t, "", &fake{})
if _, err := x.font(context.Background(), desktop.Args{}); err == nil || !strings.Contains(err.Error(), "xresources") {
t.Fatalf("font: %v", err)
}
if _, err := x.colours(context.Background(), desktop.Args{}); err == nil || !strings.Contains(err.Error(), "xresources") {
t.Fatalf("colours: %v", err)
}
}
// 1656/AC4: tools.go holds no copy of the defaults.
func TestTheToolsHoldNoCopyOfTheDefaults(t *testing.T) {
src, err := os.ReadFile("tools.go")
if err != nil {
t.Fatal(err)
}
for _, lit := range []string{"JetBrainsMono", "#a8a8a8", "#9e1828", "xterm-256color", "Face =", "Size =", "func Resources"} {
if strings.Contains(string(src), lit) {
t.Errorf("tools.go holds %q", lit)
}
}
}
+3 -2
View File
@@ -9,6 +9,7 @@ import (
"context"
"fmt"
"os"
"path/filepath"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
@@ -17,7 +18,7 @@ import (
)
func main() {
if err := stdio.Serve("", tools(xterm{d: desktop.Machine("i3", "sway")})); err != nil {
if err := stdio.Serve("", tools(xterm{d: desktop.Machine("i3", "sway"), resources: filepath.Join(desktop.Home(), ResourcesFile)})); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
@@ -53,7 +54,7 @@ func tools(x xterm) []stdio.Tool {
"resources set the default again at the next login.",
Input: desktop.Schema(map[string]any{
"face": desktop.Str("a font family, as fc-list names it"),
"size": desktop.Num("points, 6 to 48"),
"size": desktop.Num("points, 6 to 72"),
}),
Run: call(x.font),
},
+59 -44
View File
@@ -11,47 +11,50 @@ import (
"xterm/internal/desktop"
)
type xterm struct{ d desktop.Desk }
// The module's defaults: the face research 026/04 chose and both workstations' palette.
const (
Face = "JetBrainsMono Nerd Font"
Size = 12
)
// Palette is the colours both workstations ran, by resource name, in the order they are written.
var Palette = [][2]string{
{"background", "#000000"}, {"foreground", "#a8a8a8"},
{"color0", "#000000"}, {"color8", "#666666"},
{"color1", "#9e1828"}, {"color9", "#bc5766"},
{"color2", "#008800"}, {"color10", "#61a171"},
{"color3", "#d2bb4b"}, {"color11", "#e7db52"},
{"color4", "#414171"}, {"color12", "#5085af"},
{"color5", "#963c59"}, {"color13", "#a97a99"},
{"color6", "#418179"}, {"color14", "#6ba4a4"},
{"color7", "#bebebe"}, {"color15", "#ffffff"},
type xterm struct {
d desktop.Desk
resources string
}
// Resources is the module's contribution to the X resources (the xresources slot `normal`), as the
// manifest carries it; the manifest test holds the two to one text.
func Resources() string {
var b strings.Builder
b.WriteString("! xterm: the face, the palette and the clipboard keys. Scoped to the XTerm class, so they\n")
b.WriteString("! outrank any `*name` resource of yours and touch no other X program.\n")
for _, kv := range [][2]string{
{"termName", "xterm-256color"}, {"saveLines", "10000"}, {"renderFont", "true"},
{"faceName", Face}, {"faceSize", strconv.Itoa(Size)},
{"boldMode", "false"}, {"alwaysBoldMode", "false"}, {"allowBoldFonts", "true"},
} {
fmt.Fprintf(&b, "XTerm*%s: %s\n", kv[0], kv[1])
// PaletteNames are the colour resources xterm_colours reports, in the order the resources write them.
// Their values are the module's config values, read from the rendered resources (Rendered), never held here.
var PaletteNames = []string{
"background", "foreground",
"color0", "color8", "color1", "color9", "color2", "color10", "color3", "color11",
"color4", "color12", "color5", "color13", "color6", "color14", "color7", "color15",
}
// ResourcesFile is where xorg writes the X resources, under the account's home: this module's resources are
// its contribution to the `normal` slot, filled with the module's config values.
const ResourcesFile = ".config/xorg/xresources"
// Rendered is the XTerm* resources of a rendered resources file, by name: `XTerm*faceSize: 12` is faceSize 12.
// The translations, which span lines, and every resource of another program are left out.
func Rendered(text string) map[string]string {
out := map[string]string{}
for _, l := range strings.Split(text, "\n") {
k, v, ok := strings.Cut(l, ":")
name, scoped := strings.CutPrefix(strings.TrimSpace(k), "XTerm*")
if ok && scoped && !strings.Contains(name, ".") {
out[name] = strings.TrimSpace(v)
}
}
for _, kv := range Palette {
fmt.Fprintf(&b, "XTerm*%s: %s\n", kv[0], kv[1])
return out
}
// defaults are the resources the module's config values rendered, as xorg placed them: what the module sets, so
// what a terminal has unless the session's own resources say otherwise.
func (x xterm) defaults() (map[string]string, error) {
text, err := os.ReadFile(x.resources)
if err != nil {
return nil, fmt.Errorf("the module's resources are not where xorg places them (%s): is xorg assigned to this machine, "+
"and has its xresources file been written? %w", x.resources, err)
}
b.WriteString("XTerm*VT100.Translations: #override \\n\\\n")
b.WriteString(" Ctrl Shift ~Alt <Key> C: copy-selection(CLIPBOARD) \\n\\\n")
b.WriteString(" Ctrl Shift ~Alt <Key> V: insert-selection(CLIPBOARD)\n")
return b.String()
got := Rendered(string(text))
if len(got) == 0 {
return nil, fmt.Errorf("%s holds no XTerm* resource: the xterm module's resources are not in it", x.resources)
}
return got, nil
}
// openScript runs in the new terminal: into the directory, then the login shell, or a command
@@ -124,6 +127,10 @@ func (x xterm) font(ctx context.Context, a desktop.Args) (any, error) {
if err != nil {
return nil, err
}
def, err := x.defaults()
if err != nil {
return nil, err
}
env := s.Env(x.d.Base)
face := a.Opt("face", "")
size, sized, err := a.Number("size")
@@ -142,8 +149,8 @@ func (x xterm) font(ctx context.Context, a desktop.Args) (any, error) {
res = append(res, "XTerm*faceName: "+face)
}
if sized {
if size < 6 || size > 48 {
return nil, fmt.Errorf("size is from 6 to 48 points")
if size < 6 || size > 72 {
return nil, fmt.Errorf("size is from 6 to 72 points")
}
res = append(res, "XTerm*faceSize: "+strconv.FormatFloat(size, 'f', -1, 64))
}
@@ -156,9 +163,13 @@ func (x xterm) font(ctx context.Context, a desktop.Args) (any, error) {
return nil, q.Err()
}
inForce := Effective(q.Stdout, "faceName")
var points any = def["faceSize"]
if n, err := strconv.ParseFloat(def["faceSize"], 64); err == nil {
points = n
}
answer := map[string]any{
"session": s.Display, "face": inForce, "size": Effective(q.Stdout, "faceSize"),
"default": map[string]any{"face": Face, "size": Size},
"default": map[string]any{"face": def["faceName"], "size": points},
}
if inForce != "" {
m := x.d.Run(ctx, env, nil, "fc-match", "-f", "%{family[0]}|%{style[0]}|%{file}", inForce)
@@ -175,7 +186,7 @@ func (x xterm) font(ctx context.Context, a desktop.Args) (any, error) {
var rgbSpec = regexp.MustCompile(`^rgb:([0-9a-fA-F]{1,4})/([0-9a-fA-F]{1,4})/([0-9a-fA-F]{1,4})$`)
// Hex writes an X colour (`rgb:9e/18/28`, `#9e1828`) as #rrggbb; anything else (a colour name) as it is.
// Hex writes an X colour (`rgb:9e/18/28`, `#rrggbb`) as #rrggbb; anything else (a colour name) as it is.
func Hex(v string) string {
v = strings.TrimSpace(v)
if m := rgbSpec.FindStringSubmatch(v); m != nil {
@@ -191,6 +202,10 @@ func Hex(v string) string {
}
func (x xterm) colours(ctx context.Context, a desktop.Args) (any, error) {
def, err := x.defaults()
if err != nil {
return nil, err
}
res, s, err := x.d.InSession(ctx, "xrdb", "-query")
if err != nil {
return nil, err
@@ -205,9 +220,9 @@ func (x xterm) colours(ctx context.Context, a desktop.Args) (any, error) {
Differs bool `json:"differs,omitempty"`
}
var out []colour
for _, kv := range Palette {
v := Hex(Effective(res.Stdout, kv[0]))
out = append(out, colour{kv[0], v, kv[1], v != kv[1]})
for _, name := range PaletteNames {
v, d := Hex(Effective(res.Stdout, name)), Hex(def[name])
out = append(out, colour{name, v, d, v != d})
}
cursor := Hex(Effective(res.Stdout, "cursorColor"))
return map[string]any{"session": s.Display, "colours": out, "cursor": cursor}, nil
+26 -11
View File
@@ -19,7 +19,8 @@ type manifest struct {
Name string `json:"name"`
Serves []string `json:"serves"`
} `json:"claims"`
Tools []string `json:"tools"`
Tools []string `json:"tools"`
Config map[string]configKey `json:"config"`
Environment struct {
Variables map[string]string `json:"variables"`
} `json:"environment"`
@@ -37,6 +38,18 @@ type manifest struct {
} `json:"build"`
}
type configKey struct {
Type string `json:"type"`
Description string `json:"description"`
Pattern string `json:"pattern"`
Minimum *float64 `json:"minimum"`
Maximum *float64 `json:"maximum"`
Predefined string `json:"predefined"`
Default any `json:"default"`
Required bool `json:"required"`
Trusted bool `json:"trusted"`
}
func readManifest(t *testing.T) manifest {
raw, err := os.ReadFile(filepath.Join("..", "..", "module.json"))
if err != nil {
@@ -99,10 +112,8 @@ func TestItsResourcesAreOneContributionInTheNormalSlotScopedToXTerm(t *testing.T
if len(m.Shell) != 1 || m.Shell[0].For != "xresources" || m.Shell[0].Slot != "normal" {
t.Fatalf("%+v", m.Shell)
}
if m.Shell[0].Code != Resources() {
t.Fatal("the manifest's resources are not the ones the bundle describes: regenerate the manifest from Resources()")
}
for _, l := range strings.Split(strings.TrimSpace(Resources()), "\n") {
code := m.Shell[0].Code
for _, l := range strings.Split(strings.TrimSpace(code), "\n") {
switch {
case strings.HasPrefix(l, "!"), strings.HasPrefix(l, " "):
case strings.HasPrefix(l, "#"):
@@ -111,8 +122,8 @@ func TestItsResourcesAreOneContributionInTheNormalSlotScopedToXTerm(t *testing.T
t.Errorf("a resource of another program, or of every one: %q", l)
}
}
if !strings.Contains(Resources(), "XTerm*faceName: JetBrainsMono Nerd Font") || !strings.Contains(Resources(), "copy-selection(CLIPBOARD)") {
t.Fatal("the chosen face and the clipboard keys")
if !strings.Contains(code, "XTerm*faceName: ${config:face}") || !strings.Contains(code, "copy-selection(CLIPBOARD)") {
t.Fatal("the face and the clipboard keys")
}
}
@@ -202,7 +213,11 @@ func TestFontAndColoursAreReadAsAnXTermSeesThem(t *testing.T) {
}
}
f := &fake{query: q}
got, err := xterm{d: f.desk(true)}.colours(context.Background(), desktop.Args{})
rendered := filepath.Join(t.TempDir(), "xresources")
if err := os.WriteFile(rendered, []byte(renderedWith(t, nil)), 0o644); err != nil {
t.Fatal(err)
}
got, err := xterm{d: f.desk(true), resources: rendered}.colours(context.Background(), desktop.Args{})
if err != nil {
t.Fatal(err)
}
@@ -210,14 +225,14 @@ func TestFontAndColoursAreReadAsAnXTermSeesThem(t *testing.T) {
if !strings.Contains(j, `{"name":"color1","value":"#ff0000","default":"#9e1828","differs":true}`) || !strings.Contains(j, `{"name":"foreground","value":"#a8a8a8","default":"#a8a8a8"}`) {
t.Fatalf("%s", j)
}
if _, err := (xterm{d: f.desk(true)}).font(context.Background(), desktop.Args{"face": "Missing Font"}); err == nil {
if _, err := (xterm{d: f.desk(true), resources: rendered}).font(context.Background(), desktop.Args{"face": "Missing Font"}); err == nil {
t.Fatal("a face that is not installed")
}
if _, err := (xterm{d: f.desk(true)}).font(context.Background(), desktop.Args{"size": 99.0}); err == nil {
if _, err := (xterm{d: f.desk(true), resources: rendered}).font(context.Background(), desktop.Args{"size": 99.0}); err == nil {
t.Fatal("size out of range")
}
f.ran, f.stdin = nil, nil
got, err = xterm{d: f.desk(true)}.font(context.Background(), desktop.Args{"face": "Inter", "size": 13.0})
got, err = xterm{d: f.desk(true), resources: rendered}.font(context.Background(), desktop.Args{"face": "Inter", "size": 13.0})
if err != nil {
t.Fatal(err)
}
+28 -28
View File
@@ -1,31 +1,31 @@
! xterm: the face, the palette and the clipboard keys. Scoped to the XTerm class, so they
! outrank any `*name` resource of yours and touch no other X program.
XTerm*termName: xterm-256color
XTerm*saveLines: 10000
XTerm*renderFont: true
XTerm*faceName: JetBrainsMono Nerd Font
XTerm*faceSize: 12
XTerm*boldMode: false
XTerm*alwaysBoldMode: false
XTerm*allowBoldFonts: true
XTerm*background: #000000
XTerm*foreground: #a8a8a8
XTerm*color0: #000000
XTerm*color8: #666666
XTerm*color1: #9e1828
XTerm*color9: #bc5766
XTerm*color2: #008800
XTerm*color10: #61a171
XTerm*color3: #d2bb4b
XTerm*color11: #e7db52
XTerm*color4: #414171
XTerm*color12: #5085af
XTerm*color5: #963c59
XTerm*color13: #a97a99
XTerm*color6: #418179
XTerm*color14: #6ba4a4
XTerm*color7: #bebebe
XTerm*color15: #ffffff
XTerm*termName: ${config:term-name}
XTerm*saveLines: ${config:save-lines}
XTerm*renderFont: ${config:render-font}
XTerm*faceName: ${config:face}
XTerm*faceSize: ${config:size}
XTerm*boldMode: ${config:bold-mode}
XTerm*alwaysBoldMode: ${config:always-bold-mode}
XTerm*allowBoldFonts: ${config:allow-bold-fonts}
XTerm*background: ${config:background}
XTerm*foreground: ${config:foreground}
XTerm*color0: ${config:color0}
XTerm*color8: ${config:color8}
XTerm*color1: ${config:color1}
XTerm*color9: ${config:color9}
XTerm*color2: ${config:color2}
XTerm*color10: ${config:color10}
XTerm*color3: ${config:color3}
XTerm*color11: ${config:color11}
XTerm*color4: ${config:color4}
XTerm*color12: ${config:color12}
XTerm*color5: ${config:color5}
XTerm*color13: ${config:color13}
XTerm*color6: ${config:color6}
XTerm*color14: ${config:color14}
XTerm*color7: ${config:color7}
XTerm*color15: ${config:color15}
XTerm*VT100.Translations: #override \n\
Ctrl Shift ~Alt <Key> C: copy-selection(CLIPBOARD) \n\
Ctrl Shift ~Alt <Key> V: insert-selection(CLIPBOARD)
${config:copy-binding}: copy-selection(CLIPBOARD) \n\
${config:paste-binding}: insert-selection(CLIPBOARD)
+169
View File
@@ -21,6 +21,175 @@
"xterm_font",
"xterm_colours"
],
"config": {
"face": {
"type": "string",
"description": "The font family xterm draws with, as fc-list names it: JetBrainsMono Nerd Font, because research 026/04 chose it for its glyphs the prompt uses. A family the machine does not have falls back to the nearest installed face (xterm_font says which).",
"pattern": "[A-Za-z0-9 ._-]{1,64}",
"default": "JetBrainsMono Nerd Font"
},
"size": {
"type": "number",
"description": "The size of xterm's text, in points: 12 reads well on a screen of about 100 DPI, and a denser screen needs a larger size.",
"minimum": 6,
"maximum": 72,
"predefined": "font-size",
"default": 12
},
"save-lines": {
"type": "number",
"description": "How many lines of scrollback xterm keeps per terminal: 10000 holds a long build log, and each line costs memory in every open terminal.",
"minimum": 0,
"maximum": 1000000,
"default": 10000
},
"term-name": {
"type": "string",
"description": "The TERM value xterm sets for the programs it runs: xterm-256color, because the palette has 256 colours and every program knows the name.",
"pattern": "[A-Za-z0-9._-]{1,64}",
"default": "xterm-256color"
},
"render-font": {
"type": "boolean",
"description": "Whether xterm draws with fontconfig faces (true) or the old X core fonts (false): true, because the face is a fontconfig family and core fonts cannot show it.",
"default": true
},
"bold-mode": {
"type": "boolean",
"description": "Whether xterm draws bold text by overstriking the normal face (true): false, because the face has a real bold style and overstriking blurs it.",
"default": false
},
"always-bold-mode": {
"type": "boolean",
"description": "Whether xterm overstrikes bold even where the face has a bold style (true): false, for the same reason as bold-mode.",
"default": false
},
"allow-bold-fonts": {
"type": "boolean",
"description": "Whether xterm may use the face's bold style for bold text (true): true, so bold text is drawn with the real bold weight.",
"default": true
},
"background": {
"type": "string",
"description": "The terminal's background colour, as #rrggbb: black, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#000000"
},
"foreground": {
"type": "string",
"description": "The terminal's text colour, as #rrggbb: a light grey, easier on the eyes than white on black.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#a8a8a8"
},
"color0": {
"type": "string",
"description": "Palette entry 0, the normal black, as #rrggbb: #000000, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#000000"
},
"color1": {
"type": "string",
"description": "Palette entry 1, the normal red, as #rrggbb: #9e1828, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#9e1828"
},
"color2": {
"type": "string",
"description": "Palette entry 2, the normal green, as #rrggbb: #008800, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#008800"
},
"color3": {
"type": "string",
"description": "Palette entry 3, the normal yellow, as #rrggbb: #d2bb4b, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#d2bb4b"
},
"color4": {
"type": "string",
"description": "Palette entry 4, the normal blue, as #rrggbb: #414171, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#414171"
},
"color5": {
"type": "string",
"description": "Palette entry 5, the normal magenta, as #rrggbb: #963c59, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#963c59"
},
"color6": {
"type": "string",
"description": "Palette entry 6, the normal cyan, as #rrggbb: #418179, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#418179"
},
"color7": {
"type": "string",
"description": "Palette entry 7, the normal white, as #rrggbb: #bebebe, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#bebebe"
},
"color8": {
"type": "string",
"description": "Palette entry 8, the bright black, as #rrggbb: #666666, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#666666"
},
"color9": {
"type": "string",
"description": "Palette entry 9, the bright red, as #rrggbb: #bc5766, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#bc5766"
},
"color10": {
"type": "string",
"description": "Palette entry 10, the bright green, as #rrggbb: #61a171, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#61a171"
},
"color11": {
"type": "string",
"description": "Palette entry 11, the bright yellow, as #rrggbb: #e7db52, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#e7db52"
},
"color12": {
"type": "string",
"description": "Palette entry 12, the bright blue, as #rrggbb: #5085af, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#5085af"
},
"color13": {
"type": "string",
"description": "Palette entry 13, the bright magenta, as #rrggbb: #a97a99, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#a97a99"
},
"color14": {
"type": "string",
"description": "Palette entry 14, the bright cyan, as #rrggbb: #6ba4a4, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#6ba4a4"
},
"color15": {
"type": "string",
"description": "Palette entry 15, the bright white, as #rrggbb: #ffffff, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#ffffff"
},
"copy-binding": {
"type": "string",
"description": "The key combination that copies the selection to the clipboard, in X translation syntax: Ctrl Shift ~Alt <Key> C, so a plain Ctrl+C still interrupts the program. It holds no colon, backslash or line break, which would end the resource.",
"pattern": "[^:\\\\\\n]+",
"default": "Ctrl Shift ~Alt <Key> C"
},
"paste-binding": {
"type": "string",
"description": "The key combination that pastes the clipboard, in X translation syntax: Ctrl Shift ~Alt <Key> V, the pair of copy-binding. It holds no colon, backslash or line break, which would end the resource.",
"pattern": "[^:\\\\\\n]+",
"default": "Ctrl Shift ~Alt <Key> V"
}
},
"items": [
{
"collection": "environment",