hq/04-ISSUES/242-the-mesh-has-no-backups/00-report.md

---

status: resolved opened: 2026-10-05 located-in: [mesh-catalog modules/restic (the node-backup holder) and every store module's contribution, mesh-controller internal/catalogue (the node-backup seat)] fixed-by: novox/mesh-catalog PR #49 (96ee0d7), novox/mesh-catalog PR #92 (f144f6e) replay-none: the fix is the backup holder and each store's contribution, not core logic, and the symptom was an absence: there was never a backup to replay. The seat's backed-up verb reports whether it holds. amended-design: 03-DESIGN/01-to-be/43-backups-against-mistakes.md

242 — The mesh has no backups

What was observed

When seven databases were dropped on 2026-10-04 (issue 241), the newest copy of any of them was a leftover of the migration: data directories and one dump, nine to twelve days old, found by searching the control node's disk. Nothing in the mesh takes a backup, no record says what should be backed up, and nothing would have said so until the day one was needed.

What survived did so by accident: git history because every repository is also cloned on the operator's machines; the file-sync service's files because they live in the object store, which nothing dropped; the mailboxes because they are files outside the database.

Scope, set by the operator (2026-10-05)

Mistakes, not disasters. Backups protect against a dropped database, a deleted bucket, a bad migration — not against a dead disk or a lost site; that loss is accepted. So no off-site copy is needed, and question 4's "spread across the mesh" and "encrypted to whom" fall away. Worked out in research 030; proposed as ADR 0214.

Questions this must answer

1. What is backed up — every store's databases (postgres, mssql, mongodb), the object store's buckets, the mail spool, the forge's repositories and data, the vault and the controller's own records, each module's state directories? Is it declared by the module that owns the data, the way a module declares its listens and its jails?
2. How often, and how long kept — a daily schedule? Rotation: how many daily, weekly, monthly?
3. Full or incremental — full dumps for databases, deltas (snapshots, deduplicating archives) for large object stores and file trees?
4. Where — never only on the machine whose disk it copies. Spread across the mesh (the home server holding the control node's, and the other way round), an external target, or both? Encrypted to whom?
5. Who runs it — a seat (mesh-backup?) whose holder schedules and stores, with the verbs a person needs: what was backed up and when, restore one database beside the live one?
6. How it is proven — a backup never restored is a hope. A scheduled restore of the newest copy into a throwaway instance, compared with the live one, and a failure that reaches the operator.

Why it matters beyond this incident

Issue 241's recovery cost a night and lost the forge's records of twelve days. With a nightly backup held on another machine, it would have been a ten-minute restore of yesterday.

Resolved — 2026-10-08

The mesh takes backups. ADR 0214 and to-be 43 answered the questions above: the backup module holds the node-backup seat on every machine, each store contributes its dumps (novox/mesh-catalog PR #49), and every module declares the data it holds, which the holder measures (ADR 0233, novox/mesh-catalog PR #92). Read live on 2026-10-08 through the seat's backed-up verb on the control node: every declared store and module — the forge, the mail server, the vault, the object store, the three database stores, the bus's snapshots and the rest — has a last good night of 2026-10-07 and up to four restore points. What is still being built (the scheduled restore drill of question 6) is to-be 43's, which stays in-progress; the symptom here, no backup at all, no longer holds.