jochen
7fcca88c01
Lessons from closing only-office's old directories: a directory keeps its id until emptied on the operator's word, then leaves the manifest in a second fix; and a fifth try is refused, not queued.
128 lines
7.2 KiB
Markdown
128 lines
7.2 KiB
Markdown
# Playbook 06 — Writing a module
|
|
|
|
**Trigger.** Something that runs today must run on the mesh, or a new capability must be
|
|
declarable.
|
|
|
|
**Who runs it.** Whoever is porting or writing it.
|
|
|
|
*Written 2026-09-01 from doing this for the first time end to end. Every step below exists
|
|
because skipping it cost something.*
|
|
|
|
## Before anything: read what runs
|
|
|
|
**A module is written from the thing, not from memory of the thing.** For a port, that means its
|
|
current compose file, its environment, and where its data actually sits. Assumptions about any of
|
|
the three have been wrong every time they were not checked.
|
|
|
|
Three questions, answered from the machine:
|
|
|
|
| | why it decides something |
|
|
|---|---|
|
|
| **what containers, and how do they find each other?** | more than one means a `network`; names between them must match what the software is configured to dial |
|
|
| **where is its data?** | a bind mount moves with a path; a named volume does not; an anonymous volume is already losing data on every redeploy |
|
|
| **which values are secret, and which are merely settings?** | a secret goes in `own-secrets` or a grant; a setting goes in the manifest and may be overridden per node |
|
|
|
|
## The steps
|
|
|
|
1. **Name what it provides and requires**, if anything. A name is what a consumer is coupled to,
|
|
not the role it plays (ADR 0027):
|
|
`postgres-database`, not `database`. Most modules provide nothing and require nothing — an
|
|
application is usually a leaf.
|
|
|
|
2. **Declare capabilities, not dependencies, for facts about the machine.** `container-runtime`,
|
|
`package-manager`, `seat`. A capability is detected and refused against; it is not something a
|
|
module can install.
|
|
|
|
3. **Write the resources in the order they must happen.** They are applied in the order written
|
|
and orphans are removed in reverse, so a `network` is written before the containers that join
|
|
it and removed after them.
|
|
|
|
4. **Put every secret in a file, never in `env`.** A declaration travels over the broker in plain
|
|
text: a password in `env` is a password the broker sees. **The mesh delivers parts; a module
|
|
that needs them combined combines them.**
|
|
|
|
**A sealed file holds the password and nothing else** — no key, no `=`, no newline that means
|
|
anything. So `env-file` must never point at one. It points at a file the module *declares*,
|
|
whose content leaves a hole:
|
|
|
|
```
|
|
own-secrets superuser → /var/lib/postgres/superuser.secret the password, alone
|
|
a file /var/lib/postgres/superuser.env, mode 0600,
|
|
content: POSTGRES_PASSWORD=${secret:superuser}
|
|
the container env-file: [/var/lib/postgres/superuser.env]
|
|
```
|
|
|
|
The node-engine fills the hole on the machine, which is the only place both halves exist — the mesh
|
|
discarded the value
|
|
([credentials and their rotation](../../03-DESIGN/01-to-be/13-credentials-and-their-rotation.md)).
|
|
A **provisioner** is the exception: it reads a password file, so it mounts the `.secret`
|
|
directly.
|
|
|
|
Every example module in `mesh-controller` had this wrong and shipped: `own-secrets` pointing at a
|
|
path *named* `.env`, mounted as `env-file`, holding a bare password. Docker reads that as a
|
|
malformed line and the container starts **with no password set at all** — not a failure to
|
|
start, a service running on the wrong credential. They parsed and they resolved. Two tests in
|
|
`examples/modules` now refuse both halves of it.
|
|
|
|
Add `restart-on` naming the env file, or the container keeps the credential it started with
|
|
through every rotation.
|
|
|
|
5. **Pin every image by digest.** A tag moves. The manifest in a repository names artifacts; the
|
|
manifest the mesh holds names digests, and they are not the same document.
|
|
|
|
6. **Decide generate or accept.** A new module's credential is generated. **An adopted one keeps
|
|
the credential it already has** — `secret accept` — because minting a new password for a
|
|
database that already exists locks the application out of its own data.
|
|
|
|
7. **Add a provisioner only if the software cannot read a file.** A proxy that watches a
|
|
directory needs nothing. PostgreSQL needs `CREATE ROLE`, an object store needs a bucket and a
|
|
policy, an identity provider needs a realm and a client — those need a small program beside
|
|
them. It reads what the mesh granted and reconciles; it does not decide anything.
|
|
|
|
8. **Prove it in the lab, against the real software.** Not that a container started — that the
|
|
thing works: the credential authenticates, a wrong one is refused, the containers reach each
|
|
other, the data survives a restart.
|
|
|
|
## What the first port actually cost
|
|
|
|
Six attempts, one real bug. Recorded because the ratio is the lesson: **the mesh was right every
|
|
time and the scaffolding was not.**
|
|
|
|
- A shape existed in the language and no host implemented it, so every declaration carrying one
|
|
was refused whole — correctly, and the node-engine said exactly that. **Nobody was reading the node-engine's
|
|
log.** Read it first; it is the only place that says why a machine did nothing.
|
|
- A blind find-and-replace renamed a provision in quotes and missed the same word bare.
|
|
- A command was tested only for the invocations that should fail, so it rejected every real one
|
|
and the suite stayed green.
|
|
- A test asserted on a helper rather than on the code that calls it, three separate times. **A
|
|
test that cannot fail when the behaviour is deleted is not defending the behaviour.**
|
|
|
|
## Rules
|
|
|
|
- **Read the node-engine's log before theorising.** A declaration that was sent and not applied says so
|
|
there and nowhere else.
|
|
- **A failing test is kept, not skipped.** It is the reproduction.
|
|
- **Never rotate during an adoption.** Rotation is a separate act, afterwards, deliberately.
|
|
- **A data directory is never removed by the mesh** (ADR 0030),
|
|
and that protects against the mesh only — not against a disk or a mistaken command.
|
|
|
|
## Moving a module's directory
|
|
|
|
When a fix moves what a container mounts to a new path (issue 1843: only-office's working directories
|
|
moved into a root-only box), the old directory does not go away by itself, and it is closed in three steps.
|
|
|
|
1. **The move.** The new directory gets a new id at its new path. The old one keeps **its old id at its old
|
|
path** for as long as it holds data, with the mode it should now have: the node-engine leaves a directory
|
|
it finds at a path under a new id as it found it, and sets no mode when the owner differs (issue 339), so
|
|
only the old id closes it. The node-engine removes a directory that is no longer declared only when it is
|
|
empty; a non-empty one is kept on disk with no further cleanup.
|
|
2. **The deletion, on the operator's word.** Its contents are data, so they are removed only after the
|
|
operator's yes, through `<node>/node-files.delete`. That tool moves into the node's trash: the disk space
|
|
comes back only when the trash is emptied (issue 1870: no tool does that yet).
|
|
3. **The declaration leaves** in a second small fix, once the directory is empty (issue 1854).
|
|
|
|
Plan the three in the first fix's issue, so the second fix is not found later by accident.
|
|
|
|
A service that starts on empty working directories may need minutes before it answers (OnlyOffice: about
|
|
two): the live check after the delivery waits for it before calling the move broken.
|