diff --git a/04-ISSUES/268-letta-printed-its-passwords-into-its-log/00-report.md b/04-ISSUES/268-letta-printed-its-passwords-into-its-log/00-report.md new file mode 100644 index 0000000..34bf93e --- /dev/null +++ b/04-ISSUES/268-letta-printed-its-passwords-into-its-log/00-report.md @@ -0,0 +1,138 @@ +--- +status: located +opened: 2026-10-06 +located-in: [mesh-catalog modules/letta, mesh-catalog modules/docker, mesh-controller cmd/mesh-controller] +fixed-by: +amended-design: +--- + +# 268. letta printed its passwords into its log + +## Symptom + +The `letta` container on the home server writes two secrets into its own log every time it starts: + +- `Creating engine postgresql://:@:/`: the database password + the mesh granted it from the `postgres` provision; +- `▶ Using secure mode with password: `: the letta server password, one of the module's + own secrets. + +The same database URI is printed twice more on each start: by the image's `startup.sh` (`External +Postgres configuration detected, using …`) and by its migration step (`Using database: …`). +The container had restarted about a hundred times, so each line was in the log about a hundred +times. The runtime keeps that log in its own file, up to ten files of 100 MB each. Anyone who reads +it can read both passwords: the docker module's `docker_logs` tool, anything with the runtime's +socket, and every agent transcript that ever called `docker_logs` on this container. + +No value appears in this record. + +## Cause + +**letta prints what it is given.** letta 0.6.8 prints `LETTA_PG_URI` whole in three places, and +prints its server password in `--secure` mode. These are bare `print` calls. No log level or flag +turns them off. The newest release (0.16.8) still prints the server password and the migration +step's URI, so upgrading does not fix it. + +**The module handed it both.** It composed the database password into `LETTA_PG_URI` in an +env-file. letta reads its settings only from the environment +([ADR 0086](../../02-DECISIONS/0086-a-secret-reaches-a-process-as-a-file.md)'s exception, declared on +the container as `secrets-in-environment`). The declared reason even said that `startup.sh` echoes +the URI. The exception was written down to justify the environment, and nobody acted on the log line. + +**Nothing in the mesh looks at what a container prints.** +[Issue 041](../041-a-sealed-credential-ends-up-in-the-process-environment/00-report.md) closed the +environment path. `docker_inspect` leaves environment values out. But a program's own output carried +the same values to more readers, and no check compared a container's log with the secrets it was +given. + +The pattern is wider than letta. Ten more modules compose a granted password into a connection URI +the same way. Whether each program prints it is the program's business, so a manifest lint cannot +decide it. Only reading the log can. + +## Fix + +**letta** (mesh-catalog, `modules/letta`): + +- `LETTA_PG_URI` names no password. libpq, through letta's psycopg2 driver, reads the password from + a `pgpass` file the module writes at `0600` and mounts read-only, named by `PGPASSFILE`. All three + prints of the URI now carry no secret. The file is mounted directly into the container, so it is + part of the container's spec by content + ([issue 103](../103-a-container-is-not-recreated-when-a-file-it-reads-changes/00-report.md)), and a + rotated password recreates the container. +- The container starts through a small script the module writes. The script rewrites the one line + of the image's code that prints the server password, then executes the image's own `startup.sh`. + The image is pinned by digest, so the rewrite is exact. **If the server password is still printed + after the rewrite, or a password is back in `LETTA_PG_URI`, the script refuses to start letta and + says why.** A letta that does not start is diagnosable. A letta that leaks is silent. +- Both own secrets now say they are read at start (`taken: at-start`, + [ADR 0114](../../02-DECISIONS/0114-a-shared-credential-rotates-over-two-credentials.md)). Without + that, `rotate` refuses to replace the server password. + +Two options were rejected. Filtering the container's output through a pipe would lose the +process's signals and exit status, and would have to recognise every secret by a pattern. Building a +derived image would put a build of a third-party image into the pipeline to change one line. + +**The safety net** (mesh-catalog, `modules/docker`): a new tool, `docker_secrets_in_logs`. It reads +the recent log of every container the mesh holds and compares it with three things: the values in +that container's environment whose names say they are secrets, the password in any URI the +environment holds, and the shape `scheme://user:password@` anywhere in a line. A finding names the +container, the module and the variable, with a count and the first and last time it appeared. It +never includes the value or the line. A container whose log cannot be read is listed as not read, +never as clean. `docker_logs` redacts the same values before answering, because its answers end up in +agent transcripts. + +The tool has a limit. A secret delivered only as a mounted file is not known to it, because the tools +run as the operator account, which cannot read the host's `0600` files. Such a secret is caught only +when it is printed inside a URI. + +**Rotation** (mesh-controller): before this change, `rotate --consumer ` was the +narrowest way to rotate a pair credential. It replaced the credential of every module on that machine +that consumes the provision, and restarted all of them. `--module` (and `module` beside `provision` +on the verb) narrows the rotation to one consuming module. + +## Rotation, once the fix runs + +The two secrets have already been exposed, so they are replaced after the fix is live. Each step +below needs the operator's approval. + +1. **Roll out the fix.** Merge the catalogue and controller changes and let the pipeline build. Then + push the home server. The container's spec changes (its arguments, its mounts, its env-file), so + the host removes the `letta` container and creates it again. **The runtime's log file goes with + the removed container**, and this purges the retained copies under the runtime's own driver. + Check: `docker_logs` on `letta` shows the URI with no password and `Using secure mode (the + password is not printed)`, and `docker_secrets_in_logs` names nothing for `letta`. +2. **The database password.** Run `mesh-controller.rotate` with `provision: postgres-database`, + `consumer: ` and `module: letta`. The new credential is sent to both ends together. + The `pgpass` file changes, so letta is recreated with the new value. Before the controller change + is deployed, the only form available is the one without `module`, which also rotates every other + module on the machine that consumes `postgres-database`. The command lists them before it acts. +3. **The server password.** Run `mesh-controller.rotate` with `node: `, + `module: letta` and `secret: server-password`. If the mesh made the value, it makes a new one and + pushes it. The env-file and the runtime's config file change, and both letta containers start + again on the new value. If the value was *given* to the mesh (`secret accept`, when the server + already had clients), the controller refuses with the reason + ([ADR 0113](../../02-DECISIONS/0113-the-vault-makes-every-secret.md)). In that case, generate a new + value into a `0600` file on the controller's machine, run `secret accept letta + server-password --from `, remove the file, and push. In both cases, every client outside + the module that calls letta with that password (a workflow reaches it by its public name) needs + the new value too. +4. **Copies the runtime did not hold.** The container logs through the runtime's own file driver, + not the journal. Confirm with `docker_inspect` (`LogConfig.Type`). Then check whether a backup on + the home server includes the runtime's data directory, and if it does, which snapshots predate + step 1. Agent transcripts that called `docker_logs` on `letta` hold the old values. Steps 2 and 3 + make those values useless, and deleting the transcripts is optional. + +## How it is checked + +- The start script was tested against the image's own `app.py`, extracted from the pinned digest. It + rewrites the line, and the result compiles. It refuses to start when a print of the password + remains or when the URI carries a password. It allows an unchanged rerun. +- `mesh-controller module check` passes for `letta` and `docker`. The catalogue tests pass. +- The docker module's tests: a scan of a log shaped like this leak names the server password, the + password inside an environment URI, and a URI found by its shape, by name only. The answer, marshalled + whole, contains no value. A masked `***` is not counted. An unreadable log is reported, not called + clean. `docker_logs` returns the same lines redacted. +- The controller's tests: the verb passes `module` beside `provision` through as `--module`, and the + narrowing touches only that module's credentials. +- On the mesh, after step 1: `docker_secrets_in_logs` on the home server. It is run again after any + change to a module that composes a secret into a URI.