From f6f5563de23fb5c8c47a871ff328b3f1278b1b87 Mon Sep 17 00:00:00 2001 From: jochen Date: Tue, 6 Oct 2026 02:15:00 +0200 Subject: [PATCH] Issue 268: letta printed its passwords into its log Record the leak, its cause, the fix and the rotation steps the operator approves once the fix runs, so the exposed secrets are replaced in order. --- .../00-report.md | 138 ++++++++++++++++++ 1 file changed, 138 insertions(+) create mode 100644 04-ISSUES/268-letta-printed-its-passwords-into-its-log/00-report.md diff --git a/04-ISSUES/268-letta-printed-its-passwords-into-its-log/00-report.md b/04-ISSUES/268-letta-printed-its-passwords-into-its-log/00-report.md new file mode 100644 index 0000000..34bf93e --- /dev/null +++ b/04-ISSUES/268-letta-printed-its-passwords-into-its-log/00-report.md @@ -0,0 +1,138 @@ +--- +status: located +opened: 2026-10-06 +located-in: [mesh-catalog modules/letta, mesh-catalog modules/docker, mesh-controller cmd/mesh-controller] +fixed-by: +amended-design: +--- + +# 268. letta printed its passwords into its log + +## Symptom + +The `letta` container on the home server writes two secrets into its own log every time it starts: + +- `Creating engine postgresql://:@:/`: the database password + the mesh granted it from the `postgres` provision; +- `▶ Using secure mode with password: `: the letta server password, one of the module's + own secrets. + +The same database URI is printed twice more on each start: by the image's `startup.sh` (`External +Postgres configuration detected, using …`) and by its migration step (`Using database: …`). +The container had restarted about a hundred times, so each line was in the log about a hundred +times. The runtime keeps that log in its own file, up to ten files of 100 MB each. Anyone who reads +it can read both passwords: the docker module's `docker_logs` tool, anything with the runtime's +socket, and every agent transcript that ever called `docker_logs` on this container. + +No value appears in this record. + +## Cause + +**letta prints what it is given.** letta 0.6.8 prints `LETTA_PG_URI` whole in three places, and +prints its server password in `--secure` mode. These are bare `print` calls. No log level or flag +turns them off. The newest release (0.16.8) still prints the server password and the migration +step's URI, so upgrading does not fix it. + +**The module handed it both.** It composed the database password into `LETTA_PG_URI` in an +env-file. letta reads its settings only from the environment +([ADR 0086](../../02-DECISIONS/0086-a-secret-reaches-a-process-as-a-file.md)'s exception, declared on +the container as `secrets-in-environment`). The declared reason even said that `startup.sh` echoes +the URI. The exception was written down to justify the environment, and nobody acted on the log line. + +**Nothing in the mesh looks at what a container prints.** +[Issue 041](../041-a-sealed-credential-ends-up-in-the-process-environment/00-report.md) closed the +environment path. `docker_inspect` leaves environment values out. But a program's own output carried +the same values to more readers, and no check compared a container's log with the secrets it was +given. + +The pattern is wider than letta. Ten more modules compose a granted password into a connection URI +the same way. Whether each program prints it is the program's business, so a manifest lint cannot +decide it. Only reading the log can. + +## Fix + +**letta** (mesh-catalog, `modules/letta`): + +- `LETTA_PG_URI` names no password. libpq, through letta's psycopg2 driver, reads the password from + a `pgpass` file the module writes at `0600` and mounts read-only, named by `PGPASSFILE`. All three + prints of the URI now carry no secret. The file is mounted directly into the container, so it is + part of the container's spec by content + ([issue 103](../103-a-container-is-not-recreated-when-a-file-it-reads-changes/00-report.md)), and a + rotated password recreates the container. +- The container starts through a small script the module writes. The script rewrites the one line + of the image's code that prints the server password, then executes the image's own `startup.sh`. + The image is pinned by digest, so the rewrite is exact. **If the server password is still printed + after the rewrite, or a password is back in `LETTA_PG_URI`, the script refuses to start letta and + says why.** A letta that does not start is diagnosable. A letta that leaks is silent. +- Both own secrets now say they are read at start (`taken: at-start`, + [ADR 0114](../../02-DECISIONS/0114-a-shared-credential-rotates-over-two-credentials.md)). Without + that, `rotate` refuses to replace the server password. + +Two options were rejected. Filtering the container's output through a pipe would lose the +process's signals and exit status, and would have to recognise every secret by a pattern. Building a +derived image would put a build of a third-party image into the pipeline to change one line. + +**The safety net** (mesh-catalog, `modules/docker`): a new tool, `docker_secrets_in_logs`. It reads +the recent log of every container the mesh holds and compares it with three things: the values in +that container's environment whose names say they are secrets, the password in any URI the +environment holds, and the shape `scheme://user:password@` anywhere in a line. A finding names the +container, the module and the variable, with a count and the first and last time it appeared. It +never includes the value or the line. A container whose log cannot be read is listed as not read, +never as clean. `docker_logs` redacts the same values before answering, because its answers end up in +agent transcripts. + +The tool has a limit. A secret delivered only as a mounted file is not known to it, because the tools +run as the operator account, which cannot read the host's `0600` files. Such a secret is caught only +when it is printed inside a URI. + +**Rotation** (mesh-controller): before this change, `rotate --consumer ` was the +narrowest way to rotate a pair credential. It replaced the credential of every module on that machine +that consumes the provision, and restarted all of them. `--module` (and `module` beside `provision` +on the verb) narrows the rotation to one consuming module. + +## Rotation, once the fix runs + +The two secrets have already been exposed, so they are replaced after the fix is live. Each step +below needs the operator's approval. + +1. **Roll out the fix.** Merge the catalogue and controller changes and let the pipeline build. Then + push the home server. The container's spec changes (its arguments, its mounts, its env-file), so + the host removes the `letta` container and creates it again. **The runtime's log file goes with + the removed container**, and this purges the retained copies under the runtime's own driver. + Check: `docker_logs` on `letta` shows the URI with no password and `Using secure mode (the + password is not printed)`, and `docker_secrets_in_logs` names nothing for `letta`. +2. **The database password.** Run `mesh-controller.rotate` with `provision: postgres-database`, + `consumer: ` and `module: letta`. The new credential is sent to both ends together. + The `pgpass` file changes, so letta is recreated with the new value. Before the controller change + is deployed, the only form available is the one without `module`, which also rotates every other + module on the machine that consumes `postgres-database`. The command lists them before it acts. +3. **The server password.** Run `mesh-controller.rotate` with `node: `, + `module: letta` and `secret: server-password`. If the mesh made the value, it makes a new one and + pushes it. The env-file and the runtime's config file change, and both letta containers start + again on the new value. If the value was *given* to the mesh (`secret accept`, when the server + already had clients), the controller refuses with the reason + ([ADR 0113](../../02-DECISIONS/0113-the-vault-makes-every-secret.md)). In that case, generate a new + value into a `0600` file on the controller's machine, run `secret accept letta + server-password --from `, remove the file, and push. In both cases, every client outside + the module that calls letta with that password (a workflow reaches it by its public name) needs + the new value too. +4. **Copies the runtime did not hold.** The container logs through the runtime's own file driver, + not the journal. Confirm with `docker_inspect` (`LogConfig.Type`). Then check whether a backup on + the home server includes the runtime's data directory, and if it does, which snapshots predate + step 1. Agent transcripts that called `docker_logs` on `letta` hold the old values. Steps 2 and 3 + make those values useless, and deleting the transcripts is optional. + +## How it is checked + +- The start script was tested against the image's own `app.py`, extracted from the pinned digest. It + rewrites the line, and the result compiles. It refuses to start when a print of the password + remains or when the URI carries a password. It allows an unchanged rerun. +- `mesh-controller module check` passes for `letta` and `docker`. The catalogue tests pass. +- The docker module's tests: a scan of a log shaped like this leak names the server password, the + password inside an environment URI, and a URI found by its shape, by name only. The answer, marshalled + whole, contains no value. A masked `***` is not counted. An unreadable log is reported, not called + clean. `docker_logs` returns the same lines redacted. +- The controller's tests: the verb passes `module` beside `provision` through as `--module`, and the + narrowing touches only that module's credentials. +- On the mesh, after step 1: `docker_secrets_in_logs` on the home server. It is run again after any + change to a module that composes a secret into a URI.