Merge pull request 'Issue 268: letta printed its passwords into its log' (#126) from issues/268-letta-printed-its-passwords into main
This commit was merged in pull request #126.
This commit is contained in:
@@ -0,0 +1,138 @@
|
||||
---
|
||||
status: located
|
||||
opened: 2026-10-06
|
||||
located-in: [mesh-catalog modules/letta, mesh-catalog modules/docker, mesh-controller cmd/mesh-controller]
|
||||
fixed-by:
|
||||
amended-design:
|
||||
---
|
||||
|
||||
# 268. letta printed its passwords into its log
|
||||
|
||||
## Symptom
|
||||
|
||||
The `letta` container on the home server writes two secrets into its own log every time it starts:
|
||||
|
||||
- `Creating engine postgresql://<user>:<password>@<host>:<port>/<database>`: the database password
|
||||
the mesh granted it from the `postgres` provision;
|
||||
- `▶ Using secure mode with password: <password>`: the letta server password, one of the module's
|
||||
own secrets.
|
||||
|
||||
The same database URI is printed twice more on each start: by the image's `startup.sh` (`External
|
||||
Postgres configuration detected, using …`) and by its migration step (`Using database: …`).
|
||||
The container had restarted about a hundred times, so each line was in the log about a hundred
|
||||
times. The runtime keeps that log in its own file, up to ten files of 100 MB each. Anyone who reads
|
||||
it can read both passwords: the docker module's `docker_logs` tool, anything with the runtime's
|
||||
socket, and every agent transcript that ever called `docker_logs` on this container.
|
||||
|
||||
No value appears in this record.
|
||||
|
||||
## Cause
|
||||
|
||||
**letta prints what it is given.** letta 0.6.8 prints `LETTA_PG_URI` whole in three places, and
|
||||
prints its server password in `--secure` mode. These are bare `print` calls. No log level or flag
|
||||
turns them off. The newest release (0.16.8) still prints the server password and the migration
|
||||
step's URI, so upgrading does not fix it.
|
||||
|
||||
**The module handed it both.** It composed the database password into `LETTA_PG_URI` in an
|
||||
env-file. letta reads its settings only from the environment
|
||||
([ADR 0086](../../02-DECISIONS/0086-a-secret-reaches-a-process-as-a-file.md)'s exception, declared on
|
||||
the container as `secrets-in-environment`). The declared reason even said that `startup.sh` echoes
|
||||
the URI. The exception was written down to justify the environment, and nobody acted on the log line.
|
||||
|
||||
**Nothing in the mesh looks at what a container prints.**
|
||||
[Issue 041](../041-a-sealed-credential-ends-up-in-the-process-environment/00-report.md) closed the
|
||||
environment path. `docker_inspect` leaves environment values out. But a program's own output carried
|
||||
the same values to more readers, and no check compared a container's log with the secrets it was
|
||||
given.
|
||||
|
||||
The pattern is wider than letta. Ten more modules compose a granted password into a connection URI
|
||||
the same way. Whether each program prints it is the program's business, so a manifest lint cannot
|
||||
decide it. Only reading the log can.
|
||||
|
||||
## Fix
|
||||
|
||||
**letta** (mesh-catalog, `modules/letta`):
|
||||
|
||||
- `LETTA_PG_URI` names no password. libpq, through letta's psycopg2 driver, reads the password from
|
||||
a `pgpass` file the module writes at `0600` and mounts read-only, named by `PGPASSFILE`. All three
|
||||
prints of the URI now carry no secret. The file is mounted directly into the container, so it is
|
||||
part of the container's spec by content
|
||||
([issue 103](../103-a-container-is-not-recreated-when-a-file-it-reads-changes/00-report.md)), and a
|
||||
rotated password recreates the container.
|
||||
- The container starts through a small script the module writes. The script rewrites the one line
|
||||
of the image's code that prints the server password, then executes the image's own `startup.sh`.
|
||||
The image is pinned by digest, so the rewrite is exact. **If the server password is still printed
|
||||
after the rewrite, or a password is back in `LETTA_PG_URI`, the script refuses to start letta and
|
||||
says why.** A letta that does not start is diagnosable. A letta that leaks is silent.
|
||||
- Both own secrets now say they are read at start (`taken: at-start`,
|
||||
[ADR 0114](../../02-DECISIONS/0114-a-shared-credential-rotates-over-two-credentials.md)). Without
|
||||
that, `rotate` refuses to replace the server password.
|
||||
|
||||
Two options were rejected. Filtering the container's output through a pipe would lose the
|
||||
process's signals and exit status, and would have to recognise every secret by a pattern. Building a
|
||||
derived image would put a build of a third-party image into the pipeline to change one line.
|
||||
|
||||
**The safety net** (mesh-catalog, `modules/docker`): a new tool, `docker_secrets_in_logs`. It reads
|
||||
the recent log of every container the mesh holds and compares it with three things: the values in
|
||||
that container's environment whose names say they are secrets, the password in any URI the
|
||||
environment holds, and the shape `scheme://user:password@` anywhere in a line. A finding names the
|
||||
container, the module and the variable, with a count and the first and last time it appeared. It
|
||||
never includes the value or the line. A container whose log cannot be read is listed as not read,
|
||||
never as clean. `docker_logs` redacts the same values before answering, because its answers end up in
|
||||
agent transcripts.
|
||||
|
||||
The tool has a limit. A secret delivered only as a mounted file is not known to it, because the tools
|
||||
run as the operator account, which cannot read the host's `0600` files. Such a secret is caught only
|
||||
when it is printed inside a URI.
|
||||
|
||||
**Rotation** (mesh-controller): before this change, `rotate <provision> --consumer <machine>` was the
|
||||
narrowest way to rotate a pair credential. It replaced the credential of every module on that machine
|
||||
that consumes the provision, and restarted all of them. `--module` (and `module` beside `provision`
|
||||
on the verb) narrows the rotation to one consuming module.
|
||||
|
||||
## Rotation, once the fix runs
|
||||
|
||||
The two secrets have already been exposed, so they are replaced after the fix is live. Each step
|
||||
below needs the operator's approval.
|
||||
|
||||
1. **Roll out the fix.** Merge the catalogue and controller changes and let the pipeline build. Then
|
||||
push the home server. The container's spec changes (its arguments, its mounts, its env-file), so
|
||||
the host removes the `letta` container and creates it again. **The runtime's log file goes with
|
||||
the removed container**, and this purges the retained copies under the runtime's own driver.
|
||||
Check: `docker_logs` on `letta` shows the URI with no password and `Using secure mode (the
|
||||
password is not printed)`, and `docker_secrets_in_logs` names nothing for `letta`.
|
||||
2. **The database password.** Run `mesh-controller.rotate` with `provision: postgres-database`,
|
||||
`consumer: <the home server>` and `module: letta`. The new credential is sent to both ends together.
|
||||
The `pgpass` file changes, so letta is recreated with the new value. Before the controller change
|
||||
is deployed, the only form available is the one without `module`, which also rotates every other
|
||||
module on the machine that consumes `postgres-database`. The command lists them before it acts.
|
||||
3. **The server password.** Run `mesh-controller.rotate` with `node: <the home server>`,
|
||||
`module: letta` and `secret: server-password`. If the mesh made the value, it makes a new one and
|
||||
pushes it. The env-file and the runtime's config file change, and both letta containers start
|
||||
again on the new value. If the value was *given* to the mesh (`secret accept`, when the server
|
||||
already had clients), the controller refuses with the reason
|
||||
([ADR 0113](../../02-DECISIONS/0113-the-vault-makes-every-secret.md)). In that case, generate a new
|
||||
value into a `0600` file on the controller's machine, run `secret accept <the home server> letta
|
||||
server-password --from <file>`, remove the file, and push. In both cases, every client outside
|
||||
the module that calls letta with that password (a workflow reaches it by its public name) needs
|
||||
the new value too.
|
||||
4. **Copies the runtime did not hold.** The container logs through the runtime's own file driver,
|
||||
not the journal. Confirm with `docker_inspect` (`LogConfig.Type`). Then check whether a backup on
|
||||
the home server includes the runtime's data directory, and if it does, which snapshots predate
|
||||
step 1. Agent transcripts that called `docker_logs` on `letta` hold the old values. Steps 2 and 3
|
||||
make those values useless, and deleting the transcripts is optional.
|
||||
|
||||
## How it is checked
|
||||
|
||||
- The start script was tested against the image's own `app.py`, extracted from the pinned digest. It
|
||||
rewrites the line, and the result compiles. It refuses to start when a print of the password
|
||||
remains or when the URI carries a password. It allows an unchanged rerun.
|
||||
- `mesh-controller module check` passes for `letta` and `docker`. The catalogue tests pass.
|
||||
- The docker module's tests: a scan of a log shaped like this leak names the server password, the
|
||||
password inside an environment URI, and a URI found by its shape, by name only. The answer, marshalled
|
||||
whole, contains no value. A masked `***` is not counted. An unreadable log is reported, not called
|
||||
clean. `docker_logs` returns the same lines redacted.
|
||||
- The controller's tests: the verb passes `module` beside `provision` through as `--module`, and the
|
||||
narrowing touches only that module's credentials.
|
||||
- On the mesh, after step 1: `docker_secrets_in_logs` on the home server. It is run again after any
|
||||
change to a module that composes a secret into a URI.
|
||||
Reference in New Issue
Block a user