letta: keep its passwords out of its log; docker: find and hide secrets a container printed (hq issue 268) #82

2026-10-06T00:15:12Z

mesh-admin commented

2026-10-06 00:15:12 +00:00

hq issue 268 (novox/hq, branch issues/268-letta-printed-its-passwords).

letta printed its database password (LETTA_PG_URI, printed whole by startup.sh, alembic/env.py and server.py) and its server password (--secure banner in app.py) into its container log on every start. These are bare print calls with no log level, and letta 0.16.8 still prints both.

LETTA_PG_URI no longer contains the password. libpq (psycopg2) reads it from a mounted pgpass file at 0600 (PGPASSFILE). A mounted file is part of the container's spec, so a rotation recreates the container.
start.sh (a module file, the container's args) rewrites the one print of the server password in the image pinned by digest, then execs the image's startup.sh. It refuses to start if a print of the password remains or if LETTA_PG_URI carries a password again.
Both own secrets are now taken: at-start, so rotate can replace the server password.

docker: new tool docker_secrets_in_logs. For every container the mesh holds, it compares the recent log lines with that container's secret-named env values, the passwords inside URIs in its env, and any scheme://user:password@ in the log. It reports container, module and variable name only, never values. Logs it cannot read are reported as unread. docker_logs now redacts the same values.

Checked: the start script was run against the real app.py extracted from the pinned image (it rewrites the line, the result compiles, and it refuses a leaking start). module check passes for letta and docker. The catalogue tests pass. go vet and go test -race pass in modules/docker.

After merge and push, the old container and its json log are removed. The two exposed secrets must then be rotated: steps are in the hq issue and need the operator's approval. Do not merge without review.

hq issue 268 (novox/hq, branch issues/268-letta-printed-its-passwords). **letta** printed its database password (LETTA_PG_URI, printed whole by startup.sh, alembic/env.py and server.py) and its server password (`--secure` banner in app.py) into its container log on every start. These are bare `print` calls with no log level, and letta 0.16.8 still prints both. - `LETTA_PG_URI` no longer contains the password. libpq (psycopg2) reads it from a mounted `pgpass` file at 0600 (`PGPASSFILE`). A mounted file is part of the container's spec, so a rotation recreates the container. - `start.sh` (a module file, the container's args) rewrites the one print of the server password in the image pinned by digest, then execs the image's `startup.sh`. It **refuses to start** if a print of the password remains or if `LETTA_PG_URI` carries a password again. - Both own secrets are now `taken: at-start`, so `rotate` can replace the server password. **docker**: new tool `docker_secrets_in_logs`. For every container the mesh holds, it compares the recent log lines with that container's secret-named env values, the passwords inside URIs in its env, and any `scheme://user:password@` in the log. It reports container, module and variable name only, never values. Logs it cannot read are reported as unread. `docker_logs` now redacts the same values. Checked: the start script was run against the real `app.py` extracted from the pinned image (it rewrites the line, the result compiles, and it refuses a leaking start). `module check` passes for letta and docker. The catalogue tests pass. `go vet` and `go test -race` pass in modules/docker. After merge and push, the old container and its json log are removed. The two exposed secrets must then be rotated: steps are in the hq issue and need the operator's approval. Do not merge without review.

mesh-admin added 2 commits 2026-10-06 00:15:12 +00:00

letta: keep its database and server passwords out of its log (hq issue 268) f9f27d4878

letta 0.6.8 prints LETTA_PG_URI whole (startup.sh, alembic, server.py)
and its server password when it starts in secure mode, so both were in
the container's log on every one of its restarts. Newer letta still
prints both, and neither is a log level.

The URI now names no password: libpq reads it from a mounted pgpass
file (PGPASSFILE). The one print of the server password is rewritten by
a start script before the server starts, and the script refuses to start
letta if that print, or a password in the URI, is still there - a letta
that does not start says why; one that leaks says nothing.

Both own secrets say they are read at start, so `rotate` can replace the
server password the mesh made.

docker: find and hide secrets a container printed into its log (hq issue 268) bbb67e41a0

letta printed two passwords into its log for weeks and nothing noticed,
and docker_logs handed them to whoever asked. docker_secrets_in_logs
compares each container's recent lines with the secret-named values of
its environment, the passwords in its URIs, and any URI carrying a
password, and names what it found by container, module and variable -
never the value. docker_logs redacts the same values before answering.

mesh-admin merged commit 7fad19a764 into main

2026-10-06 00:25:32 +00:00

mesh-admin deleted branch fix/letta-secrets-out-of-logs

2026-10-06 00:25:32 +00:00

mesh-admin referenced this pull request

2026-10-06 00:33:47 +00:00

supabase: keep logflare's API key out of its log (hq issue 268) #85

mesh-admin commented

2026-10-06 23:08:49 +00:00

Delivery novox/mesh-catalog@7fad19a7642b — held since 2026-10-06T23:05:28Z

Held for a person: merged without a passing check: only a person decides that it goes on — mesh-delivery.release with why lets it go on.

Delivery plan — builds docker, letta → ace, g14, novox, shanks; no bus step

tier 0: docker, letta
ace: receives docker, letta
g14: receives docker
novox: receives docker
shanks: receives docker

Transitions

2026-10-06 23:05 (new) → proposed (announced): novox/mesh-catalog#82 merged; its head was not heard by this owner, so it is made from the forge's word
2026-10-06 23:05 proposed → held (merged-unchecked): merged without a passing check — a verdict known with it is taken first: only a person decides that it goes on

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

**Delivery** `novox/mesh-catalog@7fad19a7642b` — **held** since 2026-10-06T23:05:28Z Held for a person: merged without a passing check: only a person decides that it goes on — `mesh-delivery.release` with why lets it go on. **Delivery plan** — builds docker, letta → ace, g14, novox, shanks; no bus step - tier 0: docker, letta - ace: receives docker, letta - g14: receives docker - novox: receives docker - shanks: receives docker **Transitions** - 2026-10-06 23:05 (new) → proposed (announced): novox/mesh-catalog#82 merged; its head was not heard by this owner, so it is made from the forge's word - 2026-10-06 23:05 proposed → held (merged-unchecked): merged without a passing check — a verdict known with it is taken first: only a person decides that it goes on The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-catalog#82