supabase: keep logflare's API key out of its log (hq issue 268) #85

2026-10-06T00:33:47Z

mesh-admin commented

2026-10-06 00:33:47 +00:00

The leak

docker_secrets_in_logs found supabase-analytics printing LOGFLARE_API_KEY, 79 lines in about 13 minutes. Every hit is a Plug.Cowboy crash report:

[error] #PID<…> running LogflareWeb.Endpoint (connection #PID<…>, stream id …) terminated
Server: supabase-analytics:4000 (http)
Request: POST /api/logs?source_name=cloudflare.logs.prod&api_key=[redacted: LOGFLARE_API_KEY]
** (exit) an exception was raised:
    ** (CaseClauseError) no case clause matching: {:error, {:shutdown, {:failed_to_start_child, "source-backend-1", {:bad_return_value, {:error, :not_connected}}}}}
        (logflare 1.4.0) lib/logflare/backends.ex:196: Logflare.Backends.start_source_sup/1

(The same appears for deno-relay-logs and the other sources.) The callers are the vector sinks this module writes, which put the key in the URL as ?api_key=…. Each ingest fails, and logflare prints the whole URL of the failed request, query string included. This is an error-level report, so changing LOGFLARE_LOG_LEVEL would not hide it unless errors were turned off entirely.

The fix

vector.yml: all seven sinks send the key in the x-api-key request header, which logflare 1.4.0 checks before the api_key parameter (set_verify_user.ex, verify_api_access.ex). The URLs keep only source_name.
analytics now starts through a mesh-written start script, following the letta pattern from #82. It mounts vector.yml read-only and refuses to start, with a message saying why, if any non-comment line still contains api_key=. It then runs the image's own run.sh. The container restarts when either file changes.
own secret logflare is marked "taken": "applied", not at-start. logflare writes LOGFLARE_API_KEY into its default user once (SingleTenant.create_default_user, which returns :already_created from then on) and never updates it. Rotating the key by restarting would leave vector and studio with a key logflare rejects, so the mesh refuses rotate by name.

module check over all 113 manifests: ok. I ran the start script against the old and new vector.yml: it refuses the old one and starts with the new one.

Not fixed here (separate defect)

The crash itself is Postgrex … tcp connect (db:5432): non-existing domain. Logflare's source backends were stored with a URL whose host is db (the upstream compose service name). They were written once, when the sources were created, and are never refreshed from POSTGRES_BACKEND_URL, which says supabase-db. Analytics ingests nothing until those stored backend rows point at supabase-db. With this PR, that failure no longer carries the key.

## The leak `docker_secrets_in_logs` found `supabase-analytics` printing `LOGFLARE_API_KEY`, 79 lines in about 13 minutes. Every hit is a Plug.Cowboy crash report: ``` [error] #PID<…> running LogflareWeb.Endpoint (connection #PID<…>, stream id …) terminated Server: supabase-analytics:4000 (http) Request: POST /api/logs?source_name=cloudflare.logs.prod&api_key=[redacted: LOGFLARE_API_KEY] ** (exit) an exception was raised: ** (CaseClauseError) no case clause matching: {:error, {:shutdown, {:failed_to_start_child, "source-backend-1", {:bad_return_value, {:error, :not_connected}}}}} (logflare 1.4.0) lib/logflare/backends.ex:196: Logflare.Backends.start_source_sup/1 ``` (The same appears for `deno-relay-logs` and the other sources.) The callers are the vector sinks this module writes, which put the key in the URL as `?api_key=…`. Each ingest fails, and logflare prints the whole URL of the failed request, query string included. This is an error-level report, so changing `LOGFLARE_LOG_LEVEL` would not hide it unless errors were turned off entirely. ## The fix - **vector.yml**: all seven sinks send the key in the `x-api-key` request header, which logflare 1.4.0 checks before the `api_key` parameter (`set_verify_user.ex`, `verify_api_access.ex`). The URLs keep only `source_name`. - **analytics** now starts through a mesh-written start script, following the letta pattern from #82. It mounts `vector.yml` read-only and refuses to start, with a message saying why, if any non-comment line still contains `api_key=`. It then runs the image's own `run.sh`. The container restarts when either file changes. - **own secret `logflare`** is marked `"taken": "applied"`, not `at-start`. logflare writes `LOGFLARE_API_KEY` into its default user once (`SingleTenant.create_default_user`, which returns `:already_created` from then on) and never updates it. Rotating the key by restarting would leave vector and studio with a key logflare rejects, so the mesh refuses `rotate` by name. `module check` over all 113 manifests: ok. I ran the start script against the old and new vector.yml: it refuses the old one and starts with the new one. ## Not fixed here (separate defect) The crash itself is `Postgrex … tcp connect (db:5432): non-existing domain`. Logflare's source backends were stored with a URL whose host is `db` (the upstream compose service name). They were written once, when the sources were created, and are never refreshed from `POSTGRES_BACKEND_URL`, which says `supabase-db`. Analytics ingests nothing until those stored backend rows point at `supabase-db`. With this PR, that failure no longer carries the key.

mesh-admin added 1 commit 2026-10-06 00:33:47 +00:00

supabase: keep logflare's API key out of its log (hq issue 268) b87dc29706

vector handed logflare its API key as ?api_key= in every sink URL, and
logflare 1.4.0 prints a failed request's whole URL in its Plug.Cowboy
error report. Its ingest fails on every request here, so the key was in
the analytics log about every ten seconds. The report is an error, so no
log level hides it.

Every sink now sends the key in the x-api-key header, which logflare
reads first. A start script refuses to start logflare while the vector
config it is given still puts the key in a URL.

The key is marked "applied", not "at-start": logflare writes it into its
default user once and never updates it, so the mesh must not rotate it
by restarting.

mesh-admin merged commit 495bb88111 into main

2026-10-06 00:34:14 +00:00

mesh-admin deleted branch fix/supabase-logflare-key-out-of-logs

2026-10-06 00:34:14 +00:00

jschoubben referenced this issue from a commit

2026-10-06 09:49:00 +00:00

supabase: make logflare's stored key and backends follow its environment

mesh-admin referenced this pull request

2026-10-06 09:49:19 +00:00

supabase: make logflare's stored key and backends follow its environment #87

mesh-admin commented

2026-10-06 23:08:34 +00:00

Delivery novox/mesh-catalog@495bb88111c0 — held since 2026-10-06T23:05:28Z

Held for a person: merged without a passing check: only a person decides that it goes on — mesh-delivery.release with why lets it go on.

Delivery plan — builds supabase → ace; no bus step

tier 0: supabase
ace: receives supabase
supabase keeps data (ADR 0233): what it holds is backed up before it moves

Transitions

2026-10-06 23:05 (new) → proposed (announced): novox/mesh-catalog#85 merged; its head was not heard by this owner, so it is made from the forge's word
2026-10-06 23:05 proposed → held (merged-unchecked): merged without a passing check — a verdict known with it is taken first: only a person decides that it goes on

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

**Delivery** `novox/mesh-catalog@495bb88111c0` — **held** since 2026-10-06T23:05:28Z Held for a person: merged without a passing check: only a person decides that it goes on — `mesh-delivery.release` with why lets it go on. **Delivery plan** — builds supabase → ace; no bus step - tier 0: supabase - ace: receives supabase - supabase keeps data (ADR 0233): what it holds is backed up before it moves **Transitions** - 2026-10-06 23:05 (new) → proposed (announced): novox/mesh-catalog#85 merged; its head was not heard by this owner, so it is made from the forge's word - 2026-10-06 23:05 proposed → held (merged-unchecked): merged without a passing check — a verdict known with it is taken first: only a person decides that it goes on The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-catalog#85