supabase: make logflare's stored key and backends follow its environment #87

2026-10-06T09:49:19Z

mesh-admin commented

2026-10-06 09:49:19 +00:00

Follow-up to #85 (hq issue 268). Fixes two things on ace's supabase-analytics, through logflare's own code instead of hand edits to its database.

Cause (confirmed)

logflare 1.4.0 copies two settings into its database once and never reads them again:

Sources.create_source → create_backend_adaptor saves POSTGRES_BACKEND_URL into each source's source_backends.config when the source is created. On ace every source still points at db:5432, so each ingest fails with tcp connect (db:5432): non-existing domain - :nxdomain and then CaseClauseError ... failed_to_start_child "source-backend-1" ... :not_connected (live log, 2026-10-06). Nothing is stored.
SingleTenant.create_default_user saves LOGFLARE_API_KEY as the default user's api_key only when it inserts that user ({:error, :already_created} afterwards). That is why #85 marked the secret applied, and why the key that leaked into the log before #85 cannot be rotated today.

Reproduced locally with the pinned image: sources created with backend host db, then started with supabase-db and a new key → old key gets 500 (nxdomain), new key gets 401.

Fix

analytics-start.sh now runs logflare's migrations (logflare eval Logflare.Release.migrate, which run.sh does anyway), then logflare eval 'Code.eval_file("/run/logflare/reconcile.exs")', then run.sh.
New file analytics-reconcile.exs: with Ecto.Migrator.with_repo(Logflare.Repo, ...) (the pattern Logflare.Release uses), it
- sets the default user's api_key to LOGFLARE_API_KEY and clears old_api_key through Users.update_user_all_fields (User changeset);
- points every postgres source backend at POSTGRES_BACKEND_URL / POSTGRES_BACKEND_SCHEMA through Backends.update_source_backend_config (its changeset and adaptor validation). An empty SourceRegistry is started so its after-save restart finds nothing to restart.
- Idempotent: changes nothing that matches. Logs each change as host:port/db only, never the key or a password. Exits 1 (container does not start, reason is the last log line) when the key or URL is unset or a changeset is refused. On a fresh database (no default user yet) it does nothing and logflare seeds from env as before.
own-secrets.logflare is taken: at-start, so secret rotate works.
analytics restarts on analytics-env and analytics-reconcile as well; studio restarts on studio-env (it reads LOGFLARE_API_KEY from it and had no restart-on, so a rotated key would not have reached it).

Tested (throwaway postgres:15 + the pinned logflare image, removed after)

Sources created with backend host db, key A. 2. Moved db, new key B, new start script: reconcile logged the key replaced and 9 backends moved db:5432 → supabase-db:5432; key A → 401, key B → 200, event row present in the gotrue log_events table; no key or password in the container log. 3. Restart: "API key is LOGFLARE_API_KEY already", "9 source backend(s) already at supabase-db:5432/_supabase". 4. Key unset → exit 1 with reason; invalid backend URL → exit 1 with the changeset error. 5. Fresh empty schema → "no default user yet", logflare seeds itself, ingest 200.

mesh-controller module check (main, 9cf47f4): supabase ok, all 115 manifests ok.

Rollout, then rotation

Merge, let the pipeline deliver, push ace. analytics restarts; its log should show the backends moved (the key line will say "already", since the env still has the current key).
Then secret rotate {node: ace, module: supabase, secret: logflare} and push ace: vector, analytics and studio restart with the new key; analytics logs "the default user's API key was replaced with LOGFLARE_API_KEY". The leaked key then gets 401.

Destructive fallback (documented, not done)

If reconcile ever cannot fix logflare's state: stop supabase-analytics and drop and recreate the _analytics schema in _supabase via a module change (a run-once step or migration, not by hand), so logflare re-seeds its user, sources and backends from env on next start. It loses every stored log, endpoint and source token, which is why it is not the chosen path.

Follow-up to #85 (hq issue 268). Fixes two things on ace's supabase-analytics, through logflare's own code instead of hand edits to its database. ## Cause (confirmed) logflare 1.4.0 copies two settings into its database once and never reads them again: - `Sources.create_source` → `create_backend_adaptor` saves `POSTGRES_BACKEND_URL` into each source's `source_backends.config` when the source is created. On ace every source still points at `db:5432`, so each ingest fails with `tcp connect (db:5432): non-existing domain - :nxdomain` and then `CaseClauseError ... failed_to_start_child "source-backend-1" ... :not_connected` (live log, 2026-10-06). **Nothing is stored.** - `SingleTenant.create_default_user` saves `LOGFLARE_API_KEY` as the default user's `api_key` only when it inserts that user (`{:error, :already_created}` afterwards). That is why #85 marked the secret `applied`, and why the key that leaked into the log before #85 cannot be rotated today. Reproduced locally with the pinned image: sources created with backend host `db`, then started with `supabase-db` and a new key → old key gets 500 (nxdomain), new key gets 401. ## Fix - `analytics-start.sh` now runs logflare's migrations (`logflare eval Logflare.Release.migrate`, which run.sh does anyway), then `logflare eval 'Code.eval_file("/run/logflare/reconcile.exs")'`, then `run.sh`. - New file `analytics-reconcile.exs`: with `Ecto.Migrator.with_repo(Logflare.Repo, ...)` (the pattern `Logflare.Release` uses), it - sets the default user's `api_key` to `LOGFLARE_API_KEY` and clears `old_api_key` through `Users.update_user_all_fields` (User changeset); - points every postgres source backend at `POSTGRES_BACKEND_URL` / `POSTGRES_BACKEND_SCHEMA` through `Backends.update_source_backend_config` (its changeset and adaptor validation). An empty `SourceRegistry` is started so its after-save restart finds nothing to restart. - Idempotent: changes nothing that matches. Logs each change as host:port/db only, never the key or a password. Exits 1 (container does not start, reason is the last log line) when the key or URL is unset or a changeset is refused. On a fresh database (no default user yet) it does nothing and logflare seeds from env as before. - `own-secrets.logflare` is `taken: at-start`, so `secret rotate` works. - `analytics` restarts on `analytics-env` and `analytics-reconcile` as well; `studio` restarts on `studio-env` (it reads `LOGFLARE_API_KEY` from it and had no restart-on, so a rotated key would not have reached it). ## Tested (throwaway postgres:15 + the pinned logflare image, removed after) 1. Sources created with backend host `db`, key A. 2. Moved db, new key B, new start script: reconcile logged the key replaced and 9 backends moved `db:5432` → `supabase-db:5432`; key A → 401, key B → 200, event row present in the gotrue log_events table; no key or password in the container log. 3. Restart: "API key is LOGFLARE_API_KEY already", "9 source backend(s) already at supabase-db:5432/_supabase". 4. Key unset → exit 1 with reason; invalid backend URL → exit 1 with the changeset error. 5. Fresh empty schema → "no default user yet", logflare seeds itself, ingest 200. `mesh-controller module check` (main, 9cf47f4): supabase ok, all 115 manifests ok. ## Rollout, then rotation 1. Merge, let the pipeline deliver, push ace. analytics restarts; its log should show the backends moved (the key line will say "already", since the env still has the current key). 2. Then `secret rotate {node: ace, module: supabase, secret: logflare}` and push ace: vector, analytics and studio restart with the new key; analytics logs "the default user's API key was replaced with LOGFLARE_API_KEY". The leaked key then gets 401. ## Destructive fallback (documented, not done) If reconcile ever cannot fix logflare's state: stop supabase-analytics and drop and recreate the `_analytics` schema in `_supabase` via a module change (a run-once step or migration, not by hand), so logflare re-seeds its user, sources and backends from env on next start. It loses every stored log, endpoint and source token, which is why it is not the chosen path.

mesh-admin added 1 commit 2026-10-06 09:49:20 +00:00

supabase: make logflare's stored key and backends follow its environment 908d45864a

logflare 1.4.0 copies LOGFLARE_API_KEY into its default user and
POSTGRES_BACKEND_URL into every source's backend once, when it creates
them, and never reads either again. On ace every source still points at
db:5432, which resolves nowhere, so analytics stored no logs; and the key
that leaked into the log before #85 could not be replaced, because the
mesh had to mark it "applied".

The start script now runs logflare's migrations and then reconcile.exs
through `logflare eval`, before logflare starts: it uses logflare's own
Users and Backends contexts to set the default user's key to
LOGFLARE_API_KEY (clearing old_api_key) and to point each postgres
source backend at POSTGRES_BACKEND_URL. It changes nothing that already
matches, says what it changed without printing the key or a password,
and stops the start when it cannot finish.

With the key taken at every start, the secret is "at-start" and `secret
rotate` works. analytics restarts on its env file and studio on its env
file too, so a rotated key reaches every reader.

mesh-admin merged commit aabc8aa039 into main

2026-10-06 09:50:29 +00:00

mesh-admin deleted branch fix/supabase-logflare-reconcile

2026-10-06 09:50:29 +00:00

mesh-admin commented

2026-10-06 23:09:02 +00:00

Delivery novox/mesh-catalog@aabc8aa0399e — held since 2026-10-06T23:05:28Z

Held for a person: merged without a passing check: only a person decides that it goes on — mesh-delivery.release with why lets it go on.

Delivery plan — builds supabase → ace; no bus step

tier 0: supabase
ace: receives supabase
supabase keeps data (ADR 0233): what it holds is backed up before it moves

Transitions

2026-10-06 23:05 (new) → proposed (announced): novox/mesh-catalog#87 merged; its head was not heard by this owner, so it is made from the forge's word
2026-10-06 23:05 proposed → held (merged-unchecked): merged without a passing check — a verdict known with it is taken first: only a person decides that it goes on

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

**Delivery** `novox/mesh-catalog@aabc8aa0399e` — **held** since 2026-10-06T23:05:28Z Held for a person: merged without a passing check: only a person decides that it goes on — `mesh-delivery.release` with why lets it go on. **Delivery plan** — builds supabase → ace; no bus step - tier 0: supabase - ace: receives supabase - supabase keeps data (ADR 0233): what it holds is backed up before it moves **Transitions** - 2026-10-06 23:05 (new) → proposed (announced): novox/mesh-catalog#87 merged; its head was not heard by this owner, so it is made from the forge's word - 2026-10-06 23:05 proposed → held (merged-unchecked): merged without a passing check — a verdict known with it is taken first: only a person decides that it goes on The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-catalog#87