diff --git a/04-ISSUES/282-a-secret-on-a-command-line-is-kept-in-the-runtimes-events/00-report.md b/04-ISSUES/282-a-secret-on-a-command-line-is-kept-in-the-runtimes-events/00-report.md new file mode 100644 index 00000000..dd233ffd --- /dev/null +++ b/04-ISSUES/282-a-secret-on-a-command-line-is-kept-in-the-runtimes-events/00-report.md @@ -0,0 +1,137 @@ +--- +status: located +opened: 2026-10-07 +located-in: [mesh-catalog modules/mosquitto, mesh-catalog modules/docker, mesh-catalog modules/keycloak, mesh-catalog modules/minio, mesh-catalog modules/gitea] +fixed-by: +amended-design: +--- + +# 282. A secret on a command line is kept in the runtime's events + +## Symptom + +Found on 2026-10-07 while research 032 read the home server's container runtime, read-only. The +`mosquitto` module administers its broker by running `mosquitto_ctrl` inside the broker's container +with `docker exec`, and it passed the broker's **admin password as an argument** (`-P `) on +every call: the provisioner's check every minute, and every call of its tools. The container runtime +records the command line of every exec in its event stream, as the event's action +(`exec_create: `). So: + +- `docker events` showed the admin password to anyone who may ask the runtime; +- the docker module's `docker_events`, asked with `execs: true`, returned it, and any agent transcript + of that call holds a copy. + +No value appears in this record. + +## Cause + +**The module chose argv and wrote down why.** Its client said that `mosquitto_ctrl` 2.x offers no +way to give the password but `-P` or an interactive prompt, called the exposure "momentary and +confined to this single-purpose runtime container", and went ahead. Both halves were wrong: + +- `mosquitto_ctrl` takes its connection options from a file (`-o `), and its `createClient` + and `setClientPassword` prompt for a password on stdin when none is given. Both work without a + terminal (checked against the 2.1.2 the module pins). +- An exec's command line is not confined to the container. The runtime records it, outside the + container, for as long as it keeps its events, and hands it to every reader of its event stream. + +**Nothing in the mesh looked at command lines.** [Issue 268](../268-letta-printed-its-passwords-into-its-log/00-report.md) +taught the docker module to find a secret in a container's own log and to redact it from +`docker_logs`. The same values on an exec's command line went through `docker_events` unredacted, +and no check compared exec command lines with anything. + +## Other modules that put a secret on a command line + +The catalogue was searched for every place a module runs a program (`docker exec`, `exec.Command`, +`execFile`, `spawn`, dump commands in manifests, container arguments) and for password-taking flags +(`-P`, `-p`, `-a`, `--password`, `--secret-key`, `NAME=value`): + +| module | what | seen by | here | +|---|---|---|---| +| `mosquitto` | the broker's admin password as `-P`, and a client's password as `createClient -p` or `setClientPassword `, on every `docker exec` | the runtime's events; the machine's process table | fixed | +| `mosquitto` | the admin password as the last argument of `dynsec init`, in the throwaway container that seeds the broker's store | that container's description while it ran; the process table | fixed | +| `keycloak` | the repair script gave `kcadm` the temporary admin's password (`--password`) and the mesh's admin password (`--new-password`) as arguments, inside the container | the machine's process table, for as long as the JVM ran | fixed | +| `minio` | `mc alias set` with the root user and password as arguments (and `mc` then kept them in its alias file) | the machine's process table | fixed | +| `minio` | `mc admin user svcacct add --secret-key ` | the machine's process table, for the second `mc` runs | **open** | +| `gitea` | its run-once admin step runs `gitea admin user create --password "$(cat …)"` inside its container | the machine's process table, while the CLI runs | **open** | + +Checked and clean: the `mssql` and `nats` dumps hand their secret over on stdin; the `mongodb` dump +writes its password into an options file inside the container with a shell builtin; the `postgres` +and `supabase` dumps carry no password; `keycloak`'s own `docker exec` carries its passwords on stdin; +`nextcloud`, `mailu`, the forge's delivery notes and the `nats` monitor carry no secret. + +Only an exec is recorded by the runtime. The other rows are visible in the machine's process table +to every local user while the program runs, and are kept by anything that records process starts. + +## Fix + +Proposed in mesh-catalog pull request #100. Not merged at the time of writing. + +**mosquitto.** `mosquitto_ctrl` runs under a fixed shell that holds no secret. The shell reads the +admin's name and password from the first two lines of stdin into an options file only it can read, +runs `mosquitto_ctrl -o `, and removes the file as it exits. The rest of stdin answers the +tool's own password prompt, so a client's password is never an argument either. `dynsec init` reads +its password from stdin the same way. **Before anything runs, the module refuses an argument list +that carries the admin password or a password being set**, and says which, by name. The admin tool +refuses the three dynsec commands that would put a password on a command line. + +**The admin password can now be rotated.** It was declared as a bare path, which `rotate` refuses. +Saying `taken: at-start` alone would have been false: the broker's store keeps the admin's hash, and a +new value written by the mesh would leave the module locked out. The module's run-once bootstrap step +now runs again when the secret changes (`restart-on` names it). It keeps the value it last applied in +a 0600 file of its own state. When the mesh's value is another one, it connects with the applied +value and sets the new one, online, with the password on stdin. Then it checks the new one with a +connect and records it. The broker keeps running and keeps every client. A re-key that cannot be done +fails the step and says why. + +**docker.** `docker_events` redacts every exec's command line before it answers. It hides the values +`docker_logs` hides, and also, whatever their source, what a command line carries by its shape: the +word after a flag that takes a password, a `NAME=value` whose name says secret, and the password a +dynsec command sets. A new tool, `docker_secrets_in_events`, reads the exec events of a window and +names each secret it finds by container, module, what it was and the program, with counts and the +first and last time. It never returns the value or the command line. `docker_inspect` redacts a +container's own command line the same way. + +**keycloak** gives `kcadm` each password in `KC_CLI_PASSWORD`, set for the one command that needs it. +A command's environment can be read by its own user only. **minio** gives `mc` its admin alias in +`MC_HOST_mesh`, in `mc`'s environment, and refuses an argument list that carries the root password. + +**What stays open.** `mc admin user svcacct add` takes a consumer's secret key only as an argument. +Moving it off the command line means making the admin API's encrypted request in the module's own +code. `gitea admin user create` takes the admin password only as an argument. Moving it means creating +the admin another way. Neither is recorded by the runtime, and both are short-lived. + +## The leaked value + +The leaked value must be replaced after the fix is on the home server, because only then can the +mesh rotate it. On that apply the bootstrap step runs again, because its code changed, and records +the current value as applied. The rotation is then one call to the controller's `rotate` verb, by +machine, module and secret name (`mosquitto`, `admin`), with a why naming this issue. It waits for the +operator's approval. + +Where the old value went: + +- **The runtime's events.** They are held in memory and are bounded. On the home server the oldest + are about a minute old, so the value leaves the stream within a minute of the last call that + carried it. The runtime runs without debug logging, so its own log did not record execs. +- **The broker's log.** It does not carry the value. The broker logs an administrative command only + at the information level, which this broker does not log, and masks a password even there. + `docker_secrets_in_logs` found nothing in any container's log on the home server. +- **The module's own log.** It does not carry the value. A failed call's message was built from the + tool's output, never from its arguments. This was checked by reading the code, not the log, so that + the check could not copy the value. +- **Backups.** They do not carry the value. Events are never written to disk, and no backup reads them. +- **Agent transcripts.** Any transcript of a `docker_events` call with `execs: true` on the home server + before the fix holds the value, including the session that found this. The rotation is what makes + those copies harmless. + +## How it is checked + +| Rule | Checked by | +|---|---| +| mosquitto puts no password on a command line | Module test: a stand-in `mosquitto_ctrl` records its arguments, its options file and its stdin. No password reaches the arguments, the options file is 0600, and a password being set arrives at the prompt. Lab: the exec events of a real broker carry no value. | +| An argument list carrying a secret is refused before it runs | Module test: the guard refuses the admin password and a password being set, by name, and its message carries no value. The admin tool refuses `createClient -p`, `setClientPassword` with a password, and `init`. | +| The admin password rotates | Lab, with made-up values: a first run records the current value, an unchanged run does nothing, and a run after the value changed re-keys the running broker. The new value then administers the broker and the old one is refused. Live: the rotation above, then a provisioning pass. | +| `docker_events` never answers a secret an exec carried | Module tests: a broker's admin password after `-P` is redacted and named, and so are a client password after `createClient -p`, `setClientPassword`'s password, `redis-cli -a`, `PGPASSWORD=`, `--password=` and a URI password. A port, a path and a plain command are left as they are. | +| `docker_secrets_in_events` names, never quotes | Module test: the scan counts each exec once, names container, module, what it was and the program, and its answer carries no value. | +| keycloak's repair passes no password as an argument | Module test: the script contains no `--password "$…"`, no `--new-password` and no `-p "$…"`. |