Merge pull request 'Issue 282: a secret on a command line is kept in the runtime's events' (#160) from issue/282-a-secret-on-a-command-line into main
This commit was merged in pull request #160.
This commit is contained in:
@@ -0,0 +1,137 @@
|
||||
---
|
||||
status: located
|
||||
opened: 2026-10-07
|
||||
located-in: [mesh-catalog modules/mosquitto, mesh-catalog modules/docker, mesh-catalog modules/keycloak, mesh-catalog modules/minio, mesh-catalog modules/gitea]
|
||||
fixed-by:
|
||||
amended-design:
|
||||
---
|
||||
|
||||
# 282. A secret on a command line is kept in the runtime's events
|
||||
|
||||
## Symptom
|
||||
|
||||
Found on 2026-10-07 while research 032 read the home server's container runtime, read-only. The
|
||||
`mosquitto` module administers its broker by running `mosquitto_ctrl` inside the broker's container
|
||||
with `docker exec`, and it passed the broker's **admin password as an argument** (`-P <password>`) on
|
||||
every call: the provisioner's check every minute, and every call of its tools. The container runtime
|
||||
records the command line of every exec in its event stream, as the event's action
|
||||
(`exec_create: <the arguments joined by spaces>`). So:
|
||||
|
||||
- `docker events` showed the admin password to anyone who may ask the runtime;
|
||||
- the docker module's `docker_events`, asked with `execs: true`, returned it, and any agent transcript
|
||||
of that call holds a copy.
|
||||
|
||||
No value appears in this record.
|
||||
|
||||
## Cause
|
||||
|
||||
**The module chose argv and wrote down why.** Its client said that `mosquitto_ctrl` 2.x offers no
|
||||
way to give the password but `-P` or an interactive prompt, called the exposure "momentary and
|
||||
confined to this single-purpose runtime container", and went ahead. Both halves were wrong:
|
||||
|
||||
- `mosquitto_ctrl` takes its connection options from a file (`-o <file>`), and its `createClient`
|
||||
and `setClientPassword` prompt for a password on stdin when none is given. Both work without a
|
||||
terminal (checked against the 2.1.2 the module pins).
|
||||
- An exec's command line is not confined to the container. The runtime records it, outside the
|
||||
container, for as long as it keeps its events, and hands it to every reader of its event stream.
|
||||
|
||||
**Nothing in the mesh looked at command lines.** [Issue 268](../268-letta-printed-its-passwords-into-its-log/00-report.md)
|
||||
taught the docker module to find a secret in a container's own log and to redact it from
|
||||
`docker_logs`. The same values on an exec's command line went through `docker_events` unredacted,
|
||||
and no check compared exec command lines with anything.
|
||||
|
||||
## Other modules that put a secret on a command line
|
||||
|
||||
The catalogue was searched for every place a module runs a program (`docker exec`, `exec.Command`,
|
||||
`execFile`, `spawn`, dump commands in manifests, container arguments) and for password-taking flags
|
||||
(`-P`, `-p`, `-a`, `--password`, `--secret-key`, `NAME=value`):
|
||||
|
||||
| module | what | seen by | here |
|
||||
|---|---|---|---|
|
||||
| `mosquitto` | the broker's admin password as `-P`, and a client's password as `createClient -p` or `setClientPassword <user> <password>`, on every `docker exec` | the runtime's events; the machine's process table | fixed |
|
||||
| `mosquitto` | the admin password as the last argument of `dynsec init`, in the throwaway container that seeds the broker's store | that container's description while it ran; the process table | fixed |
|
||||
| `keycloak` | the repair script gave `kcadm` the temporary admin's password (`--password`) and the mesh's admin password (`--new-password`) as arguments, inside the container | the machine's process table, for as long as the JVM ran | fixed |
|
||||
| `minio` | `mc alias set` with the root user and password as arguments (and `mc` then kept them in its alias file) | the machine's process table | fixed |
|
||||
| `minio` | `mc admin user svcacct add --secret-key <a consumer's secret>` | the machine's process table, for the second `mc` runs | **open** |
|
||||
| `gitea` | its run-once admin step runs `gitea admin user create --password "$(cat …)"` inside its container | the machine's process table, while the CLI runs | **open** |
|
||||
|
||||
Checked and clean: the `mssql` and `nats` dumps hand their secret over on stdin; the `mongodb` dump
|
||||
writes its password into an options file inside the container with a shell builtin; the `postgres`
|
||||
and `supabase` dumps carry no password; `keycloak`'s own `docker exec` carries its passwords on stdin;
|
||||
`nextcloud`, `mailu`, the forge's delivery notes and the `nats` monitor carry no secret.
|
||||
|
||||
Only an exec is recorded by the runtime. The other rows are visible in the machine's process table
|
||||
to every local user while the program runs, and are kept by anything that records process starts.
|
||||
|
||||
## Fix
|
||||
|
||||
Proposed in mesh-catalog pull request #100. Not merged at the time of writing.
|
||||
|
||||
**mosquitto.** `mosquitto_ctrl` runs under a fixed shell that holds no secret. The shell reads the
|
||||
admin's name and password from the first two lines of stdin into an options file only it can read,
|
||||
runs `mosquitto_ctrl -o <that file>`, and removes the file as it exits. The rest of stdin answers the
|
||||
tool's own password prompt, so a client's password is never an argument either. `dynsec init` reads
|
||||
its password from stdin the same way. **Before anything runs, the module refuses an argument list
|
||||
that carries the admin password or a password being set**, and says which, by name. The admin tool
|
||||
refuses the three dynsec commands that would put a password on a command line.
|
||||
|
||||
**The admin password can now be rotated.** It was declared as a bare path, which `rotate` refuses.
|
||||
Saying `taken: at-start` alone would have been false: the broker's store keeps the admin's hash, and a
|
||||
new value written by the mesh would leave the module locked out. The module's run-once bootstrap step
|
||||
now runs again when the secret changes (`restart-on` names it). It keeps the value it last applied in
|
||||
a 0600 file of its own state. When the mesh's value is another one, it connects with the applied
|
||||
value and sets the new one, online, with the password on stdin. Then it checks the new one with a
|
||||
connect and records it. The broker keeps running and keeps every client. A re-key that cannot be done
|
||||
fails the step and says why.
|
||||
|
||||
**docker.** `docker_events` redacts every exec's command line before it answers. It hides the values
|
||||
`docker_logs` hides, and also, whatever their source, what a command line carries by its shape: the
|
||||
word after a flag that takes a password, a `NAME=value` whose name says secret, and the password a
|
||||
dynsec command sets. A new tool, `docker_secrets_in_events`, reads the exec events of a window and
|
||||
names each secret it finds by container, module, what it was and the program, with counts and the
|
||||
first and last time. It never returns the value or the command line. `docker_inspect` redacts a
|
||||
container's own command line the same way.
|
||||
|
||||
**keycloak** gives `kcadm` each password in `KC_CLI_PASSWORD`, set for the one command that needs it.
|
||||
A command's environment can be read by its own user only. **minio** gives `mc` its admin alias in
|
||||
`MC_HOST_mesh`, in `mc`'s environment, and refuses an argument list that carries the root password.
|
||||
|
||||
**What stays open.** `mc admin user svcacct add` takes a consumer's secret key only as an argument.
|
||||
Moving it off the command line means making the admin API's encrypted request in the module's own
|
||||
code. `gitea admin user create` takes the admin password only as an argument. Moving it means creating
|
||||
the admin another way. Neither is recorded by the runtime, and both are short-lived.
|
||||
|
||||
## The leaked value
|
||||
|
||||
The leaked value must be replaced after the fix is on the home server, because only then can the
|
||||
mesh rotate it. On that apply the bootstrap step runs again, because its code changed, and records
|
||||
the current value as applied. The rotation is then one call to the controller's `rotate` verb, by
|
||||
machine, module and secret name (`mosquitto`, `admin`), with a why naming this issue. It waits for the
|
||||
operator's approval.
|
||||
|
||||
Where the old value went:
|
||||
|
||||
- **The runtime's events.** They are held in memory and are bounded. On the home server the oldest
|
||||
are about a minute old, so the value leaves the stream within a minute of the last call that
|
||||
carried it. The runtime runs without debug logging, so its own log did not record execs.
|
||||
- **The broker's log.** It does not carry the value. The broker logs an administrative command only
|
||||
at the information level, which this broker does not log, and masks a password even there.
|
||||
`docker_secrets_in_logs` found nothing in any container's log on the home server.
|
||||
- **The module's own log.** It does not carry the value. A failed call's message was built from the
|
||||
tool's output, never from its arguments. This was checked by reading the code, not the log, so that
|
||||
the check could not copy the value.
|
||||
- **Backups.** They do not carry the value. Events are never written to disk, and no backup reads them.
|
||||
- **Agent transcripts.** Any transcript of a `docker_events` call with `execs: true` on the home server
|
||||
before the fix holds the value, including the session that found this. The rotation is what makes
|
||||
those copies harmless.
|
||||
|
||||
## How it is checked
|
||||
|
||||
| Rule | Checked by |
|
||||
|---|---|
|
||||
| mosquitto puts no password on a command line | Module test: a stand-in `mosquitto_ctrl` records its arguments, its options file and its stdin. No password reaches the arguments, the options file is 0600, and a password being set arrives at the prompt. Lab: the exec events of a real broker carry no value. |
|
||||
| An argument list carrying a secret is refused before it runs | Module test: the guard refuses the admin password and a password being set, by name, and its message carries no value. The admin tool refuses `createClient -p`, `setClientPassword` with a password, and `init`. |
|
||||
| The admin password rotates | Lab, with made-up values: a first run records the current value, an unchanged run does nothing, and a run after the value changed re-keys the running broker. The new value then administers the broker and the old one is refused. Live: the rotation above, then a provisioning pass. |
|
||||
| `docker_events` never answers a secret an exec carried | Module tests: a broker's admin password after `-P` is redacted and named, and so are a client password after `createClient -p`, `setClientPassword`'s password, `redis-cli -a`, `PGPASSWORD=`, `--password=` and a URI password. A port, a path and a plain command are left as they are. |
|
||||
| `docker_secrets_in_events` names, never quotes | Module test: the scan counts each exec once, names container, module, what it was and the program, and its answer carries no value. |
|
||||
| keycloak's repair passes no password as an argument | Module test: the script contains no `--password "$…"`, no `--new-password` and no `-p "$…"`. |
|
||||
Reference in New Issue
Block a user