The mesh noticed 48 core failures in six days and told nobody (ADR 0227).
messenger holds the operator-channel seat: it consumes the controller's
condition events and sends them to Telegram and the desktop notifier,
deduplicated by key, reminded once, edited on clear, capped at 20 an hour
with the rest folded, and refusing anything carrying an address, a path or
a secret. mesh-watcher, on a machine other than the control node, sends to
Telegram directly when the self-check heartbeat or the bus goes silent.
vector handed logflare its API key as ?api_key= in every sink URL, and
logflare 1.4.0 prints a failed request's whole URL in its Plug.Cowboy
error report. Its ingest fails on every request here, so the key was in
the analytics log about every ten seconds. The report is an error, so no
log level hides it.
Every sink now sends the key in the x-api-key header, which logflare
reads first. A start script refuses to start logflare while the vector
config it is given still puts the key in a URL.
The key is marked "applied", not "at-start": logflare writes it into its
default user once and never updates it, so the mesh must not rotate it
by restarting.
Consumers were held to an S3 access key's 20 characters whatever they
required. Each provider now says what its backend keeps: minio 20,
PostgreSQL and MongoDB and DNS 63, Gitea 40, a mailbox 64, SQL Server 128,
Keycloak 255, unbounded where the store has no limit, and none for the
resolver and route provisions, which keep no name of their consumers.
Needs the controller that reads the field (mesh-controller, ADR 0225).
letta printed two passwords into its log for weeks and nothing noticed,
and docker_logs handed them to whoever asked. docker_secrets_in_logs
compares each container's recent lines with the secret-named values of
its environment, the passwords in its URIs, and any URI carrying a
password, and names what it found by container, module and variable -
never the value. docker_logs redacts the same values before answering.
letta 0.6.8 prints LETTA_PG_URI whole (startup.sh, alembic, server.py)
and its server password when it starts in secure mode, so both were in
the container's log on every one of its restarts. Newer letta still
prints both, and neither is a log level.
The URI now names no password: libpq reads it from a mounted pgpass
file (PGPASSFILE). The one print of the server password is rewritten by
a start script before the server starts, and the script refuses to start
letta if that print, or a password in the URI, is still there - a letta
that does not start says why; one that leaks says nothing.
Both own secrets say they are read at start, so `rotate` can replace the
server password the mesh made.
On 2.10.29 such a consumer was moved past a message now and then without
handing it over; the controller's events consumer has seven filters, and a
merge on the stream never reached it. The test reproduces the skip on 2.10.29
and keeps the image's release equal to the server it tests.
The bootstrap once created the temporary admin and then failed on a held port; marked only after
it succeeded, the cleanup did not know the admin existed and left it.
Twice the identity provider's admin kept an older password than the one the
mesh minted (an adopted, then a moved database), and the provisioner failed
every consumer until it was repaired by hand (hq issue 179). The module now
checks the admin's login and repairs a refusal itself through the server's
bootstrap command, verifies, brakes a failed repair and announces it, and
stops asking the server while refused. Ported to Go to change it.
A provider failed every consumer for a day and said so only in its journal
(hq issue 179). The provisioner loop now emits provisioner.failing after five
minutes without a success — create, check or secret — and repeats it every
fifteen; provisioner.recovered on the next success, on withdrawal, and on the
first success after a restart, so the controller can name it in status
(hq ADR 0224).
Requiring wildcard-resolution gave each a consumer identity; mesh_<machine>_networkmanager is
over the 20 characters a backend keeps, and the anchor's declaration, which carries every
consumer's grant, could not be composed.
Two files say one fact, the machine's name, and nothing owned /etc/hostname.
The name written is the operator's hostname setting, with no default: three
of four machines call themselves something other than their mesh name, and
renaming one is the operator's call. It takes effect at the next boot.
The program that manages a machine's network is the one that would rewrite
the resolver file, so its module now writes it: networkmanager,
systemd-networkd and dhcpcd render the same template from the resolver's
holders. resolv-conf declares nothing for one release, so every machine
hands the file over in one apply; it is removed once unassigned everywhere.
letta crash-loops on 'type "vector" does not exist': pgvector is not a
trusted extension, so only the provider's superuser can create it, and
the provisioner never did. A contribution may now name extensions; the
provider creates each (IF NOT EXISTS, available ones only) in the
consumer's database on every pass. Go per the standing rule for a
TypeScript module that changes. letta asks for vector.
Next.js binds the address HOSTNAME names; docker sets HOSTNAME to the container's id, so studio
answered only on its network address while its image's health check asks localhost, and it read
unhealthy while working. HOSTNAME=:: as the upstream compose file sets it.
musl asks every nameserver at once and takes the first reply, so a public
resolver's NXDOMAIN for a mesh name beat the mesh's answer in every Alpine
container (hq ADR 0223). resolv-conf now renders /etc/resolv.conf from the
holders of mesh-dns-resolver, this machine first when it holds one; dnsmasq's
comments say the seat may have several holders.
The controller's private network writes insecure-registries into daemon.json, a file this
module owns. The runtime's module states it instead, through ${seat:mesh-artifact-store:reach}
(hq ADR 0222), so the controller can stop generating its registry-trust resources.
The resolver answered a machine's name only by wildcard, which says there is no such name when
asked for an IPv6 address; musl reads that as final, so Alpine containers could not find a
machine at all. A host record per machine answers that the name exists and has none, as the
hosts file the per-machine resolvers read used to.
The resolv-conf comment pointed operators at it and at NetworkManager as
alternative claimants; it now says the uplink's holder is required beside it
(hq ADR 0220).
The tools were TypeScript; the mesh's modules are Go. The write to /etc/hosts is now staged and
moved into place rather than written over the live file.
resolv-conf would have taken over dnsmasq's write into daemon.json — the same defect issue 190
names. docker, on every machine, now writes live-restore and reloads its own service. Log rotation
is left as each machine has it.
- dnsmasq holds mesh-dns-resolver: provides wildcard-resolution mesh-wide, forwards every declared
zone (zones fact), listens on the private address and loopback only, reads no hosts file and no
operator's files, and no longer writes the container runtime's dns.
- resolv-conf names the mesh's resolver by address, then 1.1.1.1, timeout 1, one attempt; it now
holds the runtime's live-restore, which dnsmasq held and every node needs.
- resolved-split-dns routes the suffix to the mesh's resolver by address, not 127.0.0.1.
- hosts: new module holding node-hosts-file — the machine's own lines in its block of /etc/hosts,
the operator's lines kept, changed by entries/add/remove through sudo -n.
The node-build-agent seat now promises these verbs, and a holder that does not
name them cannot hold it. The binary serving them is mesh-controller's
cmd/mesh-builder; merge only once a controller carrying the seat's new verbs runs,
since an older one refuses a claim naming verbs its row lacks.
The dry run stood until the controller held each kept archive by a manifest
(mesh-controller #53) and an apply stopped reopening the collection window
(mesh-host #23, hq issue 224). The controller's collection command now
reports 134 of 134 kept archives held. hq issue 253.
With the poll the only announcer of a merge, its cost showed: every 30 s it
asked every repository for its pull requests, a pass outlasted the tick, and
passes piled up beside each other — a merge was announced four and a half
minutes late, and two passes at once could each announce it. A pass now asks
only repositories updated since a minute before the last look, and the next
pass starts when this one ends. hq issue 250.
The journal verb ran journalctl as the operator account, which outside the
journal's group sees only its own entries: every system service read
'-- No entries --', and a person reached for a shell. The read now
escalates with sudo -n like the acts; ported to Go with every test. hq
issue 255.
mesh-admin
jochen
jschoubben