jschoubben
e41eed0852
The certificate is genuine, from Mesh Internal CA, and nothing on the workstation trusts it — verbatim the error the report gives. The public name on the same proxy verifies cleanly, which puts the fault exactly where the report puts it. What is in the way is not an assignment. `ca-trust` is merged in the catalogue and has never been registered with the mesh — 39 of 76 manifests are — so there is no module to assign. It dry-runs clean and needs no artifact built. Two findings from reproducing it, both their own issues: 157 — every routed name is published with an `.internal` alias that nothing serves. The hosts file says keycloak.novox.be.internal; the proxy serves keycloak.novox.internal and refuses the other by name. The first three names I tried came from the hosts file and failed with a TLS alert rather than a verification error, which pointed at a regression that had not happened. 158 — the proxy re-logs all 52 routes every two seconds, 31 times a minute. The one line that explained 157 sat between two of them. Also recorded, because it was nearly filed as a defect and is not one: step-ca publishes roots as /roots.pem, which is PEM, so ca-trust's fetch and its refuse-a-non-certificate guard are both right. Its other endpoint /roots returns JSON that contains the text the guard greps for, so the guard is sound only because of which path is published.
3.4 KiB
status, opened, located-in
| status | opened | located-in | |
|---|---|---|---|
| diagnosing | 2026-09-26 |
|
129 — nothing makes a machine trust the mesh's own certificate authority
What was observed
On an enrolled, adopted workstation — on the private network, resolving the mesh's names through the mesh's resolver — every HTTPS name the mesh serves internally fails verification:
curl https://<a name the mesh routes internally>/
curl: (60) SSL certificate OpenSSL verify result: unable to get local issuer certificate (20)
The route proxy presents a certificate issued by the mesh's internal authority (step-ca, the
internal-acme-ca provision). The machine's trust store holds the predecessor's authority
and a developer tool's local root, and nothing of the mesh's. No module installs the mesh's
root, and no fact carries it: step-ca's only consumers are proxies, which obtain certificates
over ACME and never need the root on the machine they run on.
Issue 048 found the same shape for the mesh's registry and resolved it by treating the private network as the transport security (ADR 0082): the runtime pulls in the clear, over the tunnel. That answer does not carry over. A browser, git over HTTPS, a package manager and every TLS client a person or a module uses verify the certificate chain, and there is no "insecure registries" for them — nor should there be.
Consequences today, all silent until someone tries:
- a person on a workstation cannot open any internal HTTPS name without a warning;
- git over HTTPS to the mesh's forge fails, so the working clone URL is ssh-only;
- a module on a non-hub machine that calls another module's internal HTTPS name fails verification unless its image happens to carry the root;
- the predecessor's authority cannot be retired from any machine while anything there still speaks TLS to a mesh name, because it is the only authority those machines trust.
What would have prevented it
- A mesh fact carrying the internal authority's root (public material; the controller or the step-ca module is its source), written onto every machine on the private network — the same reasoning that has the private network write the registry trust and the names: being on the network is what makes a machine one that speaks to the mesh's names.
- A resource that puts it where the machine's TLS clients look — on Arch,
/etc/ca-certificates/trust-source/anchors/— and refreshes the extracted bundles (update-ca-trust). The refresh is the open design question: it is a command, and the link may not carry an action (ADR 0005). A declared one-shot unit, or a host primitive for "trust this anchor", are the obvious candidates. - Removal symmetric to arrival: undeclared, the anchor goes and the bundles are refreshed again, so a machine leaving the mesh stops trusting it.
Evidence to carry into diagnosis
step-camodule: providesacme-ca/internal-acme-ca, listens on 9000 for proxies; no resource writes its root anywhere but its own state directory.- The private network's generator writes
/etc/hostsand the registry trust, and nothing about certificates. - On the workstation, the trust anchors present are the predecessor's authority and a local
development root;
trust listshows no entry for the mesh.