jschoubben
e41eed0852
The certificate is genuine, from Mesh Internal CA, and nothing on the workstation trusts it — verbatim the error the report gives. The public name on the same proxy verifies cleanly, which puts the fault exactly where the report puts it. What is in the way is not an assignment. `ca-trust` is merged in the catalogue and has never been registered with the mesh — 39 of 76 manifests are — so there is no module to assign. It dry-runs clean and needs no artifact built. Two findings from reproducing it, both their own issues: 157 — every routed name is published with an `.internal` alias that nothing serves. The hosts file says keycloak.novox.be.internal; the proxy serves keycloak.novox.internal and refuses the other by name. The first three names I tried came from the hosts file and failed with a TLS alert rather than a verification error, which pointed at a regression that had not happened. 158 — the proxy re-logs all 52 routes every two seconds, 31 times a minute. The one line that explained 157 sat between two of them. Also recorded, because it was nearly filed as a defect and is not one: step-ca publishes roots as /roots.pem, which is PEM, so ca-trust's fetch and its refuse-a-non-certificate guard are both right. Its other endpoint /roots returns JSON that contains the text the guard greps for, so the guard is sound only because of which path is published.
83 lines
3.9 KiB
Markdown
83 lines
3.9 KiB
Markdown
# 129 — diagnosis
|
|
|
|
*2026-09-30, from the workstation the issue was opened on.*
|
|
|
|
## Still live, and reproduced exactly
|
|
|
|
The certificate is genuine, the authority is the mesh's, and nothing on the machine trusts it:
|
|
|
|
```
|
|
$ openssl s_client -connect keycloak.novox.internal:443 -servername keycloak.novox.internal
|
|
subject=CN=keycloak.novox.internal
|
|
issuer=O=Mesh Internal CA, CN=Mesh Internal CA Intermediate CA
|
|
Verify return code: 20 (unable to get local issuer certificate)
|
|
|
|
$ curl https://keycloak.novox.internal/
|
|
curl: (60) SSL certificate OpenSSL verify result: unable to get local issuer certificate (20)
|
|
```
|
|
|
|
`trust list` holds no entry for the mesh. The anchors present are two `mkcert` development roots and
|
|
the predecessor's lab root — the report's account of the trust store is unchanged.
|
|
|
|
The public name on the same proxy verifies cleanly (`CN=keycloak.novox.be`, Let's Encrypt, return code
|
|
0), which places the fault exactly where the report puts it: not in the proxy, not in the authority,
|
|
and not in the certificate.
|
|
|
|
**The name matters, and the report's "every HTTPS name the mesh serves internally" is too broad.** The
|
|
served internal name is `<label>.<node>.internal`. The hosts file also carries
|
|
`<label>.<public-domain>.internal`, which nothing serves and which fails differently — that is
|
|
[issue 157](../157-a-routed-names-internal-alias-is-served-by-nothing/00-report.md), found while
|
|
reproducing this, and it cost the first several minutes of this diagnosis.
|
|
|
|
## The authority serves what the module needs
|
|
|
|
`step-ca` is up and healthy, and publishes `roots: /roots.pem` for both `acme-ca` and
|
|
`internal-acme-ca`. That endpoint returns PEM:
|
|
|
|
```
|
|
$ curl -sk https://127.0.0.1:9000/roots.pem
|
|
-----BEGIN CERTIFICATE-----
|
|
MIIBvzCCAWWgAwIBAgIQYa2CkdJk16JyG/dVy2qoEzAKBggqhkjOPQQDAjA+…
|
|
```
|
|
|
|
So `${bound:internal-acme-ca:roots}` in the `ca-trust` module composes to a URL that returns a
|
|
certificate, and the module's own check — refuse a body that is not one — is checking the right thing.
|
|
|
|
Worth recording because it was nearly filed as a defect: step-ca *also* serves `/roots`, which returns
|
|
`{"crts":["-----BEGIN CERTIFICATE-----\n…"]}`. That body contains the literal text the module greps
|
|
for, so had the module used `/roots` it would have installed JSON into the anchors directory and
|
|
reported success. It does not use it. The guard is sound only because the published path is the PEM
|
|
one, which is worth knowing before anybody changes either.
|
|
|
|
## What is actually in the way
|
|
|
|
**The module is not registered.** The report and the work plan both say it exists and is merged, which
|
|
it does — `mesh-catalog modules/ca-trust`, on `main`. But the mesh has never been told about it:
|
|
|
|
```
|
|
$ mesh-controller module list | grep -iE 'ca-trust|step-ca'
|
|
step-ca 1 built 67f5f4cf on novox
|
|
```
|
|
|
|
39 of the catalogue's 76 manifests are registered. `ca-trust` is one of the 37 that are not, so it
|
|
cannot be assigned to anything — "assign it to one machine" has no module to name.
|
|
|
|
A dry run confirms it registers cleanly and needs no artifact built: it declares a directory, a script,
|
|
a unit and a service, and no image.
|
|
|
|
```
|
|
$ mesh-controller build <catalogue> --path modules/ca-trust --dry-run
|
|
… the manifest, parsed and validated
|
|
```
|
|
|
|
## So the remaining work is three steps, not one
|
|
|
|
1. **Register it** — build it from the catalogue, which pins nothing because it has no artifacts.
|
|
2. **Assign it** to a machine. The workstation this was observed on is the honest first choice: it is
|
|
where a person meets the fault, and it is where the check can be made with a plain client.
|
|
3. **Verify** `curl https://<label>.<node>.internal/` with no flags, and `trust list` naming the mesh.
|
|
|
|
Then removal, which the module declares and nothing has exercised: unassigning must take the anchor
|
|
away and refresh the bundles ([ADR 0147](../../02-DECISIONS/0147-a-module-anchors-the-meshs-authority.md)),
|
|
and that is the half most likely to be wrong, because it is the half nobody reaches by accident.
|