novox/mesh-controller
3
0
Fork 0
Code Issues Pull Requests 1 Releases Wiki Activity

route-proxy: an ACME account belongs to the authority that issued it

Browse Source 
autocert keeps its account key at one fixed name, `acme_account+key`, in whatever
directory it is given, and reuses it for ever. That is right while the authority
stays the same and silently wrong the moment it does not. Re-initialising the
internal CA makes a new authority with a new root: it has never heard of the
account in the cache, rejects every use of it, and autocert has no path back from
that. Nothing is re-registered, no order ever reaches the CA, and issuance stops
with nothing saying why — until somebody guesses that deleting the cache directory
by hand is the answer.

Name the directory after the authority instead of sharing one between all of them:
a digest of the ACME directory URL and the root this proxy was told to verify it
with. A re-initialised CA has a new root, the mesh delivers it as a changed bundle,
the proxy restarts on that file and lands in a directory with no account in it, so
autocert registers afresh and orders again. The healing is that "is this account
still valid" never has to be asked — an account is only ever found where it is
still valid, which needs no error codes, no probe at startup and no network call
that can itself fail.

It closes a latent one of the same shape: pointing ACME_DIRECTORY at production
after testing against staging reused the staging account, because the cache had no
idea the two were different.

Trailing whitespace around the delivered root is not a new authority — the mesh
writes that file, and a newline coming or going must not throw away an account.
Old directories are left on disk, unused: they hold the only copy of certificates
that may still be valid, and this program is not the thing that should decide a
certificate is finished with.

A proxy already holding certificates orders them once more on the first start after
this, because its account moves. Free against the lab's own CA and against staging;
one issuance per name against a public authority.

novox/hq ADR 0056

Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
This commit is contained in:
jschoubben
2026-09-10 21:02:00 +02:00
parent 54cb915c4e
commit 76fcbda9a7
2 changed files with 116 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
examples/route-proxy/authority_test.go
+73
View File
@@ -0,0 +1,73 @@
package main
import (
"path/filepath"
"strings"
"testing"
)
// The lab's first root, and the one a re-initialised CA generates in its place.
const (
firstRoot = "-----BEGIN CERTIFICATE-----\nMIIBeFIRST\n-----END CERTIFICATE-----\n"
secondRoot = "-----BEGIN CERTIFICATE-----\nMIIBeSECOND\n-----END CERTIFICATE-----\n"
)
// A re-initialised CA does not need somebody to delete the cache by hand.
//
// autocert keeps its account key at one fixed name and reuses it for ever. When an internal CA is
// re-initialised it has never heard of that account, rejects every use of it, and autocert has no
// path back: nothing is re-registered, no order reaches the CA, and issuance stops with nothing
// saying why. Naming the cache after the authority means the account is only ever found where it is
// still valid — the new root lands in a directory with no account in it, and autocert registers.
func TestANewCARootMeansANewAccountCache(t *testing.T) {
const directory = "https://anchor.internal/acme/acme/directory"
before := forThisAuthority("/var/lib/route-proxy/acme", directory, []byte(firstRoot))
after := forThisAuthority("/var/lib/route-proxy/acme", directory, []byte(secondRoot))
if before == after {
t.Fatalf("a re-initialised CA reuses the account it was rejected for: %s", before)
}
}
// And the SAME authority keeps the account it registered, restart after restart.
//
// This is the whole reason ACME_CACHE is required in the first place: an account and its
// certificates that did not persist would be re-ordered on every restart, which works silently until
// a rate limit says it does not. Whitespace around the delivered root is not a new authority — the
// mesh writes that file, and a trailing newline coming or going must not throw away an account.
func TestTheSameAuthorityKeepsItsAccount(t *testing.T) {
const directory = "https://anchor.internal/acme/acme/directory"
first := forThisAuthority("/var/lib/route-proxy/acme", directory, []byte(firstRoot))
again := forThisAuthority("/var/lib/route-proxy/acme", directory, []byte("\n"+firstRoot+"\n\n"))
if first != again {
t.Errorf("the same authority was given two caches, so every restart orders again:\n%s\n%s",
first, again)
}
}
// Staging and production are different authorities, and were sharing one account.
//
// The latent fault of the same shape: pointing ACME_DIRECTORY at production after testing against
// staging reused the staging account, because the cache had no idea they were different.
func TestStagingAndProductionDoNotShareAnAccount(t *testing.T) {
staging := forThisAuthority("/acme", stagingDirectory, nil)
production := forThisAuthority("/acme", "https://acme-v02.api.letsencrypt.org/directory", nil)
if staging == production {
t.Errorf("two issuers share one account: %s", staging)
}
}
// It stays inside the directory the mesh gave it, and is a plain name.
//
// The mesh owns ACME_CACHE and mounts it; a name derived from a certificate that escaped it — or
// that carried a separator out of the PEM — would put an account somewhere nothing persists.
func TestTheAccountCacheStaysWhereTheMeshPutIt(t *testing.T) {
const cache = "/var/lib/route-proxy/acme"
got := forThisAuthority(cache, "https://anchor.internal/acme/acme/directory", []byte(firstRoot))
if !strings.HasPrefix(got, cache+"/") {
t.Fatalf("the account cache is not under %s: %s", cache, got)
}
name := strings.TrimPrefix(got, cache+"/")
if name != filepath.Base(got) || strings.ContainsAny(name, "/.") {
t.Errorf("the account cache is not a plain name: %q", name)
}
}
examples/route-proxy/main.go
+43 -5
View File
@@ -20,9 +20,12 @@
package main
import (
"bytes"
"context"
"crypto/sha256"
"crypto/tls"
"crypto/x509"
"encoding/hex"
"encoding/json"
"fmt"
"log"
@@ -31,6 +34,7 @@ import (
"net/http/httputil"
"net/url"
"os"
"path/filepath"
"sort"
"strings"
"sync"
@@ -198,20 +202,22 @@ func run() error {
// nothing trusts yet — the lab's, or an internal step-ca. Trusting it is a deliberate act and
// names a file, rather than the client being told to skip verification: *skip* would also
// apply on the day this points at a public issuer, and nothing would say so.
var root []byte
if bundle := strings.TrimSpace(os.Getenv("ACME_CA_BUNDLE")); bundle != "" {
pem, err := os.ReadFile(bundle)
read, err := os.ReadFile(bundle)
if err != nil {
return fmt.Errorf("ACME_CA_BUNDLE names %s and it cannot be read: %w", bundle, err)
}
root = read
// An empty bundle means the issuer's root is already in the system trust store — a public
// authority whose root ships with the OS, pointed at by a provider that serves an empty
// root (novox/hq ADR 0056). The mesh always writes the bundle file, so it exists and holds
// nothing; that is the signal to fall back to the system roots, the same as if nothing had
// named a bundle at all. A file that holds bytes but no certificate is still a
// misconfiguration and is refused, because there the operator meant to trust something.
if strings.TrimSpace(string(pem)) != "" {
if strings.TrimSpace(string(root)) != "" {
pool := x509.NewCertPool()
if !pool.AppendCertsFromPEM(pem) {
if !pool.AppendCertsFromPEM(root) {
return fmt.Errorf("%s holds no certificate this can trust", bundle)
}
client.HTTPClient = &http.Client{
@@ -220,13 +226,16 @@ func run() error {
}
}
}
// Where this authority's account and certificates are kept. Per authority, not per proxy — see
// forThisAuthority, which is what makes a re-initialised CA heal itself.
mine := forThisAuthority(cache, issuer(), root)
manager := &autocert.Manager{
Cache: autocert.DirCache(cache),
Cache: autocert.DirCache(mine),
Prompt: autocert.AcceptTOS,
HostPolicy: onlyWhatTheMeshSaid(held),
Client: client,
}
log.Printf("issuing from %s, for whatever the mesh routes here", issuer())
log.Printf("issuing from %s into %s, for whatever the mesh routes here", issuer(), mine)
// Port 80 answers the HTTP-01 challenge and goes on proxying everything else. The challenge
// must be answered *at the name being certified*, which is why issuance happens on the node
@@ -245,6 +254,35 @@ func run() error {
return server.ListenAndServeTLS("", "")
}
// forThisAuthority is where one ACME authority's account and certificates are kept.
//
// **A cached ACME account belongs to the authority that issued it, and nothing in the cache says
// so** (novox/hq ADR 0056). autocert keeps its account key at one fixed name — `acme_account+key` —
// in whatever directory it is given, and reuses it for ever. That is right while the authority stays
// the same and silently wrong the moment it does not: an internal CA that is re-initialised is a new
// authority with a new root, it has never heard of the account in the cache, and every attempt to
// use it is rejected. autocert has no path back from that. Nothing is retried, nothing is
// re-registered, no order ever reaches the CA — issuance simply stops, with no error anybody sees,
// until a person deletes the directory by hand and finds out that was the answer.
//
// So the directory is named after the authority instead of being shared by all of them. The name is
// a digest of the two things that identify one: the directory URL, and the root this proxy was told
// to verify it with. Re-initialising the CA produces a new root; the mesh delivers it as a changed
// bundle; this proxy restarts on that file and lands in a directory with no account in it, so
// autocert registers afresh and orders again. **The healing is that the question "is this account
// still valid" never has to be asked** — an account is only ever found where it is still valid.
//
// It also fixes a latent one of the same shape: pointing ACME_DIRECTORY at production after testing
// against staging reused the staging account, because the cache had no idea they were different.
//
// The old directories stay on disk, unused. Left rather than deleted: they are the only copy of
// certificates that may still be valid, and this program is not the thing that should decide a
// certificate is finished with.
func forThisAuthority(cache, directory string, root []byte) string {
sum := sha256.Sum256([]byte(directory + "\x00" + string(bytes.TrimSpace(root))))
return filepath.Join(cache, hex.EncodeToString(sum[:])[:16])
}
// newTable is an empty routing table.
func newTable() *table {
return &table{to: map[string]*httputil.ReverseProxy{}, targets: map[string]string{}}