Bring build-agent and route-proxy in whole (issue 1019)

This commit is contained in:
2026-10-17 18:15:33 +02:00
parent c43772fd94
commit 1586bd1134
17 changed files with 1032 additions and 0 deletions
+29
View File
@@ -0,0 +1,29 @@
ARG ALPINE_BASE=alpine:3.20
ARG GO_BASE=golang:1.25
# The route-proxy module's runtime image: the reference reverse proxy compiled into a container.
#
# **The proxy source sits beside this module.** The canonical proxy, the contract written as something
# that runs, lives in this same repository, mesh-controller, at examples/route-proxy (novox/hq 08-connectivity
# §3), and the repository is the build context (a module lives whole in one repository, novox/hq ADR 0420),
# so this Dockerfile compiles ./examples/route-proxy from the repository root. The build seat hands it only
# that program's build source, its import closure, go.mod, go.sum and the vendored packages it uses
# (novox/hq ADR 0267, the artifact's `compiles`), so a merge elsewhere in the repository does not
# move this image, and a COPY of anything outside it fails by name.
#
# docker build -f modules/route-proxy/Dockerfile -t mesh-route-proxy:development .
#
# The mesh pins the digest of what this produces; the committed module.json carries the placeholder
# digest every mesh-built image does, replaced at publish.
FROM ${GO_BASE} AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -buildvcs=false -ldflags=-buildid= -o /mesh-route-proxy ./examples/route-proxy
# A small runtime with the public CA roots the ACME client needs to reach a real authority, and run
# as root so it can bind :80 and :443 — the two privileged ports a public front door listens on.
FROM ${ALPINE_BASE}
RUN apk add --no-cache ca-certificates
COPY --from=build /mesh-route-proxy /usr/local/bin/mesh-route-proxy
ENTRYPOINT ["/usr/local/bin/mesh-route-proxy"]
+92
View File
@@ -0,0 +1,92 @@
# route-proxy — the mesh's public ingress
The shipping form of the reference reverse proxy (novox/hq ADR 0007, novox/hq 08-connectivity §3
"Exposure"). It is the `traefik` replacement: the module that **provides `route`**.
## What it is
**A route is a grant.** A module that must be reachable declares `requires: ["route"]` and
contributes the name it wants and the port it listens on; this module provides `route`, receives
every consumer that asked as a file, and forwards by the `Host` header of each request to
`http://<at>:<port>` — where `at` is where the mesh says that consumer's machine is (empty means
this machine, reached over loopback).
The control plane does not write the proxy's configuration. It delivers the facts — the
contributions file at `receives.route` — and the proxy turns them into whatever it runs. That
boundary is why swapping this proxy for another costs nothing to the modules that publish through
it.
| field | value | why |
|---|---|---|
| `provides` | `route` (scope `mesh`) | a module on any node may require it |
| `serves.route` | `{}` | a route hands back a name, not a credential — the consumer already knows the name it asked for |
| `receives.route` | `…/routes/mesh.json` | where the mesh writes every contribution; the proxy reads it as `$ROUTES` and re-reads on change |
| `listens` | `80` and `443`, both `from: anywhere` | the one machine with a public opening; the firewall opens these for free (ADR 0045) |
No provisioner: the proxy neither mints a credential for anybody nor answers a provision other
than `route`. It reads the file the mesh writes, and its own broker credential is for its tools.
## Which issuer: Let's Encrypt, stated here (novox/hq ADR 0226)
**The public issuer is this module's own fact, not a provision.** Until 2026-10-06 it came from a
second module, `public-acme`, that ran nothing and offered `acme-ca` pointing at Let's Encrypt; every
node running a proxy was assigned it too, and nothing else ever consumed it. It is retired: the proxy
names the production directory in its own `acme.env`, and the only issuer it is still bound to is the
mesh's own (`internal-acme-ca`, from `step-ca`).
**`acme.env` is byte for byte what the binding rendered.** The proxy keeps each authority's account
and certificates in a directory under `/var/lib/route-proxy/acme` named after a digest of the
directory URL — `https://acme-v02.api.letsencrypt.org:443/directory`, port included, as the binding
spelled it — and of the root bundle the `trust` container copies in. Any change to either is a new
authority to the proxy: a new account, and every routed name ordered again against Let's Encrypt's
rate limits. So the URL keeps the `:443`, `ACME_ROOTS_PATH` stays empty, and the `trust` artifact's
pinned image is not moved without a decision; mesh-controller's
`TestRouteProxyKeepsItsPublicAccountDirectory` holds all three.
## How it ships the Go proxy
The proxy is a Go program, unlike the TypeScript tool-runtime modules. Its source sits in this same
repository, mesh-controller, at `examples/route-proxy`, the contract written as something that runs. This
module ships the packaging: a multi-stage [`Dockerfile`](Dockerfile) whose build context is the repository
root and which compiles `./examples/route-proxy` into `mesh-route-proxy`. The committed `module.json` carries the
placeholder digest every mesh-built image does (`@sha256:0000…`); the mesh pins the real digest at
publish.
In the lab, `mesh-lab/scripts/build-route-proxy-image.sh` builds this into the local daemon as
`mesh-route-proxy:development`, and a scenario stocks it and serves it by digest.
## Certificates, and the endpoint (resolves 04-ISSUES/004)
TLS is opt-in and terminates here for public names, via ACME (autocert). A proxy with no
`TLS_LISTEN` serves plain HTTP; with it, it obtains a publicly-trusted certificate for exactly the
names the mesh routes here (`HostPolicy` = only routed names, so a scan cannot spend the account's
issuance quota) and answers the HTTP-01 challenge on `:80` at the name being certified.
**`ACME_DIRECTORY` defaults to Let's Encrypt *staging*, not production.** This closes
novox/hq 04-ISSUES/004 (certificate issuance targets production):
the issuer is a per-node choice, and the safe endpoint is the default. Production issuance is
rate-limited per domain and per account and does not replenish quickly; defaulting to production
would leave the safe path depending on somebody remembering to opt out of it, on exactly the work
most likely to iterate. A staging certificate is trusted by no browser, so the mistake announces
itself on the first request rather than a fortnight later at the rate limit.
**The module states production.** The binary's staging default stands for anything that runs it
without the module — the lab, a developer's machine — and the module's `acme.env` names production
for every node it is assigned to, because a node assigned the public proxy is one that serves public
traffic.
| env | default | meaning |
|---|---|---|
| `ROUTES` | `/routes/mesh.json` | the contributions file the mesh writes (`receives.route`) |
| `LISTEN` | `:80` | HTTP, and the ACME HTTP-01 challenge |
| `TLS_LISTEN` | `:443` | HTTPS; unset to serve plain HTTP only |
| `ACME_CACHE` | `/acme` | where issued certificates persist; required when `TLS_LISTEN` is set, so a restart does not re-order |
| `ACME_DIRECTORY` | Let's Encrypt **staging** in the binary; production in the module's `acme.env` | the issuer |
| `ACME_CA_BUNDLE` | *(unset)* | a file of roots to trust for the issuer's own API — set only for a private/lab authority whose API certificate the world does not yet trust |
## Answered by the provider helper (issue 792)
A small Go program beside the proxy (`cmd/route-provider`) runs the SDK's provider helper for `route`, with no credential,
and the offer says `"answered-by": "helper"`: a handed consumer (gitea's origins file) is filled by its machine's client
from this module's entry, which carries the public names the controller composed for it. The module is **not** marked
moved: its own files, `acme.env` and the broker secret, are composed as before, and the proxy itself is unchanged.
@@ -0,0 +1,37 @@
// route-provider: route-proxy's Go half, one binary the node's runtime launches and speaks MCP to over stdio
// through the Go SDK. It serves no tools; it runs the SDK's provider helper that answers the consumers of `route`
// (issue 792). The TypeScript adapter beside it keeps writing the proxy's files.
//
// stdout is the MCP channel; everything this module says, it says on stderr.
package main
import (
"context"
"fmt"
"os"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/provider"
)
func say(format string, args ...any) {
fmt.Fprintf(os.Stderr, "[route-proxy] "+format+"\n", args...)
}
func main() {
node := os.Getenv("MESH_NODE")
if node == "" {
say("MESH_NODE is not set; answering no consumers")
} else {
p := newProvider(node, provider.StateBus(ModuleName), say)
go func() {
if err := p.Run(context.Background()); err != nil {
say("the provider helper stopped: %v", err)
}
}()
}
if err := stdio.Serve("", nil); err != nil {
say("%v", err)
os.Exit(1)
}
}
@@ -0,0 +1,36 @@
package main
// route-proxy handed to the provisioner (novox/hq ADR 0349 clause 12, issue 792): the module runs the SDK's provider
// helper for `route`, so a consumer of a route — a handed gitea reading `${bound:route:name-web}` — is told its public
// name from the entry in this module's bucket, the one its machine's client fills its files from.
//
// **It composes nothing.** A route's public name is the label its consumer chose joined to the node's public domain,
// which only the controller knows (the domain is placement), so the controller composes it and carries it in the
// consumer's plan entry (`names`); the helper copies it into the entry's values unchanged. There is no credential for
// a route and nothing to make: the traefik files stay the TypeScript adapter's, written from the contributions the
// mesh still gives it.
import (
"git.novox.be/novox/mesh-sdk/go/provider"
)
// ProvisionName is what route-proxy provides, as its manifest says.
const ProvisionName = "route"
// ModuleName is this module.
const ModuleName = "route-proxy"
// newProvider is route-proxy answering its consumers over bus on node: the route, served as the manifest serves it
// (nothing of its own), with no credential, and nothing to make or retire for a consumer.
func newProvider(node string, bus provider.Bus, logf func(string, ...any)) *provider.Provider {
return provider.New(provider.Options{
Node: node,
Module: ModuleName,
Offers: map[string]provider.Offer{ProvisionName: {
Credential: provider.NoCredential,
Serves: map[string]any{},
}},
Bus: bus,
Logf: logf,
})
}
@@ -0,0 +1,125 @@
package main
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"os"
"sync"
"testing"
"git.novox.be/novox/mesh-sdk/go/provider"
)
// bucketBus is the helper's bus over in-memory buckets.
type bucketBus struct {
mu sync.Mutex
buckets map[string]map[string][]byte
}
func (b *bucketBus) Watch(_ context.Context, bucket string, handle func(string, []byte)) error {
b.mu.Lock()
defer b.mu.Unlock()
for k, v := range b.buckets[bucket] {
handle(k, v)
}
return nil
}
func (b *bucketBus) Put(_ context.Context, bucket, key string, value []byte) error {
b.mu.Lock()
defer b.mu.Unlock()
if b.buckets[bucket] == nil {
b.buckets[bucket] = map[string][]byte{}
}
b.buckets[bucket][key] = append([]byte(nil), value...)
return nil
}
func (b *bucketBus) Delete(_ context.Context, bucket, key string) error {
b.mu.Lock()
defer b.mu.Unlock()
delete(b.buckets[bucket], key)
return nil
}
// A consumer of a route is answered with the names its plan's entry carries, and nothing of a credential (issue 792):
// the entry's values are the controller's composition, copied unchanged, and no vault is asked.
func TestAConsumerOfARouteIsToldItsNamesFromItsPlanEntry(t *testing.T) {
bus := &bucketBus{buckets: map[string]map[string][]byte{}}
plan, _ := json.Marshal(map[string]any{"node": "g14", "module": "gitea", "provisioning": map[string]any{
"provisions": []any{map[string]any{"provision": ProvisionName, "provider": "novox",
"provider-module": ModuleName, "at": "10.7.0.1", "from": "novox", "as": "mesh_g14_gitea",
"names": map[string]any{"name": "git.example.org", "internal-name": "git.novox.internal",
"name-web": "git.example.org"}}}}})
if err := bus.Put(context.Background(), provider.AssignmentsBucket, "g14.gitea", plan); err != nil {
t.Fatal(err)
}
p := newProvider("novox", bus, t.Logf)
out, err := p.Pass(context.Background())
if err != nil {
t.Fatal(err)
}
if len(out) != 1 || out[0].Status != provider.Answered {
t.Fatalf("the pass did %+v", out)
}
var e provider.Entry
if err := json.Unmarshal(bus.buckets[provider.BucketOf(ModuleName)]["g14.gitea.route"], &e); err != nil {
t.Fatal(err)
}
if e.Values["name-web"] != "git.example.org" || e.Values["name"] != "git.example.org" ||
e.Values["internal-name"] != "git.novox.internal" || e.VaultName != "" || e.User != "" || e.Refused != "" {
t.Fatalf("the entry is %+v", e)
}
}
// route-proxy's own files and resources stay as they were (issue 792): the module answers its route through the
// helper but is not marked moved, so its acme.env, its broker secret and its containers are composed as today and
// the public ingress does not change. Everything of the manifest but its build, what it provides and the state it
// keeps is pinned by this digest; changing the ingress means changing it here, on purpose.
func TestRouteProxyIsNotMovedAndKeepsItsOwnFiles(t *testing.T) {
raw, err := os.ReadFile("../../module.json")
if err != nil {
t.Fatal(err)
}
var m map[string]any
if err := json.Unmarshal(raw, &m); err != nil {
t.Fatal(err)
}
if _, moved := m["moved"]; moved {
t.Fatal("route-proxy is marked moved: its own files would be handed")
}
provides, _ := m["provides"].([]any)
if len(provides) != 1 || provides[0].(map[string]any)["answered-by"] != "helper" {
t.Fatalf("route is not said to be answered by the helper: %v", provides)
}
for _, k := range []string{"build", "provides", "state", "reads"} {
delete(m, k)
}
pinned, err := json.Marshal(m) // sorted keys, compact
if err != nil {
t.Fatal(err)
}
sum := sha256.Sum256(pinned)
if got := hex.EncodeToString(sum[:]); got != "57a39de0943d8b33b9db6cb998ce97f8793602cca6cdfa5849c1c392390454a3" {
t.Fatalf("route-proxy's own manifest changed: digest %s", got)
}
// The two acme templates are pinned the same way (issue 849): they are what the proxy's certificates are asked
// with, so a change to either is a change to the public ingress.
for file, want := range map[string]string{
"acme.env": "d66eba145c893080e30e013329fc00e51395f0c31fc8621a20c7b26abc11b52f",
"internal-acme.env": "baf4da75379183758c8ae505be4dd8a79d5b089ae1e7c8ded4d232df40251df5",
} {
text, err := os.ReadFile("../../files/" + file)
if err != nil {
t.Fatal(err)
}
sum := sha256.Sum256(text)
if got := hex.EncodeToString(sum[:]); got != want {
t.Fatalf("route-proxy's files/%s changed: digest %s, pinned %s; if the change is meant, check that the "+
"public and internal ingress still ask for the certificates they should, then pin the new digest here",
file, got, want)
}
}
}
+3
View File
@@ -0,0 +1,3 @@
ACME_DIRECTORY=https://acme-v02.api.letsencrypt.org:443/directory
ACME_ROOTS=https://acme-v02.api.letsencrypt.org:443
ACME_ROOTS_PATH=
@@ -0,0 +1,3 @@
INTERNAL_ACME_DIRECTORY=https://${bound:internal-acme-ca:at}:${bound:internal-acme-ca:port}${bound:internal-acme-ca:path}
INTERNAL_ACME_ROOTS=https://${bound:internal-acme-ca:at}:${bound:internal-acme-ca:port}${bound:internal-acme-ca:roots}
INTERNAL_ACME_ROOTS_PATH=${bound:internal-acme-ca:roots}
+5
View File
@@ -0,0 +1,5 @@
module route-proxy
go 1.25.0
require git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261016174426-a4e7f480b3b0
+2
View File
@@ -0,0 +1,2 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261016174426-a4e7f480b3b0 h1:EK84so3Y8JdFT3BHrzj9cPQQrDUPa/d+IdOGnerlmEI=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261016174426-a4e7f480b3b0/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
+253
View File
@@ -0,0 +1,253 @@
{
"module": "route-proxy",
"version": "1",
"slug": "rproxy",
"capabilities": [
"container-runtime"
],
"provides": [
{
"name": "route",
"scope": "mesh",
"identity": false,
"answered-by": "helper"
}
],
"serves": {
"route": {}
},
"receives": {
"route": "${dir:routes-dir}/mesh.json"
},
"requires": [
"internal-acme-ca"
],
"binds": {
"internal-acme-ca": "${dir:state}/internal-acme-ca.json"
},
"own-secrets": {
"broker": "${dir:mesh-state}/broker"
},
"listens": [
{
"name": "http",
"port": 80,
"protocol": "tcp",
"from": "anywhere",
"why": "public HTTP, and the ACME HTTP-01 challenge answered at the name being certified"
},
{
"name": "https",
"port": 443,
"protocol": "tcp",
"from": "anywhere",
"why": "public HTTPS for every name the mesh routes here"
}
],
"data": {
"own": [
{
"id": "acme",
"path": "${dir:acme-cache}",
"class": "rebuildable",
"why": "issued certificates, issued again"
},
{
"id": "ca",
"path": "${dir:ca-dir}",
"class": "cache",
"why": "the authority's roots, fetched again at every start"
}
]
},
"resources": [
{
"id": "state",
"type": "directory",
"mode": "0700",
"place": "."
},
{
"id": "mesh-state",
"type": "directory",
"mode": "0700",
"place": "mesh"
},
{
"id": "routes-dir",
"type": "directory",
"path": "/var/lib/route-proxy/routes",
"mode": "0700"
},
{
"id": "acme-cache",
"type": "directory",
"path": "/var/lib/route-proxy/acme",
"mode": "0700"
},
{
"id": "ca-dir",
"type": "directory",
"path": "/var/lib/route-proxy/ca",
"mode": "0755"
},
{
"id": "acme-env",
"type": "file",
"path": "${dir:state}/acme.env",
"mode": "0600",
"template": "files/acme.env"
},
{
"id": "internal-acme-env",
"type": "file",
"path": "${dir:state}/internal-acme.env",
"mode": "0600",
"template": "files/internal-acme.env"
},
{
"id": "trust",
"type": "container",
"name": "route-proxy-trust",
"artifact": "trust",
"run-once": true,
"network": "host",
"env-file": [
"${dir:state}/acme.env"
],
"volumes": [
"${dir:ca-dir}:/ca"
],
"args": [
"sh",
"-c",
"if [ -z \"$ACME_ROOTS_PATH\" ]; then cp /etc/ssl/certs/ca-certificates.crt /ca/root.crt; exit 0; fi; for i in $(seq 1 60); do wget -q -T 10 --no-check-certificate -O /ca/root.crt \"$ACME_ROOTS\" && grep -q 'BEGIN CERTIFICATE' /ca/root.crt && exit 0; sleep 2; done; echo \"the authority at $ACME_ROOTS did not serve its roots within two minutes\" >&2; exit 1"
],
"restart-on": [
"acme-env"
]
},
{
"id": "internal-trust",
"type": "container",
"name": "route-proxy-internal-trust",
"artifact": "trust",
"run-once": true,
"network": "host",
"env-file": [
"${dir:state}/internal-acme.env"
],
"volumes": [
"${dir:ca-dir}:/ca"
],
"args": [
"sh",
"-c",
"if [ -z \"$INTERNAL_ACME_ROOTS_PATH\" ]; then cp /etc/ssl/certs/ca-certificates.crt /ca/internal-root.crt; exit 0; fi; for i in $(seq 1 60); do wget -q -T 10 --no-check-certificate -O /ca/internal-root.crt \"$INTERNAL_ACME_ROOTS\" && grep -q 'BEGIN CERTIFICATE' /ca/internal-root.crt && exit 0; sleep 2; done; echo \"the authority at $INTERNAL_ACME_ROOTS did not serve its roots within two minutes\" >&2; exit 1"
],
"restart-on": [
"internal-acme-env"
]
},
{
"id": "server",
"type": "container",
"name": "route-proxy",
"artifact": "server",
"network": "host",
"env-file": [
"${dir:state}/acme.env",
"${dir:state}/internal-acme.env"
],
"volumes": [
"${dir:routes-dir}:/routes:ro",
"${dir:acme-cache}:/acme",
"${dir:ca-dir}:/ca:ro",
"${dir:mesh-state}/broker:/run/secrets/broker:ro"
],
"env": {
"ROUTES": "/routes/mesh.json",
"LISTEN": ":80",
"TLS_LISTEN": ":443",
"ACME_CACHE": "/acme",
"ACME_CA_BUNDLE": "/ca/root.crt",
"INTERNAL_ACME_CA_BUNDLE": "/ca/internal-root.crt",
"MESH_BROKER_FILE": "/run/secrets/broker"
},
"restart-on": [
"trust",
"acme-env",
"internal-trust",
"internal-acme-env"
],
"logging": "journald",
"health": {
"kind": "tcp",
"endpoint": "http",
"interval": "10s",
"timeout": "5s",
"looks": 3,
"grace": "60s"
}
}
],
"build": {
"artifacts": [
{
"name": "server",
"kind": "image",
"from": "Dockerfile",
"compiles": "examples/route-proxy",
"context": {
"seat": "git",
"repository": "novox/mesh-controller",
"ref": "main"
}
},
{
"name": "trust",
"kind": "upstream",
"from": "alpine@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b"
},
{
"name": "provider",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/route-provider",
"binary": "route-provider",
"loads": [
"route-provider"
]
},
{
"name": "files",
"kind": "archive",
"from": "files"
}
],
"on": [
{
"arg": "GO_BASE",
"image": "golang@sha256:6c2a5538f964f1c82f97ad14988bf05de100d922d159d0e398b54c7b0ca0c6c9"
},
{
"arg": "ALPINE_BASE",
"image": "alpine@sha256:d9e853e87e55526f6b2917df91a2115c36dd7c696a35be12163d44e6e2a4b6bc"
}
]
},
"jails": [
{
"name": "route-proxy",
"failregex": "^.*TLS handshake error from <HOST>:\\d+: (?:no public route for|acme/autocert: missing server name)\n ^.*refused: no route for .*, asked from <HOST>:\\d+$",
"jail": "backend = systemd\njournalmatch = CONTAINER_NAME=route-proxy\nport = http,https\nmaxretry = 10\nfindtime = 1d\nbantime = 1d"
}
],
"state": [
"bound"
],
"reads": [
"mesh-controller.assignments"
]
}