novox/mesh-controller
13
0
Fork 0
Code Issues Pull Requests 5 Releases Wiki Activity

Bring build-agent and route-proxy in whole (issue 1019)

Browse Source
This commit is contained in:
jschoubben
2026-10-17 18:15:33 +02:00
parent c43772fd94
commit 1586bd1134
17 changed files with 1032 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
modules/build-agent/Dockerfile
+25
View File
@@ -0,0 +1,25 @@
ARG GO_BASE
ARG ALPINE_BASE
# build-agent's image: the build machine itself, compiled into a container (novox/hq ADR 0190).
#
# **The source sits beside this module.** The program, cmd/mesh-builder with internal/builder and
# internal/catalogue, lives in this same repository, mesh-controller, which is also the build context
# (declared under build.artifacts[].context; a module lives whole in one repository, novox/hq ADR 0420),
# so this Dockerfile compiles ./cmd/mesh-builder from the repository root. The build seat hands it only
# that program's build source (novox/hq ADR 0267, the artifact's `compiles`): a merge elsewhere in the
# repository does not move this image.
FROM ${GO_BASE} AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -buildvcs=false -ldflags=-buildid= -o /mesh-builder ./cmd/mesh-builder
# Unlike mesh-controller's own FROM scratch (ADR 0006: nothing to audit but one binary), the build
# machine's whole job is shelling out to git and docker — it needs a real userland to do that in,
# not a second copy of either tool vendored into this image. apk installs both from the base's own
# packages, not fetched on its own at build time.
FROM ${ALPINE_BASE}
RUN apk add --no-cache docker-cli git
COPY --from=build /mesh-builder /usr/local/bin/mesh-builder
ENTRYPOINT ["/usr/local/bin/mesh-builder"]
modules/build-agent/README.md
+34
View File
@@ -0,0 +1,34 @@
# build-agent
The mesh's build machine as a role every machine can hold (novox/hq ADR 0190). It holds the node seat
`node-build-agent`: every holder pulls one build at a time from the role's one work queue when it is
idle, so a tier of many images is built by as many machines as hold the seat and are online, and a
machine that is off builds nothing and blocks nothing. The controller asks the role, never a machine;
the outcome names the machine that built it.
What a holding machine needs is what the builder always needed, said here once: a container runtime
(the socket is mounted), the artifact store as a provision, a workspace, and the bus credential. The code is `cmd/mesh-builder`, in this same repository beside the module, compiled from
the repository's main (`build.artifacts[].context`).
Assign it to every machine with a container runtime. It replaces `builder`, the one-holder form of the
same thing; retire that once this is assigned where it was.
A send that replaces the agent's container does not throw away the build it runs: the container declares
`busy` as `/usr/local/bin/mesh-builder busy` (novox/hq ADR 0335). The node-engine asks it before replacing it; from that
ask the agent takes no new build, the node-engine leaves it running while it says a build is in hand, and
replaces it once it says idle, or after 15 minutes at the latest.
## The package registry (novox/hq ADR 0336)
The package registry (`npm-package-registry`, provided by the gitea module) is an optional requirement.
Genesis assigns the build agent before anything provides it, so the agent is assignable on a mesh with no
registry, and `mesh-controller.plan` and `node` list the requirement as unbound until the gitea module is
assigned; it then binds with no further act.
Until it binds, the agent on genesis's node reads the `genesis-package-registry` own-secret, which the
installer seals to it at the packages step: the forge it raised, the port, the npm path, the account
`mesh-builder` and its password. Issued outside the mesh, so no other machine is given one. A real binding
wins over it, and each build's log says which the build resolved from. A build that needs the registry (a
`package` artifact, an image whose Dockerfile names `.npmrc`, a TypeScript artifact with an `@novox/`
dependency) on an agent with neither fails before building, naming the machine. Go artifacts and images
that do not name `.npmrc` build without it. `manifest_test.go` holds the manifest to this.
modules/build-agent/files/build-agent.env
+8
View File
@@ -0,0 +1,8 @@
MESH_BROKER_FILE=/run/mesh/broker
MESH_NODE=${machine:name}
MESH_REGISTRY=${bound:artifact-store:at}:${bound:artifact-store:port}
MESH_PACKAGE_BINDING=/run/mesh/package-registry.json
MESH_NPM_TOKEN_FILE=/run/mesh/package-registry.secret
MESH_GENESIS_PACKAGE_REGISTRY=/run/mesh/genesis-package-registry.json
MESH_WORKSPACE=${dir:workspace}
MESH_JUDGE_CACHE=${dir:judge-cache}
modules/build-agent/go.mod
+3
View File
@@ -0,0 +1,3 @@
module build-agent
go 1.22
modules/build-agent/manifest_test.go
+174
View File
@@ -0,0 +1,174 @@
// The build agent's manifest (novox/hq ADR 0336, issue 121). The build agent is assigned at genesis's
// step 13, before anything provides the package registry, so that requirement is optional
// (ADR 0330); until it binds, the agent on genesis's node reads the genesis document the installer
// seals to it at the packages step. The program is mesh-controller's cmd/mesh-builder; only the
// packaging lives here, and this is its test.
package buildagent
import (
"encoding/json"
"os"
"strings"
"testing"
)
type manifest struct {
Requires []json.RawMessage `json:"requires"`
Binds map[string]string `json:"binds"`
Secrets map[string]string `json:"secrets"`
OwnSecrets map[string]json.RawMessage `json:"own-secrets"`
Resources []map[string]any `json:"resources"`
}
func read(t *testing.T) manifest {
t.Helper()
raw, err := os.ReadFile("module.json")
if err != nil {
t.Fatal(err)
}
var m manifest
if err := json.Unmarshal(raw, &m); err != nil {
t.Fatal(err)
}
return m
}
// requirement is one entry of `requires`: its name, and whether it is optional.
func requirements(t *testing.T, m manifest) map[string]bool {
t.Helper()
out := map[string]bool{}
for _, raw := range m.Requires {
var name string
if json.Unmarshal(raw, &name) == nil {
out[name] = false
continue
}
var long struct {
Name string `json:"name"`
Optional bool `json:"optional"`
}
if err := json.Unmarshal(raw, &long); err != nil {
t.Fatalf("requires: %s: %v", raw, err)
}
out[long.Name] = long.Optional
}
return out
}
func TestThePackageRegistryIsOptionalAndTheArtifactStoreStaysHard(t *testing.T) {
r := requirements(t, read(t))
optional, has := r["npm-package-registry"]
if !has || !optional {
t.Errorf("npm-package-registry is not an optional requirement (ADR 0336 §1): %v", r)
}
if optional, has := r["artifact-store"]; !has || optional {
t.Errorf("artifact-store is not a hard requirement: %v", r)
}
}
// The binding and the credential of the registry are delivered only while it is bound: ADR 0330 §6
// composes neither for an absent requirement, so no file of the module may name them through a
// placeholder unless it is itself only-when-bound.
func TestTheRegistrysBindsAndSecretsAreOnlyWhenBound(t *testing.T) {
m := read(t)
if m.Binds["npm-package-registry"] == "" || m.Secrets["npm-package-registry"] == "" {
t.Fatalf("the agent is not told where the registry is once it binds: binds %v, secrets %v", m.Binds, m.Secrets)
}
for _, r := range m.Resources {
if r["type"] != "file" {
continue
}
content := templateText(t, r)
if strings.Contains(content, "${bound:npm-package-registry:") || strings.Contains(content, "${secret:npm-package-registry}") {
if r["only-when-bound"] != "npm-package-registry" {
t.Errorf("%v names the registry's binding and is delivered while it is absent", r["id"])
}
}
}
}
// The genesis document is an own-secret issued outside the mesh — by the installer at genesis's packages
// step (ADR 0336 §3) — so the mesh never makes one of its own on any other machine, and the agent is
// told where it lands.
func TestTheGenesisDocumentIsDeclaredAndIssuedOutside(t *testing.T) {
m := read(t)
raw, has := m.OwnSecrets["genesis-package-registry"]
if !has {
t.Fatalf("the build agent declares no genesis-package-registry own-secret: %v", m.OwnSecrets)
}
var own struct {
Path string `json:"path"`
IssuedBy string `json:"issued-by"`
}
if err := json.Unmarshal(raw, &own); err != nil || own.IssuedBy != "outside" {
t.Fatalf("the genesis document is not issued outside the mesh: %s (%v)", raw, err)
}
if !strings.HasPrefix(own.Path, "${dir:mesh-state}/") {
t.Fatalf("the genesis document is not where the container reads /run/mesh: %s", own.Path)
}
inside := "/run/mesh/" + strings.TrimPrefix(own.Path, "${dir:mesh-state}/")
for _, r := range m.Resources {
if r["id"] == "agent-env" {
content := templateText(t, r)
if !strings.Contains(content, "MESH_GENESIS_PACKAGE_REGISTRY="+inside+"\n") {
t.Errorf("the agent is not told where the genesis document is (%s):\n%s", inside, content)
}
return
}
}
t.Fatal("the build agent has no environment file")
}
// **The judge cache is the agent's own, beside its workspace and never in it** (novox/hq ADR 0358 §6, issue 704): a
// directory resource of its own, kept as the agent's data, mounted into the agent at the same path as on the host —
// the container runtime resolves a judge build's mounts on the host — and handed to the agent as MESH_JUDGE_CACHE.
func TestTheJudgeCacheIsADirectoryOfItsOwnMountedAtTheSamePath(t *testing.T) {
raw, err := os.ReadFile("module.json")
if err != nil {
t.Fatal(err)
}
var m struct {
Data struct {
Own []struct {
ID string `json:"id"`
Path string `json:"path"`
Class string `json:"class"`
} `json:"own"`
} `json:"data"`
Resources []struct {
ID string `json:"id"`
Type string `json:"type"`
Template string `json:"template"`
Volumes []string `json:"volumes"`
} `json:"resources"`
}
if err := json.Unmarshal(raw, &m); err != nil {
t.Fatal(err)
}
dir, data, env, mounted := false, false, false, false
for _, d := range m.Data.Own {
if d.ID == "judge-cache" && d.Path == "${dir:judge-cache}" && d.Class == "cache" {
data = true
}
}
for _, r := range m.Resources {
switch {
case r.ID == "judge-cache" && r.Type == "directory":
dir = true
case r.ID == "agent-env":
env = strings.Contains(templateText(t, map[string]any{"id": r.ID, "template": r.Template}), "MESH_JUDGE_CACHE=${dir:judge-cache}\n")
case r.ID == "server":
for _, v := range r.Volumes {
if v == "${dir:judge-cache}:${dir:judge-cache}" {
mounted = true
}
if strings.HasPrefix(v, "${dir:workspace}") && strings.Contains(v, "judge-cache") {
t.Errorf("the judge cache is mounted inside the workspace: %s", v)
}
}
}
}
if !dir || !data || !env || !mounted {
t.Fatalf("the judge cache: directory %v, data %v, env %v, mounted at the same path %v", dir, data, env, mounted)
}
}
modules/build-agent/module.json
+139
View File
@@ -0,0 +1,139 @@
{
"module": "build-agent",
"version": "1",
"slug": "agent",
"capabilities": [
"container-runtime"
],
"claims": [
{
"name": "node-build-agent",
"scope": "node",
"serves": ["current", "kill", "pause", "resume"]
}
],
"reads": [
"mesh-delivery.pulls"
],
"requires": [
"artifact-store",
{
"name": "npm-package-registry",
"optional": true
}
],
"binds": {
"npm-package-registry": "${dir:mesh-state}/package-registry.json"
},
"secrets": {
"npm-package-registry": "${dir:mesh-state}/package-registry.secret"
},
"own-secrets": {
"broker": "${dir:mesh-state}/broker",
"genesis-package-registry": {
"path": "${dir:mesh-state}/genesis-package-registry.json",
"issued-by": "outside"
}
},
"data": {
"own": [
{
"id": "workspace",
"path": "${dir:workspace}",
"class": "cache",
"why": "a build's working copy, cloned again for every build"
},
{
"id": "judge-cache",
"path": "${dir:judge-cache}",
"class": "cache",
"why": "the Go cache every merge check's judge is built with and the release-branch judges kept per head; mounted into judge builds only, never into a check's containers (novox/hq ADR 0358 §6)"
}
]
},
"resources": [
{
"id": "mesh-state",
"type": "directory",
"mode": "0700",
"place": "mesh"
},
{
"id": "workspace",
"type": "directory",
"mode": "0700"
},
{
"id": "judge-cache",
"type": "directory",
"mode": "0700"
},
{
"id": "agent-env",
"type": "file",
"path": "${dir:mesh-state}/build-agent.env",
"mode": "0600",
"template": "files/build-agent.env"
},
{
"id": "server",
"type": "container",
"name": "mesh-build-agent",
"artifact": "server",
"env-file": [
"${dir:mesh-state}/build-agent.env"
],
"volumes": [
"${dir:mesh-state}:/run/mesh:ro",
"${dir:workspace}:${dir:workspace}",
"${dir:judge-cache}:${dir:judge-cache}",
"/var/run/docker.sock:/var/run/docker.sock"
],
"restart-on": [
"agent-env"
],
"network": "host",
"health": {
"kind": "exec",
"command": "/usr/local/bin/mesh-builder ready",
"interval": "10s",
"timeout": "5s",
"looks": 3,
"grace": "60s"
},
"busy": {
"command": "/usr/local/bin/mesh-builder busy"
}
}
],
"build": {
"artifacts": [
{
"name": "server",
"kind": "image",
"from": "Dockerfile",
"compiles": "cmd/mesh-builder",
"context": {
"seat": "git",
"repository": "novox/mesh-controller",
"ref": "main"
}
},
{
"name": "files",
"kind": "archive",
"from": "files"
}
],
"on": [
{
"arg": "GO_BASE",
"image": "golang@sha256:8ac98ca534ac3f51e1f420a1dd2c15e74c75cfa0f23f3ad27eb5d7236c349a0c"
},
{
"arg": "ALPINE_BASE",
"image": "alpine@sha256:d9e853e87e55526f6b2917df91a2115c36dd7c696a35be12163d44e6e2a4b6bc"
}
]
}
}
modules/build-agent/templates_test.go
+64
View File
@@ -0,0 +1,64 @@
package buildagent
import (
"encoding/json"
"os"
"path/filepath"
"testing"
)
// The module's directory, where its manifest and its files/ are.
const moduleDir = "."
// templateText is the text a file resource places on the machine: its template's, in the module's files/
// (novox/hq ADR 0349 §2). A resource that names no template, or a template that is not there, fails the test.
func templateText(t *testing.T, r map[string]any) string {
t.Helper()
name, _ := r["template"].(string)
if name == "" {
t.Fatalf("resource %v names no template", r["id"])
}
raw, err := os.ReadFile(filepath.Join(moduleDir, filepath.FromSlash(name)))
if err != nil {
t.Fatalf("resource %v names %s: %v", r["id"], name, err)
}
return string(raw)
}
// No file resource carries its text inline: each names its template in files/, and the build packs files/
// into the one archive the controller reads the templates back from (novox/hq ADR 0349 §2).
func TestEveryFileTakesItsTextFromATemplate(t *testing.T) {
raw, err := os.ReadFile(filepath.Join(moduleDir, "module.json"))
if err != nil {
t.Fatal(err)
}
var m struct {
Resources []map[string]any `json:"resources"`
Build struct {
Artifacts []map[string]any `json:"artifacts"`
} `json:"build"`
}
if err := json.Unmarshal(raw, &m); err != nil {
t.Fatal(err)
}
for _, r := range m.Resources {
if _, inline := r["content"]; inline {
t.Errorf("resource %v keeps its text inline: keep it in files/ and name it as the template", r["id"])
}
if r["type"] == "file" {
templateText(t, r)
}
}
packs := 0
for _, a := range m.Build.Artifacts {
if a["kind"] == "archive" && a["from"] == "files" {
packs++
if a["name"] != "files" {
t.Errorf("the templates archive is named %v, not files", a["name"])
}
}
}
if packs != 1 {
t.Errorf("the build packs files/ into %d archives, not one", packs)
}
}
modules/route-proxy/Dockerfile
+29
View File
@@ -0,0 +1,29 @@
ARG ALPINE_BASE=alpine:3.20
ARG GO_BASE=golang:1.25
# The route-proxy module's runtime image: the reference reverse proxy compiled into a container.
#
# **The proxy source sits beside this module.** The canonical proxy, the contract written as something
# that runs, lives in this same repository, mesh-controller, at examples/route-proxy (novox/hq 08-connectivity
# §3), and the repository is the build context (a module lives whole in one repository, novox/hq ADR 0420),
# so this Dockerfile compiles ./examples/route-proxy from the repository root. The build seat hands it only
# that program's build source, its import closure, go.mod, go.sum and the vendored packages it uses
# (novox/hq ADR 0267, the artifact's `compiles`), so a merge elsewhere in the repository does not
# move this image, and a COPY of anything outside it fails by name.
#
# docker build -f modules/route-proxy/Dockerfile -t mesh-route-proxy:development .
#
# The mesh pins the digest of what this produces; the committed module.json carries the placeholder
# digest every mesh-built image does, replaced at publish.
FROM ${GO_BASE} AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -buildvcs=false -ldflags=-buildid= -o /mesh-route-proxy ./examples/route-proxy
# A small runtime with the public CA roots the ACME client needs to reach a real authority, and run
# as root so it can bind :80 and :443 — the two privileged ports a public front door listens on.
FROM ${ALPINE_BASE}
RUN apk add --no-cache ca-certificates
COPY --from=build /mesh-route-proxy /usr/local/bin/mesh-route-proxy
ENTRYPOINT ["/usr/local/bin/mesh-route-proxy"]
modules/route-proxy/README.md
+92
View File
@@ -0,0 +1,92 @@
# route-proxy — the mesh's public ingress
The shipping form of the reference reverse proxy (novox/hq ADR 0007, novox/hq 08-connectivity §3
"Exposure"). It is the `traefik` replacement: the module that **provides `route`**.
## What it is
**A route is a grant.** A module that must be reachable declares `requires: ["route"]` and
contributes the name it wants and the port it listens on; this module provides `route`, receives
every consumer that asked as a file, and forwards by the `Host` header of each request to
`http://<at>:<port>` — where `at` is where the mesh says that consumer's machine is (empty means
this machine, reached over loopback).
The control plane does not write the proxy's configuration. It delivers the facts — the
contributions file at `receives.route` — and the proxy turns them into whatever it runs. That
boundary is why swapping this proxy for another costs nothing to the modules that publish through
it.
| field | value | why |
|---|---|---|
| `provides` | `route` (scope `mesh`) | a module on any node may require it |
| `serves.route` | `{}` | a route hands back a name, not a credential — the consumer already knows the name it asked for |
| `receives.route` | `…/routes/mesh.json` | where the mesh writes every contribution; the proxy reads it as `$ROUTES` and re-reads on change |
| `listens` | `80` and `443`, both `from: anywhere` | the one machine with a public opening; the firewall opens these for free (ADR 0045) |
No provisioner: the proxy neither mints a credential for anybody nor answers a provision other
than `route`. It reads the file the mesh writes, and its own broker credential is for its tools.
## Which issuer: Let's Encrypt, stated here (novox/hq ADR 0226)
**The public issuer is this module's own fact, not a provision.** Until 2026-10-06 it came from a
second module, `public-acme`, that ran nothing and offered `acme-ca` pointing at Let's Encrypt; every
node running a proxy was assigned it too, and nothing else ever consumed it. It is retired: the proxy
names the production directory in its own `acme.env`, and the only issuer it is still bound to is the
mesh's own (`internal-acme-ca`, from `step-ca`).
**`acme.env` is byte for byte what the binding rendered.** The proxy keeps each authority's account
and certificates in a directory under `/var/lib/route-proxy/acme` named after a digest of the
directory URL — `https://acme-v02.api.letsencrypt.org:443/directory`, port included, as the binding
spelled it — and of the root bundle the `trust` container copies in. Any change to either is a new
authority to the proxy: a new account, and every routed name ordered again against Let's Encrypt's
rate limits. So the URL keeps the `:443`, `ACME_ROOTS_PATH` stays empty, and the `trust` artifact's
pinned image is not moved without a decision; mesh-controller's
`TestRouteProxyKeepsItsPublicAccountDirectory` holds all three.
## How it ships the Go proxy
The proxy is a Go program, unlike the TypeScript tool-runtime modules. Its source sits in this same
repository, mesh-controller, at `examples/route-proxy`, the contract written as something that runs. This
module ships the packaging: a multi-stage [`Dockerfile`](Dockerfile) whose build context is the repository
root and which compiles `./examples/route-proxy` into `mesh-route-proxy`. The committed `module.json` carries the
placeholder digest every mesh-built image does (`@sha256:0000…`); the mesh pins the real digest at
publish.
In the lab, `mesh-lab/scripts/build-route-proxy-image.sh` builds this into the local daemon as
`mesh-route-proxy:development`, and a scenario stocks it and serves it by digest.
## Certificates, and the endpoint (resolves 04-ISSUES/004)
TLS is opt-in and terminates here for public names, via ACME (autocert). A proxy with no
`TLS_LISTEN` serves plain HTTP; with it, it obtains a publicly-trusted certificate for exactly the
names the mesh routes here (`HostPolicy` = only routed names, so a scan cannot spend the account's
issuance quota) and answers the HTTP-01 challenge on `:80` at the name being certified.
**`ACME_DIRECTORY` defaults to Let's Encrypt *staging*, not production.** This closes
novox/hq 04-ISSUES/004 (certificate issuance targets production):
the issuer is a per-node choice, and the safe endpoint is the default. Production issuance is
rate-limited per domain and per account and does not replenish quickly; defaulting to production
would leave the safe path depending on somebody remembering to opt out of it, on exactly the work
most likely to iterate. A staging certificate is trusted by no browser, so the mistake announces
itself on the first request rather than a fortnight later at the rate limit.
**The module states production.** The binary's staging default stands for anything that runs it
without the module — the lab, a developer's machine — and the module's `acme.env` names production
for every node it is assigned to, because a node assigned the public proxy is one that serves public
traffic.
| env | default | meaning |
|---|---|---|
| `ROUTES` | `/routes/mesh.json` | the contributions file the mesh writes (`receives.route`) |
| `LISTEN` | `:80` | HTTP, and the ACME HTTP-01 challenge |
| `TLS_LISTEN` | `:443` | HTTPS; unset to serve plain HTTP only |
| `ACME_CACHE` | `/acme` | where issued certificates persist; required when `TLS_LISTEN` is set, so a restart does not re-order |
| `ACME_DIRECTORY` | Let's Encrypt **staging** in the binary; production in the module's `acme.env` | the issuer |
| `ACME_CA_BUNDLE` | *(unset)* | a file of roots to trust for the issuer's own API — set only for a private/lab authority whose API certificate the world does not yet trust |
## Answered by the provider helper (issue 792)
A small Go program beside the proxy (`cmd/route-provider`) runs the SDK's provider helper for `route`, with no credential,
and the offer says `"answered-by": "helper"`: a handed consumer (gitea's origins file) is filled by its machine's client
from this module's entry, which carries the public names the controller composed for it. The module is **not** marked
moved: its own files, `acme.env` and the broker secret, are composed as before, and the proxy itself is unchanged.
modules/route-proxy/cmd/route-provider/main.go
+37
View File
@@ -0,0 +1,37 @@
// route-provider: route-proxy's Go half, one binary the node's runtime launches and speaks MCP to over stdio
// through the Go SDK. It serves no tools; it runs the SDK's provider helper that answers the consumers of `route`
// (issue 792). The TypeScript adapter beside it keeps writing the proxy's files.
//
// stdout is the MCP channel; everything this module says, it says on stderr.
package main
import (
"context"
"fmt"
"os"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/provider"
)
func say(format string, args ...any) {
fmt.Fprintf(os.Stderr, "[route-proxy] "+format+"\n", args...)
}
func main() {
node := os.Getenv("MESH_NODE")
if node == "" {
say("MESH_NODE is not set; answering no consumers")
} else {
p := newProvider(node, provider.StateBus(ModuleName), say)
go func() {
if err := p.Run(context.Background()); err != nil {
say("the provider helper stopped: %v", err)
}
}()
}
if err := stdio.Serve("", nil); err != nil {
say("%v", err)
os.Exit(1)
}
}
modules/route-proxy/cmd/route-provider/provider.go
+36
View File
@@ -0,0 +1,36 @@
package main
// route-proxy handed to the provisioner (novox/hq ADR 0349 clause 12, issue 792): the module runs the SDK's provider
// helper for `route`, so a consumer of a route — a handed gitea reading `${bound:route:name-web}` — is told its public
// name from the entry in this module's bucket, the one its machine's client fills its files from.
//
// **It composes nothing.** A route's public name is the label its consumer chose joined to the node's public domain,
// which only the controller knows (the domain is placement), so the controller composes it and carries it in the
// consumer's plan entry (`names`); the helper copies it into the entry's values unchanged. There is no credential for
// a route and nothing to make: the traefik files stay the TypeScript adapter's, written from the contributions the
// mesh still gives it.
import (
"git.novox.be/novox/mesh-sdk/go/provider"
)
// ProvisionName is what route-proxy provides, as its manifest says.
const ProvisionName = "route"
// ModuleName is this module.
const ModuleName = "route-proxy"
// newProvider is route-proxy answering its consumers over bus on node: the route, served as the manifest serves it
// (nothing of its own), with no credential, and nothing to make or retire for a consumer.
func newProvider(node string, bus provider.Bus, logf func(string, ...any)) *provider.Provider {
return provider.New(provider.Options{
Node: node,
Module: ModuleName,
Offers: map[string]provider.Offer{ProvisionName: {
Credential: provider.NoCredential,
Serves: map[string]any{},
}},
Bus: bus,
Logf: logf,
})
}
modules/route-proxy/cmd/route-provider/provider_test.go
+125
View File
@@ -0,0 +1,125 @@
package main
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"os"
"sync"
"testing"
"git.novox.be/novox/mesh-sdk/go/provider"
)
// bucketBus is the helper's bus over in-memory buckets.
type bucketBus struct {
mu sync.Mutex
buckets map[string]map[string][]byte
}
func (b *bucketBus) Watch(_ context.Context, bucket string, handle func(string, []byte)) error {
b.mu.Lock()
defer b.mu.Unlock()
for k, v := range b.buckets[bucket] {
handle(k, v)
}
return nil
}
func (b *bucketBus) Put(_ context.Context, bucket, key string, value []byte) error {
b.mu.Lock()
defer b.mu.Unlock()
if b.buckets[bucket] == nil {
b.buckets[bucket] = map[string][]byte{}
}
b.buckets[bucket][key] = append([]byte(nil), value...)
return nil
}
func (b *bucketBus) Delete(_ context.Context, bucket, key string) error {
b.mu.Lock()
defer b.mu.Unlock()
delete(b.buckets[bucket], key)
return nil
}
// A consumer of a route is answered with the names its plan's entry carries, and nothing of a credential (issue 792):
// the entry's values are the controller's composition, copied unchanged, and no vault is asked.
func TestAConsumerOfARouteIsToldItsNamesFromItsPlanEntry(t *testing.T) {
bus := &bucketBus{buckets: map[string]map[string][]byte{}}
plan, _ := json.Marshal(map[string]any{"node": "g14", "module": "gitea", "provisioning": map[string]any{
"provisions": []any{map[string]any{"provision": ProvisionName, "provider": "novox",
"provider-module": ModuleName, "at": "10.7.0.1", "from": "novox", "as": "mesh_g14_gitea",
"names": map[string]any{"name": "git.example.org", "internal-name": "git.novox.internal",
"name-web": "git.example.org"}}}}})
if err := bus.Put(context.Background(), provider.AssignmentsBucket, "g14.gitea", plan); err != nil {
t.Fatal(err)
}
p := newProvider("novox", bus, t.Logf)
out, err := p.Pass(context.Background())
if err != nil {
t.Fatal(err)
}
if len(out) != 1 || out[0].Status != provider.Answered {
t.Fatalf("the pass did %+v", out)
}
var e provider.Entry
if err := json.Unmarshal(bus.buckets[provider.BucketOf(ModuleName)]["g14.gitea.route"], &e); err != nil {
t.Fatal(err)
}
if e.Values["name-web"] != "git.example.org" || e.Values["name"] != "git.example.org" ||
e.Values["internal-name"] != "git.novox.internal" || e.VaultName != "" || e.User != "" || e.Refused != "" {
t.Fatalf("the entry is %+v", e)
}
}
// route-proxy's own files and resources stay as they were (issue 792): the module answers its route through the
// helper but is not marked moved, so its acme.env, its broker secret and its containers are composed as today and
// the public ingress does not change. Everything of the manifest but its build, what it provides and the state it
// keeps is pinned by this digest; changing the ingress means changing it here, on purpose.
func TestRouteProxyIsNotMovedAndKeepsItsOwnFiles(t *testing.T) {
raw, err := os.ReadFile("../../module.json")
if err != nil {
t.Fatal(err)
}
var m map[string]any
if err := json.Unmarshal(raw, &m); err != nil {
t.Fatal(err)
}
if _, moved := m["moved"]; moved {
t.Fatal("route-proxy is marked moved: its own files would be handed")
}
provides, _ := m["provides"].([]any)
if len(provides) != 1 || provides[0].(map[string]any)["answered-by"] != "helper" {
t.Fatalf("route is not said to be answered by the helper: %v", provides)
}
for _, k := range []string{"build", "provides", "state", "reads"} {
delete(m, k)
}
pinned, err := json.Marshal(m) // sorted keys, compact
if err != nil {
t.Fatal(err)
}
sum := sha256.Sum256(pinned)
if got := hex.EncodeToString(sum[:]); got != "57a39de0943d8b33b9db6cb998ce97f8793602cca6cdfa5849c1c392390454a3" {
t.Fatalf("route-proxy's own manifest changed: digest %s", got)
}
// The two acme templates are pinned the same way (issue 849): they are what the proxy's certificates are asked
// with, so a change to either is a change to the public ingress.
for file, want := range map[string]string{
"acme.env": "d66eba145c893080e30e013329fc00e51395f0c31fc8621a20c7b26abc11b52f",
"internal-acme.env": "baf4da75379183758c8ae505be4dd8a79d5b089ae1e7c8ded4d232df40251df5",
} {
text, err := os.ReadFile("../../files/" + file)
if err != nil {
t.Fatal(err)
}
sum := sha256.Sum256(text)
if got := hex.EncodeToString(sum[:]); got != want {
t.Fatalf("route-proxy's files/%s changed: digest %s, pinned %s; if the change is meant, check that the "+
"public and internal ingress still ask for the certificates they should, then pin the new digest here",
file, got, want)
}
}
}
modules/route-proxy/files/acme.env
+3
View File
@@ -0,0 +1,3 @@
ACME_DIRECTORY=https://acme-v02.api.letsencrypt.org:443/directory
ACME_ROOTS=https://acme-v02.api.letsencrypt.org:443
ACME_ROOTS_PATH=
modules/route-proxy/files/internal-acme.env
+3
View File
@@ -0,0 +1,3 @@
INTERNAL_ACME_DIRECTORY=https://${bound:internal-acme-ca:at}:${bound:internal-acme-ca:port}${bound:internal-acme-ca:path}
INTERNAL_ACME_ROOTS=https://${bound:internal-acme-ca:at}:${bound:internal-acme-ca:port}${bound:internal-acme-ca:roots}
INTERNAL_ACME_ROOTS_PATH=${bound:internal-acme-ca:roots}
modules/route-proxy/go.mod
+5
View File
@@ -0,0 +1,5 @@
module route-proxy
go 1.25.0
require git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261016174426-a4e7f480b3b0
modules/route-proxy/go.sum
+2
View File
@@ -0,0 +1,2 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261016174426-a4e7f480b3b0 h1:EK84so3Y8JdFT3BHrzj9cPQQrDUPa/d+IdOGnerlmEI=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261016174426-a4e7f480b3b0/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
modules/route-proxy/module.json
+253
View File
@@ -0,0 +1,253 @@
{
"module": "route-proxy",
"version": "1",
"slug": "rproxy",
"capabilities": [
"container-runtime"
],
"provides": [
{
"name": "route",
"scope": "mesh",
"identity": false,
"answered-by": "helper"
}
],
"serves": {
"route": {}
},
"receives": {
"route": "${dir:routes-dir}/mesh.json"
},
"requires": [
"internal-acme-ca"
],
"binds": {
"internal-acme-ca": "${dir:state}/internal-acme-ca.json"
},
"own-secrets": {
"broker": "${dir:mesh-state}/broker"
},
"listens": [
{
"name": "http",
"port": 80,
"protocol": "tcp",
"from": "anywhere",
"why": "public HTTP, and the ACME HTTP-01 challenge answered at the name being certified"
},
{
"name": "https",
"port": 443,
"protocol": "tcp",
"from": "anywhere",
"why": "public HTTPS for every name the mesh routes here"
}
],
"data": {
"own": [
{
"id": "acme",
"path": "${dir:acme-cache}",
"class": "rebuildable",
"why": "issued certificates, issued again"
},
{
"id": "ca",
"path": "${dir:ca-dir}",
"class": "cache",
"why": "the authority's roots, fetched again at every start"
}
]
},
"resources": [
{
"id": "state",
"type": "directory",
"mode": "0700",
"place": "."
},
{
"id": "mesh-state",
"type": "directory",
"mode": "0700",
"place": "mesh"
},
{
"id": "routes-dir",
"type": "directory",
"path": "/var/lib/route-proxy/routes",
"mode": "0700"
},
{
"id": "acme-cache",
"type": "directory",
"path": "/var/lib/route-proxy/acme",
"mode": "0700"
},
{
"id": "ca-dir",
"type": "directory",
"path": "/var/lib/route-proxy/ca",
"mode": "0755"
},
{
"id": "acme-env",
"type": "file",
"path": "${dir:state}/acme.env",
"mode": "0600",
"template": "files/acme.env"
},
{
"id": "internal-acme-env",
"type": "file",
"path": "${dir:state}/internal-acme.env",
"mode": "0600",
"template": "files/internal-acme.env"
},
{
"id": "trust",
"type": "container",
"name": "route-proxy-trust",
"artifact": "trust",
"run-once": true,
"network": "host",
"env-file": [
"${dir:state}/acme.env"
],
"volumes": [
"${dir:ca-dir}:/ca"
],
"args": [
"sh",
"-c",
"if [ -z \"$ACME_ROOTS_PATH\" ]; then cp /etc/ssl/certs/ca-certificates.crt /ca/root.crt; exit 0; fi; for i in $(seq 1 60); do wget -q -T 10 --no-check-certificate -O /ca/root.crt \"$ACME_ROOTS\" && grep -q 'BEGIN CERTIFICATE' /ca/root.crt && exit 0; sleep 2; done; echo \"the authority at $ACME_ROOTS did not serve its roots within two minutes\" >&2; exit 1"
],
"restart-on": [
"acme-env"
]
},
{
"id": "internal-trust",
"type": "container",
"name": "route-proxy-internal-trust",
"artifact": "trust",
"run-once": true,
"network": "host",
"env-file": [
"${dir:state}/internal-acme.env"
],
"volumes": [
"${dir:ca-dir}:/ca"
],
"args": [
"sh",
"-c",
"if [ -z \"$INTERNAL_ACME_ROOTS_PATH\" ]; then cp /etc/ssl/certs/ca-certificates.crt /ca/internal-root.crt; exit 0; fi; for i in $(seq 1 60); do wget -q -T 10 --no-check-certificate -O /ca/internal-root.crt \"$INTERNAL_ACME_ROOTS\" && grep -q 'BEGIN CERTIFICATE' /ca/internal-root.crt && exit 0; sleep 2; done; echo \"the authority at $INTERNAL_ACME_ROOTS did not serve its roots within two minutes\" >&2; exit 1"
],
"restart-on": [
"internal-acme-env"
]
},
{
"id": "server",
"type": "container",
"name": "route-proxy",
"artifact": "server",
"network": "host",
"env-file": [
"${dir:state}/acme.env",
"${dir:state}/internal-acme.env"
],
"volumes": [
"${dir:routes-dir}:/routes:ro",
"${dir:acme-cache}:/acme",
"${dir:ca-dir}:/ca:ro",
"${dir:mesh-state}/broker:/run/secrets/broker:ro"
],
"env": {
"ROUTES": "/routes/mesh.json",
"LISTEN": ":80",
"TLS_LISTEN": ":443",
"ACME_CACHE": "/acme",
"ACME_CA_BUNDLE": "/ca/root.crt",
"INTERNAL_ACME_CA_BUNDLE": "/ca/internal-root.crt",
"MESH_BROKER_FILE": "/run/secrets/broker"
},
"restart-on": [
"trust",
"acme-env",
"internal-trust",
"internal-acme-env"
],
"logging": "journald",
"health": {
"kind": "tcp",
"endpoint": "http",
"interval": "10s",
"timeout": "5s",
"looks": 3,
"grace": "60s"
}
}
],
"build": {
"artifacts": [
{
"name": "server",
"kind": "image",
"from": "Dockerfile",
"compiles": "examples/route-proxy",
"context": {
"seat": "git",
"repository": "novox/mesh-controller",
"ref": "main"
}
},
{
"name": "trust",
"kind": "upstream",
"from": "alpine@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b"
},
{
"name": "provider",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/route-provider",
"binary": "route-provider",
"loads": [
"route-provider"
]
},
{
"name": "files",
"kind": "archive",
"from": "files"
}
],
"on": [
{
"arg": "GO_BASE",
"image": "golang@sha256:6c2a5538f964f1c82f97ad14988bf05de100d922d159d0e398b54c7b0ca0c6c9"
},
{
"arg": "ALPINE_BASE",
"image": "alpine@sha256:d9e853e87e55526f6b2917df91a2115c36dd7c696a35be12163d44e6e2a4b6bc"
}
]
},
"jails": [
{
"name": "route-proxy",
"failregex": "^.*TLS handshake error from <HOST>:\\d+: (?:no public route for|acme/autocert: missing server name)\n ^.*refused: no route for .*, asked from <HOST>:\\d+$",
"jail": "backend = systemd\njournalmatch = CONTAINER_NAME=route-proxy\nport = http,https\nmaxretry = 10\nfindtime = 1d\nbantime = 1d"
}
],
"state": [
"bound"
],
"reads": [
"mesh-controller.assignments"
]
}