Compare commits

..
Author SHA1 Message Date
jschoubben 4e2b8f32ce Pin mesh-sdk a3718126, whose bus retry never asks again a call its callee answered (issue 774)
mesh/delivery closed: the pull request closed unmerged
mesh/delivery-group group fix/sweep-2026-10-30 rejected: composed together: the checker's verdict at the members' heads (9a9df8453781): g14
mesh/check failed: g14: novox/mesh-catalog#565: modules/mesh-vault/mesh-vault: a compiled program added in 2066021fca5f and removed later: it stays in…
2026-10-30 18:43:13 +01:00
jschoubben dd366f8a3f Pin the group's final mesh-sdk 206f794e with its captured conformance files, and read the engine's superseded word on a secret move (issue 774)
The captured SDK (testdata/beside, conformance/events) is taken again at the pinned commit, as CAPTURED
says. The controller's mirror of MoveSecretAnswer gains `superseded` (mesh-host b9c24a7): the vault holds
another value, the live one, and the machine's older value is not moved and not asked again. The field
test names it. A superseded row is still counted with the moved ones: counting it apart needs a field and
its words in nodeMove, more than the one line asked.
2026-10-30 18:37:56 +01:00
jschoubben 93d167f0e8 Say rightly which own secrets the core ask leaves out: broker-address and broker-management are no bus accounts (issue 774)
The comment of c421629f named the controller's broker-address and broker-management as process accounts.
They are not: no process of the controller declares a bus; broker-address is the bus's address, read by
broker.FromEnvironment, and broker-management the retired AMQP management login, read by nothing. The
filter never touched them; only the words were wrong.
2026-10-30 18:36:18 +01:00
jschoubben cf4ca55596 Judge apply-failing by the client's failing-since, so a failure said in new words is measured from when it began (issue 774)
mesh-provisioner-client's report now carries failing-since (mesh-catalog 3aae6c08), kept while a module
stays failed however its words change. S21 measures a failed placing from it, falling back to the
report's time from an older client; the hysteresis of 6ffcea4f (an open condition stays while any
failure stands) is kept. This is the thorough fix that commit named.

Research 072 piece: milestone 3 (health leaves the controller): the judgement reads the owner's state.
2026-10-30 18:35:37 +01:00
jschoubben 77c54445dc Pin the group's mesh-sdk ef1ec1c9 and capture its conformance files there, so the sweep builds on kvlimit and the named readers (issue 774)
The sweep's controller changes use mesh-sdk's go/kvlimit and the ReadsAnnouncedBy that grants named
readers. At ef1ec1c9 mesh-config is a named reader of the controller's lease (its take-over reads it,
ADR 0561 §3), so the read test now expects it granted and a module not named refused. The captured SDK
(testdata/beside, conformance/events) is taken again at the pinned commit, as CAPTURED says; its file
is unchanged.
2026-10-30 18:27:54 +01:00
jschoubben 8c3d372322 Compose no placed user from the node-engine's manifest in the controller's bus composer too, as mesh-sdk #136 does (issue 774)
mesh-sdk #136 (42988a7) keeps the node-engine's manifest out of every placed user in buscompose: where a
placement names it, no user, no process and nothing the tool runner carries; the node's account alone
writes the engine's state, and a second writer has the writers table refuse the whole list (met twice on
2026-10-30: mesh-host_acts, then mesh-host_health). The controller still composes the bus's users with
its own copy (internal/broker Users), which lacked the guard. The same two predicates, Composed and
CarriedByRunner, now guard it.

Read on the bus at 18:30: the bus follows the grants part's list (mesh-grants_user-list.sealed), so the
controller's copy is not what the bus loads; it still composes the bus module's files and the controller's
own grants, where a refusal would stop that composition.

Quick fix: the same guard here. Thorough fix: delete the controller's copy of the bus composer and use
buscompose (about 18,000 lines of internal/broker with its tests, its genesis list and its golden file).
Chose quick: the controller is being deprecated and the grants part already holds the list; the copy goes
with the controller's composition of the bus module, not in a sweep.

Research 072 piece: 03-domains, mesh-grants (membership and grants leave the controller): until the copy is
removed, it composes as the SDK does.
2026-10-30 18:23:42 +01:00
jschoubben 519434241a Queue one state per call for the bus, bounded far above a burst, so a burst of read calls no longer drops call records (issue 774)
At 17:44:53 the controller said 30 calls (tools, nodes, modules) were "kept in memory only: the bus is
not taking calls' records". No write had failed (no "could not be kept" line): a burst of about 200 read
calls in one second queued a begin and a finish each into a channel of 256 while the one writer put
them one by one, and the channel overflowed.

The queue now holds the latest state of each waiting call (a finish replaces its own waiting begin, as
only the last state is kept anyway), in first-queued order, bounded at 8192 calls; the writer takes the
whole queue at each wake. A burst now costs one write per call and drops nothing.

Quick fix: a larger channel. Thorough fix: coalesce by call, so the writes a burst asks are halved and
the bound is spent on calls, not states. Chose thorough; both are small.

Research 072 piece: §2's "read calls said" row: the calls bucket is the controller's own state on the
bus, kept from events (each call's change), never re-read.
2026-10-30 18:21:58 +01:00
jschoubben c421629f6e Ask the vault for a bus process's account only once it is issued, so the vault never makes a random value under an account's name (issue 774)
At 16:36:51 the vault handed own/novox/nats/broker-nats-tools to the controller as a plain random own
secret, and owes the bus-account hashes of it and of the controller's broker-address and
broker-management for good ("holds no account the controller hands (not JSON)"). The controller decided
that: its send asks the vault (`get`) for every own secret of a core module, a bus process's account
among them, before the account is issued; the vault, holding none, makes one.

The send now leaves out of that ask a process's account nobody issued (the same rule as its warning
line); the machine keeps the controller's stand-in until the account is issued and handed to the vault.
The vault side (not minting an account-shaped name, and replacing the random value already held) is
mesh-catalog's mesh-vault, with build-catalog.

Quick fix: this filter on the ask. Thorough fix: process accounts minted by the vault itself (ADR 0113 §6,
research 072 milestone 6), so no part asks for one before it exists. Chose quick: milestone 6 is not
started, and the filter removes the cause on the controller's side now.

Research 072 piece: milestone 6 (identity and bus accounts to the vault): until then the controller asks
the vault only for what exists.
2026-10-30 18:19:26 +01:00
jschoubben 07cecd287c Leave modules the mesh provides itself out of the lines that say a module lacks a fingerprint (issue 774)
mesh-wireguard is provided by the mesh (sdkdelivery.ProvidedByMesh): no repository builds it and no
fingerprint will ever name it, yet every composition said it composes "with the node's own: no
fingerprint of the new form names them", and the switch's list counted it "not known". Both now leave
such a module out; its resources still go with the node's own composition, as before.

Quick and thorough are one here: the SDK names the modules the mesh provides, and every line that judges
fingerprints filters by it.

Research 072 piece: §6 milestone 2 (the composer reacts; its words say only what a change can fix).
2026-10-30 18:19:26 +01:00
jschoubben e6ecc2d15e Put mesh-delivery's merge words through mesh-sdk's kvlimit, so a word too large for its bucket is refused once and never retried (issue 774)
The controller's other writer of the 16 KiB bucket of merges, the keeper of mesh-delivery's merge words,
called every put failure "not now". A word too large would be retried as the forge's merges were. It now
puts through kvlimit.Put: a value that does not fit is refused before it is put and returned as an answer
(the events consumer takes it, max-deliver 5 then a dead letter), and only another failure is retried.
It is the one write of the controller whose failure was retried as "not now" (searched: every Put,
Create and Update wrapped in ErrTryAgain).

Needs the group's mesh-sdk head (go/kvlimit, 7b02f63 or later).

Quick fix: none smaller. Thorough fix: every bucket write through kvlimit; this is the only one whose
refusal was retried, the others fail once and say so. No new test: kvlimit is tested in mesh-sdk, and an
integration test here would push the repository further over its budget (rules-are-weighed).

Research 072 piece: 03-domains (Control shrinks): the merge words stay the controller's only until the
registration of a moved module leaves it.
2026-10-30 18:16:09 +01:00
jschoubben d543ff7126 Judge each read for its reader with mesh-sdk's ReadsAnnouncedBy, so the parts named for the lease are granted it, and say a refusal once (issue 774)
2536 reads of mesh-controller.lease were refused on 2026-10-30 (mesh-vault, mesh-grants, the assigner,
artifact-store-tools, the watchman, mesh-config). The controller composed the bus's users with its own
copy of a reader-less judgement (internal/broker ReadsAnnounced), which knows no named readers: the lease
is announced readable by nobody and read by the parts named for it (ADR 0547 clause 9, ADR 0561 §3). The
copy is deleted; BusRecords judges each read for its module with buscompose.ReadsAnnouncedBy. Each
refusal was also printed on every composition, about 420 times a read; it is now said once per process.

mesh-config's read of the lease stays refused: it reads the lease for its take-over (hand-back ends at
the lease write, ADR 0561 §3) and is not among the named readers. That is mesh-sdk's NamedReaders,
routed to the lead.

Quick fix: mark the lease readable in the controller's announcement (tried and reverted: it would open
the lease to every module and contradict ADR 0561 §3). Thorough fix: one judgement, the SDK's, for every
composer. Chose thorough.

Research 072 piece: 03-domains, mesh-grants (bus membership and grants leave the controller): the
controller composes reads with the same SDK function the grants part uses.
2026-10-30 18:15:17 +01:00
jschoubben d63eb83ed3 Clear a condition whose check no longer exists, so a removed row or probe leaves no alarm open for good (issue 774)
About 35 conditions from 10-20 to 10-27 still showed as current. Read on the bus at 18:06: S23's three
failed plans (last observed 10-24) and D14's four stalled deliveries (last observed 10-24) come from a
signals row and a probe this controller no longer has; nothing reconciles their source, so nothing clears
them. The other old ones are observed every tick and are not stale by the controller: S9's dead letters
are still held, S15's hand acts are still kept, and DM's 26 stuck keys are the watchman's (it never writes
a key whose watched key aged out as moved) — routed to mesh-delivery.

Each watchdogs tick now clears every open condition whose source has a row's or a probe's form (S12, D4,
DW, D-root) and is in neither the signals table nor the probe registry, saying the check was removed.
Sources of other forms are left to their own word.

Quick fix: clear the seven by hand. Thorough fix: the controller judges its own retired checks on every
tick, so the next removed row or probe clears what it raised. Chose thorough: rows and probes are being
moved out of the controller one by one (ADR 0564), each removal would leave this again.

Research 072 piece: milestone 3 (health leaves the controller): the conditions keeper drops what no check
of its own observes, so the move to mesh-conditions starts from a bucket of live conditions.
2026-10-30 18:10:29 +01:00
jschoubben 6ffcea4f1d Keep apply-failing raised while any failure stands, so a failure said again in new words never clears it (issue 774)
machine.novox.apply-failing cleared at 16:56 while the vault refused every secret there: the provisioner
client says a failure again whenever its words change, with a new time, and S21 measured each module's
failure from that newest time, so every new wording hid the failure for one apply interval (raised again
17:01). The same holds for the node-engine's own failed apply, whose since restarts with new words.

The bound now gates raising only: once the condition is open (read from the keeper's open conditions on
the bus), it stays while any module's apply fails on that machine, and clears when none does.

Quick fix: this hysteresis on the open condition in the controller. Thorough fix: the provisioner client
carries the start of a failure run (failing-since) in its report, so every reader measures from the true
start; that is mesh-catalog's mesh-provisioner-client, routed to the lead. Chose the quick fix here,
because the check moves to mesh-watchdogs (ADR 0564) and the condition's own state already proves the run.

Research 072 piece: milestone 3 (health leaves the controller): the judgement reads bus state, the open
condition, rather than a report's time.
2026-10-30 18:08:09 +01:00
jschoubben 48d9f22fa5 Retire the controller's merges consumer and read merges from the forge bridge's stream, so a held merge can never again queue every merge or look like a deaf controller (issue 774)
The controller-merges consumer (no max-deliver, one at a time) held the de-spiegel merge e25e1beb from
10-29 09:39: keepForgeMerge put an announcement with its Paths into delivery-merges (16 KiB values), got
10054, retried "not now" forever, and 167 merges queued behind it. Every five-minute hold outran the deaf
bound, so core.controller.deaf flapped.

Merges are mesh-delivery's: the gate merges on the Approve and the forge bridge says each merge on its
stream FORGE (pull request, head, merge commit), kept a week. The controller's copy was a leftover, read
only at a moved module's registration (ADR 0420 §7, ADR 0432). That read now finds the merge on FORGE;
the consumer, its follow, its handler and its hold delays are removed, and every assertion removes the
retired consumer from the bus.

Kind fix for the watchdog: the event loop records when a message it held back is due again, and deafness
counts from then, so no hold longer than the bound (on any consumer) reads as deafness.

Quick fix: strip Paths from the kept announcement and add max-deliver with a dead letter. Thorough fix:
remove the duplicate duty and make the deaf watchdog hold-aware. Chose thorough: the controller is being
deprecated and the merges already have an owner; the hold-vs-deaf confusion is a kind, not one consumer.

Research 072 piece: 03-domains (Control shrinks to placement and composition; the delivery parts own
what a merge is) and milestone 3 (health signals stop sharing the controller's loop assumptions).
2026-10-30 18:05:46 +01:00
79 changed files with 1458 additions and 3435 deletions
+22 -5
View File
@@ -30,6 +30,11 @@ const packageManagerKey = "the package manager ("
// says a re-apply only when how it went changed, so the same failure is one report however long it lasts.
// Names the refusal's or the failure's words; cleared by the clean apply the node-engine then says. Not while
// the machine is silent, asleep or not known: S1 says silent, and asleep is not lost.
//
// **The bound gates raising, never keeping** (sweep 2026-10-30, issue 774). A report says a failure again whenever its
// words change, with a new time: on novox the vault's refusal of every secret replaced a bus refusal at 16:56, and the
// condition cleared for five minutes while every module still failed. Once raised, it stays raised while any failure
// stands on that machine, judged by the open condition the keeper holds on the bus, not by a report's time.
func watchApplyFailing(f *signalFacts) []conditions.Observation {
heard := map[string]bool{}
for _, m := range f.machines {
@@ -40,7 +45,7 @@ func watchApplyFailing(f *signalFacts) []conditions.Observation {
var out []conditions.Observation
said := map[string]bool{}
for _, d := range f.doing {
if !d.Wrong() || d.Since == nil || f.now.Sub(*d.Since) <= applyEvery || !heard[d.Node] {
if !d.Wrong() || d.Since == nil || !heard[d.Node] || (f.now.Sub(*d.Since) <= applyEvery && !f.applyFailingOpen[d.Node]) {
continue
}
out = append(out, applyFailing(d, f.now))
@@ -51,7 +56,8 @@ func watchApplyFailing(f *signalFacts) []conditions.Observation {
// on a machine whose node-engine applied what it was sent.
failed := map[string][]inventory.ProvReport{}
for _, r := range f.provFailed {
if r.Status != inventory.ProvFailed || f.now.Sub(r.At) <= applyEvery || !heard[r.Node] || said[r.Node] {
if r.Status != inventory.ProvFailed || !heard[r.Node] || said[r.Node] ||
(f.now.Sub(r.Since()) <= applyEvery && !f.applyFailingOpen[r.Node]) {
continue
}
failed[r.Node] = append(failed[r.Node], r)
@@ -62,17 +68,28 @@ func watchApplyFailing(f *signalFacts) []conditions.Observation {
return out
}
// applyFailingOpen are the machines whose S21 condition is open among the open conditions.
func applyFailingOpen(open []conditions.Condition) map[string]bool {
out := map[string]bool{}
for _, c := range open {
if c.Kind == kindApplyFailing && c.Subject.Machine != "" {
out[c.Subject.Machine] = true
}
}
return out
}
// provisioningFailing is one machine's S21 condition for the modules whose files its provisioner client failed to
// place.
func provisioningFailing(m string, failed []inventory.ProvReport, now time.Time) conditions.Observation {
sort.Slice(failed, func(a, b int) bool { return failed[a].Module < failed[b].Module })
var names, words []string
since := failed[0].At
since := failed[0].Since()
for _, r := range failed {
names = append(names, r.Module)
words = append(words, r.Module+": "+oneLine(orNoReason(r.Reason)))
if r.At.Before(since) {
since = r.At
if r.Since().Before(since) {
since = r.Since()
}
}
o := conditions.Observation{Scope: conditions.ScopeMachine, ID: m, Kind: kindApplyFailing, Machine: m,
@@ -372,7 +372,7 @@ func TestTheAccountsAModuleSpeaksWithOnANode(t *testing.T) {
t.Error("sealed no password")
}
return c.sealErr
}, func() error { acts.did("after"); return nil })
})
if (err != nil) != (c.sealErr != nil) {
t.Fatalf("error %v, want %v", err, c.sealErr)
}
@@ -385,52 +385,6 @@ func TestTheAccountsAModuleSpeaksWithOnANode(t *testing.T) {
})
}
// Issue 774: where a vault is installed the controller mints nothing; `module issue` and an assignment's issue are
// forwarded to the vault's issue-account, and what the account needs beside its value still runs. A refusal fails
// `module issue` and only says itself on an assignment's issue: the vault mints on the want when it hears it.
for _, c := range []struct {
name string
only bool
answer string
refuse error
want string
fails bool
}{
{"module issue: forwarded, minted by the vault", false, `{"minted":true,"account":{"secret":"own/ace/mesh-build-go/broker","generation":1}}`, nil, "after", false},
{"an assignment's issue: forwarded, held already", true, `{"minted":false,"why":"held already"}`, nil, "after", false},
{"an assignment's issue: refused, said", true, "", errors.New("nobody wants it yet"), "after", false},
{"module issue: refused, failed", false, "", errors.New("nobody wants it"), "", true},
} {
t.Run("forwarded to the vault: "+c.name, func(t *testing.T) {
was := vaultIssuer
t.Cleanup(func() { vaultIssuer = was })
var asked []string
vaultIssuer = func(context.Context, accountIssuer) (vaultAsker, bool, error) {
return func(verb string, args map[string]any) (json.RawMessage, error) {
asked = append(asked, verb+" "+fmt.Sprint(args["user"]))
return json.RawMessage(c.answer), c.refuse
}, true, nil
}
acts := &anAccountStore{}
ctx, got := withIssued(t.Context())
if c.only {
ctx = onlyIfAbsent(ctx)
}
err := issueAccount(ctx, acts, inventory.BusUser{Username: "ace.mesh-build-go", Kind: inventory.BusModule},
func(string) error { acts.did("seal"); return nil }, func() error { acts.did("after"); return nil })
if (err != nil) != c.fails || strings.Join(acts.acts, " ") != c.want {
t.Fatalf("error %v, acts %q; want failed %v, acts %q (nothing minted, locked or sealed here)", err,
strings.Join(acts.acts, " "), c.fails, c.want)
}
if strings.Join(asked, ",") != "issue-account ace.mesh-build-go" {
t.Errorf("the vault was asked %v", asked)
}
if minted := len(got.list()) == 1; minted != strings.Contains(c.answer, `"minted":true`) {
t.Errorf("said issued %v", got.list())
}
})
}
// The accounts of a module that no longer resolves on its node are stale, its provisioner client's among them; a
// module still resolved (assigned or required), an account of another kind, and a node whose resolution is unknown
// are left alone (issue 1945: ace's mesh-build-go rows from 17:19, marked sealed by migration 0108).
+1 -1
View File
@@ -250,7 +250,7 @@ func prepareSend(ctx context.Context, open *stores, node string) ([]string, erro
if n.Stated && n.Contract >= compose.VaultBoxContract {
boxes, err := coreBoxesFor(ctx, inv, node, resolved.Plan.Modules, func(m catalogue.Manifest) bool {
return handing.Handing(m)
}, true)
}, true, in.BusMinted)
if err != nil {
return nil, err
}
+2 -2
View File
@@ -124,11 +124,11 @@ func composeModulesByNode(ctx context.Context, open *stores, nodes []string) (ma
errsOf[node] = append(errsOf[node], fmt.Errorf("%s: %w", node, err))
return
}
if len(moved) > 0 {
if moved := builtByTheMesh(moved); len(moved) > 0 {
fmt.Printf("compositions: %s's %s compose with the node's own: no fingerprint of the new form names them\n",
node, strings.Join(moved, ", "))
}
if len(left) > 0 {
if left := builtByTheMesh(left); len(left) > 0 {
fmt.Printf("compositions: %s's %s are left out of the node's own: the controller knows no fingerprint of the "+
"new form for them, and the node's want or have names their own composition (issue 1945)\n",
node, strings.Join(left, ", "))
+12 -4
View File
@@ -69,7 +69,15 @@ type coreBoxes struct {
// coreModulesOf is every module of the plan on the core's list whose own secrets this machine's declaration carries,
// with their plain own secret names, sorted: one handed to the provisioner has its secrets from the vault through the
// client, and one issued outside the mesh is never the vault's to seal here.
func coreModulesOf(modules []catalogue.Manifest, handed func(catalogue.Manifest) bool) map[string][]string {
//
// **Nor a process's bus account nobody issued yet** (sweep 2026-10-30, issue 774). Asked with `get`, the vault holds
// none and makes a random value under the account's name: own/novox/nats/broker-nats-tools at 16:36:51. Until the
// controller issues the account and hands it to the vault, the machine has the controller's stand-in (prepareSend), as
// every module below the vault's contract does. (The controller's own broker-address and broker-management are no
// accounts: the bus's address, which broker.FromEnvironment reads, and the retired AMQP management login; no process
// declares them, so this leaves them asked as before.)
func coreModulesOf(modules []catalogue.Manifest, handed func(catalogue.Manifest) bool, node string,
minted []string) map[string][]string {
out := map[string][]string{}
for _, m := range modules {
if !catalogue.CarriesVaultSealed(m.Module) || (handed != nil && handed(m)) {
@@ -77,7 +85,7 @@ func coreModulesOf(modules []catalogue.Manifest, handed func(catalogue.Manifest)
}
var names []string
for name, own := range m.OwnSecrets.Plain() {
if own.IssuedBy != catalogue.IssuedOutside {
if own.IssuedBy != catalogue.IssuedOutside && processAccountWarning(m, node, name, minted) == "" {
names = append(names, name)
}
}
@@ -208,8 +216,8 @@ var coreVault = func(ctx context.Context) coreAsk {
// coreBoxesFor is coreBoxesOf for one machine, read from and kept in the store, and said on a send.
func coreBoxesFor(ctx context.Context, inv *inventory.Inventory, node string, modules []catalogue.Manifest,
handed func(catalogue.Manifest) bool, allocating bool) (coreBoxes, error) {
core := coreModulesOf(modules, handed)
handed func(catalogue.Manifest) bool, allocating bool, minted []string) (coreBoxes, error) {
core := coreModulesOf(modules, handed, node, minted)
if len(core) == 0 {
if allocating {
settleVaultSilent(ctx, node)
+28 -1
View File
@@ -124,8 +124,35 @@ func TestTheCoresModulesAreTheListsWithTheirOwnSecrets(t *testing.T) {
{Module: "postgres", OwnSecrets: catalogue.OwnSecrets{"superuser": {Path: "/c"}}},
{Module: "mesh-provisioner", OwnSecrets: catalogue.OwnSecrets{"broker": {Path: "/d"}}},
}
got := coreModulesOf(modules, func(m catalogue.Manifest) bool { return m.Module == "mesh-provisioner" })
got := coreModulesOf(modules, func(m catalogue.Manifest) bool { return m.Module == "mesh-provisioner" }, "anchor", nil)
if want := map[string][]string{"nats": {"broker"}}; !reflect.DeepEqual(got, want) {
t.Fatalf("got %v, want %v", got, want)
}
}
// Sweep 2026-10-30 (issue 774): a bus process's account is asked of the vault only once it is issued; before, the vault
// would make a random value under the account's name.
func TestAnUnissuedProcessAccountIsNotAskedOfTheVault(t *testing.T) {
nats, err := catalogue.ParseManifest([]byte(`{"module":"nats","version":"1","emits":["x"],
"own-secrets":{"broker":{"path":"/a"},"broker-nats-tools":{"path":"/b","owner":"nats-tools"}},
"resources":[{"id":"account","type":"user","name":"nats-tools"},
{"id":"nats-tools","type":"process","run":["./nats-tools"],"user":"nats-tools","bus":{"emits":["x"]}}]}`))
if err != nil {
t.Fatal(err)
}
if len(nats.ProcessBuses()) == 0 {
t.Fatal("the fixture declares no bus process")
}
for name, c := range map[string]struct {
minted []string
want []string
}{
"not issued": {nil, []string{"broker"}},
"issued": {[]string{processUser("anchor", "nats", "nats-tools")}, []string{"broker", "broker-nats-tools"}},
} {
got := coreModulesOf([]catalogue.Manifest{nats}, nil, "anchor", c.minted)
if !reflect.DeepEqual(got["nats"], c.want) {
t.Errorf("%s: asks %v, want %v", name, got["nats"], c.want)
}
}
}
@@ -0,0 +1,31 @@
//go:build integration
package main
import (
"context"
"testing"
"github.com/nats-io/nats.go/jetstream"
sdkdelivery "git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/process"
"github.com/novox/mesh-controller/internal/broker"
)
// sayForgeMerge is the forge bridge saying a merge on its stream FORGE, which it makes when missing.
func sayForgeMerge(ctx context.Context, t *testing.T, js *broker.JetStream, repo string, number int, head, merge string) {
t.Helper()
api, err := jetstream.New(js.Conn())
if err != nil {
t.Fatal(err)
}
if _, err := api.CreateOrUpdateStream(ctx, jetstream.StreamConfig{Name: forgeStream,
Subjects: []string{process.ModuleStreamSubject(sdkdelivery.ForgeBridgeModule, "forge.>")}}); err != nil {
t.Fatal(err)
}
subject, body := forgeMerged(t, repo, number, head, merge)
if _, err := api.Publish(ctx, subject, body); err != nil {
t.Fatal(err)
}
}
+106
View File
@@ -0,0 +1,106 @@
package main
import (
"context"
"encoding/json"
"strings"
"testing"
"github.com/nats-io/nats.go/jetstream"
sdkdelivery "git.novox.be/novox/mesh-sdk/go/delivery"
)
// fakeForge is the forge bridge's stream as forgeMergeAt reads it: messages by sequence, each on a subject.
type fakeForge struct {
subjects []string
data [][]byte
}
func (f fakeForge) last(_ context.Context, subject string) (uint64, []byte, error) {
for i := len(f.subjects) - 1; i >= 0; i-- {
if f.subjects[i] == subject {
return uint64(i + 1), f.data[i], nil
}
}
return 0, nil, jetstream.ErrMsgNotFound
}
func (f fakeForge) next(_ context.Context, subject string, from uint64) (uint64, []byte, error) {
for i := int(from) - 1; i < len(f.subjects); i++ {
if i >= 0 && f.subjects[i] == subject {
return uint64(i + 1), f.data[i], nil
}
}
return 0, nil, jetstream.ErrMsgNotFound
}
func forgeMerged(t *testing.T, repo string, number int, head, merge string) (string, []byte) {
t.Helper()
owner, name, _ := strings.Cut(repo, "/")
subject, err := forgeMergeSubject(owner, name)
if err != nil {
t.Fatal(err)
}
body, err := json.Marshal(sdkdelivery.Forge{V: 1, Kind: sdkdelivery.ForgeMerged, Repository: repo, Branch: "main",
Number: number, Head: head, Merge: merge, Root: "t0", Trees: map[string]string{".": "t1"}})
if err != nil {
t.Fatal(err)
}
return subject, body
}
// **A moved module's merge is read from the forge bridge's stream** (sweep 2026-10-30, issue 774): the controller keeps
// no copy of the forge's merges, so its registration finds the pull request and head of a merge commit on FORGE, the
// newest first and then every merge the stream keeps; a merge of another repository or commit is never taken for it.
func TestAMergeIsFoundOnTheForgeBridgesStream(t *testing.T) {
var f fakeForge
add := func(repo string, number int, head, merge string) {
s, b := forgeMerged(t, repo, number, head, merge)
f.subjects, f.data = append(f.subjects, s), append(f.data, b)
}
add("novox/mesh-host", 6, "h6", "c6c6c6c6c6")
add("novox/mesh-catalog", 9, "h9", "c7c7c7c7c7")
add("novox/mesh-host", 7, "h7", "c7c7c7c7c7")
f.subjects, f.data = append(f.subjects, f.subjects[0]), append(f.data, []byte("not a forge message"))
add("novox/mesh-host", 8, "h8", "c8c8c8c8c8")
for name, c := range map[string]struct {
repo, commit string
number int
found bool
}{
"the newest merge": {"novox/mesh-host", "c8c8c8c8c8", 8, true},
"an older merge, found by scanning": {"novox/mesh-host", "c6c6c6c6c6", 6, true},
"a short commit": {"novox/mesh-host", "c7c7c7c", 7, true},
"a commit of another repository": {"novox/mesh-catalog", "c6c6c6c6c6", 0, false},
"a repository with no merge": {"novox/hq", "c6c6c6c6c6", 0, false},
} {
owner, name2, _ := strings.Cut(c.repo, "/")
k, found, err := forgeMergeAt(t.Context(), f, owner, name2, c.commit)
if err != nil {
t.Fatalf("%s: %v", name, err)
}
if found != c.found || k.Number != c.number {
t.Errorf("%s: found %v #%d, want %v #%d", name, found, k.Number, c.found, c.number)
}
if found && (k.Owner != owner || k.Repo != name2 || k.HeadSHA == "") {
t.Errorf("%s: the merge reads as %+v", name, k.SourceMoved)
}
}
}
// Sweep 2026-10-30 (issue 774): a module the mesh provides itself is never named as lacking a fingerprint.
func TestAModuleTheMeshProvidesIsNotSaidToLackAFingerprint(t *testing.T) {
for _, c := range []struct {
in, want []string
}{
{[]string{"mesh-wireguard"}, nil},
{[]string{"claude-code", "mesh-wireguard", "power"}, []string{"claude-code", "power"}},
{nil, nil},
} {
if got := builtByTheMesh(c.in); strings.Join(got, ",") != strings.Join(c.want, ",") {
t.Errorf("%v: %v, want %v", c.in, got, c.want)
}
}
}
+13 -53
View File
@@ -2,15 +2,9 @@ package main
import (
"context"
"errors"
"fmt"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/catalogue"
sdkdelivery "git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
)
@@ -21,19 +15,17 @@ import (
// controller mints such an account and writes its hash to the bus's user list, as before, and then hands the whole
// value at once to the vault's `accept` with origin `issued`: sealed here to the key the vault made for that call, so
// the vault alone opens it (ADR 0349 clause 8). The vault keeps it as the module's own secret on that machine and
// raises `secret-changed`; a re-issue is the same hand with a new value, which replaces the old in place. Until issue 774 an
// account not covered was minted and kept as it was and handed to nobody.
//
// **Since issue 774 every account the controller mints is handed, covered or not**, once a vault is installed: until
// the grants part's take-over the controller still mints (the bus loads its list), and a vault that holds every value
// at once needs no node to report a hash for it to adopt (ADR 0547 clause 9). From the take-over the vault mints every
// account itself (issueAccount forwards to its issue-account), so this hand is reached only before it, and by
// `rollout mint`'s re-issue of a module's account onto a new bus. The vault never lets such a hand replace an account it
// minted.
// raises `secret-changed`; a re-issue is the same hand with a new value, which replaces the old in place. An account
// not covered is minted and kept as it was and handed to nobody.
// vaultAskerForIssue is the vault as `module issue` asks it. A test replaces it.
var vaultAskerForIssue = func(ctx context.Context) vaultAsker { return theVault(ctx) }
// accountIsCovered says whether the bus accounts minted for m are the vault's to hold (ADR 0374 clause 1).
func accountIsCovered(m catalogue.Manifest) bool {
return catalogue.CarriesVaultSealed(m.Module) || m.Moved
}
// vaultInstalled says whether the vault is assigned to any machine. Until it is, as in genesis (ADR 0113 §15 to §18,
// mesh-host's bootstrap, which issues the bus's and the provisioner's accounts before the vault step), there is no
// vault to hand to: the value is kept in the controller's store as before, and the vault takes it afterwards by the
@@ -52,45 +44,13 @@ func vaultInstalled(ctx context.Context, inv *inventory.Inventory) (bool, error)
return false, nil
}
// grantsTookOver says whether the grants part took over the bus's user list: its key `user-list.sealed` says
// FollowGrants (novox/hq ADR 0547 clauses 7 and 8), which the bus node's engine follows. No bucket or no key yet is no
// take-over; a bus that cannot be read is an error, never a guess. A test replaces it.
var grantsTookOver = func(ctx context.Context) (bool, error) {
over := false
err := onTheBus(func(conn *nats.Conn) error {
js, err := jetstream.New(conn)
if err != nil {
return err
}
kv, err := js.KeyValue(ctx, broker.GrantsUserListBucket)
if errors.Is(err, jetstream.ErrBucketNotFound) {
return nil
} else if err != nil {
return err
}
e, err := kv.Get(ctx, broker.GrantsUserListKey)
if errors.Is(err, jetstream.ErrKeyNotFound) {
return nil
} else if err != nil {
return err
}
g, err := sdkdelivery.DecodeGrantsUserList(e.Value())
if err != nil {
return err
}
over = g.Follow == sdkdelivery.FollowGrants
return nil
})
if err != nil {
return false, fmt.Errorf("whether the grants part took over the bus's user list cannot be read: %w", err)
}
return over, nil
}
// handIssuedToVault hands the value just minted for module's own secret name on node to the vault, once one is
// installed, covered or not (issue 774), and does nothing before. Loud when the vault does not take it: the bus's user
// list already holds the new hash, so a vault that kept the old value would serve a password that no longer logs in.
// handIssuedToVault hands the value just minted for module's own secret name on node to the vault, when the module's
// accounts are covered, and does nothing otherwise. Loud when the vault does not take it: the bus's user list
// already holds the new hash, so a vault that kept the old value would serve a password that no longer logs in.
func handIssuedToVault(ctx context.Context, inv *inventory.Inventory, m catalogue.Manifest, node, name, held string) error {
if !accountIsCovered(m) {
return nil
}
if installed, err := vaultInstalled(ctx, inv); err != nil {
return fmt.Errorf("whether the vault is installed cannot be read, so %s of %s was not handed to it: %w", name,
m.Module, err)
+8 -69
View File
@@ -8,50 +8,21 @@ import (
"strings"
"testing"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/catalogue"
sdkdelivery "git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/testbus"
)
// grantsTookOverOnTheBus is the reader of the grants part's take-over a test reads on its own bus.
var grantsTookOverOnTheBus = grantsTookOver
// Every other test mints without a vault to hand to: a stand-in that takes anything, replaced by the tests below; and
// before the grants part's take-over, which no test bus says.
// Every other test mints without a vault to hand to: a stand-in that takes anything, replaced by the tests below.
func init() {
vaultAskerForIssue = func(context.Context) vaultAsker { return newStandInVault(nil).ask }
grantsTookOver = func(context.Context) (bool, error) { return false, nil }
}
// theGrantsPartTookOver says the grants part's take-over is on, or off, for the rest of the test.
func theGrantsPartTookOver(t *testing.T, over bool) {
t.Helper()
was := grantsTookOver
grantsTookOver = func(context.Context) (bool, error) { return over, nil }
t.Cleanup(func() { grantsTookOver = was })
}
// An account is handed to the vault when it is minted (novox/hq ADR 0374, issue 787): the whole value, sealed to the
// key the vault made for the call, with origin issued. Since issue 774 every account is, covered or not, until the
// grants part's take-over, read on the bus as the bus node reads it; from the take-over nothing is minted here and the
// issue is forwarded to the vault's issue-account.
func TestModuleIssueHandsEveryAccountToTheVaultUntilTheTakeOver(t *testing.T) {
// A covered account is handed to the vault when it is minted (novox/hq ADR 0374, issue 787): the whole value, sealed
// to the key the vault made for the call, with origin issued. A module that is neither core nor moved hands nothing.
func TestModuleIssueHandsACoveredAccountToTheVaultAndOnlyThat(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
theVaultIsInstalled(t, open)
js, err := broker.Dial(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(js.Close)
servingBus.Store(js)
t.Cleanup(func() { servingBus.Store(nil) })
was := grantsTookOver
grantsTookOver = grantsTookOverOnTheBus
t.Cleanup(func() { grantsTookOver = was })
inv := open.inventory
known := broker.Broker{Fingerprint: "ab"}
@@ -95,11 +66,11 @@ func TestModuleIssueHandsEveryAccountToTheVaultUntilTheTakeOver(t *testing.T) {
t.Errorf("the vault was told %s, without %s", said, want)
}
}
if n := strings.Count(said, `"origin":"issued"`); n != 4 {
t.Errorf("%d accounts were handed with origin issued, want 4 (the core's two, the moved and the plain module's): %s", n, said)
if n := strings.Count(said, `"origin":"issued"`); n != 3 {
t.Errorf("%d accounts were handed with origin issued, want 3 (the core's two and the moved module's): %s", n, said)
}
if !strings.Contains(said, "kept-bot") {
t.Errorf("an account that is neither core nor moved was not handed to the vault: %s", said)
if strings.Contains(said, "kept-bot") {
t.Errorf("an account that is neither core nor moved was handed to the vault: %s", said)
}
if strings.Contains(said, "first-core-password") || strings.Contains(said, "first-moved-password") {
t.Errorf("a value crossed the bus in the clear: %s", said)
@@ -112,38 +83,6 @@ func TestModuleIssueHandsEveryAccountToTheVaultUntilTheTakeOver(t *testing.T) {
!strings.Contains(got, `"url":"nats://10.0.0.1:4222"`) || !strings.Contains(got, `"fingerprint":"ab"`) {
t.Errorf("the vault was handed %q after a re-issue, not the new account whole", got)
}
// The take-over: the grants part's key says the bus node follows its list. From then the vault mints.
api, err := jetstream.New(js.Conn())
if err != nil {
t.Fatal(err)
}
kv, err := api.CreateOrUpdateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: broker.GrantsUserListBucket})
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = api.DeleteKeyValue(context.Background(), broker.GrantsUserListBucket) })
client := broker.Principal{Kind: broker.KindProvisioner, Node: "laptop", Module: "nats"}.Username()
for _, follow := range []string{sdkdelivery.FollowController, sdkdelivery.FollowGrants} {
value, err := sdkdelivery.EncodeGrantsUserList(sdkdelivery.GrantsUserList{Follow: follow})
if err != nil {
t.Fatal(err)
}
if _, err := kv.Put(ctx, broker.GrantsUserListKey, value); err != nil {
t.Fatal(err)
}
before, _, _ := inv.BusUserHash(ctx, client)
if err := issueProvisionerClient(ctx, inv, core, "laptop", known, "10.0.0.1:4222"); err != nil {
t.Fatal(err)
}
after, _, _ := inv.BusUserHash(ctx, client)
if minted := before != after; minted != (follow == sdkdelivery.FollowController) {
t.Errorf("following %s's list, the controller minted %s: %v", follow, client, minted)
}
}
if said := vault.said(); strings.Count(said, `"verb":"issue-account"`) != 1 || !strings.Contains(said, `"user":"`+client+`"`) {
t.Errorf("the vault was not asked once, after the take-over, to issue %s: %s", client, said)
}
}
// A vault that does not take the value fails the issue loudly: the bus's user list holds the new hash already.
+54 -1
View File
@@ -21,8 +21,13 @@ func holdingOne(names ...string) func(broker.Consumer) (uint64, error) {
}
func deafObservations(now time.Time, held func(broker.Consumer) (uint64, error)) (string, int) {
return deafHeld(now, time.Time{}, held)
}
// deafHeld is deafObservations with the loop's last hold due at due.
func deafHeld(now, due time.Time, held func(broker.Consumer) (uint64, error)) (string, int) {
f := calm(now)
loop, err := readLoop(now.Add(-loopDeafAfter-time.Minute), loopConsumers(), held)
loop, err := readLoop(now.Add(-loopDeafAfter-time.Minute), due, loopConsumers(), held)
if err != nil {
panic(err)
}
@@ -65,3 +70,51 @@ func TestDeafLookReadsExactlyTheControllersOwnConsumers(t *testing.T) {
}
}
}
// Sweep 2026-10-30 (issue 774): a message the loop held back with a delay longer than the deaf bound is not a deaf
// controller until it is due and still not taken; the merges consumer's five-minute holds flapped the alarm this way.
func TestAMessageHeldBackIsNotDeafUntilItIsDue(t *testing.T) {
now := time.Now()
held := holdingOne("CONTROL/" + broker.ControllerName)
for name, c := range map[string]struct {
due time.Time
deaf int
}{
"nothing held": {time.Time{}, 1},
"held, due in three minutes": {now.Add(3 * time.Minute), 0},
"held, due a moment ago": {now.Add(-time.Second), 0},
"held, due past the deaf bound, unsent": {now.Add(-loopDeafAfter - time.Second), 1},
} {
if _, n := deafHeld(now, c.due, held); n != c.deaf {
t.Errorf("%s: %d deaf observation(s), want %d", name, n, c.deaf)
}
}
}
// Sweep 2026-10-30 (issue 774): the controller holds no consumer of the forge's merges any more, and removes the one
// it held at every assertion.
func TestTheMergesConsumerIsRetired(t *testing.T) {
for _, c := range broker.MeshConsumers() {
if c.Name == broker.MergesConsumerName {
t.Fatalf("the controller still holds %s on %s", c.Name, c.Stream)
}
}
r := &removed{}
if err := broker.AssertMeshConsumers(r); err != nil {
t.Fatal(err)
}
if !r.gone[broker.EventsStream+"/"+broker.MergesConsumerName] {
t.Fatalf("the retired merges consumer is not removed: removed %v", r.gone)
}
}
type removed struct{ gone map[string]bool }
func (*removed) EnsureConsumer(broker.Consumer) error { return nil }
func (r *removed) RemoveConsumer(stream, name string) error {
if r.gone == nil {
r.gone = map[string]bool{}
}
r.gone[stream+"/"+name] = true
return nil
}
+44 -132
View File
@@ -1018,21 +1018,8 @@ const issueLockedFor = 2 * time.Minute
// left ace's mesh-build-go with no credential), under the store's lock on the account, so two issuers never interleave
// their look, mint and seal: the hash the bus holds is always the password sealed. A seal that fails leaves the
// account unsealed, issued again by the next issue.
//
// **From the grants part's take-over the controller mints nothing** (issue 774, research 072's gap "vault mints after
// the lease"; the controller is deprecated and hands its duties over): the bus then loads the grants part's list, whose
// hashes are the vault's, so the issue is forwarded to the vault's issue-account and only after runs (what the account
// needs beside its value: its durable consumers). **Before it** the bus loads the controller's own list, whose hashes
// come from its store alone, so an account the vault minted would be refused: the controller mints as before and hands
// every account it mints to an installed vault at once (handIssuedToVault), so none waits on a node reporting its hash.
func issueAccount(ctx context.Context, accounts accountIssuer, u inventory.BusUser, seal func(password string) error,
after func() error) error {
func issueAccount(ctx context.Context, accounts accountIssuer, u inventory.BusUser, seal func(password string) error) error {
only, _ := ctx.Value(mintOnlyIfAbsent{}).(bool)
if ask, installed, err := vaultIssuer(ctx, accounts); err != nil {
return fmt.Errorf("whether the vault issues bus accounts yet cannot be read, so %s is not issued: %w", u.Username, err)
} else if installed {
return forwardToTheVault(ctx, ask, u.Username, only, after)
}
if !only {
password, err := accounts.MintBusPassword(ctx, u)
if err != nil {
@@ -1075,67 +1062,6 @@ func issueAccount(ctx context.Context, accounts accountIssuer, u inventory.BusUs
return nil
}
// vaultIssuer answers the vault to forward an issue to, and whether the issue is the vault's: one is installed
// (vaultInstalled) and the grants part took over the bus's user list (grantsTookOver). A store that is not the
// controller's inventory (a test's) has none. A test replaces it.
var vaultIssuer = func(ctx context.Context, accounts accountIssuer) (vaultAsker, bool, error) {
inv, ok := accounts.(*inventory.Inventory)
if !ok {
return nil, false, nil
}
installed, err := vaultInstalled(ctx, inv)
if err != nil || !installed {
return nil, false, err
}
if over, err := grantsTookOver(ctx); err != nil || !over {
return nil, false, err
}
return vaultAskerForIssue(ctx), true, nil
}
// issuedByTheVault is the vault's answer to issue-account (mesh-vault heldBackOrMinted).
type issuedByTheVault struct {
Minted bool `json:"minted"`
Why string `json:"why"`
Account struct {
Secret string `json:"secret"`
Generation int `json:"generation"`
} `json:"account"`
}
// forwardToTheVault asks the vault to issue user (its verb issue-account) and then runs after. The vault mints only
// what the grants part wants and never anew a value it holds, so asking twice, or with the want's own mint, is harmless.
// An issue on an assignment (only) is not failed by a refusal: the vault mints on the want itself when it is heard, and
// the refusal is said; `module issue` fails with it.
func forwardToTheVault(ctx context.Context, ask vaultAsker, user string, only bool, after func() error) error {
raw, err := ask("issue-account", map[string]any{"user": user})
if err != nil {
if !only {
return fmt.Errorf("the vault is the one issuer of bus accounts and did not issue %s: %w", user, err)
}
fmt.Printf(" %s is the vault's to mint, on the grants part's want of it; issue-account said: %v\n", user, err)
} else {
var got issuedByTheVault
if err := json.Unmarshal(raw, &got); err != nil {
return fmt.Errorf("the vault's answer to issue-account %s does not read: %w", user, err)
}
if got.Minted {
fmt.Printf("bus user %s minted by the vault as %s (generation %d)\n", user, got.Account.Secret, got.Account.Generation)
if g, _ := ctx.Value(issuedNowKey{}).(*issuedNow); g != nil {
g.mu.Lock()
g.users = append(g.users, user)
g.mu.Unlock()
}
} else {
fmt.Printf("bus user %s not minted anew by the vault: %s\n", user, got.Why)
}
}
if after == nil {
return nil
}
return after()
}
// issueOnTheNewBus gives an assigned module its credential on the bus being built.
//
// **Three things differ from a management call, and each is the point of the move.** The credential
@@ -1164,7 +1090,7 @@ func issueOnTheNewBus(ctx context.Context, inv *inventory.Inventory, m catalogue
if err := issueAccount(ctx, inv, inventory.BusUser{Username: user, Kind: busKindOf(m.Module), Node: node,
Module: m.Module}, func(password string) error {
return issueWith(ctx, inv, m, node, busAddress, known, reachable, user, password)
}, func() error { return ensureModuleConsumer(m, node, busAddress) }); err != nil {
}); err != nil {
return err
}
}
@@ -1192,7 +1118,7 @@ func issueProcess(ctx context.Context, inv *inventory.Inventory, m catalogue.Man
return issueAccount(ctx, inv, inventory.BusUser{Username: user, Kind: inventory.BusProcess, Node: node, Module: m.Module},
func(password string) error {
return sealProcess(ctx, inv, m, pr, node, busAddress, known, reachable, user, password)
}, func() error { return ensureProcessQueues(m, pr, node, busAddress) })
})
}
// sealProcess seals a process's minted account to its node, hands it to the vault and makes its work-queue consumers.
@@ -1209,40 +1135,30 @@ func sealProcess(ctx context.Context, inv *inventory.Inventory, m catalogue.Mani
if err := handIssuedToVault(ctx, inv, m, node, name, string(held)); err != nil {
return err
}
if err := ensureProcessQueues(m, pr, node, busAddress); err != nil {
return err
if len(pr.Queues) > 0 {
if busAddress == "" {
fmt.Printf(" %s's process %s consumes its work queue; its consumer is created when the bus is "+
"reachable (`push --behind`, then `rollout mint` again is harmless)\n", m.Module, pr.Process)
} else {
js, err := broker.Dial(busAddress)
if err != nil {
return fmt.Errorf("the credential is minted and the mesh cannot reach the bus to create "+
"%s's process %s's work-queue consumer: %w", m.Module, pr.Process, err)
}
defer js.Close()
who := broker.Principal{Kind: broker.KindProcess, Node: node, Module: m.Module, Process: pr.Process}
for _, q := range pr.Queues {
if err := js.EnsureConsumer(broker.QueueConsumer(who, q)); err != nil {
return err
}
}
}
}
fmt.Printf("bus user %s minted for %s's process %s, scoped to its bus declaration\n", user, m.Module, pr.Process)
fmt.Printf(" sealed to %s as %s. It arrives with the next push — `push %s` to send it\n", node, name, node)
return nil
}
// ensureProcessQueues makes a process's work-queue consumers where the bus can be reached: what its account needs beside
// its value, whoever minted it.
func ensureProcessQueues(m catalogue.Manifest, pr broker.ProcessPrincipal, node, busAddress string) error {
if len(pr.Queues) == 0 {
return nil
}
if busAddress == "" {
fmt.Printf(" %s's process %s consumes its work queue; its consumer is created when the bus is "+
"reachable (`push --behind`, then `rollout mint` again is harmless)\n", m.Module, pr.Process)
return nil
}
js, err := broker.Dial(busAddress)
if err != nil {
return fmt.Errorf("the mesh cannot reach the bus to create %s's process %s's work-queue consumer: %w",
m.Module, pr.Process, err)
}
defer js.Close()
who := broker.Principal{Kind: broker.KindProcess, Node: node, Module: m.Module, Process: pr.Process}
for _, q := range pr.Queues {
if err := js.EnsureConsumer(broker.QueueConsumer(who, q)); err != nil {
return err
}
}
return nil
}
// heldProcessAccount is a process's account, in the provisioner client's form (novox/hq ADR 0436 §2):
// {url, fingerprint, node, module, user, password} — never claims, since a process holds no seat (§3).
func heldProcessAccount(known broker.Broker, reachable, node, module, user, password string) ([]byte, error) {
@@ -1275,7 +1191,7 @@ func issueProvisionerClient(ctx context.Context, inv *inventory.Inventory, m cat
return issueAccount(ctx, inv, inventory.BusUser{Username: user, Kind: inventory.BusProvisioner, Node: node,
Module: m.Module}, func(password string) error {
return sealProvisionerClient(ctx, inv, m, node, known, reachable, user, password)
}, nil)
})
}
// sealProvisionerClient seals the provisioner client's minted account to its machine and hands it to the vault.
@@ -1347,8 +1263,28 @@ func issueWith(ctx context.Context, inv *inventory.Inventory, m catalogue.Manife
if err := handIssuedToVault(ctx, inv, m, node, "broker", string(held)); err != nil {
return err
}
if err := ensureModuleConsumer(m, node, busAddress); err != nil {
return err
// And how it hears what it consumes. Derived from its declaration, and only when it declared
// something: a module that consumes nothing needs no consumer, and creating one would be a
// durable subscription nobody reads.
if consumer, needed := broker.ConsumerFor(broker.Principal{
Kind: broker.KindModule, Node: node, Module: m.Module,
Emits: m.EmitsAll(), Consumes: m.Consumes, Serves: m.Tools,
}); needed {
if busAddress == "" {
fmt.Printf(" %s consumes; its consumer is created when the bus is reachable (`push --behind`, then "+
"`rollout mint` again is harmless)\n", m.Module)
} else {
js, err := broker.Dial(busAddress)
if err != nil {
return fmt.Errorf("the credential is minted and the mesh cannot reach the bus to create "+
"how %s hears what it consumes: %w", m.Module, err)
}
defer js.Close()
if err := js.EnsureConsumer(consumer); err != nil {
return err
}
}
}
fmt.Printf("bus user %s minted for %s, scoped to what it emits and consumes\n", user, m.Module)
@@ -1358,30 +1294,6 @@ func issueWith(ctx context.Context, inv *inventory.Inventory, m catalogue.Manife
return nil
}
// ensureModuleConsumer makes how a module hears what it consumes, where the bus can be reached: what its account needs
// beside its value, whoever minted it. Derived from its declaration, and only when it declared something: a module that
// consumes nothing needs no consumer, and creating one would be a durable subscription nobody reads.
func ensureModuleConsumer(m catalogue.Manifest, node, busAddress string) error {
consumer, needed := broker.ConsumerFor(broker.Principal{
Kind: broker.KindModule, Node: node, Module: m.Module,
Emits: m.EmitsAll(), Consumes: m.Consumes, Serves: m.Tools,
})
if !needed {
return nil
}
if busAddress == "" {
fmt.Printf(" %s consumes; its consumer is created when the bus is reachable (`push --behind`, then "+
"`rollout mint` again is harmless)\n", m.Module)
return nil
}
js, err := broker.Dial(busAddress)
if err != nil {
return fmt.Errorf("the mesh cannot reach the bus to create how %s hears what it consumes: %w", m.Module, err)
}
defer js.Close()
return js.EnsureConsumer(consumer)
}
// whereItComesFrom is the provenance a module handed over by hand records, and what a record must
// say to be worth anything later.
//
+98 -42
View File
@@ -16,6 +16,8 @@ import (
sdkdelivery "git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/kvlimit"
"git.novox.be/novox/mesh-sdk/go/process"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
@@ -179,7 +181,12 @@ func (k keepingDeliveryMerges) KeepDeliveryMerge(ctx context.Context, emitter st
if err != nil {
return fmt.Errorf("%w: the bucket of mesh-delivery's merges: %w", link.ErrTryAgain, err)
}
if _, err := kv.Put(ctx, deliveryMergeKey(d.Owner, d.Repo, d.Number, d.HeadSHA), body); err != nil {
// Through kvlimit (sweep 2026-10-30, issue 774): a word too large for the bucket is refused before it is put, and
// is an answer, never "not now" — retried, it held the forge's merges for a day (10054 on de-spiegel's merge).
if _, err := kvlimit.Put(ctx, kv, deliveryMergeKey(d.Owner, d.Repo, d.Number, d.HeadSHA), body); err != nil {
if errors.Is(err, kvlimit.ErrTooLarge) {
return err
}
return fmt.Errorf("%w: %w", link.ErrTryAgain, err)
}
if k.kept != nil {
@@ -343,64 +350,113 @@ func claimantsOf(entries []inventory.Entry, seat string) []string {
return out
}
// forgeMergeKey is where the forge's announcement of a merge is kept, by the merge commit: the one thing a build of a merge
// names, from which the pull request and its head are found (novox/hq ADR 0420 §7).
func forgeMergeKey(owner, repo, commit string) string {
return fmt.Sprintf("merge/%s/%s/%s", owner, repo, commit)
}
// keepForgeMerge keeps the forge's announcement of a merge in the bucket of the merges, a week at most as that bucket
// keeps them. The controller plans nothing from a merge any more (ADR 0539 §17); this is all it keeps of one.
var keepForgeMerge = func(ctx context.Context, m link.SourceMoved) error {
serving := servingBus.Load()
if serving == nil {
return fmt.Errorf("%w: the bus is not held", link.ErrTryAgain)
}
api, err := jetstream.New(serving.Conn())
if err != nil {
return fmt.Errorf("%w: %w", link.ErrTryAgain, err)
}
kv, err := api.KeyValue(ctx, broker.DeliveryMergesBucket)
if err != nil {
return fmt.Errorf("%w: the bucket of the merges: %w", link.ErrTryAgain, err)
}
body, err := json.Marshal(keptMerge{SourceMoved: m})
if err != nil {
return err
}
if _, err := kv.Put(ctx, forgeMergeKey(m.Owner, m.Repo, m.Commit), body); err != nil {
return fmt.Errorf("%w: %w", link.ErrTryAgain, err)
}
return nil
}
// keptForgeMerge is the forge's announcement of a merge, by its commit; false when none was kept. A variable so a test can
// hand it one.
// keptForgeMerge is the forge's word on the merge of a repository at a commit: the pull request and the head it merged,
// read from the forge bridge's own stream, FORGE, where mesh-delivery-forge-bridge says every merge as
// `forge.<owner>.<name>.merged` and keeps it a week (novox/hq ADR 0437). False when the stream holds no merge at that
// commit. A variable so a test can hand it one.
//
// **The controller keeps no copy of the forge's merges any more** (sweep 2026-10-30, issue 774). It once kept each in
// its bucket of merges off a consumer of its own on EVENTS that never gave up a message; one announcement over the
// bucket's value size held that consumer for a day and queued every merge behind it, and the deaf watchdog read the
// queue as a deaf controller. The forge bridge's stream is the record of the merges (research 072, 03-domains: the
// delivery parts own what a merge is); the controller reads it where it needs one, at a moved module's registration.
var keptForgeMerge = func(ctx context.Context, owner, repo, commit string) (keptMerge, bool, error) {
serving := servingBus.Load()
if serving == nil {
return keptMerge{}, false, errors.New("only the serving controller keeps what the forge announced")
return keptMerge{}, false, errors.New("only the serving controller reads the forge's merges")
}
api, err := jetstream.New(serving.Conn())
if err != nil {
return keptMerge{}, false, err
}
kv, err := api.KeyValue(ctx, broker.DeliveryMergesBucket)
stream, err := api.Stream(ctx, forgeStream)
if err != nil {
return keptMerge{}, false, fmt.Errorf("the forge bridge's stream %s cannot be read: %w", forgeStream, err)
}
return forgeMergeAt(ctx, streamMerges{stream}, owner, repo, commit)
}
// forgeStream is the forge bridge's stream as the bus names it (its declared name, forge, in capitals).
var forgeStream = catalogue.StreamBusName(sdkdelivery.ForgeStream)
// forgeMergeSubject is the subject of the forge bridge's merged message for a repository.
func forgeMergeSubject(owner, repo string) (string, error) {
event, err := sdkdelivery.ForgeEvent(owner+"/"+repo, sdkdelivery.ForgeMerged)
if err != nil {
return "", err
}
return process.ModuleStreamSubject(sdkdelivery.ForgeBridgeModule, event), nil
}
// mergeMessages is what forgeMergeAt reads of a stream: the last message on a subject, and the first at or after a
// sequence. Each answers jetstream.ErrMsgNotFound when there is none.
type mergeMessages interface {
last(ctx context.Context, subject string) (uint64, []byte, error)
next(ctx context.Context, subject string, from uint64) (uint64, []byte, error)
}
type streamMerges struct{ s jetstream.Stream }
func (m streamMerges) last(ctx context.Context, subject string) (uint64, []byte, error) {
msg, err := m.s.GetLastMsgForSubject(ctx, subject)
if err != nil {
return 0, nil, err
}
return msg.Sequence, msg.Data, nil
}
func (m streamMerges) next(ctx context.Context, subject string, from uint64) (uint64, []byte, error) {
msg, err := m.s.GetMsg(ctx, from, jetstream.WithGetMsgSubject(subject))
if err != nil {
return 0, nil, err
}
return msg.Sequence, msg.Data, nil
}
// forgeMergeAt finds the merged message of owner/repo at commit: the newest first, which is the merge a take-in
// follows nearly always, then every one the stream keeps, oldest first. A message that does not read as a forge
// message is passed over.
func forgeMergeAt(ctx context.Context, msgs mergeMessages, owner, repo, commit string) (keptMerge, bool, error) {
subject, err := forgeMergeSubject(owner, repo)
if err != nil {
return keptMerge{}, false, err
}
e, err := kv.Get(ctx, forgeMergeKey(owner, repo, commit))
if errors.Is(err, jetstream.ErrKeyNotFound) {
// Read for the four fields a registration needs, not checked whole (sdkdelivery.DecodeForge): a merged message a
// newer bridge writes, or one the SDK's check refuses for a part the controller never reads, still names its pull
// request, head and commit.
match := func(data []byte) (keptMerge, bool) {
var f sdkdelivery.Forge
if json.Unmarshal(data, &f) != nil || f.Kind != sdkdelivery.ForgeMerged || f.Number <= 0 || f.Head == "" ||
!sameCommitID(f.Merge, commit) {
return keptMerge{}, false
}
return keptMerge{SourceMoved: link.SourceMoved{Owner: owner, Repo: repo, Base: f.Branch, Number: f.Number,
HeadSHA: f.Head, Commit: f.Merge}}, true
}
lastSeq, data, err := msgs.last(ctx, subject)
if errors.Is(err, jetstream.ErrMsgNotFound) {
return keptMerge{}, false, nil
}
if err != nil {
return keptMerge{}, false, err
}
var k keptMerge
if err := json.Unmarshal(e.Value(), &k); err != nil {
return keptMerge{}, false, err
if k, ok := match(data); ok {
return k, true, nil
}
return k, true, nil
for from := uint64(1); from < lastSeq; {
seq, data, err := msgs.next(ctx, subject, from)
if errors.Is(err, jetstream.ErrMsgNotFound) || (err == nil && seq >= lastSeq) {
return keptMerge{}, false, nil
}
if err != nil {
return keptMerge{}, false, err
}
if k, ok := match(data); ok {
return k, true, nil
}
from = seq + 1
}
return keptMerge{}, false, nil
}
// keptMerge is a merge as the record keeps it: the forge's announcement.
+2 -10
View File
@@ -99,17 +99,9 @@ func TestAModuleBuiltFromANewRepositoryIsRegisteredOnlyOnTheOperatorsApproveOnTh
return err == nil && kept
})
// The forge's merge of that pull request, as the controller heard it, and the build it asked for.
// The forge bridge's merge of that pull request on its stream, and the build it asked for.
const commit = "c0ffee0123456789"
event, _ := json.Marshal(map[string]any{"owner": "novox", "repo": "mesh-host", "base": "main", "number": 7,
"head_sha": a.merge.HeadSHA, "merge_commit_sha": commit})
var forgeMerge link.SourceMoved
if err := json.Unmarshal(event, &forgeMerge); err != nil {
t.Fatal(err)
}
if err := keepForgeMerge(ctx, forgeMerge); err != nil {
t.Fatal(err)
}
sayForgeMerge(ctx, t, js, "novox/mesh-host", 7, a.merge.HeadSHA, commit)
id := fmt.Sprintf("build-%d", time.Now().Add(time.Hour).UnixNano())
if err := inv.RecordBuildRequest(ctx, inventory.BuildRequest{ID: id, Repository: "novox/mesh-host", Seat: "git",
Path: "modules/sudo", Ref: "main", Commit: commit, For: "merge"}); err != nil {
+2 -10
View File
@@ -100,17 +100,9 @@ func TestANewModuleFromANewRepositoryIsRegisteredOnlyOnTheOperatorsApproveOnTheB
return err == nil && kept
})
// The forge's merge of that pull request, as the controller heard it, and the build it asked for.
// The forge bridge's merge of that pull request on its stream, and the build it asked for.
const commit = "c0ffee0123456789"
event, _ := json.Marshal(map[string]any{"owner": "novox", "repo": "mesh-delivery", "base": "main", "number": 7,
"head_sha": a.merge.HeadSHA, "merge_commit_sha": commit})
var forgeMerge link.SourceMoved
if err := json.Unmarshal(event, &forgeMerge); err != nil {
t.Fatal(err)
}
if err := keepForgeMerge(ctx, forgeMerge); err != nil {
t.Fatal(err)
}
sayForgeMerge(ctx, t, js, "novox/mesh-delivery", 7, a.merge.HeadSHA, commit)
id := fmt.Sprintf("build-%d", time.Now().Add(time.Hour).UnixNano())
if err := inv.RecordBuildRequest(ctx, inventory.BuildRequest{ID: id, Repository: "novox/mesh-delivery", Seat: "git",
Path: "modules/mesh-delivery-view", Ref: "main", Commit: commit, For: "merge"}); err != nil {
@@ -11,7 +11,6 @@ import (
"git.novox.be/novox/mesh-sdk/go/catalogue"
sdkdelivery "git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// ADR 0501 §6 (issue 1851, A5): the live case of 2026-10-07 (novox/hq issue 300), on keys. A merge of the catalogue adds
@@ -33,12 +32,6 @@ func TestAMergeAddingAModuleEndsRegisteredFromItsKeyAndAsksNoBuild(t *testing.T)
watchNewModules(watching, open)
const commit = "3da80a4b00000000"
m := link.SourceMoved{Owner: "novox", Repo: "mesh-catalog", Base: "main", Commit: commit,
Paths: []string{"modules/systemd-resolved/module.json", "modules/systemd-resolved/cmd/main.go"},
ModuleDirs: []string{"modules/systemd-resolved"}, ModuleDirsSaid: true}
if err := (following{open: open}).SourceMoved(ctx, m); err != nil {
t.Fatal(err)
}
// The indexer writes the module's trunk value; the builder builds it, then its key is done.
fp := fpOf('7')
manifest := b.indexed(t, "systemd-resolved", "1", "novox/mesh-catalog", "modules/systemd-resolved", commit, fp)
@@ -21,9 +21,7 @@ import (
// account against the store — no bus to dial, busAddress "" — for mesh-mail's real manifest shape (review
// round 1 finding 1: no own secret `broker` at all, its tools served by the tool runner), and issuing it
// again is the path every bus account takes (`rollout mint`'s KindProcess case, and a re-run of `module
// issue`): it rotates the password, and the old hash no longer matches. Issue 774: every account minted is handed to the
// vault, covered or not, until the grants part's take-over; from it the issue is forwarded to the vault's issue-account,
// and the controller's hash stays as it was.
// issue`): it rotates the password, and the old hash no longer matches.
func TestModuleIssueMintsAndDeliversAProcesssAccount(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
@@ -31,6 +29,7 @@ func TestModuleIssueMintsAndDeliversAProcesssAccount(t *testing.T) {
theVaultIsInstalled(t, open)
m := mailSenderWithProcess()
m.Moved = true // so handIssuedToVault's accountIsCovered is true, and the hand-off is exercised whole
register(t, open, m)
vault := newStandInVault(t)
@@ -85,13 +84,4 @@ func TestModuleIssueMintsAndDeliversAProcesssAccount(t *testing.T) {
if hash1 == hash2 {
t.Fatal("issuing the process's account again did not rotate its password")
}
theGrantsPartTookOver(t, true)
issue()
if hash3, held, err := inv.BusUserHash(ctx, user); err != nil || !held || hash3 != hash2 {
t.Fatalf("the controller minted again after the grants part's take-over (held %v, %v)", held, err)
}
if said := vault.said(); !strings.Contains(said, `"verb":"issue-account"`) || !strings.Contains(said, `"user":"`+user+`"`) {
t.Errorf("the vault was not asked to issue %s after the take-over: %s", user, said)
}
}
+42
View File
@@ -7,6 +7,7 @@ import (
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
)
@@ -72,3 +73,44 @@ func TestAFailedPlacingRaisesApplyFailing(t *testing.T) {
t.Fatalf("a failed placing is not raised as anchor's apply-failing: %+v", got)
}
}
// Sweep 2026-10-30 (issue 774): a failure said again in new words carries a new time, and the condition it raised stays
// raised while it stands; only a machine with no open condition waits out the bound, and a cleared failure clears it.
func TestApplyFailingStaysRaisedWhileAnyFailureStands(t *testing.T) {
now := time.Date(2026, 10, 30, 16, 57, 0, 0, time.UTC)
since := now.Add(-time.Minute)
for name, c := range map[string]struct {
open bool
prov string
doing bool
raised bool
since time.Duration // the client's failing-since, before now; zero when it says none
}{
"a fresh placing failure, nothing open": {false, inventory.ProvFailed, false, false, 0},
"a fresh placing failure, condition open": {true, inventory.ProvFailed, false, true, 0},
"placing applied again, condition open": {true, inventory.ProvApplied, false, false, 0},
"a fresh failed apply, condition open": {true, "", true, true, 0},
"a fresh failed apply, nothing open": {false, "", true, false, 0},
"new words of a failure standing ten minutes, nothing open": {false, inventory.ProvFailed, false, true, 10 * time.Minute},
} {
f := calm(now)
if c.open {
f.applyFailingOpen = applyFailingOpen([]conditions.Condition{{Kind: kindApplyFailing,
Subject: conditions.Subject{Scope: conditions.ScopeMachine, ID: "anchor", Machine: "anchor"}}})
}
if c.prov != "" {
f.provFailed = []inventory.ProvReport{{Node: "anchor", Module: "postgres", Status: c.prov,
Reason: "the vault refused it", At: now.Add(-time.Minute)}}
if c.since > 0 {
f.provFailed[0].FailingSince = now.Add(-c.since)
}
}
if c.doing {
f.doing = []inventory.Doing{{Node: "anchor", Outcome: inventory.OutcomeRefused, Refused: "no", At: since,
Since: &since}}
}
if got := watchApplyFailing(f); (len(got) == 1) != c.raised || len(got) > 1 {
t.Errorf("%s: %d observation(s), raised want %v", name, len(got), c.raised)
}
}
}
@@ -207,13 +207,8 @@ func giveVerifiedSendersBack(ctx context.Context, d *doctor) {
}
// reissueVerifiedSendersOn is reissueVerifiedSenders on the bus: memberships read from the AssignmentsStream and
// published to it. Once the stream is gone (the bus module's tools delete it at the lease write, research 072 row 33)
// the memberships are the grants part's alone, and there is nothing here to issue again.
// published to it.
func reissueVerifiedSendersOn(ctx context.Context, bus link.OverNATS, records broker.Records, where broker.Placements) int {
if _, err := bus.JS.StreamInfo(broker.AssignmentsStream, nats.Context(ctx)); errors.Is(err, nats.ErrStreamNotFound) {
fmt.Printf(" the assignments stream is gone: the grants part issues the memberships, none is given a verified sender here\n")
return 0
}
return reissueVerifiedSenders(records, where,
func(node, module string) ([]byte, uint64, error) {
m, err := bus.JS.GetLastMsg(broker.AssignmentsStream, broker.MembershipSubject(node, module))
@@ -6,8 +6,6 @@ import (
"encoding/json"
"testing"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/testbus"
@@ -19,10 +17,7 @@ func TestNatsAMembershipIssuedWithoutItsVerifiedSenderIsIssuedAgainWhenTheJudgem
t.Fatal(err)
}
t.Cleanup(js.Close)
// The assignments stream is no longer the mesh's (research 072 row 33): it is asserted by nothing, and stays on a
// bus only until the lease write deletes it. Made here as it is left there.
if _, err := js.Context().AddStream(&nats.StreamConfig{Name: broker.AssignmentsStream,
Subjects: []string{"mesh.assignment.*.*"}, MaxMsgsPerSubject: 1, AllowDirect: true}); err != nil {
if err := broker.AssertMeshStreams(js); err != nil {
t.Fatal(err)
}
bus := link.OverNATS{Conn: js.Conn(), JS: js.Context()}
+54
View File
@@ -0,0 +1,54 @@
package main
import (
"context"
"fmt"
"regexp"
"github.com/novox/mesh-controller/internal/conditions"
)
// **A condition whose check no longer exists is cleared** (sweep 2026-10-30, issue 774). A signals row or a self-check
// probe raises and clears its conditions by reconciling what it sees; one removed from the controller reconciles
// nothing again, so what it raised stayed open for good and read as current: S23's failed plans (the plans went with
// ADR 0539) and D14's stalled deliveries, from 2026-10-20 on. Each watchdogs tick clears every open condition whose
// source is a row's or a probe's name that neither the signals table nor the probe registry holds any more, saying so.
// A source of another form (an event, a module's health, a verb) is never judged here: its own word clears it.
// checkSource is the form of a row's (S1) or a probe's (D4, DW, D-root) name.
var checkSource = regexp.MustCompile(`^(S[0-9]+|D[0-9]+|D[A-Z]+|D-[a-z]+)$`)
// liveChecks are the sources this controller still reconciles: its signals rows and its self-check's probes.
func liveChecks() map[string]bool {
out := map[string]bool{}
for _, r := range signalsTable {
out[r.Row] = true
}
for _, p := range probeRegistry {
out[p.ID] = true
}
return out
}
// retiredCheckConditions are the open conditions raised by a row or a probe the controller no longer has.
func retiredCheckConditions(open []conditions.Condition, live map[string]bool) []conditions.Condition {
var out []conditions.Condition
for _, c := range open {
if checkSource.MatchString(c.Source) && !live[c.Source] {
out = append(out, c)
}
}
return out
}
// clearRetiredChecks clears each of them, saying why.
func clearRetiredChecks(ctx context.Context, k *conditions.Keeper, retired []conditions.Condition) []string {
var problems []string
for _, c := range retired {
why := fmt.Sprintf("%s, which raised it, is no longer a check of the controller", c.Source)
if _, err := k.ClearSaying(ctx, c.Key, why, "No longer checked: the check that raised this was removed"); err != nil {
problems = append(problems, c.Key+": "+err.Error())
}
}
return problems
}
@@ -0,0 +1,31 @@
package main
import (
"testing"
"github.com/novox/mesh-controller/internal/conditions"
)
// Sweep 2026-10-30 (issue 774): an open condition raised by a row or a probe the controller no longer has is cleared;
// one raised by a check it still has, or by a source of another form, is left to its own word.
func TestAConditionWhoseCheckIsGoneIsCleared(t *testing.T) {
live := liveChecks()
for source, gone := range map[string]bool{
"S23": true,
"D14": true,
"S21": false,
"S9": false,
"D-root": false,
"DM": false,
"DB": false,
"D4": false,
"health": false,
"provisioner.failing": false,
"done keys": false,
} {
got := retiredCheckConditions([]conditions.Condition{{Key: "k." + source, Source: source}}, live)
if (len(got) == 1) != gone {
t.Errorf("%s: judged retired %v, want %v", source, len(got) == 1, gone)
}
}
}
+5 -1
View File
@@ -284,13 +284,17 @@ func watchLoop(f *signalFacts) []conditions.Observation {
if since.IsZero() {
since = f.started
}
// A message the loop held back waits by the loop's own choice until it is due: deafness is counted from then.
if f.loop.due.After(since) {
since = f.loop.due
}
if f.now.Sub(since) <= loopDeafAfter {
return nil
}
return []conditions.Observation{{Scope: conditions.ScopeCore, ID: "controller", Kind: "controller-deaf",
Token: "deaf", Machine: f.host, Severity: conditions.Urgent,
Summary: fmt.Sprintf("the controller's event loop has taken nothing for %s while its consumers hold %d "+
"message(s): reports, builds and merges are not being acted on", ago(f.now.Sub(since)), f.loop.pending),
"message(s): reports and events are not being acted on", ago(f.now.Sub(since)), f.loop.pending),
Said: fmt.Sprintf("%d pending (%s), last taken %s", f.loop.pending, f.loop.where, since.UTC().Format(time.RFC3339))}}
}
+14 -1
View File
@@ -160,7 +160,7 @@ func switchList(ctx context.Context, inv *inventory.Inventory, keys switchReads)
var out []switchLine
for _, e := range entries {
m := e.Manifest.Module
if e.Provided || e.Source.Repository == "" {
if e.Provided || e.Source.Repository == "" || sdkdelivery.ProvidedByMesh(m) {
continue
}
trunk, err := keys.ModuleAt(ctx, m, "")
@@ -202,3 +202,16 @@ func heldArtifacts(ctx context.Context, inv *inventory.Inventory, module, builtF
// empty-on-error: not an error; the mesh holds no build of that commit on record, which the line says
return nil, nil
}
// builtByTheMesh are the modules of a list that a build of the mesh's makes: without the ones the mesh provides itself
// (sdkdelivery.ProvidedByMesh, the private network), which no fingerprint ever names, so a line saying one has none
// says nothing (sweep 2026-10-30, issue 774: "mesh-wireguard ... no fingerprint of the new form" on every composition).
func builtByTheMesh(modules []string) []string {
var out []string
for _, m := range modules {
if !sdkdelivery.ProvidedByMesh(m) {
out = append(out, m)
}
}
return out
}
-9
View File
@@ -93,15 +93,6 @@ func (f following) Announceable(ctx context.Context) ([]link.Announcement, error
return out, nil
}
// SourceMoved is the forge announcing a merge. It is put into the open batch (novox/hq ADR 0276), which
// becomes one walk when its merge window closes: hearMerge, and cutBatchesHeld in batches.go.
func (f following) SourceMoved(ctx context.Context, m link.SourceMoved) error {
// A merge plans nothing here any more: mesh-delivery merges on the Approve, the indexer writes the fingerprints and
// the builder builds them (ADR 0539 §5, §17). The controller keeps the announcement, for the build of a merge that
// moves a module between repositories to be shown the operator's Approve (ADR 0420 §7).
return keepForgeMerge(ctx, m)
}
// actingOnMerges keeps one merge acted on at a time (novox/hq issue 266).
var actingOnMerges sync.Mutex
@@ -49,12 +49,6 @@ func (v *standInVault) ask(verb string, args map[string]any) (json.RawMessage, e
// The vault asks the desk's prompt and the provider's tool itself, in its own name (issue 723): one step from
// the controller, which carries no key and no sealed value.
switch verb {
case "issue-account":
// The vault is the one issuer once installed (issue 774): the controller forwards, and keeps nothing.
if v.refuse != nil {
return nil, v.refuse
}
return json.Marshal(map[string]any{"minted": true, "account": map[string]any{"user": args["user"], "generation": 1}})
case "secret-ask":
if v.refuse != nil {
return nil, v.refuse
+14 -4
View File
@@ -58,6 +58,11 @@ type signalFacts struct {
// provFailed is every module whose files a machine's provisioner client last said it failed to place (S21,
// novox/hq ADR 0362 §3): that machine's failed apply.
provFailed []inventory.ProvReport
// applyFailingOpen are the machines whose S21 condition is open: kept raised while any failure stands, whatever
// the newest report's time (sweep 2026-10-30).
applyFailingOpen map[string]bool
// retiredChecks are the open conditions raised by a row or a probe the controller no longer has (sweep 2026-10-30).
retiredChecks []conditions.Condition
loop loopFacts
loopErr error
@@ -147,7 +152,9 @@ type machineFacts struct {
func (m machineFacts) asleep() bool { return m.power.Away() }
type loopFacts struct {
took time.Time
took time.Time
// due is when a message the loop held back is handed over again (link.Loop.Due): the loop is not deaf before it.
due time.Time
pending uint64
where string
}
@@ -252,6 +259,7 @@ func (w *watchdogs) see(running context.Context, f *signalFacts) {
problems = append(problems, "S8: "+err.Error())
}
}
problems = append(problems, clearRetiredChecks(running, w.keeper, f.retiredChecks)...)
if err := w.keeper.EndSilences(running); err != nil {
problems = append(problems, err.Error())
}
@@ -315,6 +323,8 @@ func (w *watchdogs) gather(ctx context.Context) *signalFacts {
} else {
f.standings = providerStandings(open)
f.providerWords = providerConditions(open)
f.applyFailingOpen = applyFailingOpen(open)
f.retiredChecks = retiredCheckConditions(open, liveChecks())
}
f.advisories = link.Advisories.Since(now.Add(-advisoryQuiet))
f.lostConsumers, f.advisoriesErr = w.lostConsumers(ctx, f.advisories)
@@ -458,7 +468,7 @@ func (w *watchdogs) gatherLoop() (loopFacts, error) {
if w.js == nil {
return loopFacts{took: link.Loop.Last()}, errors.New("this controller is not on the bus")
}
return readLoop(link.Loop.Last(), loopConsumers(), func(c broker.Consumer) (uint64, error) {
return readLoop(link.Loop.Last(), link.Loop.Due(), loopConsumers(), func(c broker.Consumer) (uint64, error) {
info, err := w.js.Context().ConsumerInfo(c.Stream, c.Name)
if err != nil {
return 0, err
@@ -468,8 +478,8 @@ func (w *watchdogs) gatherLoop() (loopFacts, error) {
}
// readLoop sums what the given consumers hold, as held reads each.
func readLoop(took time.Time, consumers []broker.Consumer, held func(broker.Consumer) (uint64, error)) (loopFacts, error) {
f := loopFacts{took: took}
func readLoop(took, due time.Time, consumers []broker.Consumer, held func(broker.Consumer) (uint64, error)) (loopFacts, error) {
f := loopFacts{took: took, due: due}
var where []string
for _, c := range consumers {
n, err := held(c)
+1 -1
View File
@@ -3,7 +3,7 @@ module github.com/novox/mesh-controller
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030001116-2da0de75c3f0
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030174137-a37181261c24
github.com/jackc/pgx/v5 v5.10.0
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
+2 -2
View File
@@ -1,7 +1,7 @@
git.novox.be/novox/mesh-host v0.0.0-20261018171808-9d9e3a46c4f5 h1:E5stfIeCN6uYgA9TCv+4K8JgaURg3zQppvUgJLaDMP4=
git.novox.be/novox/mesh-host v0.0.0-20261018171808-9d9e3a46c4f5/go.mod h1:4wyoCr6IIIIarEUuROThWDj7DAvJvIidqMjWcpSnOwE=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030001116-2da0de75c3f0 h1:lKU3HjNXGiVbLW2hUgFjU/TMTUYMlNnnmZxG5T7biQA=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030001116-2da0de75c3f0/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030174137-a37181261c24 h1:s0XXZtR7CFPBbnaDygWnE4/Q827iEFt0YfEzujk4Q9g=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030174137-a37181261c24/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
-17
View File
@@ -41,23 +41,6 @@ func ControllerAnnouncement(node string) announce.States {
return s
}
// ReadsAnnounced splits a reader's `reads` (`<owner>.<name>`) by what each owner announced: granted when the owner
// announced the state readable, or announced nothing yet (the transition); refused when the owner announced and did
// not announce that state readable.
func ReadsAnnounced(reads []string, announced map[string]announce.States) (granted, refused []string) {
for _, r := range reads {
owner, name, ok := strings.Cut(r, ".")
s, said := announced[owner]
switch {
case !ok || !said || s.Readable(name):
granted = append(granted, r)
default:
refused = append(refused, r)
}
}
return granted, refused
}
// NotAnnounced are the owners some module reads the state of and that have announced nothing yet, sorted: read from the
// readers' `reads` as before, until they announce.
func NotAnnounced(r Records) []string {
-17
View File
@@ -7,23 +7,6 @@ import (
"git.novox.be/novox/mesh-sdk/go/announce"
)
// D3 (ADR 0495 clause 3, ADR 0513 §3; issue 1790): a reader's `reads` is granted where the owner announced the state
// readable, refused where it announced and did not, and granted as before where the owner announced nothing yet.
func TestAReadIsGrantedFromItsOwnersAnnouncement(t *testing.T) {
announced := map[string]announce.States{
"owner": {Module: "owner", States: []announce.State{{Name: "open", Readable: true}, {Name: "closed"}}},
}
granted, refused := ReadsAnnounced([]string{"owner.open", "owner.closed", "owner.unsaid", "quiet.any"}, announced)
if !slices.Equal(granted, []string{"owner.open", "quiet.any"}) || !slices.Equal(refused, []string{"owner.closed", "owner.unsaid"}) {
t.Fatalf("granted %v, refused %v", granted, refused)
}
r := Records{Announced: announced, Assigned: map[string][]Declared{"one": {{Module: "reader", Reads: granted,
ReadsRefused: refused}}}}
if got := NotAnnounced(r); !slices.Equal(got, []string{"quiet"}) {
t.Fatalf("not yet announced: %v", got)
}
}
// D1 (ADR 0495 clause 2): the controller announces every bucket of its own, and readable exactly the record mirrors (the
// declared packages the pacman module reads among them, ADR 0507 §5), the chain's buckets, the checks running and the
// plans building.
+39
View File
@@ -0,0 +1,39 @@
package broker
import "testing"
// Sweep 2026-10-30 (issue 774), the guard of mesh-sdk #136 in the controller's copy: where a placement names the
// node-engine's manifest, it adds no user, no process and nothing the tool runner carries; a module beside it does.
func TestTheNodeEnginesManifestComposesNoPlacedUser(t *testing.T) {
engine := Declared{Module: EngineModule, Serves: []string{"health"}, State: []Bucket{{Name: "health"}},
Processes: []ProcessPrincipal{{Process: "apply"}}}
beside := Declared{Module: "power", Serves: []string{"power_draw"}}
for name, c := range map[string]struct {
assigned []Declared
module bool
}{
"with the runner": {[]Declared{{Module: RuntimeModule, NoAccount: true}, engine, beside}, false},
"without the runner": {[]Declared{engine, beside}, true},
} {
users, err := Users(Records{Nodes: []string{"anchor"}, Assigned: map[string][]Declared{"anchor": c.assigned}})
if err != nil {
t.Fatalf("%s: %v", name, err)
}
besideSeen := false
for _, u := range users {
if u.Module == EngineModule && (u.Kind == KindModule || u.Kind == KindProcess) {
t.Errorf("%s: the node-engine's manifest composed a %v user", name, u.Kind)
}
besideSeen = besideSeen || u.Module == beside.Module
for _, carried := range u.Carries {
if carried.Module == EngineModule {
t.Errorf("%s: the tool runner carries the node-engine's manifest", name)
}
besideSeen = besideSeen || carried.Module == beside.Module
}
}
if !besideSeen {
t.Errorf("%s: the module beside it is composed nowhere", name)
}
}
}
+1 -7
View File
@@ -21,13 +21,7 @@ func TestAModuleStreamOnARealBus(t *testing.T) {
t.Fatal(err)
}
defer js.Close()
var events Stream
for _, s := range MeshStreams() {
if s.Name == EventsStream {
events = s
}
}
for _, s := range []Stream{events, mailStream.Stream()} {
for _, s := range []Stream{MeshStreams()[3], mailStream.Stream()} {
_ = js.js.DeleteStream(s.Name)
if err := js.EnsureStream(s); err != nil {
t.Fatalf("%s: %v", s.Name, err)
@@ -14,9 +14,7 @@ import (
// queues are left on the bus until a cleanup removes them, and no module may take their names meanwhile. So the catalogue
// reserves every name of the broker's own, and may reserve these retired ones beside them.
func TestTheCataloguesReservedStreamNamesAreTheMeshsOwn(t *testing.T) {
// And ASSIGNMENTS (research 072 row 33): the bus module's tools delete it at the lease write, and the controller
// asserts it no more, so it is never made again.
retired := []string{"CHECKS_OTHER", "CHECKS_READY", "ASSIGNMENTS"}
retired := []string{"CHECKS_OTHER", "CHECKS_READY"}
var own []string
for _, s := range MeshStreams() {
own = append(own, s.Name)
+42 -25
View File
@@ -171,9 +171,14 @@ func MeshStreams() []Stream {
Why: "one declaration per node, always the newest; a node that sees sequence n refuses " +
"n-1 by construction (issue 107)",
},
// The assignments stream (ADR 0160) is no longer the mesh's: a runtime reads its membership from the grants
// part's state memberships (ADR 0571 clause 7), and the bus module's tools delete the stream at the lease write
// (research 072 row 33), so it is not declared here, which would make it again at the controller's next start.
{
Name: AssignmentsStream,
Subjects: []string{"mesh.assignment.*.*"},
Retention: RetentionLastPerSubject,
Direct: true,
Why: "one membership per assignment, always the newest: what the mesh issued this module " +
"on this machine to serve and to reach (ADR 0160); read directly by the runtime it is for",
},
{
Name: EventsStream,
// A seat's own events ride here too: they are 1:many like any event, and the
@@ -334,13 +339,36 @@ func Overlaps() []string {
// ack subject is derived from (nats.go: `$JS.ACK.<stream>.controller.>`).
const ControllerName = "controller"
// MergesConsumerName is the controller's durable consumer of the forge's merged event on EVENTS, apart from the
// events consumer (novox/hq ADR 0407 decision 3, issue 973). A merge is never dropped, so this consumer has no
// max-deliver, which would send a merge held five times to DEAD_LETTERS to wait for a person, and no healer
// resets it, which would drop the merges it holds. A consumer of its own because both of those are properties of
// a consumer, not of a subject.
// MergesConsumerName was the controller's durable consumer of the forge's merged event on EVENTS (novox/hq ADR 0407
// decision 3, issue 973), with no max-deliver and one message at a time. **Retired** (sweep 2026-10-30, issue 774):
// the forge bridge's stream FORGE is the record of the merges, and the controller reads it where it needs a merge. On
// 2026-10-29 one announcement too large for the controller's bucket of merges was held on it for over a day, with
// 163 merges queued behind it, and the deaf watchdog read the queue as a deaf controller. RetiredConsumers removes it.
const MergesConsumerName = "controller-merges"
// RetiredConsumers are durable consumers the controller once held and holds no more: each assertion removes them,
// so a retired consumer cannot keep a queue nobody takes from (research 072, 03-domains: the controller shrinks).
var RetiredConsumers = []Consumer{{Name: MergesConsumerName, Stream: EventsStream}}
// Remover is the part of a JetStream connection that removes a retired consumer.
type Remover interface {
RemoveConsumer(stream, name string) error
}
// removeRetired removes every retired consumer, when the connection can.
func removeRetired(e Ensurer) error {
r, ok := e.(Remover)
if !ok {
return nil
}
for _, c := range RetiredConsumers {
if err := r.RemoveConsumer(c.Stream, c.Name); err != nil {
return fmt.Errorf("removing the retired consumer %s on %s: %w", c.Name, c.Stream, err)
}
}
return nil
}
// ControllerSeat is the role the control plane holds, and ControllerStates are the facts it states
// under it (novox/hq ADR 0134).
//
@@ -508,23 +536,6 @@ func ControllerConsumers() []Consumer {
Why: "the controller is the single consumer of what nodes say; explicit ack and no " +
"max-deliver, because the store window's bound is the controller's own",
},
// The forge's merges (novox/hq ADR 0407 decision 3, issue 973): never dropped, so no max-deliver and no
// reset; a merge the controller cannot record yet is held with a growing delay until it can. **Asserted before
// the events consumer**, which stops filtering merges: the order of this list is the order of assertion.
{
Name: MergesConsumerName,
Stream: "EVENTS",
// Within what the controller may subscribe already (`_DELIVER.controller.>`).
DeliverSubject: "_DELIVER." + ControllerName + ".merges",
Filters: []string{ControllerFollows[3]},
Push: true,
AckWaitSeconds: 30,
MaxAckPending: 1,
FromNow: true,
TakesOverFrom: ControllerName,
Why: "the forge's merges, one at a time; no max-deliver and never reset, because a merge is never " +
"dropped and no person releases one",
},
{
Name: ControllerName,
Stream: "EVENTS",
@@ -576,6 +587,9 @@ func NoticesConsumer() Consumer {
// one on DEAD_LETTER_NOTICES, which the keeper of dead letters asserts and retries by itself, so a fault
// there never stops the controller serving (novox/hq issue 330).
func AssertServingConsumers(e Ensurer) error {
if err := removeRetired(e); err != nil {
return err
}
for _, c := range MeshConsumers() {
if c.Stream == DeadLetterNoticesStream {
continue
@@ -598,6 +612,9 @@ type Ensurer interface {
// After the streams, necessarily: a consumer on a stream that does not exist is refused, and the
// refusal names the stream rather than the order.
func AssertMeshConsumers(e Ensurer) error {
if err := removeRetired(e); err != nil {
return err
}
for _, c := range MeshConsumers() {
if err := e.EnsureConsumer(c); err != nil {
return fmt.Errorf("asserting consumer %s on %s: %w", c.Name, c.Stream, err)
+4 -5
View File
@@ -123,11 +123,10 @@ func subjectMatches(filter, subject string) bool {
// Each relationship's retention is the thing that makes it what it is (design 29 §4).
func TestEachStreamCarriesTheRetentionItsShapeNeeds(t *testing.T) {
want := map[string]Retention{
"CONTROL": RetentionWorkQueue,
"NODES": RetentionLastPerSubject,
"EVENTS": RetentionLimits,
// No ASSIGNMENTS: the bus module's tools delete it at the lease write and nothing makes it again (research 072
// row 33); a runtime reads its membership from the grants part's state.
"CONTROL": RetentionWorkQueue,
"NODES": RetentionLastPerSubject,
"EVENTS": RetentionLimits,
"ASSIGNMENTS": RetentionLastPerSubject,
// What a consumer gave up on (novox/hq issue 330): the server's notice taken once, the message
// kept until somebody acts.
"DEAD_LETTER_NOTICES": RetentionWorkQueue,
+18 -1
View File
@@ -181,6 +181,11 @@ func Users(r Records) ([]Principal, error) {
}
}
for _, d := range r.Assigned[node] {
// **The node-engine's manifest is no placed module's** (novox/hq ADR 0563 §2, §3; sweep 2026-10-30, issue
// 774): the guard mesh-sdk #136 put in buscompose, kept here while the controller composes with its own copy.
if !Composed(d) {
continue
}
// **A process account is its own, delivered in its own secret** (broker-<process id>, novox/hq
// ADR 0436 §1, §2), never the module's `broker` — so it is composed whether or not the module
// itself has an account, and is never carried by the machine's runtime, which runs as the
@@ -227,7 +232,7 @@ func Users(r Records) ([]Principal, error) {
if runtimeHere {
var carried []Declared
for _, d := range r.Assigned[node] {
if d.RunsAs == "" && runnerServes(d) {
if CarriedByRunner(d) {
// Never a restricted stream for what the tool runner carries (novox/hq ADR 0376 §1).
d.Hears = ComposedHears(d)
carried = append(carried, d)
@@ -454,3 +459,15 @@ func declaredStreamsOf(r Records) []string {
sort.Strings(out)
return out
}
// EngineModule is the node-engine's manifest's module name (mesh-sdk buscompose.EngineModule).
const EngineModule = "mesh-host"
// Composed says a placed module is composed as one: its own user, its processes and what the runner carries of it. The
// node-engine's manifest is not, though a placement may name it: each node's engine speaks as the node's account, which
// alone writes the engine's state (ADR 0563 §2, §3); granted to another account it is a second writer, and the writers
// table refuses the whole list (issue 774: mesh-host_acts, then mesh-host_health, 2026-10-30). mesh-sdk buscompose.Composed.
func Composed(d Declared) bool { return d.Module != EngineModule }
// CarriedByRunner says the machine's tool runner carries this placed module (mesh-sdk buscompose.CarriedByRunner).
func CarriedByRunner(d Declared) bool { return Composed(d) && d.RunsAs == "" && runnerServes(d) }
+34 -10
View File
@@ -6,7 +6,9 @@ import (
"git.novox.be/novox/mesh-sdk/go/announce"
"slices"
"strings"
"sync"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/broker"
)
@@ -119,22 +121,18 @@ func (i *Inventory) BusRecords(ctx context.Context) (broker.Records, error) {
// announcement does not make readable is granted nothing, and said; an owner that announced nothing yet is read as
// before. Announcements that cannot be read are said, and every read is granted as before: the bus's users are
// composed either way.
//
// **Judged for its reader, by the SDK's judgement** (sweep 2026-10-30, issue 774): mesh-sdk buscompose's
// ReadsAnnouncedBy knows the named readers (ADR 0547 clause 9, ADR 0561 §3), so the lease is granted to the parts
// named for it and refused to every other. The controller's own reader-less copy refused every named reader: 2536
// refused reads of mesh-controller.lease on 2026-10-30, each said on every composition.
if Announcements != nil {
announced, err := Announcements(ctx)
if err != nil {
fmt.Printf("the states the modules announced cannot be read, so every read is granted from `reads` as before: %v\n", err)
} else {
out.Announced = announced
for node, ds := range out.Assigned {
for k := range ds {
granted, refused := broker.ReadsAnnounced(ds[k].Reads, announced)
ds[k].Reads, ds[k].ReadsRefused = granted, refused
for _, r := range refused {
fmt.Printf("%s on %s reads %s, which its owner does not announce readable: not granted\n",
ds[k].Module, node, r)
}
}
}
judgeReads(out.Assigned, announced, sayRefusedOnce)
}
}
@@ -615,3 +613,29 @@ func hearsOf(m catalogue.Manifest, declared map[string]catalogue.Manifest, node
}
return out, touches, onEvents
}
// refusalsSaid are the refused reads already said by this process, by `<module>@<node> <read>`: a refusal is said once,
// when first composed, not on every composition (about 420 times a read on 2026-10-30). `status` names them all.
var refusalsSaid sync.Map
// sayRefusedOnce says a refused read the first time this process composes it.
func sayRefusedOnce(module, node, read string) {
if _, said := refusalsSaid.LoadOrStore(module+"@"+node+" "+read, true); said {
return
}
fmt.Printf("%s on %s reads %s, which its owner does not make readable by it: not granted\n", module, node, read)
}
// judgeReads splits each assigned module's reads into granted and refused, for that module, by what the owners announced
// and the states' named readers (buscompose.ReadsAnnouncedBy), and says each refusal.
func judgeReads(assigned map[string][]broker.Declared, announced map[string]announce.States, say func(module, node, read string)) {
for node, ds := range assigned {
for k := range ds {
granted, refused := buscompose.ReadsAnnouncedBy(ds[k].Module, ds[k].Reads, announced)
ds[k].Reads, ds[k].ReadsRefused = granted, refused
for _, r := range refused {
say(ds[k].Module, node, r)
}
}
}
}
+44
View File
@@ -0,0 +1,44 @@
package inventory
import (
"slices"
"testing"
"git.novox.be/novox/mesh-sdk/go/announce"
"github.com/novox/mesh-controller/internal/broker"
)
// Sweep 2026-10-30 (issue 774): a read is judged for its reader. The controller's lease, which its announcement marks
// readable by nobody, is granted to the parts named for it (ADR 0561 §3) and refused to every other (mesh-config named since mesh-sdk ef1ec1c9); a state its owner
// announced readable is granted; one it keeps and did not is refused; an owner that announced nothing is read as before.
// Each refusal is said, and a refusal already said is not said again.
func TestAReadIsJudgedForItsReader(t *testing.T) {
announced := map[string]announce.States{
"mesh-controller": broker.ControllerAnnouncement("novox"),
"owner": {Module: "owner", States: []announce.State{{Name: "open", Readable: true}, {Name: "closed"}}},
}
for name, c := range map[string]struct {
reader, read string
granted bool
}{
"the lease, by the vault": {"mesh-vault", "mesh-controller.lease", true},
"the lease, by the grants part": {"mesh-grants", "mesh-controller.lease", true},
"the lease, by mesh-config": {"mesh-config", "mesh-controller.lease", true},
"the lease, by a module not named": {"restic", "mesh-controller.lease", false},
"a readable state": {"reader", "owner.open", true},
"a state kept and not readable": {"reader", "owner.closed", false},
"an owner that announced nothing": {"reader", "quiet.any", true},
"a mirror the controller marks open": {"pacman", "mesh-controller.declared-packages", true},
} {
assigned := map[string][]broker.Declared{"novox": {{Module: c.reader, Reads: []string{c.read}}}}
var said []string
judgeReads(assigned, announced, func(module, node, read string) { said = append(said, read) })
d := assigned["novox"][0]
if slices.Contains(d.Reads, c.read) != c.granted || slices.Contains(d.ReadsRefused, c.read) == c.granted {
t.Errorf("%s: granted %v, refused %v", name, d.Reads, d.ReadsRefused)
}
if (len(said) == 1) == c.granted {
t.Errorf("%s: said %v", name, said)
}
}
}
+11
View File
@@ -26,6 +26,17 @@ type ProvReport struct {
WaitsFor []string `json:"waits-for,omitempty"`
Templates string `json:"templates,omitempty"`
At time.Time `json:"at"`
// FailingSince is, on a failed report, when the module's standing failure began, kept by the client while only its
// words change (mesh-catalog 3aae6c08); zero from an older client.
FailingSince time.Time `json:"failing-since,omitzero"`
}
// Since is when a failed report's failure began: FailingSince when the client says it, else At.
func (r ProvReport) Since() time.Time {
if !r.FailingSince.IsZero() {
return r.FailingSince
}
return r.At
}
// The words a client's report says.
+98 -36
View File
@@ -127,9 +127,13 @@ type CallLog struct {
// keeper keeps every call beyond this process, when Durably was given one; writes go through
// one goroutine, in order, so a call's last state is the one kept.
keeper CallKeeper
holder string
writes chan Call
keeper CallKeeper
holder string
// queued are the calls' states waiting for the keeper, one per call, in the order each call was first queued, under
// qmu (keep is called with mu held); woken is told when one is queued (callQueue).
qmu sync.Mutex
queued callQueue
woken chan struct{}
logger *log.Logger
lost int // writes the keeper could not take, said once each
keepErr error
@@ -165,8 +169,8 @@ const keepTries = 5
func (l *CallLog) Durably(ctx context.Context, keeper CallKeeper, holder string, logger *log.Logger) error {
l.mu.Lock()
l.keeper, l.holder, l.logger = keeper, holder, logger
if l.writes == nil {
l.writes = make(chan Call, 256)
if l.woken == nil {
l.woken = make(chan struct{}, 1)
go l.keepWrites()
}
l.mu.Unlock()
@@ -203,51 +207,109 @@ func orSomebody(s string) string {
return s
}
// callQueueBound is how many calls may wait for the keeper at once: far beyond a burst of reads (about 200 calls in one
// second on 2026-10-30), and a bound, so a keeper that takes nothing for long is said rather than grown into.
const callQueueBound = 8192
// callQueue holds the latest state of each call waiting for the keeper. **One state per call** (sweep 2026-10-30, issue
// 774): a call's later state replaces its earlier one still waiting, since only the last is kept anyway; a burst of
// read calls queued begin and finish each, and a channel of 256 dropped 30 of them in one second.
type callQueue struct {
order []string
byID map[string]Call
}
// put queues c, replacing a waiting state of the same call; false when the queue is full.
func (q *callQueue) put(c Call) bool {
if q.byID == nil {
q.byID = map[string]Call{}
}
if _, waiting := q.byID[c.ID]; waiting {
q.byID[c.ID] = c
return true
}
if len(q.order) >= callQueueBound {
return false
}
q.order = append(q.order, c.ID)
q.byID[c.ID] = c
return true
}
// take is every waiting state, in order, and empties the queue.
func (q *callQueue) take() []Call {
out := make([]Call, 0, len(q.order))
for _, id := range q.order {
out = append(out, q.byID[id])
}
q.order, q.byID = nil, nil
return out
}
// keep queues one call's state for the keeper. Never blocks a call: a queue that is full is a keeper
// that is not taking writes, and that is said rather than waited on.
func (l *CallLog) keep(c Call) {
if l.writes == nil {
if l.woken == nil {
return
}
l.qmu.Lock()
queued := l.queued.put(c)
if !queued {
l.lost++
}
lost := l.lost
l.qmu.Unlock()
if !queued {
if l.logger != nil {
l.logger.Printf("%s (%s.%s) is kept in memory only: %d calls wait for the bus to take their records "+
"(%d not kept)", c.ID, c.Seat, c.Verb, callQueueBound, lost)
}
return
}
select {
case l.writes <- c:
case l.woken <- struct{}{}:
default:
l.lost++
if l.logger != nil {
l.logger.Printf("%s (%s.%s) is kept in memory only: the bus is not taking calls' records (%d not kept)",
c.ID, c.Seat, c.Verb, l.lost)
}
}
}
func (l *CallLog) keepWrites() {
for c := range l.writes {
l.mu.Lock()
gate := l.epoch
l.mu.Unlock()
if gate != nil {
if _, err := gate(); err != nil {
if l.logger != nil {
l.logger.Printf("%s (%s.%s, %s) is not kept on the bus: %v — the controller holding the lease "+
"marks it abandoned", c.ID, c.Seat, c.Verb, c.State, err)
}
continue
for range l.woken {
l.qmu.Lock()
waiting := l.queued.take()
l.qmu.Unlock()
for _, c := range waiting {
l.keepOne(c)
}
}
}
// keepOne writes one call's state through the keeper, under the lease.
func (l *CallLog) keepOne(c Call) {
l.mu.Lock()
gate := l.epoch
l.mu.Unlock()
if gate != nil {
if _, err := gate(); err != nil {
if l.logger != nil {
l.logger.Printf("%s (%s.%s, %s) is not kept on the bus: %v — the controller holding the lease "+
"marks it abandoned", c.ID, c.Seat, c.Verb, c.State, err)
}
return
}
var err error
for try := 0; try < keepTries; try++ {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
err = l.keeper.Keep(ctx, c)
cancel()
if err == nil {
break
}
time.Sleep(time.Duration(try+1) * time.Second)
}
if err != nil && l.logger != nil {
l.logger.Printf("%s (%s.%s, %s) could not be kept on the bus after %d tries: %v — `calls` "+
"answers it from memory until this controller stops", c.ID, c.Seat, c.Verb, c.State, keepTries, err)
}
var err error
for try := 0; try < keepTries; try++ {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
err = l.keeper.Keep(ctx, c)
cancel()
if err == nil {
break
}
time.Sleep(time.Duration(try+1) * time.Second)
}
if err != nil && l.logger != nil {
l.logger.Printf("%s (%s.%s, %s) could not be kept on the bus after %d tries: %v — `calls` "+
"answers it from memory until this controller stops", c.ID, c.Seat, c.Verb, c.State, keepTries, err)
}
}
+32 -4
View File
@@ -254,8 +254,7 @@ func TestAHandoverRefusalIsMarkedRetryable(t *testing.T) {
// who may call the seat, and a token is the one-time right to become a machine of the mesh.
func TestAJoinTokenIsShownToItsCallerAndNotKept(t *testing.T) {
l, a := NewCallLog(), newAnswers(t)
writes := make(chan Call, 4)
l.writes = writes
l.woken = make(chan struct{}, 1) // queued for the keeper, and nothing takes them: read below
l.serveCall("mesh-controller", "token", json.RawMessage(`{"new":"laptop"}`), "_INBOX.x.1",
func(context.Context, json.RawMessage) (any, error) { return "token for laptop: s3cret-join", nil },
a.respond, nil)
@@ -266,8 +265,7 @@ func TestAJoinTokenIsShownToItsCallerAndNotKept(t *testing.T) {
if len(recent) != 1 || strings.Contains(string(recent[0].Answer), "s3cret-join") {
t.Fatalf("the token was kept in memory: %+v", recent)
}
close(writes)
for c := range writes {
for _, c := range l.queued.take() {
if strings.Contains(string(c.Answer), "s3cret-join") {
t.Fatalf("the token was sent to be kept on the bus: %s", c.Answer)
}
@@ -276,3 +274,33 @@ func TestAJoinTokenIsShownToItsCallerAndNotKept(t *testing.T) {
// A verb whose handler panics answers an error, and the controller serves the next call (review of
// novox/hq issue 327: read verbs are answered in the serving process now).
// Sweep 2026-10-30 (issue 774): a burst of calls never loses a record while the keeper is slow: each call waits once,
// with its latest state, and every call of a burst far over the old queue of 256 is kept with its last state.
func TestABurstOfCallsIsKeptWithEachCallsLastState(t *testing.T) {
var q callQueue
for i := range 2000 {
id := fmt.Sprintf("call-%d", i)
if !q.put(Call{ID: id, State: CallRunning}) || !q.put(Call{ID: id, State: CallAnswered}) {
t.Fatalf("%s was refused by a queue of %d", id, callQueueBound)
}
}
got := q.take()
if len(got) != 2000 {
t.Fatalf("%d calls wait, want 2000", len(got))
}
for i, c := range got {
if c.ID != fmt.Sprintf("call-%d", i) || c.State != CallAnswered {
t.Fatalf("call %d waits as %+v", i, c)
}
}
if len(q.take()) != 0 {
t.Fatal("taken calls wait again")
}
for i := range callQueueBound {
q.put(Call{ID: fmt.Sprint(i)})
}
if q.put(Call{ID: "one too many"}) {
t.Fatal("the queue grew past its bound")
}
}
+2 -3
View File
@@ -30,9 +30,8 @@ var Contracts = map[string]Contract{
KindBuilt: {Unordered: "a build's outcome on the retired build seat's subjects: dropped, nothing takes a build in " +
"from the bus any more (R4g, ADR 0501 §1); the subjects stay followed while the installer's carried user list " +
"names them"},
KindSourceMoved: {Ordered: "by the merge's commit against what was built from it: a merge already acted on " +
"or older than the last look is history, not a second plan (issues 250, 266)",
Tests: []string{"TestAMergeOlderThanTheLastLookIsHistory", "TestAMergeMatchesTheSourcesBuiltFromIt"}},
KindSourceMoved: {Unordered: "the forge's merge on EVENTS: no longer followed, dropped if one arrives; the forge " +
"bridge's stream FORGE is the record of the merges, read at a moved module's registration (sweep 2026-10-30)"},
KindToolsHeartbeat: {Unordered: "a word that the node tools are there, as a machine's heartbeat"},
KindHealth: {Ordered: "by the time the node-engine looked (`at`, on the machine's clock): a statement older " +
"than the one kept for that machine is refused, so an event arriving after a newer report does not undo it " +
+2 -4
View File
@@ -125,8 +125,7 @@ func askCLI(t *testing.T, conn *nats.Conn, node string, asked CLIAsked) (CLIAnsw
// bus, where `calls` answers anyone who may call the seat.
func TestAMeshCLIRecordKeepsNeitherItsLineNorItsAnswerOnTheBus(t *testing.T) {
l, a := NewCallLog(), newAnswers(t)
writes := make(chan Call, 4)
l.writes = writes
l.woken = make(chan struct{}, 1) // queued for the keeper, and nothing takes them: read below
asked, _ := json.Marshal(CLIAsked{Line: []string{"settings", "set", "x", `{"password":"s3cret"}`}, Account: "op"})
l.serveCall(CLISeat, "laptop", asked, "_INBOX.node.laptop.abcdefghijklmnopqrstuv",
func(context.Context, json.RawMessage) (any, error) {
@@ -134,8 +133,7 @@ func TestAMeshCLIRecordKeepsNeitherItsLineNorItsAnswerOnTheBus(t *testing.T) {
},
a.respond, nil)
_ = a.only()
close(writes)
for c := range writes {
for _, c := range l.queued.take() {
if carries(c.Args, "s3cret") || carries(c.Answer, "s3cret-join") {
t.Fatalf("the bus was sent %s / %s", c.Args, c.Answer)
}
+3
View File
@@ -52,6 +52,9 @@ type MoveSecretAnswer struct {
Secret string `json:"secret,omitempty"`
Generation int `json:"generation,omitempty"`
Unchanged bool `json:"unchanged,omitempty"`
// Superseded says the vault holds another value of the secret, which is the live one, and this machine's older value
// was not moved and is not asked again (mesh-host b9c24a7, sweep 2026-10-30).
Superseded bool `json:"superseded,omitempty"`
}
// MoveSecretWithin is how long the controller waits for one move: the engine asks the vault twice.
+3 -2
View File
@@ -30,8 +30,9 @@ func TestTheMovesAskAndAnswerKeepTheirFieldNames(t *testing.T) {
"provision sealed sealed_to" {
t.Fatalf("the ask's fields are %q", got)
}
answer := MoveSecretAnswer{Said: "s", Refused: "r", Secret: "own/a/b/c", Generation: 1, Unchanged: true}
if got := keys(answer); got != "generation refused said secret unchanged" {
answer := MoveSecretAnswer{Said: "s", Refused: "r", Secret: "own/a/b/c", Generation: 1, Unchanged: true,
Superseded: true}
if got := keys(answer); got != "generation refused said secret superseded unchanged" {
t.Fatalf("the answer's fields are %q", got)
}
if broker.AskMoveSecretSubject("laptop") != "mesh.node.laptop.ask.move-secret" ||
+6 -20
View File
@@ -62,7 +62,7 @@ func Nats(js *broker.JetStream) Inbound {
// whatever was asked for — and not at all when nothing was.
func (n *natsInbound) Also(kind string) error {
switch kind {
case KindModuleMoved, KindCatchUp, KindSourceMoved, KindProvisioner, KindPullUpdated, KindDecided,
case KindModuleMoved, KindCatchUp, KindProvisioner, KindPullUpdated, KindDecided,
KindDeliveryMerged:
n.follows[kind] = true
return nil
@@ -145,24 +145,6 @@ func (n *natsInbound) Receive(ctx context.Context, act func(context.Context, Con
defer func() { _ = followed.Unsubscribe() }()
}
// And the forge's merges, off a consumer of their own: no max-deliver and no reset, so a merge held for the
// plans waits for them however long (novox/hq ADR 0407 decision 3). Bound to the same channel: one loop.
if n.follows[KindSourceMoved] {
if events == nil {
events = make(chan *nats.Msg, Prefetch)
}
merges, err := standingBy(ctx, log.Default(), "EVENTS", func() (*nats.Subscription, error) {
return js.ChanSubscribe(MergedSubject, events, nats.Bind("EVENTS", broker.MergesConsumerName))
})
if err != nil {
return fmt.Errorf("subscribing to the forge's merges: %w", err)
}
if merges == nil {
return nil
}
defer func() { _ = merges.Unsubscribe() }()
}
// A connection that dropped is said, not discovered. A controller whose bus connection is gone
// is a mesh where nothing can be told anything.
gone := make(chan error, 1)
@@ -423,7 +405,11 @@ func (m *natsControl) Hold(after time.Duration) error {
if _, already := m.on.since[m.seq]; !already {
m.on.since[m.seq] = time.Now()
}
return m.msg.NakWithDelay(after)
if err := m.msg.NakWithDelay(after); err != nil {
return err
}
Loop.HeldUntil(time.Now().Add(after))
return nil
}
func (m *natsControl) forget() {
@@ -394,35 +394,3 @@ func TestNatsAReportThatIsNotNewsIsNotStated(t *testing.T) {
case <-time.After(3 * time.Second):
}
}
func TestNatsAMergeHeldMoreThanFiveTimesIsStillHandedOverAndRecorded(t *testing.T) {
first, most := MergeHeldFirst, MergeHeldMost
MergeHeldFirst, MergeHeldMost = 5*time.Millisecond, 40*time.Millisecond
t.Cleanup(func() { MergeHeldFirst, MergeHeldMost = first, most })
js := aBus(t)
up := &manyTimesBusy{busy: 8}
s := &Server{inbound: Nats(js), bus: OverNATS{Conn: js.Conn(), JS: js.Context()}, log: quiet()}
if err := s.Follows(up); err != nil {
t.Fatal(err)
}
ctx, stop := context.WithCancel(context.Background())
defer stop()
go func() { _ = s.Serve(ctx) }()
eventually(t, "the merges consumer being made", func() bool {
_, err := js.Context().ConsumerInfo("EVENTS", broker.MergesConsumerName)
return err == nil
})
body, _ := json.Marshal(SourceMoved{Owner: "novox", Repo: "one", Base: "main", Commit: "abcdef0123"})
if _, err := js.Context().Publish(MergedSubject, body); err != nil {
t.Fatal(err)
}
eventually(t, "the ninth hand-over recording the merge", func() bool { return up.called() == 9 })
eventually(t, "the merge being acknowledged", func() bool {
info, err := js.Context().ConsumerInfo("EVENTS", broker.MergesConsumerName)
return err == nil && info.NumAckPending == 0 && info.NumPending == 0
})
if info, err := js.Context().StreamInfo(broker.DeadLettersStream); err == nil && info.State.Msgs != 0 {
t.Fatalf("%d message(s) in dead letters: a merge was given up on", info.State.Msgs)
}
}
-44
View File
@@ -2,7 +2,6 @@ package link
import (
"context"
"errors"
"io"
"log"
"sync"
@@ -199,46 +198,3 @@ func (s slowly) Heard(context.Context, Report) (bool, error) { s.work(); return
type sameAgain struct{}
func (sameAgain) Heard(context.Context, Report) (bool, error) { return false, nil }
// manyTimesBusy is an upgrader whose plans are busy for its first merges, as another controller's hold is.
type manyTimesBusy struct {
toldAbout
busy int
calls int
}
func (u *manyTimesBusy) SourceMoved(context.Context, SourceMoved) error {
u.mu.Lock()
defer u.mu.Unlock()
u.calls++
if u.calls <= u.busy {
return errors.Join(ErrTryAgain, errors.New("another controller is working the plans"))
}
return nil
}
func (u *manyTimesBusy) called() int {
u.mu.Lock()
defer u.mu.Unlock()
return u.calls
}
// **A merge held more than five times is still handed over and recorded** (issue 973, novox/hq ADR 0407 decision
// 3): the merges have a durable consumer of their own with no max-deliver, so they never go to DEAD_LETTERS to wait
// for a person; the delay grows, and is capped.
// The delay a held merge waits grows from the first to the most and stays there.
func TestAHeldMergesDelayGrowsAndIsCapped(t *testing.T) {
var got []time.Duration
for _, held := range []time.Duration{0, 4 * time.Second, 10 * time.Second, 25 * time.Second, 70 * time.Second,
200 * time.Second, 20 * time.Minute} {
got = append(got, MergeRedeliverAfter(held))
}
want := []time.Duration{5 * time.Second, 5 * time.Second, 10 * time.Second, 25 * time.Second, 70 * time.Second,
200 * time.Second, 5 * time.Minute}
for i := range want {
if got[i] != want[i] {
t.Errorf("held %d: waits %v; want %v", i, got[i], want[i])
}
}
}
+3 -61
View File
@@ -49,8 +49,6 @@ type Upgrader interface {
// stop every upgrade behind it — except the store unreachable for the moment, which is asked
// again for a bounded time (novox/hq issue 083).
Upgraded(ctx context.Context, u Upgraded) error
// SourceMoved is a merge on the forge: build what that source produces, bases first.
SourceMoved(ctx context.Context, m SourceMoved) error
}
// Checker is what the controller does when the forge says a pull request's head moved: ask for it to be
@@ -105,9 +103,6 @@ const GiveUpAfter = 2 * time.Minute
// Follows says what to do about upgrades, and asks for them to be delivered.
func (s *Server) Follows(u Upgrader) error {
if err := s.inbound.Also(KindSourceMoved); err != nil {
return err
}
if err := s.inbound.Also(KindModuleMoved); err != nil {
return err
}
@@ -225,7 +220,9 @@ func (s *Server) act(ctx context.Context, m Control) {
case KindModuleMoved:
s.moved(ctx, m)
case KindSourceMoved:
s.sourceMoved(ctx, m)
// The forge's merges are no longer followed (sweep 2026-10-30): the forge bridge's stream is their record, read
// where a merge is needed (keptForgeMerge). One that arrives on an old consumer is let go of.
_ = m.Drop()
case KindCatchUp:
s.catchingUp(ctx, m)
case KindProvisioner:
@@ -521,61 +518,6 @@ func short(commit string) string {
return commit
}
// MergeHeldFirst is how long a merge the controller could not record yet waits before the bus hands it over
// again, and MergeHeldMost the longest it ever waits: the delay grows with how long the merge has been held, from
// the first to the most (novox/hq ADR 0407 decision 3). Variables so a test need not wait.
var (
MergeHeldFirst = 5 * time.Second
MergeHeldMost = 5 * time.Minute
)
// MergeRedeliverAfter is the delay for a merge held for the time given: that time, so the delay about doubles
// with each hold, between MergeHeldFirst and MergeHeldMost.
func MergeRedeliverAfter(heldFor time.Duration) time.Duration {
return min(max(heldFor, MergeHeldFirst), MergeHeldMost)
}
// sourceMoved acts on the forge's announcement of a merge. **A merge is never dropped** (issue 921, novox/hq ADR
// 0407 decision 3): one the controller could not record for now (the plans held by another controller, the store
// away: ErrTryAgain) is held and handed over again until it is recorded; the controller giving up on it would
// lose a change for good. Anything else is an answer and is taken: a build that fails is reported by the build
// itself, and re-delivering the merge would only re-fail it.
func (s *Server) sourceMoved(ctx context.Context, m Control) {
var moved SourceMoved
if err := json.Unmarshal(m.Body(), &moved); err != nil {
s.log.Printf("a merge announcement could not be read: %v", err)
_ = m.Took()
return
}
if moved.Commit == "" || moved.Repo == "" {
s.log.Printf("a merge announcement named no repository or no commit; ignored")
_ = m.Took()
return
}
err := s.upgrader.SourceMoved(ctx, moved)
// Left unsettled when the controller is ending, whatever error that gave (the plans' hold returns a bare
// context.Canceled): the bus hands it to whatever consumes next, and taking it would lose the merge.
if err != nil && ctx.Err() != nil {
return
}
if err != nil && errors.Is(err, ErrTryAgain) {
// Also left unsettled when the bus will not hold it.
delay := MergeRedeliverAfter(m.HeldFor())
if held := m.Hold(delay); held != nil {
s.log.Printf("%s/%s moved to %.8s and could not be held (%v): %v; left for the bus to hand over again",
moved.Owner, moved.Repo, moved.Commit, held, err)
return
}
s.log.Printf("%s/%s moved to %.8s and is held: %v; handed over again in %s", moved.Owner, moved.Repo,
moved.Commit, err, delay)
return
}
if err != nil {
s.log.Printf("%s/%s moved to %.8s and the mesh could not act on it: %v", moved.Owner, moved.Repo, moved.Commit, err)
}
_ = m.Took()
}
// pullUpdated asks for a pull request's merge check. Taken whatever happens: a check that could not be
// asked is said here, and the next push to the pull request asks again.
func (s *Server) pullUpdated(ctx context.Context, m Control) {
-60
View File
@@ -1,60 +0,0 @@
package link
import (
"context"
"encoding/json"
"errors"
"io"
"log"
"testing"
"time"
)
// heldControl is a merge announcement that records how it was settled.
type heldControl struct {
body []byte
took, held, drops int
}
func (c *heldControl) Kind() string { return "source-moved" }
func (c *heldControl) Body() []byte { return c.body }
func (c *heldControl) Subject() string { return "" }
func (c *heldControl) Redelivered() bool { return false }
func (c *heldControl) HeldFor() time.Duration { return 0 }
func (c *heldControl) Answer(context.Context, []byte) error { return nil }
func (c *heldControl) Took() error { c.took++; return nil }
func (c *heldControl) Hold(time.Duration) error { c.held++; return nil }
func (c *heldControl) Drop() error { c.drops++; return nil }
type sourceMover struct {
Upgrader
answer error
}
func (s sourceMover) SourceMoved(context.Context, SourceMoved) error { return s.answer }
// **A merge is acknowledged only after the controller recorded it** (novox/hq ADR 0407 decision 3, issue 1092): an
// answer of "not now" — the plan it belongs to could not be saved, the bus away, the lease lost — leaves the announcement
// unacknowledged, held to be handed over again; only an answer that is not "not now" settles it.
func TestAMergeTheControllerCouldNotRecordIsHeldAndNotAcknowledged(t *testing.T) {
body, err := json.Marshal(SourceMoved{Owner: "novox", Repo: "mesh-catalog", Base: "main", Commit: "c1"})
if err != nil {
t.Fatal(err)
}
s := &Server{log: log.New(io.Discard, "", 0)}
for name, tc := range map[string]struct {
answer error
took, held int
}{
"recorded": {nil, 1, 0},
"not now": {errors.Join(ErrTryAgain, errors.New("the plan is not saved")), 0, 1},
"an answer, not a wait": {errors.New("the mesh could not act on it"), 1, 0},
} {
s.upgrader = sourceMover{answer: tc.answer}
m := &heldControl{body: body}
s.sourceMoved(context.Background(), m)
if m.took != tc.took || m.held != tc.held || m.drops != 0 {
t.Errorf("%s: acknowledged %d times and held %d times, want %d and %d", name, m.took, m.held, tc.took, tc.held)
}
}
}
+1 -60
View File
@@ -17,8 +17,7 @@ func (r recordsWith) Built(context.Context, BuildResult) error { return r.err }
type upgradesWith struct{ err error }
func (u upgradesWith) Upgraded(context.Context, Upgraded) error { return u.err }
func (u upgradesWith) SourceMoved(context.Context, SourceMoved) error { return u.err }
func (u upgradesWith) Upgraded(context.Context, Upgraded) error { return u.err }
type replaysWith struct{ err error }
@@ -69,61 +68,3 @@ func TestACatchUpWaitsOutARestartingStore(t *testing.T) {
}
}
}
// A merge the controller could not record for now (the plans held by another controller, the store away) is held
// and handed over again, never taken: the bus keeps it until it is recorded (issue 921). Any other failure is an
// answer and is taken, as before.
func TestAMergeThatCouldNotBeRecordedNowIsHeldNotDropped(t *testing.T) {
merged := SourceMoved{Owner: "novox", Repo: "one", Base: "main", Commit: "abcdef0123"}
for _, c := range []struct {
what string
err error
want func(*settled) bool
}{
{"the plans held by another controller", errors.Join(ErrTryAgain, errors.New("another controller is working the plans")),
func(s *settled) bool { return s.unsettled() }},
{"the store away", errors.Join(ErrTryAgain, restarting), func(s *settled) bool { return s.unsettled() }},
{"cannot act", errors.New("no such module"), func(s *settled) bool { return s.acked && !s.nacked }},
{"kept", nil, func(s *settled) bool { return s.acked && !s.nacked }},
} {
s, in := serving()
s.upgrader = upgradesWith{err: c.err}
to := &settled{}
held := in.sends(t, to, KindSourceMoved, merged).(*fakeControl)
s.act(context.Background(), held)
if !c.want(to) {
t.Errorf("%s: a merge was settled as %+v", c.what, *to)
}
if c.err != nil && errors.Is(c.err, ErrTryAgain) && len(in.held) != 1 {
t.Errorf("%s: the merge was not held to be handed over again", c.what)
}
}
}
// A merge being held when the controller is replaced (its context ended) or when the bus will not hold it is left
// unsettled, never taken: the bus hands it to whatever consumes next (review of issue 921).
func TestAMergeIsLeftUnsettledWhenTheControllerEndsOrTheHoldFails(t *testing.T) {
merged := SourceMoved{Owner: "novox", Repo: "one", Base: "main", Commit: "abcdef0123"}
try := errors.Join(ErrTryAgain, errors.New("another controller is working the plans"))
// The plans' hold returns a bare context.Canceled when the controller is replaced, not ErrTryAgain.
s, in := serving()
s.upgrader = upgradesWith{err: context.Canceled}
to := &settled{}
ended, end := context.WithCancel(context.Background())
end()
s.act(ended, in.sends(t, to, KindSourceMoved, merged))
if !to.unsettled() {
t.Errorf("a merge was settled as %+v when the controller's context ended", *to)
}
s, in = serving()
s.upgrader = upgradesWith{err: try}
to = &settled{}
m := in.sends(t, to, KindSourceMoved, merged).(*fakeControl)
m.holdFails = errors.New("the bus will not hold it")
s.act(context.Background(), m)
if !to.unsettled() {
t.Errorf("a merge was settled as %+v when it could not be held", *to)
}
}
+20
View File
@@ -85,6 +85,26 @@ func nodeOfToolsAlive(subject string) (string, bool) {
type LoopActivity struct {
mu sync.Mutex
took time.Time
// due is when the latest message the loop held back with a delay is handed over again: until then the loop waits
// by its own choice, which is not deafness (sweep 2026-10-30: a merge held five minutes at a time read as a deaf
// controller every time its delay outran the deaf bound).
due time.Time
}
// HeldUntil records that the loop handed a message back to be delivered again at due.
func (l *LoopActivity) HeldUntil(due time.Time) {
l.mu.Lock()
defer l.mu.Unlock()
if due.After(l.due) {
l.due = due
}
}
// Due is when the latest message the loop held is handed over again; zero when it held none.
func (l *LoopActivity) Due() time.Time {
l.mu.Lock()
defer l.mu.Unlock()
return l.due
}
// Loop is this process's event loop.
@@ -1,8 +1,6 @@
package ownmanifest
import (
"path/filepath"
"slices"
"testing"
. "git.novox.be/novox/mesh-sdk/go/catalogue"
@@ -96,21 +94,8 @@ func TestRouteProxyKeepsItsPublicAccountDirectory(t *testing.T) {
// 2 and 4): a Go bundle of the module's own Go module stamped for alpine, unpacked by an archive resource, mounted
// read-only at /opt/mesh, called through args, and named under restart-on so a new program never leaves the old
// one running. The container's name, network and logging are what the jail and the routes rely on.
//
// And its routes are the seat mesh-provisioner's plans alone (issue 774, research 072 row 34): it reads them, and
// receives no file of contributions; no module of this repository does.
func TestRouteProxyRunsItsMountedProgramOnTheTrustImage(t *testing.T) {
m := catalogueManifest(t, "route-proxy")
if !slices.Contains(m.Reads, "mesh-provisioner.plans") {
t.Errorf("route-proxy does not read mesh-provisioner.plans, where its routes are: %v", m.Reads)
}
dirs, _ := filepath.Glob(filepath.Join("..", "..", "modules", "*"))
for _, dir := range dirs {
if r := catalogueManifest(t, filepath.Base(dir)); len(r.Receives) > 0 {
t.Errorf("%s receives %v: a route, like every provision here, is served from the seat's plans, never a "+
"received file", r.Module, r.Receives)
}
}
var bundle *Artifact
for i, a := range m.Build.Artifacts {
if a.Name == "server" {
+14 -52
View File
@@ -5,50 +5,26 @@ The shipping form of the reference reverse proxy (novox/hq ADR 0007, novox/hq 08
## What it is
**A route is a provision** (issue 774, research 072 row 34; the operator, 2026-10-31: "there is only 1 path, the other
is simply deprecated/invalid"). A module that must be reachable declares `requires: ["route"]` and contributes the
name it wants and the port it listens on; this module provides `route`. The controller composes each consumer's plan,
and the seat `mesh-provisioner` keeps it in its state `plans`: the plan's entry for the route names this proxy as
its provider and carries what the consumer contributes, composed (its names joined to the node's domains, its
endpoint's port, its policy), and where the consumer's machine is on the private network, as one opaque payload
(`contributed`) the seat never reads. The proxy follows the plans on the bus, reads each contribution with its own
contract (below), and forwards by the `Host` header of each request to `http://<at>:<port>`, or over loopback for a
consumer on its own machine.
**A route is a grant.** A module that must be reachable declares `requires: ["route"]` and
contributes the name it wants and the port it listens on; this module provides `route`, receives
every consumer that asked as a file, and forwards by the `Host` header of each request to
`http://<at>:<port>` — where `at` is where the mesh says that consumer's machine is (empty means
this machine, reached over loopback).
## Its contract: a package of its own
**route-proxy defines its own contract, and other modules use it** (the operator, 2026-10-31). It is the Go module
`git.novox.be/novox/mesh-controller/modules/route-proxy/contract`, in `contract/` here with its own `go.mod`, standard
library only, released by tags `modules/route-proxy/contract/vX.Y.Z`, the way mesh-sdk is released (`go/vX.Y.Z`); this
module depends on it by its version like any other. It holds:
- **the route a consumer asks** (`Request`, read strictly by `DecodeRequest`: no field it does not name, a name of
either kind, a port where the rule forwards, http or https, a positive body limit, a secret's file never a
credential) and its JSON schema (`route.schema.json`, `Schema`), the form of a manifest's `contributes.route`;
- **the events this proxy says** on `mesh.mod.route-proxy.event.<type>`: `route.served` and `route.withdrawn` as the
names it serves change, `route.refused` with why for a route a plan gives it and it does not serve (`Event`);
- **what it keeps on the bus**, its states `routes` and `unrouted`, their keys and records (`Route`, `Unrouted`,
`Other`, `Key`, `Split`);
- **its tools' names and arguments** (`ToolRoutes`, `ToolRoute`, `ToolUnrouted`, `RoutesArgs`, `RouteArgs`).
The controller does not write the proxy's configuration. It composes the facts into the plans and the proxy turns them
into whatever it runs. That boundary is why swapping this proxy for another costs nothing to the modules that publish
through it. **There is no received file and no second source:** a route no plan names is not served, and a module
declaring `receives.route` is refused by the catalogue check (mesh-catalog `checks/credentialform`), but for the
migration scaffolding still listed there.
The control plane does not write the proxy's configuration. It delivers the facts — the
contributions file at `receives.route` — and the proxy turns them into whatever it runs. That
boundary is why swapping this proxy for another costs nothing to the modules that publish through
it.
| field | value | why |
|---|---|---|
| `provides` | `route` (scope `mesh`, answered by the helper) | a module on any node may require it |
| `provides` | `route` (scope `mesh`) | a module on any node may require it |
| `serves.route` | `{}` | a route hands back a name, not a credential — the consumer already knows the name it asked for |
| `reads` | `mesh-provisioner.plans`, `mesh-provisioner.provisions`, `mesh-delivery-assigner.placement` | the plans are where the routes are; the helper reads both; the placement says who is inside |
| `emits` | `route.served`, `route.withdrawn`, `route.refused` | what it serves, said as it changes (the contract's `Event`) |
| `receives.route` | `…/routes/mesh.json` | where the mesh writes every contribution; the proxy reads it as `$ROUTES` and re-reads on change |
| `listens` | `80` and `443`, both `from: anywhere` | the one machine with a public opening; the firewall opens these for free (ADR 0045) |
Until the bus answers, the proxy routes nothing and says so on its log; it refuses to start without its broker
credential (`MESH_BROKER_FILE`). Who is inside the private network, and so may be served an internal-only name, is
every node's address on it as the seat mesh-placement decides it (mesh-sdk `delivery.InsideAddresses`); until that
is read, an internal-only name is served to this machine alone.
No provisioner: the proxy neither mints a credential for anybody nor answers a provision other
than `route`. It reads the file the mesh writes, and its own broker credential is for its tools.
## Which issuer: Let's Encrypt, stated here (novox/hq ADR 0226)
@@ -101,7 +77,7 @@ traffic.
| env | default | meaning |
|---|---|---|
| `MESH_BROKER_FILE` | *(required)* | the proxy's broker credential: the plans it routes from, and its membership |
| `ROUTES` | `/routes/mesh.json` | the contributions file the mesh writes (`receives.route`) |
| `LISTEN` | `:80` | HTTP, and the ACME HTTP-01 challenge |
| `TLS_LISTEN` | `:443` | HTTPS; unset to serve plain HTTP only |
| `ACME_CACHE` | `/acme` | where issued certificates persist; required when `TLS_LISTEN` is set, so a restart does not re-order |
@@ -114,17 +90,3 @@ A small Go program beside the proxy (`cmd/route-provider`) runs the SDK's provid
and the offer says `"answered-by": "helper"`: a handed consumer (gitea's origins file) is filled by its machine's client
from this module's entry, which carries the public names the controller composed for it. The module is **not** marked
moved: its own files, `acme.env` and the broker secret, are composed as before, and the proxy itself is unchanged.
## What it says on the bus, and its tools (issue 774)
The proxy keeps two states, announced readable (`conn.Keeps`, novox/hq ADR 0495), each keyed `<node>.<name>`:
| state | one key per | holds | written when |
|---|---|---|---|
| `routes` | served name | the rules it is served by (path, module, node, target, refusal, redirect, authentication); its certificate (authority, issuer, not-after, last renewal, outcome, or why it failed); the last upstream answer (status, time, error, path) | a route arrives, changes or leaves; an authority stores a certificate; a handshake fails to find one or hands out one not yet recorded; an upstream answer differs from the last one written, or the same one is a minute old |
| `unrouted` | name answered "no route for", in a request or a TLS handshake, under a domain the proxy serves | when it was last asked, how often since the start, whether it is an internal-only name asked from outside; never who asked (the journal keeps that, for fail2ban) | first asked, at most five new names a minute, then at most once a minute; deleted when the name is served; kept a week. Every other refusal is only counted, in one key per proxy (`<node>._other`: the last minute's count and the total), written once a minute. A refused write is dropped, never retried |
Nothing is written on a timer. The provider helper's binary serves the module's tools from those states, one call each:
`route_proxy_routes` (every served name with its target, certificate expiry and last answer), `route_proxy_route`
(one name in full, or that it is not served and when it was last asked for) and `route_proxy_unrouted` (the names
refused, newest first).
+3 -11
View File
@@ -1,6 +1,6 @@
// route-provider: route-proxy's Go half, one binary the node's runtime launches and speaks MCP to over stdio
// through the Go SDK. It runs the SDK's provider helper that answers the consumers of `route` (issue 792), and serves
// the module's tools, which read what the proxy keeps on the bus (issue 774, tools.go).
// through the Go SDK. It serves no tools; it runs the SDK's provider helper that answers the consumers of `route`
// (issue 792). The TypeScript adapter beside it keeps writing the proxy's files.
//
// stdout is the MCP channel, so everything this module says, it says on stderr.
package main
@@ -9,12 +9,9 @@ import (
"context"
"fmt"
"os"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/provider"
"git.novox.be/novox/mesh-controller/modules/route-proxy/contract"
)
func say(format string, args ...any) {
@@ -28,17 +25,12 @@ func main() {
} else {
p := newProvider(node, provider.StateBus(ModuleName), say)
go func() {
// The helper's watches go through the runtime's channel, which Serve holds: one asked before it does is
// refused ("before Serve"), Run returns and the helper stays stopped until the process restarts.
<-stdio.Served()
if err := p.Run(context.Background()); err != nil {
say("the provider helper stopped: %v", err)
}
}()
}
tools := front{routes: stdio.State(contract.RoutesState), unrouted: stdio.State(contract.UnroutedState),
now: time.Now}.tools()
if err := stdio.Serve("", tools); err != nil {
if err := stdio.Serve("", nil); err != nil {
say("%v", err)
os.Exit(1)
}
@@ -7,20 +7,18 @@ package main
// **It composes nothing.** A route's public name is the label its consumer chose joined to the node's public domain,
// which only the controller knows (the domain is placement), so the controller composes it and carries it in the
// consumer's plan entry (`names`); the helper copies it into the entry's values unchanged. There is no credential for
// a route and nothing to make. The proxy itself serves the same plans' entries (cmd/route-proxy plans.go), the one
// source of routes (issue 774, research 072 row 34).
// a route and nothing to make: the traefik files stay the TypeScript adapter's, written from the contributions the
// mesh still gives it.
import (
"git.novox.be/novox/mesh-sdk/go/provider"
"git.novox.be/novox/mesh-controller/modules/route-proxy/contract"
)
// ProvisionName is what route-proxy provides, as its manifest says, and ModuleName this module.
const (
ProvisionName = contract.Provision
ModuleName = contract.Module
)
// ProvisionName is what route-proxy provides, as its manifest says.
const ProvisionName = "route"
// ModuleName is this module.
const ModuleName = "route-proxy"
// newProvider is route-proxy answering its consumers over bus on node: the route, served as the manifest serves it
// (nothing of its own), with no credential, and nothing to make or retire for a consumer.
@@ -9,9 +9,7 @@ import (
"sync"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/provider"
"git.novox.be/novox/mesh-sdk/go/provisions"
)
// bucketBus is the helper's bus over in-memory buckets.
@@ -46,60 +44,33 @@ func (b *bucketBus) Delete(_ context.Context, bucket, key string) error {
return nil
}
// A consumer of a route is answered with the names its plan's entry carries, and nothing of a credential (issue 792),
// from the seat mesh-provisioner's plans and keys (novox/hq ADR 0570 clause 2), never the controller's assignments: a
// module not handed whose route the seat keys (grafana, issue 774) has a plan of provisions alone, and a helper that
// read only the assignments left its key asked for good, so the assigner withheld every new build of 38 such modules.
// A consumer of a route is answered with the names its plan's entry carries, and nothing of a credential (issue 792):
// the entry's values are the controller's composition, copied unchanged, and no vault is asked.
func TestAConsumerOfARouteIsToldItsNamesFromItsPlanEntry(t *testing.T) {
bus := &bucketBus{buckets: map[string]map[string][]byte{}}
for module, names := range map[string]map[string]string{
"gitea": {"name": "git.example.org", "internal-name": "git.novox.internal", "name-web": "git.example.org"},
"grafana": {"name": "grafana.example.org"},
} {
plan, err := provisions.EncodePlan(provisions.Plan{Node: "g14", Module: module, Plan: catalogue.Plan{
Provisions: []catalogue.PlanProvision{{Provision: ProvisionName, Provider: "novox", ProviderModule: ModuleName,
At: "10.7.0.1", From: "novox", As: "mesh_g14_" + module, Names: names}}}})
if err != nil {
t.Fatal(err)
}
key := "g14." + module
if err := bus.Put(context.Background(), provisions.PlansBucket, key, plan); err != nil {
t.Fatal(err)
}
asked, err := json.Marshal(provisions.Provision{V: provisions.Version, Provision: ProvisionName,
Consumer: provisions.End{Node: "g14", Module: module}, Provider: provisions.End{Node: "novox", Module: ModuleName},
State: provisions.Asked})
if err != nil {
t.Fatal(err)
}
if err := bus.Put(context.Background(), provisions.ProvisionsBucket, key+"."+ProvisionName, asked); err != nil {
t.Fatal(err)
}
plan, _ := json.Marshal(map[string]any{"node": "g14", "module": "gitea", "provisioning": map[string]any{
"provisions": []any{map[string]any{"provision": ProvisionName, "provider": "novox",
"provider-module": ModuleName, "at": "10.7.0.1", "from": "novox", "as": "mesh_g14_gitea",
"names": map[string]any{"name": "git.example.org", "internal-name": "git.novox.internal",
"name-web": "git.example.org"}}}}})
if err := bus.Put(context.Background(), provider.AssignmentsBucket, "g14.gitea", plan); err != nil {
t.Fatal(err)
}
p := newProvider("novox", bus, t.Logf)
out, err := p.Pass(context.Background())
if err != nil {
t.Fatal(err)
}
if len(out) != 2 || out[0].Status != provider.Answered || out[1].Status != provider.Answered {
if len(out) != 1 || out[0].Status != provider.Answered {
t.Fatalf("the pass did %+v", out)
}
for key, want := range map[string]map[string]string{
"g14.gitea.route": {"name-web": "git.example.org", "name": "git.example.org", "internal-name": "git.novox.internal"},
"g14.grafana.route": {"name": "grafana.example.org"},
} {
var e provider.Entry
if err := json.Unmarshal(bus.buckets[provider.BucketOf(ModuleName)][key], &e); err != nil {
t.Fatalf("%s: %v", key, err)
}
for k, v := range want {
if e.Values[k] != v {
t.Errorf("%s says %s %v, want %s", key, k, e.Values[k], v)
}
}
if e.State != provisions.Provided || e.VaultName != "" || e.User != "" || e.Refused != "" {
t.Errorf("the entry %s is %+v", key, e)
}
var e provider.Entry
if err := json.Unmarshal(bus.buckets[provider.BucketOf(ModuleName)]["g14.gitea.route"], &e); err != nil {
t.Fatal(err)
}
if e.Values["name-web"] != "git.example.org" || e.Values["name"] != "git.example.org" ||
e.Values["internal-name"] != "git.novox.internal" || e.VaultName != "" || e.User != "" || e.Refused != "" {
t.Fatalf("the entry is %+v", e)
}
}
@@ -108,8 +79,6 @@ func TestAConsumerOfARouteIsToldItsNamesFromItsPlanEntry(t *testing.T) {
// the public ingress does not change. Everything of the manifest but its build, what it provides and the state it
// keeps is pinned by this digest; changing the ingress means changing it here, on purpose. Re-pinned in R4d (novox/hq
// ADR 0480): the program is an archive mounted into the trust image, the container and its files are otherwise as before.
// Re-pinned for issue 774 (research 072 row 34): the received routes file, its directory, mount and ROUTES went, since
// the proxy serves from the seat's plans alone; the ingress, its certificates and its files are otherwise as before.
func TestRouteProxyIsNotMovedAndKeepsItsOwnFiles(t *testing.T) {
raw, err := os.ReadFile("../../module.json")
if err != nil {
@@ -126,17 +95,15 @@ func TestRouteProxyIsNotMovedAndKeepsItsOwnFiles(t *testing.T) {
if len(provides) != 1 || provides[0].(map[string]any)["answered-by"] != "helper" {
t.Fatalf("route is not said to be answered by the helper: %v", provides)
}
// Its states and its tools are what it says on the bus, not what it runs or writes (issue 774).
for _, k := range []string{"build", "provides", "state", "reads", "tools", "replaces"} {
for _, k := range []string{"build", "provides", "state", "reads"} {
delete(m, k)
}
// Refreshed on purpose for the events its contract names (`emits`: route.served, route.withdrawn, route.refused).
pinned, err := json.Marshal(m) // sorted keys, compact
if err != nil {
t.Fatal(err)
}
sum := sha256.Sum256(pinned)
if got := hex.EncodeToString(sum[:]); got != "3a44c56cc021be4008f521d16fa02e1c0f0d2eedab2f4d0b60f7d3177269e542" {
if got := hex.EncodeToString(sum[:]); got != "85cb955e5a4b2d8826f50aa76433064a576e8b15b71537032eb93c7388c0ee28" {
t.Fatalf("route-proxy's own manifest changed: digest %s", got)
}
// The two acme templates are pinned the same way (issue 849): they are what the proxy's certificates are asked
@@ -1,296 +0,0 @@
package main
// route-proxy's tools (issue 774): the web front's command line. Each answers from what the proxy keeps on the bus
// (route-proxy_routes and route-proxy_unrouted, written on change by the proxy itself), in one call, for every
// machine running a proxy: which sites are served and where they go, whether their certificates are good, what the
// last request through each got back, and which names were asked for and are served nowhere.
import (
"encoding/json"
"fmt"
"sort"
"strings"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-controller/modules/route-proxy/contract"
)
// The tools' names, as the manifest declares them.
const (
RoutesTool = contract.ToolRoutes
RouteTool = contract.ToolRoute
UnroutedTool = contract.ToolUnrouted
)
// reader is one of the module's states, as stdio.KeptState reads it.
type reader interface {
Keys() ([]string, error)
Get(key string) (*stdio.StateEntry, error)
}
// front is what the tools read: the routes and the refusals, and the clock a certificate's days left are counted by.
type front struct {
routes, unrouted reader
now func() time.Time
}
// RouteLine is one served name in the list: where it goes, how long its certificate holds, what it last answered.
type RouteLine struct {
Name string `json:"name"`
Proxy string `json:"proxy"`
Reach string `json:"reach"`
Targets []string `json:"targets"`
// Reachable is yes when the last answer upstream gave was below 500, no when it was 500 or more or none came,
// unknown before any request.
Reachable string `json:"reachable"`
// Certificate is its outcome, issuer and expiry in words; ExpiresInDays is the whole days until not-after.
Certificate string `json:"certificate"`
ExpiresInDays *int `json:"expires_in_days,omitempty"`
LastAnswer *contract.Answer `json:"last_answer,omitempty"`
}
func (f front) tools() []stdio.Tool {
node := map[string]any{"type": "string", "description": "only the proxy on this machine; every machine when left out"}
return []stdio.Tool{
{Name: RoutesTool,
Description: "Every site (web name, host name) the mesh's web front serves through route-proxy, the reverse proxy " +
"on the public machine: for each route the module and node it forwards to, its TLS certificate's issuer and " +
"expiry (days left), and the last answer upstream gave (status, time, error), so is X reachable and is its " +
"certificate good is answered without curl. Read from the proxy's routes state on the bus. Read-only.",
Input: map[string]any{"type": "object", "properties": map[string]any{"node": node}},
Run: func(args map[string]any) (any, error) {
at, _ := args["node"].(string)
return f.list(at)
}},
{Name: RouteTool,
Description: "One site the mesh's web front (route-proxy) is asked about, in full: every rule it is served by " +
"(path, module, node, target, refusal, redirect, authentication), its certificate (issuer, not-after, last " +
"renewal and its outcome, or why it failed) and the last upstream answer. A name with no route says so, and " +
"when it was last asked for. Answers is X reachable, why does this site not load. Read-only.",
Input: map[string]any{"type": "object", "required": []string{"name"}, "properties": map[string]any{
"name": map[string]any{"type": "string", "description": "the site's name, such as app.example.org"},
"node": node}},
Run: func(args map[string]any) (any, error) {
name, _ := args["name"].(string)
at, _ := args["node"].(string)
return f.one(name, at)
}},
{Name: UnroutedTool,
Description: "Names the mesh's web front (route-proxy) answered \"no route for\": a site asked for in a request or " +
"a TLS handshake that no route serves, newest first, with when it was last asked and how often. Only names under a " +
"domain the proxy serves are kept by name, a few new ones a minute; every other refusal (a scan) is counted per " +
"proxy, the last minute's count and the total since its start. A site that should be served and is missing " +
"its route shows here. Kept a week. Read-only.",
Input: map[string]any{"type": "object", "properties": map[string]any{"node": node}},
Run: func(args map[string]any) (any, error) {
at, _ := args["node"].(string)
return f.refusals(at)
}},
}
}
// read is every value of a state, of node's proxy alone when node is named, decoded into a fresh T each.
func read[T any](state reader, node string) ([]T, error) {
keys, err := state.Keys()
if err != nil {
return nil, err
}
var out []T
for _, key := range keys {
at, _ := contract.Split(key)
if node != "" && at != node {
continue
}
entry, err := state.Get(key)
if err != nil {
return nil, err
}
if entry == nil {
continue // deleted between the list and the read
}
var v T
if err := json.Unmarshal(entry.Value, &v); err != nil {
return nil, fmt.Errorf("%s holds what is not a route-proxy record: %w", key, err)
}
out = append(out, v)
}
return out, nil
}
func (f front) list(node string) (map[string]any, error) {
routes, err := read[contract.Route](f.routes, node)
if err != nil {
return nil, err
}
lines := make([]RouteLine, 0, len(routes))
unreachable, expiring := 0, 0
for _, r := range routes {
line := f.line(r)
if line.Reachable == "no" {
unreachable++
}
if line.ExpiresInDays != nil && *line.ExpiresInDays < 14 || r.Certificate.Outcome == contract.Failed {
expiring++
}
lines = append(lines, line)
}
sort.Slice(lines, func(i, j int) bool {
if lines[i].Name != lines[j].Name {
return lines[i].Name < lines[j].Name
}
return lines[i].Proxy < lines[j].Proxy
})
return map[string]any{"routes": lines, "served": len(lines), "unreachable": unreachable,
"certificates_failing_or_within_14_days": expiring}, nil
}
func (f front) line(r contract.Route) RouteLine {
line := RouteLine{Name: r.Name, Proxy: r.Proxy, Reach: r.Reach, LastAnswer: r.LastAnswer, Reachable: "unknown",
Targets: []string{}}
for _, t := range r.Targets {
line.Targets = append(line.Targets, describe(t))
}
if a := r.LastAnswer; a != nil {
line.Reachable = "yes"
if a.Error != "" || a.Status >= 500 || a.Status == 0 {
line.Reachable = "no"
}
}
c := r.Certificate
words := []string{}
if c.Outcome != "" {
words = append(words, c.Outcome)
}
if c.Issuer != "" {
words = append(words, "by "+c.Issuer)
}
if c.NotAfter != nil {
days := int(c.NotAfter.Sub(f.now()).Hours() / 24)
line.ExpiresInDays = &days
words = append(words, "until "+c.NotAfter.UTC().Format("2006-01-02"))
}
if c.Error != "" {
words = append(words, "("+c.Error+")")
}
if len(words) == 0 {
words = append(words, "none seen yet")
}
line.Certificate = c.Authority + ": " + strings.Join(words, " ")
return line
}
// describe is one rule in a few words: path → module@node at target, or what it answers with.
func describe(t contract.Target) string {
path := t.Path
if path == "" {
path = "/"
}
var what string
switch {
case t.Deny:
what = "refused"
case t.Redirect != "":
what = "redirected to " + t.Redirect
default:
what = fmt.Sprintf("%s@%s at %s", t.Module, t.Node, t.Target)
}
if t.Sealed != "" {
what += " (authentication cannot be checked: " + t.Sealed + ")"
} else if t.Auth {
what += " (authenticated)"
}
return path + " → " + what
}
func (f front) one(name, node string) (map[string]any, error) {
name = strings.ToLower(strings.TrimSpace(name))
if name == "" {
return nil, fmt.Errorf("name is the site to read, such as app.example.org")
}
routes, err := read[contract.Route](f.routes, node)
if err != nil {
return nil, err
}
var found []contract.Route
for _, r := range routes {
if r.Name == name {
found = append(found, r)
}
}
if len(found) > 0 {
out := map[string]any{"name": name, "routed": true, "routes": found}
if len(found) == 1 {
out["reachable"] = f.line(found[0]).Reachable
}
return out, nil
}
asked, _, err := f.asked(node)
if err != nil {
return nil, err
}
for _, a := range asked {
if a.Name == name {
return map[string]any{"name": name, "routed": false, "asked": a,
"says": fmt.Sprintf("no proxy serves %s; the proxy on %s answered \"no route\" for it, last at %s",
name, a.Proxy, a.AskedAt.UTC().Format(time.RFC3339))}, nil
}
}
return map[string]any{"name": name, "routed": false,
"says": fmt.Sprintf("no proxy serves %s, and none was asked for it this week", name)}, nil
}
func (f front) refusals(node string) (map[string]any, error) {
asked, others, err := f.asked(node)
if err != nil {
return nil, err
}
sort.Slice(asked, func(i, j int) bool { return asked[i].AskedAt.After(asked[j].AskedAt) })
if asked == nil {
asked = []contract.Unrouted{}
}
return map[string]any{"unrouted": asked, "names": len(asked), "other_refusals": others}, nil
}
// asked is the names refused and kept by name, and each proxy's count of the refusals not kept by name.
func (f front) asked(node string) ([]contract.Unrouted, []contract.Other, error) {
keys, err := f.unrouted.Keys()
if err != nil {
return nil, nil, err
}
names, others := keySet{}, keySet{}
for _, key := range keys {
if _, name := contract.Split(key); name == contract.OtherName {
others[key] = true
} else {
names[key] = true
}
}
asked, err := read[contract.Unrouted](only{f.unrouted, names}, node)
if err != nil {
return nil, nil, err
}
counted, err := read[contract.Other](only{f.unrouted, others}, node)
if counted == nil {
counted = []contract.Other{}
}
return asked, counted, err
}
// keySet is a set of keys; only is a state narrowed to them.
type keySet map[string]bool
type only struct {
reader
keys keySet
}
func (o only) Keys() ([]string, error) {
out := make([]string, 0, len(o.keys))
for k := range o.keys {
out = append(out, k)
}
sort.Strings(out)
return out, nil
}
@@ -1,99 +0,0 @@
package main
import (
"encoding/json"
"sort"
"strings"
"testing"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-controller/modules/route-proxy/contract"
)
// held is a state as the bus holds it, read-only.
type held map[string]any
func (h held) Keys() ([]string, error) {
out := []string{}
for k := range h {
out = append(out, k)
}
sort.Strings(out)
return out, nil
}
func (h held) Get(key string) (*stdio.StateEntry, error) {
v, ok := h[key]
if !ok {
return nil, nil
}
raw, err := json.Marshal(v)
return &stdio.StateEntry{Key: key, Value: raw}, err
}
// The web front's tools answer from the bus, in one call each (issue 774): every site with its target, certificate
// expiry and last answer; one site in full, or that it is not served and when it was asked for; and the names the
// proxy answered "no route for".
func TestTheWebFrontsToolsAnswerFromTheBus(t *testing.T) {
now := time.Date(2026, 10, 30, 21, 0, 0, 0, time.UTC)
expires := now.Add(30 * 24 * time.Hour)
routes := held{
"anchor.grafana.example": contract.Route{Name: "grafana.example", Proxy: "anchor", Reach: contract.ReachPublic,
Targets: []contract.Target{{Module: "grafana", Node: "home-server", Target: "http://10.0.0.2:3000"}},
Certificate: contract.Certificate{Authority: contract.ReachPublic, Issuer: "Let's Encrypt R11", NotAfter: &expires, Outcome: contract.Issued},
LastAnswer: &contract.Answer{Status: 502, At: now, Error: "dial tcp 10.0.0.2:3000: connect: connection refused"}},
"anchor.drive.example": contract.Route{Name: "drive.example", Proxy: "anchor", Reach: contract.ReachPublic,
Targets: []contract.Target{{Module: "nextcloud", Node: "home-server", Target: "http://10.0.0.2:8080"}},
Certificate: contract.Certificate{Authority: contract.ReachPublic},
LastAnswer: &contract.Answer{Status: 200, At: now}},
}
unrouted := held{
"anchor.old.example": contract.Unrouted{Name: "old.example", Proxy: "anchor", AskedAt: now.Add(-time.Hour), Times: 3},
"anchor.wiki.example": contract.Unrouted{Name: "wiki.example", Proxy: "anchor", AskedAt: now, Times: 1},
"anchor._other": contract.Other{Proxy: "anchor", Minute: now, LastMinute: 40, SinceStart: 900},
}
tools := map[string]stdio.Tool{}
for _, tool := range (front{routes: routes, unrouted: unrouted, now: func() time.Time { return now }}).tools() {
tools[tool.Name] = tool
}
for _, row := range []struct {
tool string
args map[string]any
want []string // what the answer says, in its JSON
}{
{RoutesTool, nil, []string{`"served":2`, `"unreachable":1`,
`"name":"grafana.example"`, `"reachable":"no"`, `"expires_in_days":30`, `/ → grafana@home-server at http://10.0.0.2:3000`,
`"name":"drive.example"`, `"reachable":"yes"`, `public: none seen yet`}},
{RoutesTool, map[string]any{"node": "home-server"}, []string{`"served":0`}},
{RouteTool, map[string]any{"name": "Grafana.Example"}, []string{`"routed":true`, `"reachable":"no"`,
`connection refused`, `"issuer":"Let's Encrypt R11"`}},
{RouteTool, map[string]any{"name": "wiki.example"}, []string{`"routed":false`,
`answered \"no route\" for it, last at 2026-10-30T21:00:00Z`}},
{RouteTool, map[string]any{"name": "nothing.example"}, []string{`"routed":false`, `none was asked for it`}},
{UnroutedTool, nil, []string{`"names":2`, `"unrouted":[{"name":"wiki.example"`,
`"other_refusals":[{"proxy":"anchor"`, `"last_minute":40`, `"since_start":900`}},
} {
t.Run(row.tool, func(t *testing.T) {
tool, ok := tools[row.tool]
if !ok {
t.Fatalf("%s is not served", row.tool)
}
if !strings.Contains(tool.Description, "route-proxy") {
t.Errorf("%s's description does not name the web front, so a search does not find it", row.tool)
}
answer, err := tool.Run(row.args)
if err != nil {
t.Fatal(err)
}
raw, _ := json.Marshal(answer)
for _, w := range row.want {
if !strings.Contains(string(raw), w) {
t.Errorf("%s(%v) does not say %s:\n%s", row.tool, row.args, w, raw)
}
}
})
}
}
@@ -19,57 +19,55 @@ const (
// path back: nothing is re-registered, no order reaches the CA, and issuance stops with nothing
// saying why. Naming the cache after the authority means the account is only ever found where it is
// still valid — the new root lands in a directory with no account in it, and autocert registers.
func TestTheAccountCacheIsNamedAfterItsAuthority(t *testing.T) {
t.Run("a new CA root means a new account cache", func(t *testing.T) {
const directory = "https://anchor.internal/acme/acme/directory"
before := forThisAuthority("/var/lib/route-proxy/acme", directory, []byte(firstRoot))
after := forThisAuthority("/var/lib/route-proxy/acme", directory, []byte(secondRoot))
if before == after {
t.Fatalf("a re-initialised CA reuses the account it was rejected for: %s", before)
}
})
func TestANewCARootMeansANewAccountCache(t *testing.T) {
const directory = "https://anchor.internal/acme/acme/directory"
before := forThisAuthority("/var/lib/route-proxy/acme", directory, []byte(firstRoot))
after := forThisAuthority("/var/lib/route-proxy/acme", directory, []byte(secondRoot))
if before == after {
t.Fatalf("a re-initialised CA reuses the account it was rejected for: %s", before)
}
}
// And the SAME authority keeps the account it registered, restart after restart.
//
// This is the whole reason ACME_CACHE is required in the first place: an account and its
// certificates that did not persist would be re-ordered on every restart, which works silently until
// a rate limit says it does not. Whitespace around the delivered root is not a new authority — the
// mesh writes that file, and a trailing newline coming or going must not throw away an account.
t.Run("the same authority keeps its account", func(t *testing.T) {
const directory = "https://anchor.internal/acme/acme/directory"
first := forThisAuthority("/var/lib/route-proxy/acme", directory, []byte(firstRoot))
again := forThisAuthority("/var/lib/route-proxy/acme", directory, []byte("\n"+firstRoot+"\n\n"))
if first != again {
t.Errorf("the same authority was given two caches, so every restart orders again:\n%s\n%s",
first, again)
}
})
// And the SAME authority keeps the account it registered, restart after restart.
//
// This is the whole reason ACME_CACHE is required in the first place: an account and its
// certificates that did not persist would be re-ordered on every restart, which works silently until
// a rate limit says it does not. Whitespace around the delivered root is not a new authority — the
// mesh writes that file, and a trailing newline coming or going must not throw away an account.
func TestTheSameAuthorityKeepsItsAccount(t *testing.T) {
const directory = "https://anchor.internal/acme/acme/directory"
first := forThisAuthority("/var/lib/route-proxy/acme", directory, []byte(firstRoot))
again := forThisAuthority("/var/lib/route-proxy/acme", directory, []byte("\n"+firstRoot+"\n\n"))
if first != again {
t.Errorf("the same authority was given two caches, so every restart orders again:\n%s\n%s",
first, again)
}
}
// Staging and production are different authorities, and were sharing one account.
//
// The latent fault of the same shape: pointing ACME_DIRECTORY at production after testing against
// staging reused the staging account, because the cache had no idea they were different.
t.Run("staging and production do not share an account", func(t *testing.T) {
staging := forThisAuthority("/acme", stagingDirectory, nil)
production := forThisAuthority("/acme", "https://acme-v02.api.letsencrypt.org/directory", nil)
if staging == production {
t.Errorf("two issuers share one account: %s", staging)
}
})
// Staging and production are different authorities, and were sharing one account.
//
// The latent fault of the same shape: pointing ACME_DIRECTORY at production after testing against
// staging reused the staging account, because the cache had no idea they were different.
func TestStagingAndProductionDoNotShareAnAccount(t *testing.T) {
staging := forThisAuthority("/acme", stagingDirectory, nil)
production := forThisAuthority("/acme", "https://acme-v02.api.letsencrypt.org/directory", nil)
if staging == production {
t.Errorf("two issuers share one account: %s", staging)
}
}
// It stays inside the directory the mesh gave it, and is a plain name.
//
// The mesh owns ACME_CACHE and mounts it; a name derived from a certificate that escaped it — or
// that carried a separator out of the PEM — would put an account somewhere nothing persists.
t.Run("the account cache stays where the mesh put it", func(t *testing.T) {
const cache = "/var/lib/route-proxy/acme"
got := forThisAuthority(cache, "https://anchor.internal/acme/acme/directory", []byte(firstRoot))
if !strings.HasPrefix(got, cache+"/") {
t.Fatalf("the account cache is not under %s: %s", cache, got)
}
name := strings.TrimPrefix(got, cache+"/")
if name != filepath.Base(got) || strings.ContainsAny(name, "/.") {
t.Errorf("the account cache is not a plain name: %q", name)
}
})
// It stays inside the directory the mesh gave it, and is a plain name.
//
// The mesh owns ACME_CACHE and mounts it; a name derived from a certificate that escaped it — or
// that carried a separator out of the PEM — would put an account somewhere nothing persists.
func TestTheAccountCacheStaysWhereTheMeshPutIt(t *testing.T) {
const cache = "/var/lib/route-proxy/acme"
got := forThisAuthority(cache, "https://anchor.internal/acme/acme/directory", []byte(firstRoot))
if !strings.HasPrefix(got, cache+"/") {
t.Fatalf("the account cache is not under %s: %s", cache, got)
}
name := strings.TrimPrefix(got, cache+"/")
if name != filepath.Base(got) || strings.ContainsAny(name, "/.") {
t.Errorf("the account cache is not a plain name: %q", name)
}
}
+42 -151
View File
@@ -1,58 +1,41 @@
package main
import (
"encoding/json"
"fmt"
"log"
"sort"
"strings"
"sync/atomic"
"time"
"github.com/nats-io/nats.go"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/announce"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/membership"
"git.novox.be/novox/mesh-sdk/go/process"
"git.novox.be/novox/mesh-sdk/go/provisions"
"git.novox.be/novox/mesh-controller/modules/route-proxy/contract"
)
// keptStates is every state route-proxy keeps, as both its halves write them: the routes and the refusals this
// proxy writes, readable by others, and the bound entries the provider helper writes.
func keptStates() []announce.State {
return []announce.State{
{Name: contract.BoundState},
{Name: contract.RoutesState, Readable: true},
{Name: contract.UnroutedState, Readable: true},
}
}
// What the mesh says this proxy routes, and who is inside the mesh, read on the bus.
// What the mesh issued this proxy, read on the bus (novox/hq ADR 0160, ADR 0167).
//
// **The routes are the seat mesh-provisioner's plans, alone** (issue 774, research 072 row 34): every plan's entry
// naming this proxy as a route's provider, followed, so a route added or withdrawn reaches a running proxy without a
// restart, and said on the bus as it changes (contract.EventServed, EventWithdrawn, EventRefused). **Who is inside**
// is every node's address on the private network as the seat mesh-placement decides it (mesh-sdk
// delivery.InsideAddresses), which is who may be served an internal name; until it is read, an internal name is
// served to this machine alone.
// **The proxy is told, not left to work it out.** Its membership carries the routes it is given —
// the same contributions its file is written from — and every machine's address on the private
// network, which is who may be served an internal name. Read once at connect and followed, so a
// route added or a machine joining reaches a running proxy without a restart.
// follow connects with the credential in path, follows the plans and the placement, and retries the connection for
// as long as it takes: until the bus answers, the proxy routes nothing, and says so.
func follow(path string, held *table) {
// followMembership connects with the credential in path and applies every membership the mesh
// issues this proxy. It retries the first connection for as long as it takes: a proxy that started
// before the bus keeps serving the file, and takes the bus when it answers.
func followMembership(path string, held *table, fromBus *atomic.Bool) {
for {
err := followOnce(path, held)
err := followOnce(path, held, fromBus)
if err == nil {
return
}
log.Printf("cannot follow the plans that say what this proxy routes, so it routes what it last read: %v", err)
log.Printf("cannot follow this proxy's membership, serving the file meanwhile: %v", err)
time.Sleep(30 * time.Second)
}
}
func followOnce(path string, held *table) error {
func followOnce(path string, held *table, fromBus *atomic.Bool) error {
account, err := process.ReadAccount(path)
if err != nil {
return err
@@ -61,134 +44,42 @@ func followOnce(path string, held *table) error {
if err != nil {
return fmt.Errorf("connecting to the bus at %s: %w", membership.BareAddress(account.URL), err)
}
// What this proxy keeps, said on the bus (novox/hq ADR 0495; issue 774): the names it serves and the names it
// refused, readable by any module granted a read of them; the provider helper's entries, read by the machines'
// clients through their own grant. Said before the first write, so no write of an unannounced state is refused.
if err := conn.Keeps(keptStates()...); err != nil {
log.Printf("cannot say on the bus what this proxy keeps, so it keeps nothing there: %v", err)
} else {
held.ledger.attach(account.Node, conn.State(contract.RoutesState), conn.State(contract.UnroutedState))
}
say := func(eventType string, e contract.Event) {
if err := conn.Emit(eventType, e); err != nil {
log.Printf("cannot say %s of %q on the bus: %v", eventType, e.Name, err)
subject := membership.Subject(account.Node, account.Module)
err = conn.FollowMembership(func(issued membership.Membership) {
if took := applyMembership(issued, held); took && !fromBus.Swap(true) {
log.Printf("routes now come from this proxy's membership on %s", subject)
}
}
// Who is inside first, so the first table is served to the machines it is meant for.
if err := followInside(conn.Read(delivery.PlacementRead), held); err != nil {
log.Printf("cannot follow %s, so an internal name is served to this machine alone: %v", delivery.PlacementRead, err)
}
if err := followPlans(conn.Read(provisions.PlansRead), newPlanned(account.Node, say), held, say); err != nil {
}, func(err error) { log.Print(err) })
if err != nil {
conn.Close()
return err
}
return nil
}
// planWatch is a state this proxy watches: every current key, then every change.
type planWatch interface {
Watch(key string, handle func(stdio.StateChange) error) error
}
// followPlans serves what the plans give this proxy: the whole table once every current plan is read, then again on
// every change that alters it. A table built from part of the plans would withdraw every route not yet read.
func followPlans(plans planWatch, given *planned, held *table, say func(string, contract.Event)) error {
var read atomic.Bool // every current plan is read: from here, each change is served as it comes
err := plans.Watch("", func(c stdio.StateChange) error {
var value []byte
if c.Op != "delete" && c.Op != "purge" {
value = c.Value
}
if changed := given.take(c.Key, value); changed && read.Load() {
serve(given, held, say)
}
return nil
})
if err != nil {
return fmt.Errorf("watching %s: %w", provisions.PlansRead, err)
// applyMembership serves what a membership says, and says whether it said anything about routes.
//
// A membership with no routes in it is one from a controller older than ADR 0167, and the file stays
// the source rather than every route being withdrawn because a field was absent.
func applyMembership(issued membership.Membership, held *table) bool {
raw, carries := issued.Receives["route"]
if !carries {
return false
}
read.Store(true)
serve(given, held, say)
return nil
}
// serve replaces what the proxy serves with what the plans give it, and says each name it now serves or serves no
// more.
func serve(given *planned, held *table, say func(string, contract.Event)) {
before := held.served()
routes, public := routesOf(given.contributions())
var contributions []contribution
if err := json.Unmarshal(raw, &contributions); err != nil {
log.Printf("the routes in this proxy's membership are not contributions, keeping what is served: %v", err)
return false
}
inside, err := sourcesOf(issued.Mesh)
if err != nil {
log.Printf("the mesh in this proxy's membership is unreadable, keeping what is served: %v", err)
return false
}
routes, public := routesOf(contributions)
held.set(routes, public)
after := held.served()
for _, e := range servedChanges(given.node, before, after) {
say(e.kind, e.Event)
}
log.Printf("serving %d route(s) from the seat's plans: %s", len(routes), strings.Join(held.names(), ", "))
}
// change is one event serve says.
type change struct {
kind string
contract.Event
}
// servedChanges is what changed between two tables' names, each name once, in order: a name now served, or now
// served at another reach or by another module, is served; a name no longer served is withdrawn.
func servedChanges(proxy string, before, after map[string]contract.Event) []change {
var out []change
for _, name := range sortedNames(after) {
if was, ok := before[name]; !ok || was != after[name] {
e := after[name]
e.Proxy = proxy
out = append(out, change{contract.EventServed, e})
}
}
for _, name := range sortedNames(before) {
if _, ok := after[name]; !ok {
out = append(out, change{contract.EventWithdrawn, contract.Event{Proxy: proxy, Name: name}})
}
}
return out
}
func sortedNames(m map[string]contract.Event) []string {
out := make([]string, 0, len(m))
for k := range m {
out = append(out, k)
}
sort.Strings(out)
return out
}
// followInside follows the seat mesh-placement's state and serves an internal name to every node's address on the
// private network it decides, each time it changes (mesh-sdk delivery.InsideAddresses; research 072 rows 33 and 34:
// the membership that carried them went with the assignments stream). A key that does not read keeps who is inside
// as it was.
func followInside(placement planWatch, held *table) error {
keys := delivery.NewPlacementKeys()
var read atomic.Bool
apply := func() {
inside, err := sourcesOf(delivery.InsideAddresses(keys.All()))
if err != nil {
log.Printf("the placement says an address that is not one, so who is inside stays as it was: %v", err)
return
}
held.setInside(inside)
log.Printf("internal names are served to %d machine(s)", len(inside))
}
err := placement.Watch("", func(c stdio.StateChange) error {
if _, err := keys.Put(c.Key, c.Value, c.Revision, c.Op == "delete" || c.Op == "purge"); err != nil {
log.Printf("the placement's key %s does not read, so who is inside stays as it was: %v", c.Key, err)
return nil
}
if read.Load() {
apply()
}
return nil
})
if err != nil {
return err
}
read.Store(true)
apply()
return nil
held.setInside(inside)
log.Printf("serving %d route(s) from the membership, internal names to %d machine(s): %s",
len(routes), len(inside), strings.Join(held.names(), ", "))
return true
}
@@ -1,552 +0,0 @@
package main
import (
"context"
"crypto/x509"
"encoding/json"
"encoding/pem"
"log"
"net/http"
"reflect"
"sort"
"strings"
"sync"
"time"
"golang.org/x/crypto/acme/autocert"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-controller/modules/route-proxy/contract"
)
// What this proxy says on the bus about the names it serves and the names it refuses (issue 774).
//
// **On change, never on a timer.** A route arriving or leaving, a certificate issued or failing, an upstream answer
// that differs from the last one written, a name asked for that is not served: each marks its key, and one writer
// puts the marked keys. An answer that is the same as the last one written is written again only once it is older
// than refreshAfter, so a site answering 200 all day costs one write a minute, not one per request.
//
// Until the bus is reached nothing is written; what happened meanwhile is held and written when it is.
// refreshAfter is how old a written answer, refusal or certificate failure may be before the same one is written
// again with its new time.
const refreshAfter = time.Minute
// mostAsked bounds the names held in memory as asked and not served: a scan sends any name it likes.
const mostAsked = 4096
// newPerMinute is how many names not seen before are kept by name in a minute; past it they are only counted.
const newPerMinute = 5
// kvStore is one of the module's states, as process.State is one.
type kvStore interface {
Put(key string, value any) (uint64, error)
Delete(key string) error
Keys() ([]string, error)
Get(key string) (*stdio.StateEntry, error)
}
// ledger is what this proxy keeps of what it serves and refuses, and writes to the bus.
type ledger struct {
mu sync.Mutex
now func() time.Time
node string
routes, unrouted kvStore
served map[string]*contract.Route
previous map[string]contract.Route // what the bus held for this machine when it was reached
asked map[string]*contract.Unrouted
answerWritten map[string]time.Time
askedWritten map[string]time.Time
dirtyRoutes map[string]bool // a name to put when served, to delete when not
dirtyAsked map[string]bool
// domains are the registrable domains of the names served: a refused name under one is kept by name.
domains map[string]bool
newMinute time.Time
newInMinute int
other contract.Other
otherMinute time.Time
otherInMinute int
otherDirty bool
routesFailing, askFailed bool // a failed write said once, until one succeeds
kick chan struct{}
}
func newLedger() *ledger {
return &ledger{
now: time.Now,
served: map[string]*contract.Route{},
previous: map[string]contract.Route{},
asked: map[string]*contract.Unrouted{},
answerWritten: map[string]time.Time{},
askedWritten: map[string]time.Time{},
dirtyRoutes: map[string]bool{},
dirtyAsked: map[string]bool{},
domains: map[string]bool{},
kick: make(chan struct{}, 1),
}
}
// loop writes whatever was marked, each time something was.
func (l *ledger) loop() {
for range l.kick {
l.flush()
}
}
func (l *ledger) wake() {
select {
case l.kick <- struct{}{}:
default:
}
}
// attach is the bus reached, as node: what the bus held for this machine is read back, so a restart keeps a name's
// certificate and last answer, and everything held is written.
func (l *ledger) attach(node string, routes, unrouted kvStore) {
if l == nil {
return
}
before := map[string]contract.Route{}
if keys, err := routes.Keys(); err != nil {
log.Printf("cannot read back what this proxy said it serves: %v", err)
} else {
for _, key := range keys {
at, name := contract.Split(key)
if at != node {
continue
}
entry, err := routes.Get(key)
if err != nil || entry == nil {
continue
}
var r contract.Route
if json.Unmarshal(entry.Value, &r) == nil {
before[name] = r
}
}
}
l.mu.Lock()
l.node, l.routes, l.unrouted = node, routes, unrouted
for name, r := range before {
l.previous[name] = r
l.dirtyRoutes[name] = true // put again when served, deleted when no longer
if now := l.served[name]; now != nil {
keepWhatWasKnown(now, r)
}
}
for name := range l.served {
l.dirtyRoutes[name] = true
}
for name := range l.asked {
l.dirtyAsked[name] = true
}
l.mu.Unlock()
l.wake()
}
// keepWhatWasKnown carries a certificate and a last answer read back from the bus onto a route that knows none yet.
func keepWhatWasKnown(r *contract.Route, before contract.Route) {
if r.Certificate.Outcome == "" && r.Certificate.NotAfter == nil && before.Certificate.Authority == r.Certificate.Authority {
r.Certificate = before.Certificate
}
if r.LastAnswer == nil {
r.LastAnswer = before.LastAnswer
}
}
// serving is the table this proxy now serves: a name new or changed is written, a name gone is deleted, and a name
// that was asked for unserved and now is served leaves the unrouted state.
func (l *ledger) serving(routes map[string][]rule, public map[string]bool) {
if l == nil {
return
}
l.mu.Lock()
defer l.mu.Unlock()
for name, rules := range routes {
reach := contract.ReachInternal
if public[name] {
reach = contract.ReachPublic
}
targets := targetsOf(rules)
r := l.served[name]
if r == nil {
r = &contract.Route{Name: name}
if before, ok := l.previous[name]; ok {
r.Certificate.Authority = reach
keepWhatWasKnown(r, before)
}
l.served[name] = r
l.dirtyRoutes[name] = true
}
if r.Reach != reach || !reflect.DeepEqual(r.Targets, targets) {
r.Reach, r.Targets = reach, targets
l.dirtyRoutes[name] = true
}
if r.Certificate.Authority != reach {
r.Certificate = contract.Certificate{Authority: reach}
l.dirtyRoutes[name] = true
}
if _, was := l.asked[name]; was {
delete(l.asked, name)
l.dirtyAsked[name] = true
}
}
l.domains = map[string]bool{}
for name := range routes {
if d := contract.Registrable(name); d != "" {
l.domains[d] = true
}
}
for name := range l.served {
if _, still := routes[name]; !still {
delete(l.served, name)
delete(l.answerWritten, name)
l.dirtyRoutes[name] = true
}
}
l.wake()
}
func targetsOf(rules []rule) []contract.Target {
out := make([]contract.Target, 0, len(rules))
for _, r := range rules {
out = append(out, contract.Target{Path: r.path, Module: r.from, Node: r.node, Target: r.target,
Deny: r.policy.deny, Redirect: r.policy.redirectTo, Auth: len(r.policy.users) > 0 || r.policy.sealed != "",
Sealed: r.policy.sealed})
}
return out
}
// answered is an upstream answer to a request for name: its status, or the error that stood in for one.
func (l *ledger) answered(name, path string, status int, failed error) {
if l == nil {
return
}
l.mu.Lock()
defer l.mu.Unlock()
r := l.served[name]
if r == nil {
return
}
now := l.now()
why := ""
if failed != nil {
why = failed.Error()
}
before := r.LastAnswer
r.LastAnswer = &contract.Answer{Status: status, At: now, Error: why, Path: path}
if before != nil && before.Status == status && before.Error == why && now.Sub(l.answerWritten[name]) < refreshAfter {
return
}
l.answerWritten[name] = now
l.dirtyRoutes[name] = true
l.wake()
}
// refused is a name this proxy answered "no route for": in a request, or in a handshake's server name.
//
// **Only a name under a domain this proxy serves is kept by name**, and at most newPerMinute new ones a minute: a scan
// sends any name it likes, and each kept name is a key on the bus. Every other refusal is counted, and the count is
// written once a minute under the proxy's OtherName key. Nothing here scans: forgetting old names is the writer's.
// Who asked is not kept: the journal line beside it is what the machine's intrusion prevention reads.
func (l *ledger) refused(name string, internalOnly, handshake bool) {
if l == nil {
return
}
l.mu.Lock()
defer l.mu.Unlock()
if _, isServed := l.served[name]; isServed && !internalOnly {
return
}
now := l.now()
a := l.asked[name]
if a == nil {
if !contract.Keyable(name) || !l.domains[contract.Registrable(name)] || !l.mayAddName(now) {
l.countOther(now)
return
}
a = &contract.Unrouted{Name: name}
l.asked[name] = a
}
a.AskedAt, a.InternalOnly, a.Handshake = now, internalOnly, handshake
a.Times++
if a.Times > 1 && now.Sub(l.askedWritten[name]) < refreshAfter {
return
}
l.askedWritten[name] = now
l.dirtyAsked[name] = true
l.wake()
}
// mayAddName says whether another new name may be kept this minute, and counts it.
func (l *ledger) mayAddName(now time.Time) bool {
if minute := now.Truncate(time.Minute); !minute.Equal(l.newMinute) {
l.newMinute, l.newInMinute = minute, 0
}
if l.newInMinute >= newPerMinute {
return false
}
l.newInMinute++
return true
}
// countOther counts a refusal not kept by name. A minute's count is written when the next minute's first one comes.
func (l *ledger) countOther(now time.Time) {
minute := now.Truncate(time.Minute)
if !minute.Equal(l.otherMinute) {
if l.otherInMinute > 0 {
l.other.Minute, l.other.LastMinute = l.otherMinute, l.otherInMinute
l.otherDirty = true
l.wake()
}
l.otherMinute, l.otherInMinute = minute, 0
}
l.otherInMinute++
l.other.SinceStart++
}
// forgetOldestAsked drops the names asked longest ago from memory until mostAsked are left; the bus keeps them until
// their time to live ends. Called by the writer, never on a request.
func (l *ledger) forgetOldestAsked() {
if len(l.asked) <= mostAsked {
return
}
names := make([]string, 0, len(l.asked))
for name := range l.asked {
names = append(names, name)
}
sort.Slice(names, func(i, j int) bool { return l.asked[names[i]].AskedAt.Before(l.asked[names[j]].AskedAt) })
for _, name := range names[:len(names)-mostAsked] {
delete(l.asked, name)
delete(l.askedWritten, name)
delete(l.dirtyAsked, name)
}
}
// certified is a certificate an authority issued for name, as autocert stored it.
func (l *ledger) certified(name string, leaf *x509.Certificate) {
if l == nil {
return
}
l.mu.Lock()
defer l.mu.Unlock()
r := l.served[name]
if r == nil {
return
}
now := l.now()
notAfter := leaf.NotAfter.UTC()
r.Certificate.Issuer, r.Certificate.NotAfter = issuerOf(leaf), &notAfter
r.Certificate.RenewedAt, r.Certificate.At = &now, &now
r.Certificate.Outcome, r.Certificate.Error = contract.Issued, ""
l.dirtyRoutes[name] = true
l.wake()
}
// handedOut is the certificate a handshake for name was answered with: written when it is not the one recorded.
func (l *ledger) handedOut(name string, leaf *x509.Certificate) {
if l == nil || leaf == nil {
return
}
l.mu.Lock()
defer l.mu.Unlock()
r := l.served[name]
if r == nil {
return
}
notAfter := leaf.NotAfter.UTC()
if r.Certificate.NotAfter != nil && r.Certificate.NotAfter.Equal(notAfter) && r.Certificate.Outcome != contract.Failed {
return
}
now := l.now()
r.Certificate.Issuer, r.Certificate.NotAfter, r.Certificate.At = issuerOf(leaf), &notAfter, &now
if r.Certificate.Outcome != contract.Issued || r.Certificate.Error != "" {
r.Certificate.Outcome, r.Certificate.Error = contract.Cached, ""
}
l.dirtyRoutes[name] = true
l.wake()
}
// certificateFailed is a handshake for name that found no certificate to answer with, and why.
func (l *ledger) certificateFailed(name string, failed error) {
if l == nil {
return
}
l.mu.Lock()
defer l.mu.Unlock()
r := l.served[name]
if r == nil {
return
}
now := l.now()
same := r.Certificate.Outcome == contract.Failed && r.Certificate.Error == failed.Error()
if same && r.Certificate.At != nil && now.Sub(*r.Certificate.At) < refreshAfter {
return
}
r.Certificate.Outcome, r.Certificate.Error, r.Certificate.At = contract.Failed, failed.Error(), &now
l.dirtyRoutes[name] = true
l.wake()
}
func issuerOf(leaf *x509.Certificate) string {
if cn := leaf.Issuer.CommonName; cn != "" {
if len(leaf.Issuer.Organization) > 0 {
return leaf.Issuer.Organization[0] + " " + cn
}
return cn
}
return leaf.Issuer.String()
}
// flush writes every marked key. A route's write that fails stays marked and is tried again at the next change; a
// refused name's write that fails is dropped, never tried again: the bucket refusing it is full, or the bus is away,
// and either way it is not worth a storm of retries. Each kind of failure is logged once, until a write succeeds.
func (l *ledger) flush() {
type put struct {
store kvStore
key string
value any // nil deletes
route bool
}
l.mu.Lock()
if l.routes == nil {
l.mu.Unlock()
return
}
l.forgetOldestAsked()
var puts []put
for name := range l.dirtyRoutes {
p := put{store: l.routes, key: contract.Key(l.node, name), route: true}
if r := l.served[name]; r != nil {
c := *r
c.Proxy = l.node
p.value = c
}
puts = append(puts, p)
}
for name := range l.dirtyAsked {
p := put{store: l.unrouted, key: contract.Key(l.node, name)}
if a := l.asked[name]; a != nil {
c := *a
c.Proxy = l.node
p.value = c
}
puts = append(puts, p)
}
if l.otherDirty {
o := l.other
o.Proxy = l.node
puts = append(puts, put{store: l.unrouted, key: contract.Key(l.node, contract.OtherName), value: o})
}
l.dirtyRoutes, l.dirtyAsked, l.otherDirty = map[string]bool{}, map[string]bool{}, false
l.mu.Unlock()
for _, p := range puts {
var err error
if p.value == nil {
err = p.store.Delete(p.key)
} else {
_, err = p.store.Put(p.key, p.value)
}
l.mu.Lock()
switch {
case err == nil && p.route:
l.routesFailing = false
case err == nil:
l.askFailed = false
case p.route:
if !l.routesFailing {
log.Printf("cannot write %s to the bus, kept for the next change: %v", p.key, err)
}
l.routesFailing = true
_, name := contract.Split(p.key)
l.dirtyRoutes[name] = true
default:
if !l.askFailed {
log.Printf("cannot write %s to the bus, dropped with every refused name until a write succeeds: %v",
p.key, err)
}
l.askFailed = true
}
l.mu.Unlock()
}
}
// recordingCache is an authority's certificate cache that tells the ledger of every certificate stored in it: the
// one place autocert says it was issued one.
type recordingCache struct {
autocert.Cache
held *table
}
func (c recordingCache) Put(ctx context.Context, key string, data []byte) error {
if err := c.Cache.Put(ctx, key, data); err != nil {
return err
}
name := strings.TrimSuffix(key, "+rsa")
if strings.Contains(name, "+") || !c.held.routed(name) {
return nil // the account key, a challenge token
}
if leaf := leafOf(data); leaf != nil {
c.held.ledger.certified(bareHost(name), leaf)
}
return nil
}
// leafOf is the first certificate in what autocert stores: the private key, then the chain, in PEM.
func leafOf(data []byte) *x509.Certificate {
for {
var block *pem.Block
block, data = pem.Decode(data)
if block == nil {
return nil
}
if block.Type == "CERTIFICATE" {
leaf, err := x509.ParseCertificate(block.Bytes)
if err != nil {
return nil
}
return leaf
}
}
}
// answerRecorder carries a proxied request's answer through and remembers its status and the proxy's error.
type answerRecorder struct {
http.ResponseWriter
status int
err error
}
func (a *answerRecorder) WriteHeader(status int) {
if a.status == 0 {
a.status = status
}
a.ResponseWriter.WriteHeader(status)
}
func (a *answerRecorder) Write(b []byte) (int, error) {
if a.status == 0 {
a.status = http.StatusOK
}
return a.ResponseWriter.Write(b)
}
// Unwrap lets http.ResponseController reach the connection beneath: a flush, an upgrade's hijack.
func (a *answerRecorder) Unwrap() http.ResponseWriter { return a.ResponseWriter }
// proxyFailed is the reverse proxy's answer when upstream gave none: the error kept for the ledger, and a 502.
func proxyFailed(w http.ResponseWriter, r *http.Request, err error) {
if a, ok := w.(*answerRecorder); ok {
a.err = err
}
log.Printf("http: proxy error for %s%s: %v", r.Host, r.URL.Path, err)
w.WriteHeader(http.StatusBadGateway)
}
@@ -1,291 +0,0 @@
package main
import (
"context"
"crypto/ecdsa"
"crypto/elliptic"
"crypto/rand"
"crypto/tls"
"crypto/x509"
"crypto/x509/pkix"
"encoding/json"
"encoding/pem"
"errors"
"fmt"
"math/big"
"net"
"net/http"
"net/http/httptest"
"sort"
"strings"
"testing"
"time"
"golang.org/x/crypto/acme/autocert"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-controller/modules/route-proxy/contract"
)
// fakeState is a state on the bus, in memory, counting what was written to it.
type fakeState struct {
values map[string]json.RawMessage
puts int
tries int // every Put asked, refused or not
fail error // what every Put is refused with, while set
}
func newFakeState() *fakeState { return &fakeState{values: map[string]json.RawMessage{}} }
func (f *fakeState) Put(key string, value any) (uint64, error) {
f.tries++
if f.fail != nil {
return 0, f.fail
}
raw, err := json.Marshal(value)
if err != nil {
return 0, err
}
f.values[key] = raw
f.puts++
return uint64(f.puts), nil
}
func (f *fakeState) Delete(key string) error { delete(f.values, key); return nil }
func (f *fakeState) Keys() ([]string, error) {
out := []string{}
for k := range f.values {
out = append(out, k)
}
sort.Strings(out)
return out, nil
}
func (f *fakeState) Get(key string) (*stdio.StateEntry, error) {
v, ok := f.values[key]
if !ok {
return nil, nil
}
return &stdio.StateEntry{Key: key, Value: v}, nil
}
func (f *fakeState) route(t *testing.T, key string) *contract.Route {
t.Helper()
raw, ok := f.values[key]
if !ok {
return nil
}
var r contract.Route
if err := json.Unmarshal(raw, &r); err != nil {
t.Fatal(err)
}
return &r
}
// issuedPEM is what autocert stores for a name: the key, then the certificate, signed by an issuer named here.
func issuedPEM(t *testing.T, name string, notAfter time.Time) []byte {
t.Helper()
key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
if err != nil {
t.Fatal(err)
}
template := &x509.Certificate{SerialNumber: big.NewInt(1), DNSNames: []string{name},
Subject: pkix.Name{CommonName: name}, Issuer: pkix.Name{CommonName: "R11", Organization: []string{"Let's Encrypt"}},
NotBefore: notAfter.Add(-90 * 24 * time.Hour), NotAfter: notAfter}
issuer := &x509.Certificate{SerialNumber: big.NewInt(2), Subject: template.Issuer, NotBefore: template.NotBefore,
NotAfter: notAfter, IsCA: true, BasicConstraintsValid: true}
der, err := x509.CreateCertificate(rand.Reader, template, issuer, &key.PublicKey, key)
if err != nil {
t.Fatal(err)
}
keyDER, _ := x509.MarshalECPrivateKey(key)
out := pem.EncodeToMemory(&pem.Block{Type: "EC PRIVATE KEY", Bytes: keyDER})
return append(out, pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der})...)
}
// What the proxy tells the bus (issue 774): each served name with where it goes, its certificate and the last
// upstream answer; each name refused; and only on change. Without the ledger nothing reaches the bus, and the
// operator's question "is grafana reachable" is answered with curl.
func TestTheBusIsToldWhatIsServedAndRefused(t *testing.T) {
notAfter := time.Date(2027, 1, 28, 12, 0, 0, 0, time.UTC)
gone := func() string { // an address nothing listens on: the upstream that does not answer
l, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
defer l.Close()
return l.Addr().String()
}()
_, port, _ := net.SplitHostPort(gone)
contributions := []contribution{
{From: "grafana", Node: "anchor", At: "127.0.0.1", Route: contract.Request{Name: "grafana.example", Port: int(mustFloat(t, port))}},
{From: "drive", Node: "home-server", At: "10.0.0.2", Route: contract.Request{InternalName: "drive.internal", Port: 8080}},
}
for _, row := range []struct {
name string
act func(t *testing.T, held *table)
check func(t *testing.T, routes, unrouted *fakeState)
}{
{"a served name is written with the module and node it forwards to",
func(t *testing.T, held *table) {},
func(t *testing.T, routes, _ *fakeState) {
r := routes.route(t, "anchor.grafana.example")
if r == nil || r.Proxy != "anchor" || r.Reach != contract.ReachPublic || len(r.Targets) != 1 ||
r.Targets[0].Module != "grafana" || r.Targets[0].Node != "anchor" || r.Targets[0].Target != "http://127.0.0.1:"+port {
t.Fatalf("grafana's route is not on the bus as served: %+v", r)
}
if d := routes.route(t, "anchor.drive.internal"); d == nil || d.Reach != contract.ReachInternal ||
d.Certificate.Authority != contract.ReachInternal {
t.Fatalf("an internal-only name is not on the bus as internal: %+v", d)
}
}},
{"a withdrawn route is deleted",
func(t *testing.T, held *table) {
routes, public := routesOf(contributions[1:])
held.set(routes, public)
},
func(t *testing.T, routes, _ *fakeState) {
if routes.route(t, "anchor.grafana.example") != nil {
t.Fatal("a route no longer served is still on the bus")
}
}},
{"an upstream that does not answer is written as a 502 with why, once a minute",
func(t *testing.T, held *table) {
for range 3 {
req := httptest.NewRequest(http.MethodGet, "http://grafana.example/login", nil)
handler(held).ServeHTTP(httptest.NewRecorder(), req)
}
},
func(t *testing.T, routes, _ *fakeState) {
r := routes.route(t, "anchor.grafana.example")
if r == nil || r.LastAnswer == nil || r.LastAnswer.Status != http.StatusBadGateway ||
r.LastAnswer.Error == "" || r.LastAnswer.Path != "/login" {
t.Fatalf("the failed answer is not on the bus: %+v", r)
}
if routes.puts != 3 { // the two routes, then the first answer; the same answer again is not written
t.Fatalf("the same answer was written %d times in all, want 3 writes", routes.puts)
}
}},
{"a name under a served domain is written as asked, without who asked, and leaves when it is served",
func(t *testing.T, held *table) {
req := httptest.NewRequest(http.MethodGet, "http://wiki.grafana.example/", nil)
req.RemoteAddr = "192.0.2.7:51000"
handler(held).ServeHTTP(httptest.NewRecorder(), req)
held.ledger.flush()
var a contract.Unrouted
raw := held.ledger.unrouted.(*fakeState).values["anchor.wiki.grafana.example"]
if err := json.Unmarshal(raw, &a); err != nil || a.Times != 1 || a.AskedAt.IsZero() {
t.Fatalf("the refused name is not on the bus as asked: %s", raw)
}
if strings.Contains(string(raw), "192.0.2.7") || strings.Contains(string(raw), `"from"`) {
t.Fatalf("who asked is on the bus: %s", raw)
}
routes, public := routesOf(append(contributions, contribution{From: "grafana", Node: "anchor",
Route: contract.Request{Name: "wiki.grafana.example", Port: 3000}}))
held.set(routes, public)
},
func(t *testing.T, _, unrouted *fakeState) {
if len(unrouted.values) != 0 {
t.Fatalf("a name now served is still written as unrouted: %v", unrouted.values)
}
}},
{"a handshake for a name not routed is written as asked",
func(t *testing.T, held *table) {
pick := certificateFor(held, func(*tls.ClientHelloInfo) (*tls.Certificate, error) {
return nil, errors.New("no public route for \"forgotten.grafana.example\" in this mesh, so no certificate is asked for")
}, nil)
_, _ = pick(&tls.ClientHelloInfo{ServerName: "forgotten.grafana.example"})
},
func(t *testing.T, _, unrouted *fakeState) {
var a contract.Unrouted
if err := json.Unmarshal(unrouted.values["anchor.forgotten.grafana.example"], &a); err != nil || !a.Handshake {
t.Fatalf("a handshake refused for its name is not on the bus: %v", unrouted.values)
}
}},
{"a flood of names writes at most the limit by name, counts the rest once a minute, and never retries a refused write",
func(t *testing.T, held *table) {
clock := time.Date(2026, 10, 30, 21, 0, 0, 0, time.UTC)
held.ledger.now = func() time.Time { return clock }
full := held.ledger.unrouted.(*fakeState)
for i := range 40 {
held.ledger.refused(fmt.Sprintf("x%d.grafana.example", i), false, false) // under a served domain
held.ledger.refused(fmt.Sprintf("x%d.scan.test", i), false, true) // under none
}
held.ledger.flush()
if named := len(full.values); named != newPerMinute {
t.Fatalf("a flood wrote %d names, want the limit %d: %v", named, newPerMinute, full.values)
}
full.fail = errors.New("nats: maximum bytes exceeded")
clock = clock.Add(time.Minute)
held.ledger.refused("y.grafana.example", false, false) // a new minute: one more name,
held.ledger.refused("v.scan.test", false, false) // and the last minute's count
held.ledger.flush()
tries := full.tries
held.ledger.refused("y.grafana.example", false, false)
held.ledger.refused("z.scan.test", false, false)
held.ledger.flush()
if full.tries != tries {
t.Fatalf("writes the bucket refused were tried again: %d tries, then %d", tries, full.tries)
}
full.fail = nil
clock = clock.Add(time.Minute)
held.ledger.refused("w.scan.test", false, false)
},
func(t *testing.T, _, unrouted *fakeState) {
var o contract.Other
if err := json.Unmarshal(unrouted.values["anchor._other"], &o); err != nil || o.LastMinute != 2 || o.SinceStart != 78 {
t.Fatalf("the refusals not kept by name are not counted on the bus: %s", unrouted.values["anchor._other"])
}
}},
{"a certificate an authority issued is written with its issuer, expiry and renewal",
func(t *testing.T, held *table) {
cache := recordingCache{Cache: autocert.DirCache(t.TempDir()), held: held}
if err := cache.Put(context.Background(), "grafana.example", issuedPEM(t, "grafana.example", notAfter)); err != nil {
t.Fatal(err)
}
_ = cache.Put(context.Background(), "acme_account+key", []byte("not a certificate"))
},
func(t *testing.T, routes, _ *fakeState) {
c := routes.route(t, "anchor.grafana.example").Certificate
if c.Outcome != contract.Issued || c.Issuer != "Let's Encrypt R11" || c.NotAfter == nil ||
!c.NotAfter.Equal(notAfter) || c.RenewedAt == nil {
t.Fatalf("the issued certificate is not on the bus: %+v", c)
}
}},
{"a certificate that cannot be had is written with why",
func(t *testing.T, held *table) {
pick := certificateFor(held, func(*tls.ClientHelloInfo) (*tls.Certificate, error) {
return nil, errors.New("acme: urn:ietf:params:acme:error:rateLimited")
}, nil)
_, _ = pick(&tls.ClientHelloInfo{ServerName: "grafana.example"})
},
func(t *testing.T, routes, _ *fakeState) {
c := routes.route(t, "anchor.grafana.example").Certificate
if c.Outcome != contract.Failed || c.Error != "acme: urn:ietf:params:acme:error:rateLimited" {
t.Fatalf("the failed certificate is not on the bus: %+v", c)
}
}},
} {
t.Run(row.name, func(t *testing.T) {
held := newTable()
held.ledger = newLedger()
routes, unrouted := newFakeState(), newFakeState()
held.ledger.attach("anchor", routes, unrouted)
table, public := routesOf(contributions)
held.set(table, public)
held.ledger.flush()
row.act(t, held)
held.ledger.flush()
row.check(t, routes, unrouted)
})
}
}
func mustFloat(t *testing.T, port string) float64 {
t.Helper()
var n float64
if err := json.Unmarshal([]byte(port), &n); err != nil {
t.Fatal(err)
}
return n
}
+219 -118
View File
@@ -1,14 +1,13 @@
// A reverse proxy, in the form the mesh expects one.
//
// **A route is a provision** (novox/hq ADR 0007, 08-connectivity §3; issue 774, research 072 row 34). A
// module that must be reachable declares it requires `route` and contributes the name it wants; the
// proxy provides `route` and serves every consumer whose plan, in the seat mesh-provisioner's state
// `plans`, names it as the route's provider. It is the mirror of a database grant: there the consumer
// **A route is a grant** (novox/hq ADR 0007, 08-connectivity §3). A module that must be reachable
// declares it requires `route` and contributes the name it wants; the proxy provides `route` and
// receives every consumer that asked. It is the mirror of a database grant: there the consumer
// supplies a name and receives credentials, here it supplies a target and receives a name.
//
// **It is an example, not part of the controller.** The controller decides and never touches a
// machine. A real deployment runs whatever proxy it likes — the contract is the plan's entry, not
// this program. What lives here is that contract, written as something that runs so it can be read
// **It is an example, not part of the control plane.** The control plane decides and never touches
// a machine. A real deployment runs whatever proxy it likes — the contract is the file, not this
// program. What lives here is that contract, written as something that runs so it can be read
// rather than described.
//
// **A route also carries what a request arriving at it may do** (novox/hq ADR 0108). The grant used
@@ -17,9 +16,11 @@
// an open middleware surface recreates the thing being replaced and is far harder to narrow later
// than a closed one is to widen.
//
// What it is given, in each plan's entry for the route: every contribution the consumer makes,
// composed by the controller, and where the consumer's machine is on the private network (plans.go).
// Each contribution's values carry the name and port, and optionally:
// What it is given, written by the host from an ordinary declaration:
//
// $ROUTES every consumer, the name it asked for, and where the mesh says that machine is
//
// Each contribution's values carry the name and port as before, and optionally:
//
// path the path prefix this rule is scoped to; absent means every path
// priority which rule wins where two match; higher first, and the order is total
@@ -34,8 +35,8 @@
// outright rather than served unprotected, and a secret that cannot be read makes the route refuse
// rather than open — a gate that cannot check is not a gate that opens.
//
// It follows the plans on the bus rather than being restarted, for the same reason the provisioner
// does: a route arriving or leaving is an ordinary event and must not drop the connections of every
// It re-reads on change rather than being restarted, for the same reason the provisioner does:
// a route arriving or leaving is an ordinary event and must not drop the connections of every
// other workload.
package main
@@ -47,6 +48,7 @@ import (
"crypto/tls"
"crypto/x509"
"encoding/hex"
"encoding/json"
"fmt"
"log"
"net"
@@ -59,13 +61,12 @@ import (
"sort"
"strings"
"sync"
"sync/atomic"
"time"
"golang.org/x/crypto/acme"
"golang.org/x/crypto/acme/autocert"
"golang.org/x/crypto/bcrypt"
"git.novox.be/novox/mesh-controller/modules/route-proxy/contract"
)
// Where public certificates come from when nothing says otherwise.
@@ -118,16 +119,18 @@ func onlyInternalNamesTheMeshSaid(held *table) autocert.HostPolicy {
}
}
// contribution is one route a consumer asked for: the module, its machine, where that machine is, the local name of
// its contribution, and the route, composed by the mesh and read with route-proxy's contract (plans.go).
// what the mesh writes: the contributions file, one entry per consumer.
type given struct {
Given []contribution `json:"given"`
}
type contribution struct {
From string `json:"from"`
Node string `json:"node"`
// At is where that machine is on the private network. The mesh knows it; a proxy that had to
// build it from the node name would be a naming convention copied into every provider.
At string `json:"at"`
Local string `json:"local,omitempty"`
Route contract.Request `json:"route"`
At string `json:"at"`
Values map[string]any `json:"values"`
}
// policy is what a rule does with a request that matched it.
@@ -176,13 +179,11 @@ type rule struct {
// registry is the case that needs it — image layers arrive as single requests of gigabytes, and
// a proxy's own default refuses them long before the workload is reached.
maxRequestBody int64
// from and node are the consumer that contributed the rule and its machine, which the bus is told (issue 774).
from, node string
}
// table is what the proxy is currently serving, replaced whole whenever the plans change what they give it.
// table is what the proxy is currently serving, replaced whole whenever the file changes.
//
// Replaced rather than merged: the plans are the whole truth about who has a route, so merging
// Replaced rather than merged: the file is the whole truth about who has a route, so merging
// would keep serving a name whose module was unassigned — which is the stale-route fault
// 08-connectivity lists as open, reintroduced one level down.
//
@@ -201,8 +202,6 @@ type table struct {
// machine's address on the private network, as the mesh issued it in this proxy's membership
// (novox/hq ADR 0167). Empty until it is issued, and then only the machine itself is inside.
inside sources
// ledger is what this proxy says on the bus about what it serves and refuses (issue 774); nil says nothing.
ledger *ledger
}
// sources is who may be served an internal name: the private network's addresses as the mesh
@@ -290,7 +289,6 @@ func (t *table) set(routes map[string][]rule, public map[string]bool) {
t.to = made
t.public = public
t.mu.Unlock()
t.ledger.serving(made, public)
}
// inOrder puts the rules for one host into the order they are matched in, and does so totally.
@@ -353,7 +351,7 @@ func (t *table) routed(host string) bool {
//
// **The mesh has two name spaces and two authorities** (novox/hq 03-DESIGN/01-to-be/08-connectivity
// §2): a public name is certified by a public CA, an internal one by the mesh's own. This is
// composed only from `to` and `public`, which routesOf already builds correctly — a host never
// composed only from `to` and `public`, which routesFrom already builds correctly — a host never
// lands in both a route's own `name` and only its `internal-name`, so nothing new has to be
// tracked to tell the two apart.
func (t *table) eligibleForInternalACME(host string) bool {
@@ -410,24 +408,6 @@ func (t *table) namesSeenFrom(remote string) []string {
return out
}
// served is every name served, as an event says it: how far it reaches and the module whose rule serves it first.
func (t *table) served() map[string]contract.Event {
t.mu.RLock()
defer t.mu.RUnlock()
out := make(map[string]contract.Event, len(t.to))
for name, rules := range t.to {
e := contract.Event{Name: name, Reach: contract.ReachInternal}
if t.public[name] {
e.Reach = contract.ReachPublic
}
if len(rules) > 0 {
e.Module, e.Node = rules[0].from, rules[0].node
}
out[name] = e
}
return out
}
func (t *table) names() []string {
t.mu.RLock()
defer t.mu.RUnlock()
@@ -447,21 +427,48 @@ func main() {
}
func run() error {
path := strings.TrimSpace(os.Getenv("ROUTES"))
if path == "" {
return fmt.Errorf("ROUTES is not set, so this proxy does not know what it is routing")
}
listen := strings.TrimSpace(os.Getenv("LISTEN"))
if listen == "" {
listen = ":80"
}
// **The seat's plans are the one source of routes** (issue 774, research 072 row 34). Without the bus this proxy
// knows no route, so it refuses to start rather than serving an empty table and saying nothing.
credential := strings.TrimSpace(os.Getenv("MESH_BROKER_FILE"))
if credential == "" {
return fmt.Errorf("MESH_BROKER_FILE is not set, so this proxy cannot read the plans that say what it routes")
}
held := newTable()
held.ledger = newLedger()
go held.ledger.loop()
go follow(credential, held)
// **The bus first, the file until it has spoken** (novox/hq ADR 0167). The membership carries
// the routes and who the mesh is; the file carries the routes alone, so while the proxy reads
// it an internal name is served to this machine and to nobody else — refused, never opened.
fromBus := &atomic.Bool{}
if credential := strings.TrimSpace(os.Getenv("MESH_BROKER_FILE")); credential != "" {
go followMembership(credential, held, fromBus)
} else {
log.Printf("MESH_BROKER_FILE is not set: routes come from %s alone, and a name that is only "+
"internal is served to this machine alone", path)
}
read := func() {
if fromBus.Load() {
return
}
routes, public, err := routesFrom(path)
if err != nil {
// Kept serving what it had. A file being rewritten is momentarily unreadable, and
// dropping every route because one read landed mid-write would turn an ordinary
// event into an outage.
log.Printf("cannot read %s, keeping what is already served: %v", path, err)
return
}
held.set(routes, public)
log.Printf("serving %d route(s): %s", len(routes), strings.Join(held.names(), ", "))
}
read()
go func() {
for range time.Tick(2 * time.Second) {
read()
}
}()
// TLS is opt-in. A proxy with no `TLS_LISTEN` serves plain HTTP exactly as before — which is
// what an internal-only mesh wants, and what the mesh's own certificate authority already
@@ -480,7 +487,7 @@ func run() error {
"to persist, or every restart orders them again")
}
publicManager, err := newManager(cache, issuer(), strings.TrimSpace(os.Getenv("ACME_CA_BUNDLE")),
onlyWhatTheMeshSaid(held), held)
onlyWhatTheMeshSaid(held))
if err != nil {
return err
}
@@ -492,7 +499,7 @@ func run() error {
var internalManager *autocert.Manager
if directory := strings.TrimSpace(os.Getenv("INTERNAL_ACME_DIRECTORY")); directory != "" {
internalManager, err = newManager(cache, directory, strings.TrimSpace(os.Getenv("INTERNAL_ACME_CA_BUNDLE")),
onlyInternalNamesTheMeshSaid(held), held)
onlyInternalNamesTheMeshSaid(held))
if err != nil {
return fmt.Errorf("internal certificate authority: %w", err)
}
@@ -612,7 +619,7 @@ func (p *probedResponse) replayTo(w http.ResponseWriter) {
// one of the public ones — the lab's, or the mesh's own step-ca — serves its own ACME API over TLS
// with a certificate nothing trusts yet. *Skip* would also apply the day this points at a public
// issuer, and nothing would say so; naming a bundle is a deliberate, visible act instead.
func newManager(cache, directory, bundle string, policy autocert.HostPolicy, held *table) (*autocert.Manager, error) {
func newManager(cache, directory, bundle string, policy autocert.HostPolicy) (*autocert.Manager, error) {
client := &acme.Client{DirectoryURL: directory}
var root []byte
if bundle != "" {
@@ -644,7 +651,7 @@ func newManager(cache, directory, bundle string, policy autocert.HostPolicy, hel
// different names because their directories differ.
mine := forThisAuthority(cache, directory, root)
return &autocert.Manager{
Cache: recordingCache{Cache: autocert.DirCache(mine), held: held},
Cache: autocert.DirCache(mine),
Prompt: autocert.AcceptTOS,
HostPolicy: policy,
Client: client,
@@ -693,10 +700,9 @@ func certificateFor(held *table, fromPublic func(*tls.ClientHelloInfo) (*tls.Cer
if internalManager != nil {
fromInternal = internalManager.TLSConfig().GetCertificate
}
pick := func(hello *tls.ClientHelloInfo) (*tls.Certificate, error) {
return func(hello *tls.ClientHelloInfo) (*tls.Certificate, error) {
if held.eligibleForInternalACME(hello.ServerName) {
if hello.Conn != nil && held.hiddenFrom(hello.ServerName, hello.Conn.RemoteAddr().String()) {
held.ledger.refused(bareHost(hello.ServerName), true, true)
return nil, fmt.Errorf("no public route for %q in this mesh, so no certificate is asked for",
hello.ServerName)
}
@@ -706,28 +712,6 @@ func certificateFor(held *table, fromPublic func(*tls.ClientHelloInfo) (*tls.Cer
}
return fromPublic(hello)
}
// What the bus is told of a handshake (issue 774): a routed name's certificate, or why it has none; a name not
// routed here, which a browser asking for a site the mesh forgot to route meets before any request is made.
return func(hello *tls.ClientHelloInfo) (*tls.Certificate, error) {
cert, err := pick(hello)
name := bareHost(hello.ServerName)
switch {
case name == "":
case !held.routed(name):
held.ledger.refused(name, false, true)
case err != nil:
if !strings.HasPrefix(err.Error(), "no public route for") {
held.ledger.certificateFailed(name, err)
}
case cert != nil:
leaf := cert.Leaf
if leaf == nil && len(cert.Certificate) > 0 {
leaf, _ = x509.ParseCertificate(cert.Certificate[0])
}
held.ledger.handedOut(name, leaf)
}
return cert, err
}
}
// newTable is an empty routing table.
@@ -753,9 +737,6 @@ func handler(held *table) http.Handler {
// this proxy's log for exactly that line (novox/hq ADR 0179): the address last, as the
// jail's filter expects it.
log.Printf("refused: no route for %q, asked from %s", r.Host, r.RemoteAddr)
if hidden || !held.routed(r.Host) {
held.ledger.refused(bareHost(r.Host), hidden, false)
}
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
w.WriteHeader(http.StatusNotFound)
if !hidden && held.routed(r.Host) {
@@ -805,14 +786,7 @@ func handler(held *table) http.Handler {
r.Body = http.MaxBytesReader(w, r.Body, matched.maxRequestBody)
}
// The upstream's answer is what the bus is told of this name (issue 774): its status, or why there was none.
answer := &answerRecorder{ResponseWriter: w}
matched.to.ServeHTTP(answer, r)
status := answer.status
if status == 0 && answer.err == nil && r.Header.Get("Upgrade") != "" {
status = http.StatusSwitchingProtocols
}
held.ledger.answered(bareHost(r.Host), r.URL.Path, status, answer.err)
matched.to.ServeHTTP(w, r)
})
}
@@ -867,26 +841,66 @@ func boolByte(b bool) byte {
return 0
}
// routesOf turns the routes the plans give into host → the rules for that host, and which of those hosts is a public
// name — the second is `name`, ACME-eligible; a host reached only through `internal-name` never appears there.
// routesFrom reads what the mesh wrote and turns it into host → the rules for that host, and
// which of those hosts is a public name — the second is `name`, ACME-eligible; a host reached
// only through `internal-name` never appears there.
//
// **A route may carry either name, or both** (novox/hq ADR 0138). How far an endpoint reaches decides which names the
// mesh composes, so an endpoint that reaches only the private network arrives with an `internal-name` and no `name`.
// That is a whole route, not a malformed one: it is served under its internal name and certified by the internal
// authority. Only a route with neither name has nothing to be served under (novox/hq issue 191).
// **A route may carry either name, or both** (novox/hq ADR 0138). How far an endpoint reaches
// decides which names the mesh composes, so an endpoint that reaches only the private network
// arrives with an `internal-name` and no `name`. That is a whole route, not a malformed one: it is
// served under its internal name and certified by the internal authority. Only a route with
// neither name has nothing to be served under (novox/hq issue 191).
func routesFrom(path string) (map[string][]rule, map[string]bool, error) {
raw, err := os.ReadFile(path)
if err != nil {
return nil, nil, err
}
var said given
if err := json.Unmarshal(raw, &said); err != nil {
return nil, nil, err
}
routes, public := routesOf(said.Given)
return routes, public, nil
}
// routesOf turns what the mesh gave into host → the rules for that host, and which hosts are public
// names — the same whether the contributions came in the file or in the membership.
func routesOf(contributions []contribution) (map[string][]rule, map[string]bool) {
out := map[string][]rule{}
public := map[string]bool{}
for _, c := range contributions {
// The contract admitted the route whole (contract.Request.Validate): it names a host, a rule that forwards
// gives a port, its scheme is http or https, its body limit is whole and positive, and its auth names a
// secret's file rather than holding a credential (novox/hq ADR 0108).
r := c.Route
name := strings.TrimSpace(r.Name)
internal := strings.TrimSpace(r.InternalName)
made := rule{path: r.Prefix(), priority: r.Priority}
made.policy.deny, made.policy.redirectTo = r.Deny, r.Redirect
if named := strings.TrimSpace(r.Auth); named != "" {
name, _ := c.Values["name"].(string)
name = strings.TrimSpace(name)
internal, _ := c.Values["internal-name"].(string)
internal = strings.TrimSpace(internal)
if name == "" && internal == "" {
log.Printf("%s on %s asked for a route and named nothing; skipped", c.From, c.Node)
continue
}
// What the route is called in a log line: its public name when it has one.
called := name
if called == "" {
called = internal
}
made := rule{path: asPath(c.Values["path"])}
if p, ok := asWhole(c.Values["priority"]); ok {
made.priority = p
}
made.policy.deny, _ = c.Values["deny"].(bool)
made.policy.redirectTo, _ = c.Values["redirect"].(string)
if named, carried := c.Values["auth"].(string); carried && strings.TrimSpace(named) != "" {
// **A declaration names a secret; it never holds one** (ADR 0108). Refused rather than
// tolerated, and the whole rule is dropped rather than served unprotected — the
// rejected option cannot come back by accident, which is the failure this check exists
// to make impossible.
if looksLikeACredential(named) {
log.Printf("%s on %s declared route %q with a credential in the declaration rather "+
"than the name of a secret; the whole route is refused (novox/hq ADR 0108)",
c.From, c.Node, called)
continue
}
users, err := usersFrom(named)
if err != nil {
// Fail closed: the rule is kept so the name stays routed and answers, and it
@@ -898,7 +912,13 @@ func routesOf(contributions []contribution) (map[string][]rule, map[string]bool)
}
// Only a rule that actually proxies needs somewhere to send the request.
if r.Forwards() {
if !made.policy.deny && made.policy.redirectTo == "" {
port, ok := asPort(c.Values["port"])
if !ok {
log.Printf("%s on %s asked for route %q and gave no usable port; skipped",
c.From, c.Node, called)
continue
}
// Where the mesh says that machine is. Empty means it is this one — a workload beside
// the proxy is ordinary, and reaching it over loopback is both correct and the only
// thing that works when there is no private network.
@@ -908,14 +928,33 @@ func routesOf(contributions []contribution) (map[string][]rule, map[string]bool)
}
// http unless the contribution says otherwise. A backend that terminates its own TLS
// with a certificate this proxy has no reason to trust — Mailu's webmail front is the
// first of these — is the reason `insecure` exists, and it stays the exception.
made.insecure = r.Insecure
if r.MaxRequestBody != nil {
made.maxRequestBody = *r.MaxRequestBody
// first of these — is the reason `insecure` exists, and it stays the exception: every
// other target the mesh hands this proxy is a plain workload on the private network.
scheme, _ := c.Values["scheme"].(string)
scheme = strings.ToLower(strings.TrimSpace(scheme))
if scheme == "" {
scheme = "http"
}
made.target = fmt.Sprintf("%s://%s:%d", r.SchemeOrDefault(), at, r.Port)
if scheme != "http" && scheme != "https" {
log.Printf("%s on %s asked for route %q with scheme %q, which is neither http "+
"nor https; skipped", c.From, c.Node, called, scheme)
continue
}
made.insecure, _ = c.Values["insecure"].(bool)
// A limit this proxy cannot read is a route it does not serve, named like a port that
// is not a port. Serving it without the limit would carry exactly what the module said
// not to carry, and report success doing it.
if asked, said := c.Values["max-request-body"]; said {
bytes, whole := asWhole(asked)
if !whole || bytes <= 0 {
log.Printf("%s on %s asked for route %q with a max-request-body of %v, which is "+
"not a whole positive number of bytes; skipped", c.From, c.Node, called, asked)
continue
}
made.maxRequestBody = int64(bytes)
}
made.target = fmt.Sprintf("%s://%s:%d", scheme, at, port)
}
made.from, made.node = c.From, c.Node
if name != "" {
host := strings.ToLower(name)
@@ -927,8 +966,9 @@ func routesOf(contributions []contribution) (map[string][]rule, map[string]bool)
// answered both for one route, as a convenience (reaching a service over the VPN without a
// public TLS round trip), not as an access boundary; composing it here restores exactly
// that, nothing more. Absent whenever the node composed no internal name (novox/hq ADR
// 0056's internalDomain half). And the only name, when the endpoint reaches no further than
// the private network.
// 0056's internalDomain half) — the same "nothing to join a label to" case the public name
// already has. And the only name, when the endpoint reaches no further than the private
// network.
if internal != "" {
out[strings.ToLower(internal)] = append(out[strings.ToLower(internal)], made)
}
@@ -936,6 +976,50 @@ func routesOf(contributions []contribution) (map[string][]rule, map[string]bool)
return out, public
}
// asWhole is any whole number the mesh wrote, whatever its magnitude.
//
// **Not asPort.** Priority was read with the port reader first, which caps at 65535 — so a rule
// declared at a priority above that silently became priority 0 and stopped shadowing the route it
// exists to shadow. The one real rule this has to reproduce is declared at 100000, so the bug was
// exactly load-bearing. A priority is an ordering, not a port: it has no range.
func asWhole(v any) (int, bool) {
switch n := v.(type) {
case float64:
// JSON makes a float of every number, so a non-integral one was not meant as a priority.
if n != float64(int(n)) {
return 0, false
}
return int(n), true
case int:
return n, true
}
return 0, false
}
// asPath is the path prefix a rule is scoped to, or "" for every path.
func asPath(v any) string {
p, _ := v.(string)
p = strings.TrimSpace(p)
if p == "" {
return ""
}
if !strings.HasPrefix(p, "/") {
p = "/" + p
}
return p
}
// looksLikeACredential is the check that keeps a secret out of a declaration.
//
// It errs towards refusing: a value holding a `:` (the htpasswd separator) or opening with a bcrypt
// identifier is a credential, not a path, and no filesystem path the mesh writes needs either. A
// false refusal is a loud log and a route that does not serve; a false accept is a credential
// committed to a declaration, which is the thing being prevented.
func looksLikeACredential(v string) bool {
v = strings.TrimSpace(v)
return strings.Contains(v, ":") || strings.HasPrefix(v, "$2")
}
// usersFrom reads the credentials the mesh mounted, in the one format every htpasswd already is.
func usersFrom(path string) (map[string]string, error) {
raw, err := os.ReadFile(path)
@@ -960,6 +1044,23 @@ func usersFrom(path string) (map[string]string, error) {
return users, nil
}
// asPort accepts what JSON makes of a number, which is a float even when it was written 8080.
func asPort(v any) (int, bool) {
switch n := v.(type) {
case float64:
if n < 1 || n > 65535 {
return 0, false
}
return int(n), true
case int:
if n < 1 || n > 65535 {
return 0, false
}
return n, true
}
return 0, false
}
// towards proxies to one backend and tells it what the client asked: **X-Forwarded-Proto, -Host and
// -For**, set from the request this proxy received. A backend that builds its own addresses — a forge
// writing its clone URL into a page, a login redirect — otherwise sees the plain HTTP hop from this
@@ -971,5 +1072,5 @@ func towards(where *url.URL) *httputil.ReverseProxy {
pr.SetURL(where)
pr.Out.Host = pr.In.Host
pr.SetXForwarded()
}, ErrorHandler: proxyFailed}
}}
}
@@ -1,147 +0,0 @@
package main
// The routes this proxy serves, read from the seat mesh-provisioner's plans and nothing else (issue 774; research 072
// row 34, the operator on 2026-10-31: "there is only 1 path, the other is simply deprecated/invalid").
//
// **A route is a provision, in route-proxy's own contract.** A module that must be reachable requires `route`; the
// controller composes its plan on its node, and the plan's entry for the route names this proxy as its provider and
// carries what the module contributes, composed (its names joined to the node's domains, its endpoint's port), and
// where the module's machine is on the private network, as one opaque payload the seat never reads (mesh-sdk
// catalogue.PlanProvision.Contributed). The proxy decodes each contribution with its contract (contract.DecodeRequest)
// and serves every entry naming it. There is no file and no second source: a route the plans do not name is not
// served, and a route the contract refuses is said refused (contract.EventRefused), never served half.
import (
"fmt"
"log"
"sort"
"sync"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/provisions"
"git.novox.be/novox/mesh-controller/modules/route-proxy/contract"
)
// planned is the routes the seat's plans give this proxy on its node, by plan key, kept so a plan that cannot be read
// leaves what it gave served (a bind missed would read as a route withdrawn).
type planned struct {
node string
// said is where a refusal is told: the bus's events, nothing in a test that does not ask.
said func(eventType string, e contract.Event)
mu sync.Mutex
by map[string][]contribution
}
func newPlanned(node string, said func(string, contract.Event)) *planned {
if said == nil {
said = func(string, contract.Event) {}
}
return &planned{node: node, said: said, by: map[string][]contribution{}}
}
// take is one change of a plan's key: its value, or nil when the key was deleted. It answers whether what is served
// changed, and says on the log and the bus what it could not serve.
func (p *planned) take(key string, value []byte) bool {
p.mu.Lock()
defer p.mu.Unlock()
if value == nil {
_, had := p.by[key]
delete(p.by, key)
return had
}
gave, refused, err := routesOfPlan(p.node, key, value)
if err != nil {
log.Printf("the plan %s is not readable, so what it routes is kept as it was: %v", key, err)
return false
}
for _, r := range refused {
log.Printf("the plan %s gives %s on %s a route this proxy does not serve: %s", key, r.Module, r.Node, r.Why)
p.said(contract.EventRefused, r)
}
if len(gave) == 0 {
_, had := p.by[key]
delete(p.by, key)
return had
}
p.by[key] = gave
return true
}
// contributions is every route the plans give this proxy, in the order of their plans' keys.
func (p *planned) contributions() []contribution {
p.mu.Lock()
defer p.mu.Unlock()
keys := make([]string, 0, len(p.by))
for k := range p.by {
keys = append(keys, k)
}
sort.Strings(keys)
var out []contribution
for _, k := range keys {
out = append(out, p.by[k]...)
}
return out
}
// routesOfPlan is the routes one plan gives the proxy on node: each contribution the plan's entries naming this proxy
// on node as the route's provider carry, decoded with the contract, reached on its own machine over loopback and on
// another at the address the plan says. A plan of another module, or one routing elsewhere, gives none. Refused is
// what the plan names and the proxy does not serve, each with why: an entry carrying nothing to serve, a payload or a
// route the contract does not read, or a consumer elsewhere with no address on the private network.
func routesOfPlan(node, key string, value []byte) ([]contribution, []contract.Event, error) {
plan, err := provisions.DecodePlan(value)
if err != nil {
return nil, nil, err
}
if plan.Node+"."+plan.Module != key {
return nil, nil, fmt.Errorf("it is the plan of %s.%s, under the key %s", plan.Node, plan.Module, key)
}
var out []contribution
var refused []contract.Event
refuse := func(local, why string) {
refused = append(refused, contract.Event{Proxy: node, Module: plan.Module, Node: plan.Node, Local: local, Why: why})
}
named, carried := false, false
for _, e := range plan.Plan.Provisions {
if !routesHere(node, e) {
continue
}
named = true
// What the module contributes is on the requirement's first entry alone, however many local names it has.
if len(e.Contributed) == 0 {
continue
}
carried = true
given, err := catalogue.DecodeContributed(e.Contributed)
if err != nil {
refuse("", err.Error())
continue
}
at := ""
if plan.Node != node {
if given.At == "" {
refuse("", fmt.Sprintf("it runs on %s and its plan says no address on the private network for it", plan.Node))
continue
}
at = given.At
}
for _, one := range given.Each {
route, err := contract.DecodeRequest(one.Values)
if err != nil {
refuse(one.Local, err.Error())
continue
}
out = append(out, contribution{From: plan.Module, Node: plan.Node, At: at, Local: one.Local, Route: route})
}
}
if named && !carried {
refuse("", "its plan names this proxy for its route and carries nothing to serve")
}
return out, refused, nil
}
// routesHere is whether a plan's entry is a route this proxy on node provides.
func routesHere(node string, e catalogue.PlanProvision) bool {
return e.Provision == contract.Provision && e.Provider == node && e.ProviderModule == contract.Module
}
@@ -57,7 +57,11 @@ func ask(t *testing.T, proxy, name, path string, auth [2]string) *http.Response
func proxyFor(t *testing.T, routesJSON string) string {
t.Helper()
routes, public, err := routesFrom(routesJSON)
path := filepath.Join(t.TempDir(), "routes.json")
if err := os.WriteFile(path, []byte(routesJSON), 0o644); err != nil {
t.Fatal(err)
}
routes, public, err := routesFrom(path)
if err != nil {
t.Fatal(err)
}
@@ -149,7 +153,11 @@ func TestACredentialInTheDeclarationIsRefusedRatherThanServed(t *testing.T) {
`{"given":[{"from":"c","node":"n","at":"127.0.0.1","values":{"name":"c.example","port":8080,"auth":"$2a$10$abcdefghijklmnopqrstuv"}}]}`,
}
for _, body := range inline {
routes, _, err := routesFrom(body)
path := filepath.Join(t.TempDir(), "routes.json")
if err := os.WriteFile(path, []byte(body), 0o644); err != nil {
t.Fatal(err)
}
routes, _, err := routesFrom(path)
if err != nil {
t.Fatal(err)
}
+34 -159
View File
@@ -9,16 +9,10 @@ import (
"net/http"
"net/http/httptest"
"net/url"
"reflect"
"slices"
"sort"
"strings"
"testing"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-controller/modules/route-proxy/contract"
"git.novox.be/novox/mesh-sdk/go/membership"
)
// behind is a workload the proxy can send to, and a table routing one public name and one
@@ -32,12 +26,12 @@ func behind(t *testing.T, mesh ...string) *table {
at, _ := url.Parse(workload.URL)
host, port, _ := net.SplitHostPort(at.Host)
routes, public, err := routesFrom(fmt.Sprintf(`{"given":[
routes, public, err := routesFrom(write(t, fmt.Sprintf(`{"given":[
{"from":"app","node":"anchor","at":%q,
"values":{"name":"app.example","internal-name":"app.anchor.internal","port":%s}},
{"from":"admin","node":"anchor","at":%q,
"values":{"internal-name":"admin.anchor.internal","port":%s}}
]}`, host, port, host, port))
]}`, host, port, host, port)))
if err != nil {
t.Fatal(err)
}
@@ -171,161 +165,42 @@ func TestTheMeshIsReadAsAddressesAndRanges(t *testing.T) {
}
}
// fakePlans is the seat's plans as a watch hands them: every current key, then each change after Watch returned.
type fakePlans struct {
current map[string]string
handle func(stdio.StateChange) error
}
func (f *fakePlans) Watch(_ string, handle func(stdio.StateChange) error) error {
f.handle = handle
for k, v := range f.current {
if err := handle(stdio.StateChange{Key: k, Op: "put", Value: json.RawMessage(v), Current: true}); err != nil {
return err
}
}
return nil
}
// plan is a plan of the seat mesh-provisioner for module on node, its entries as given.
func plan(node, module, entries string) string {
return fmt.Sprintf(`{"v":1,"node":%q,"module":%q,"plan":{"templates":{"source":"","digest":""},"files":null,`+
`"provisions":[%s]}}`, node, module, entries)
}
// contributed is a plan entry's opaque payload: the consumer's machine at at, and each contribution by local name.
func contributed(at string, each ...string) string {
var parts []string
for i := 0; i+1 < len(each); i += 2 {
parts = append(parts, fmt.Sprintf(`{"local":%q,"values":%s}`, each[i], each[i+1]))
}
return fmt.Sprintf(`"contributed":{"v":1,"at":%q,"each":[%s]}`, at, strings.Join(parts, ","))
}
// What the plans give is what is served, and nothing else (issue 774, research 072 row 34): each entry naming this
// proxy on its node as the route's provider, every contribution of it read with route-proxy's contract, reached over
// loopback on this machine and at the plan's address on another; an entry routing elsewhere, carrying nothing, naming
// no address or a route the contract refuses is not served, and is said refused. A change is served as it comes and
// said served or withdrawn, a plan that cannot be read keeps what it gave, and a deleted one withdraws it. The
// placement says only who is inside: an internal-only name is served to the machines it decides addresses for.
func TestThePlansAreServedAndThePlacementSaysWhoIsInside(t *testing.T) {
const route = `{"provision":"route","provider":"anchor","provider-module":"route-proxy",`
gitea := route + contributed("10.10.0.7",
"web", `{"name":"git.example","internal-name":"git.anchor.internal","port":3000}`,
"api", `{"internal-name":"git-api.anchor.internal","port":3001}`,
"refused", `{"name":"bad.example","port":3002,"host":"x"}`) + `}`
plans := &fakePlans{current: map[string]string{
"laptop.gitea": plan("laptop", "gitea", gitea+`,{"provision":"postgres-database","provider":"anchor",`+
`"provider-module":"postgres",`+contributed("10.10.0.7", "", `{"name":"gitea"}`)+`}`),
"anchor.blog": plan("anchor", "blog", route+contributed("10.10.0.1", "", `{"name":"blog.example","port":2368}`)+`}`),
"anchor.other": plan("anchor", "other", `{"provision":"route","provider":"edge","provider-module":"route-proxy",`+
contributed("", "", `{"name":"other.example","port":80}`)+`}`),
"anchor.older": plan("anchor", "older", route+`"local":""}`),
"laptop.lost": plan("laptop", "lost", route+contributed("", "", `{"name":"lost.example","port":80}`)+`}`),
}}
// What the mesh issues is what is served: the routes in the membership, internal names to the
// machines it names (novox/hq ADR 0167).
func TestAMembershipIsServedAsIssued(t *testing.T) {
held := newTable()
var said []string
say := func(kind string, e contract.Event) {
said = append(said, kind+" "+e.Proxy+" "+e.Name+" "+e.Module+" "+e.Local+" "+e.Why)
}
if err := followPlans(plans, newPlanned("anchor", say), held, say); err != nil {
t.Fatal(err)
}
served := func() map[string]string {
out := map[string]string{}
held.mu.RLock()
defer held.mu.RUnlock()
for host, rules := range held.to {
out[host] = rules[0].target
}
return out
}
want := map[string]string{"git.example": "http://10.10.0.7:3000", "git.anchor.internal": "http://10.10.0.7:3000",
"git-api.anchor.internal": "http://10.10.0.7:3001", "blog.example": "http://127.0.0.1:2368"}
if got := served(); !reflect.DeepEqual(got, want) {
t.Fatalf("serving %v, want %v", got, want)
}
if !held.eligibleForACME("git.example") || held.eligibleForACME("git-api.anchor.internal") {
t.Error("a public name is not certified publicly, or an internal-only one is")
}
// Each refusal said with why, and each name served said once.
sort.Strings(said)
for _, w := range []string{
"route.refused anchor gitea refused not a route this contract reads: json: unknown field \"host\"",
"route.refused anchor lost it runs on laptop and its plan says no address on the private network for it",
"route.refused anchor older its plan names this proxy for its route and carries nothing to serve",
"route.served anchor blog.example blog ",
"route.served anchor git-api.anchor.internal gitea ",
} {
if !slices.Contains(said, w) {
t.Errorf("not said %q, said:\n%s", w, strings.Join(said, "\n"))
}
}
if len(said) != 7 {
t.Errorf("said %d events, want 3 refusals and 4 names served:\n%s", len(said), strings.Join(said, "\n"))
}
for _, row := range []struct {
name, key, value string
served []string
withdrawn []string
say []string
}{
{"a route arrives", "anchor.new", plan("anchor", "new", route+contributed("", "", `{"name":"new.example","port":81}`)+`}`),
[]string{"new.example"}, nil, []string{"route.served anchor new.example new "}},
{"an unreadable plan keeps what it gave", "laptop.gitea", `{"v":1,`, []string{"git.example"}, nil, nil},
{"a deleted plan withdraws its routes", "laptop.gitea", "", nil, []string{"git.example", "git-api.anchor.internal"},
[]string{"route.withdrawn anchor git-api.anchor.internal ", "route.withdrawn anchor git.anchor.internal ",
"route.withdrawn anchor git.example "}},
} {
said = nil
change := stdio.StateChange{Key: row.key, Op: "put", Value: json.RawMessage(row.value)}
if row.value == "" {
change = stdio.StateChange{Key: row.key, Op: "delete"}
}
if err := plans.handle(change); err != nil {
t.Fatal(err)
}
got := served()
for _, host := range row.served {
if _, ok := got[host]; !ok {
t.Errorf("%s: %s is not served (%v)", row.name, host, got)
}
}
for _, host := range row.withdrawn {
if _, ok := got[host]; ok {
t.Errorf("%s: %s is still served", row.name, host)
}
}
if !slices.Equal(said, row.say) {
t.Errorf("%s: said %q, want %q", row.name, said, row.say)
}
}
held = behind(t)
if code, _ := askFrom(held, "admin.anchor.internal", "10.10.0.7:1"); code != http.StatusNotFound {
t.Errorf("before the placement is read, a machine of the mesh was served an internal-only name: %d", code)
}
laptop, err := delivery.PlacementValues(delivery.Placement{Node: "laptop", Decided: delivery.NodeDecided{Address: "10.10.0.7"}})
if err != nil {
t.Fatal(err)
}
placement := &fakePlans{current: map[string]string{}}
for k, v := range laptop {
placement.current[k] = string(v)
}
if err := followInside(placement, held); err != nil {
t.Fatal(err)
took := applyMembership(membership.Membership{
Receives: map[string]json.RawMessage{"route": json.RawMessage(`[
{"from":"admin","node":"anchor","at":"anchor.internal",
"values":{"internal-name":"admin.anchor.internal","port":8080}}]`)},
Mesh: []string{"10.10.0.7"},
}, held)
if !took {
t.Fatal("a membership carrying routes was not applied")
}
if code, _ := askFrom(held, "admin.anchor.internal", "10.10.0.7:1"); code == http.StatusNotFound {
t.Error("a machine the placement decides an address for was refused the internal-only route")
t.Error("a machine the membership names was refused the internal-only route")
}
if code, _ := askFrom(held, "admin.anchor.internal", "10.10.0.8:1"); code != http.StatusNotFound {
t.Error("a machine the placement does not name was served the internal-only route")
if code, _ := askFrom(held, "admin.anchor.internal", "10.10.0.9:1"); code != http.StatusNotFound {
t.Errorf("a machine the membership does not name was served the internal-only route: %d", code)
}
if err := placement.handle(stdio.StateChange{Key: "laptop.node", Op: "put", Value: json.RawMessage(`{`)}); err != nil {
t.Fatal(err)
}
// A membership that says nothing about routes is one from a controller that does not issue them,
// and changes nothing: the file stays the source rather than every route being withdrawn.
func TestAMembershipWithoutRoutesLeavesTheFileServing(t *testing.T) {
held := behind(t, "10.10.0.7")
before := held.names()
if applyMembership(membership.Membership{Mesh: []string{"10.10.0.7"}}, held) {
t.Error("a membership without routes was taken as the source of routes")
}
if code, _ := askFrom(held, "admin.anchor.internal", "10.10.0.7:1"); code == http.StatusNotFound {
t.Error("an unreadable placement key changed who is inside")
if got := held.names(); strings.Join(got, ",") != strings.Join(before, ",") {
t.Errorf("a membership without routes changed what is served: %v, was %v", got, before)
}
if applyMembership(membership.Membership{
Receives: map[string]json.RawMessage{"route": json.RawMessage(`[]`)},
Mesh: []string{"not-an-address"},
}, held) {
t.Error("a membership whose mesh cannot be read was applied")
}
}
@@ -2,40 +2,23 @@ package main
import (
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
"git.novox.be/novox/mesh-controller/modules/route-proxy/contract"
)
// routesFrom is the table a test's routes make, written as `{"given":[{"from","node","at","values"}…]}`: the
// contributions the plans give the proxy, each read with route-proxy's contract as routesOfPlan reads it, and one the
// contract refuses left out, as routesOfPlan leaves it out.
func routesFrom(body string) (map[string][]rule, map[string]bool, error) {
var said struct {
Given []struct {
From, Node, At string
Values json.RawMessage
} `json:"given"`
func write(t *testing.T, body string) string {
t.Helper()
path := filepath.Join(t.TempDir(), "routes.json")
if err := os.WriteFile(path, []byte(body), 0o644); err != nil {
t.Fatal(err)
}
if err := json.Unmarshal([]byte(body), &said); err != nil {
return nil, nil, err
}
var given []contribution
for _, g := range said.Given {
route, err := contract.DecodeRequest(g.Values)
if err != nil {
continue
}
given = append(given, contribution{From: g.From, Node: g.Node, At: g.At, Route: route})
}
routes, public := routesOf(given)
return routes, public, nil
return path
}
// plain is the table an ordinary set of routes makes: one host, one target, no policy.
@@ -68,9 +51,9 @@ func targetOf(routes map[string][]rule, host string) string {
// A route is a grant: the consumer supplies a target, and where that machine is comes from the
// mesh rather than from a naming convention the proxy has to know.
func TestARouteGoesToWhereTheMeshSaysTheConsumerIs(t *testing.T) {
routes, _, err := routesFrom(`{"contributions":1,"requirement":"route","given":[
routes, _, err := routesFrom(write(t, `{"contributions":1,"requirement":"route","given":[
{"from":"app","node":"laptop","at":"laptop.internal","values":{"name":"App.Example","port":8080}}
]}`)
]}`))
if err != nil {
t.Fatal(err)
}
@@ -85,10 +68,10 @@ func TestARouteGoesToWhereTheMeshSaysTheConsumerIs(t *testing.T) {
// target — the same convenience a predecessor proxy gave for reaching a service over the VPN
// without a public TLS round trip.
func TestARouteWithAnInternalNameIsReachableUnderBoth(t *testing.T) {
routes, public, err := routesFrom(`{"given":[
routes, public, err := routesFrom(write(t, `{"given":[
{"from":"app","node":"anchor","at":"anchor.internal",
"values":{"name":"app.example","internal-name":"app.anchor.internal","port":8080}}
]}`)
]}`))
if err != nil {
t.Fatal(err)
}
@@ -111,10 +94,10 @@ func TestARouteWithAnInternalNameIsReachableUnderBoth(t *testing.T) {
// one (novox/hq ADR 0138), and is served under that name rather than skipped as naming nothing —
// skipping it left every internal-only module unreachable by name (novox/hq issue 191).
func TestARouteWithOnlyAnInternalNameIsServed(t *testing.T) {
routes, public, err := routesFrom(`{"given":[
routes, public, err := routesFrom(write(t, `{"given":[
{"from":"app","node":"anchor","at":"anchor.internal",
"values":{"internal-name":"App.Anchor.Internal","port":8443,"scheme":"https","insecure":true}}
]}`)
]}`))
if err != nil {
t.Fatal(err)
}
@@ -138,11 +121,24 @@ func TestARouteWithOnlyAnInternalNameIsServed(t *testing.T) {
}
}
// A route with neither name has nothing to be served under, and is still skipped.
func TestARouteWithNeitherNameIsSkipped(t *testing.T) {
routes, public, err := routesFrom(write(t, `{"given":[
{"from":"app","node":"anchor","at":"anchor.internal","values":{"internal-name":" ","port":8080}}
]}`))
if err != nil {
t.Fatal(err)
}
if len(routes) != 0 || len(public) != 0 {
t.Errorf("a route that named nothing was served: %v %v", routes, public)
}
}
// A route with no internal-name composed gets no second host — the ordinary case, unchanged.
func TestARouteWithNoInternalNameGetsNoAlias(t *testing.T) {
routes, _, err := routesFrom(`{"given":[
routes, _, err := routesFrom(write(t, `{"given":[
{"from":"app","node":"anchor","values":{"name":"app.example","port":8080}}
]}`)
]}`))
if err != nil {
t.Fatal(err)
}
@@ -154,9 +150,9 @@ func TestARouteWithNoInternalNameGetsNoAlias(t *testing.T) {
// A workload beside the proxy is ordinary, and reaching it over loopback is both correct and the
// only thing that works when there is no private network.
func TestAConsumerOnTheProxysOwnMachineIsReachedOverLoopback(t *testing.T) {
routes, _, err := routesFrom(`{"given":[
routes, _, err := routesFrom(write(t, `{"given":[
{"from":"app","node":"anchor","values":{"name":"app.example","port":9000}}
]}`)
]}`))
if err != nil {
t.Fatal(err)
}
@@ -165,13 +161,28 @@ func TestAConsumerOnTheProxysOwnMachineIsReachedOverLoopback(t *testing.T) {
}
}
// Skipped rather than served wrongly. A route with no port would proxy to :0.
func TestAContributionMissingWhatARouteNeedsIsSkipped(t *testing.T) {
routes, _, err := routesFrom(write(t, `{"given":[
{"from":"a","node":"n","at":"n.internal","values":{"name":"no-port.example"}},
{"from":"b","node":"n","at":"n.internal","values":{"port":8080}},
{"from":"c","node":"n","at":"n.internal","values":{"name":"fine.example","port":8080}}
]}`))
if err != nil {
t.Fatal(err)
}
if len(routes) != 1 || targetOf(routes, "fine.example") == "" {
t.Fatalf("an unusable contribution was served: %v", routes)
}
}
// A route may name a target reached over https, for a backend that terminates its own TLS — the
// shape Mailu's webmail front needs, which this proxy reaches as a plain workload otherwise.
func TestARouteMayTargetHttps(t *testing.T) {
routes, _, err := routesFrom(`{"given":[
routes, _, err := routesFrom(write(t, `{"given":[
{"from":"mail","node":"anchor","at":"anchor.internal",
"values":{"name":"mail.example","port":7443,"scheme":"https","insecure":true}}
]}`)
]}`))
if err != nil {
t.Fatal(err)
}
@@ -183,6 +194,19 @@ func TestARouteMayTargetHttps(t *testing.T) {
}
}
// A scheme that is neither http nor https is refused rather than guessed at.
func TestARouteWithAnUnknownSchemeIsSkipped(t *testing.T) {
routes, _, err := routesFrom(write(t, `{"given":[
{"from":"a","node":"n","at":"n.internal","values":{"name":"bad.example","port":80,"scheme":"ftp"}}
]}`))
if err != nil {
t.Fatal(err)
}
if len(routes) != 0 {
t.Fatalf("a route with an unusable scheme was served: %v", routes)
}
}
// End to end: a backend terminating TLS with a certificate nothing would ordinarily trust is still
// reached when the route declared `insecure`, and the response comes back through unmodified.
func TestTheProxyReachesAnInsecureHttpsBackend(t *testing.T) {
@@ -336,10 +360,10 @@ func TestNoCertificateIsAskedForOnAnUnroutedName(t *testing.T) {
// public CA can validate a private name, and asking anyway would only spend the account's rate
// limit on an order that can never succeed.
func TestNoCertificateIsAskedForOnAnInternalAlias(t *testing.T) {
routes, public, err := routesFrom(`{"given":[
routes, public, err := routesFrom(write(t, `{"given":[
{"from":"app","node":"anchor","at":"anchor.internal",
"values":{"name":"app.example","internal-name":"app.anchor.internal","port":8080}}
]}`)
]}`))
if err != nil {
t.Fatal(err)
}
@@ -358,10 +382,10 @@ func TestNoCertificateIsAskedForOnAnInternalAlias(t *testing.T) {
// A certificate is asked of the *internal* authority only for a name that is routed here and is
// not a route's own public name — the internal-network alias, never the route it accompanies.
func TestTheInternalAuthorityOnlyCertifiesInternalOnlyAliases(t *testing.T) {
routes, public, err := routesFrom(`{"given":[
routes, public, err := routesFrom(write(t, `{"given":[
{"from":"app","node":"anchor","at":"anchor.internal",
"values":{"name":"app.example","internal-name":"app.anchor.internal","port":8080}}
]}`)
]}`))
if err != nil {
t.Fatal(err)
}
@@ -402,10 +426,10 @@ func TestWithdrawingARouteWithdrawsItsCertificate(t *testing.T) {
// default refuses them long before the workload is reached. It is configuration beside `insecure`,
// not a fifth policy — novox/hq ADR 0108 closed that set at four.
func TestARouteMayLimitTheBodyItCarries(t *testing.T) {
routes, _, err := routesFrom(`{"given":[
routes, _, err := routesFrom(write(t, `{"given":[
{"from":"registry","node":"anchor","at":"anchor.internal",
"values":{"name":"images.example","port":5000,"max-request-body":21474836480}}
]}`)
]}`))
if err != nil {
t.Fatal(err)
}
@@ -420,9 +444,9 @@ func TestARouteMayLimitTheBodyItCarries(t *testing.T) {
// Saying nothing leaves the route unlimited, which is what every route already got.
func TestARouteThatSaysNothingCarriesAnySize(t *testing.T) {
routes, _, err := routesFrom(`{"given":[
routes, _, err := routesFrom(write(t, `{"given":[
{"from":"app","node":"anchor","at":"anchor.internal","values":{"name":"app.example","port":8080}}
]}`)
]}`))
if err != nil {
t.Fatal(err)
}
@@ -431,6 +455,23 @@ func TestARouteThatSaysNothingCarriesAnySize(t *testing.T) {
}
}
// A limit that is not a whole positive number of bytes takes the route with it. Serving it without
// the limit would carry exactly what the module said not to carry, and report success doing it.
func TestARouteWithAnUnusableLimitIsSkipped(t *testing.T) {
for _, asked := range []string{`"lots"`, `-1`, `0`, `1.5`} {
routes, _, err := routesFrom(write(t, `{"given":[
{"from":"registry","node":"anchor","at":"anchor.internal",
"values":{"name":"images.example","port":5000,"max-request-body":`+asked+`}}
]}`))
if err != nil {
t.Fatal(err)
}
if len(routes["images.example"]) != 0 {
t.Errorf("a route asking for a max-request-body of %s was served anyway: %v", asked, routes)
}
}
}
// A request larger than the route carries is refused by the proxy, with the limit named, and the
// workload never sees it. A request within it is proxied normally.
func TestABodyOverTheLimitIsRefusedAndOneUnderItIsCarried(t *testing.T) {
@@ -444,10 +485,10 @@ func TestABodyOverTheLimitIsRefusedAndOneUnderItIsCarried(t *testing.T) {
at := strings.TrimPrefix(workload.URL, "http://")
host, port, _ := strings.Cut(at, ":")
routes, _, err := routesFrom(`{"given":[
{"from":"registry","node":"anchor","at":"` + host + `",
"values":{"name":"images.example","port":` + port + `,"max-request-body":8}}
]}`)
routes, _, err := routesFrom(write(t, `{"given":[
{"from":"registry","node":"anchor","at":"`+host+`",
"values":{"name":"images.example","port":`+port+`,"max-request-body":8}}
]}`))
if err != nil {
t.Fatal(err)
}
@@ -491,32 +532,3 @@ func post(url, host, body string) (*http.Response, error) {
asked.Header.Set("Content-Type", "application/octet-stream")
return http.DefaultClient.Do(asked)
}
// A contribution that cannot be served is skipped rather than served wrongly: one that names nothing to be served
// under; one with no port, which would proxy to :0; a scheme that is neither http nor https, refused rather than
// guessed at; and a body limit that is not a whole positive number of bytes, since serving it without the limit would
// carry exactly what the module said not to carry, and report success doing it.
func TestAContributionThatCannotBeServedIsSkipped(t *testing.T) {
for _, row := range []struct{ why, given, served string }{
{"neither name", `{"from":"app","node":"anchor","at":"anchor.internal","values":{"internal-name":" ","port":8080}}`, ""},
{"no port", `{"from":"a","node":"n","at":"n.internal","values":{"name":"no-port.example"}},
{"from":"b","node":"n","at":"n.internal","values":{"port":8080}},
{"from":"c","node":"n","at":"n.internal","values":{"name":"fine.example","port":8080}}`, "fine.example"},
{"an unknown scheme", `{"from":"a","node":"n","at":"n.internal","values":{"name":"bad.example","port":80,"scheme":"ftp"}}`, ""},
{"a limit in words", `{"from":"r","node":"a","at":"a.internal","values":{"name":"images.example","port":5000,"max-request-body":"lots"}}`, ""},
{"a negative limit", `{"from":"r","node":"a","at":"a.internal","values":{"name":"images.example","port":5000,"max-request-body":-1}}`, ""},
{"a zero limit", `{"from":"r","node":"a","at":"a.internal","values":{"name":"images.example","port":5000,"max-request-body":0}}`, ""},
{"a fractional limit", `{"from":"r","node":"a","at":"a.internal","values":{"name":"images.example","port":5000,"max-request-body":1.5}}`, ""},
} {
routes, public, err := routesFrom(`{"given":[` + row.given + `]}`)
if err != nil {
t.Fatal(err)
}
switch {
case row.served == "" && (len(routes) != 0 || len(public) != 0):
t.Errorf("%s: an unusable contribution was served: %v %v", row.why, routes, public)
case row.served != "" && (len(routes) != 1 || targetOf(routes, row.served) == ""):
t.Errorf("%s: an unusable contribution was served beside the usable one: %v", row.why, routes)
}
}
}
-77
View File
@@ -1,77 +0,0 @@
// Package contract is route-proxy's contract for the provision `route`: what a consumer asks a route with, what the
// proxy keeps on the bus about what it serves, the events it says, and its tools' arguments (issue 774; research 072
// row 34; the operator, 2026-10-31: "the route-proxy should define its own contract for the seat (with the message
// contracts of course) which other modules can use").
//
// **Published by the repository that holds route-proxy, as its own Go module**, versioned by tags of the form
// `modules/route-proxy/contract/vX.Y.Z`, the subdirectory form Go's module system expects, the same way mesh-sdk is
// published (`go/vX.Y.Z`). A module that asks a route, reads what the proxy serves or hears its events imports this
// package; nothing imports the repository. Standard library only, so a consumer takes nothing else with it.
//
// **How a route travels.** A module that must be reachable requires `route` and says, in its manifest's
// `contributes.route`, one Request, or several by local name; Schema is that object's JSON schema. The controller
// composes each (its label joined to the node's domains into `name` and `internal-name`, its endpoint's port into
// `port`) and the seat mesh-provisioner carries the result in the consumer's plan entry as an opaque payload
// (mesh-sdk catalogue.PlanProvision.Contributed, in the generic catalogue.ContributedForm). Neither the seat nor its
// client knows a field of a route: route-proxy decodes each contribution with DecodeRequest, and refuses, saying why
// (EventRefused), what this contract does not admit.
//
// **Two name spaces, two authorities** (novox/hq ADR 0056, ADR 0138). `name` is public: served to anyone and
// certified by the public ACME directory (Let's Encrypt). `internal-name` is served only inside the private network
// and certified by the mesh's own authority. A route carries either, or both.
package contract
// Module is the module that provides a route, and Provision what it provides: what a plan's entry names when
// route-proxy is a route's provider.
const (
Module = "route-proxy"
Provision = "route"
)
// Version is the version of the Request form this package reads and writes.
const Version = 1
// The events route-proxy says (its manifest's `emits`), on mesh.mod.route-proxy.event.<type>, each with an Event.
const (
// EventServed: a name is now served by this proxy (it was not, or it is now served from other rules).
EventServed = "route.served"
// EventWithdrawn: a name this proxy served is served no more, because no plan gives it any longer.
EventWithdrawn = "route.withdrawn"
// EventRefused: a plan gives this proxy a route it does not serve, and Why says why.
EventRefused = "route.refused"
)
// Event is the body of every event route-proxy says: the proxy's machine, the name (empty for a refusal that named
// none), how far it reaches, and who asked it: the consumer module, its machine and the local name of its
// contribution. Why is set on a refusal alone.
type Event struct {
Proxy string `json:"proxy"`
Name string `json:"name,omitempty"`
Reach string `json:"reach,omitempty"`
Module string `json:"module,omitempty"`
Node string `json:"node,omitempty"`
Local string `json:"local,omitempty"`
Why string `json:"why,omitempty"`
}
// The tools route-proxy serves (its manifest's `tools`), each answering from its states RoutesState and UnroutedState.
const (
// ToolRoutes lists every name served, with RoutesArgs.
ToolRoutes = "route_proxy_routes"
// ToolRoute is one name in full, with RouteArgs.
ToolRoute = "route_proxy_route"
// ToolUnrouted lists the names asked for and not served, with RoutesArgs.
ToolUnrouted = "route_proxy_unrouted"
)
// RoutesArgs are ToolRoutes' and ToolUnrouted's arguments: Node narrows to the proxy on one machine, every machine
// when empty.
type RoutesArgs struct {
Node string `json:"node,omitempty"`
}
// RouteArgs are ToolRoute's arguments: the name, such as app.example.org, and optionally the machine whose proxy.
type RouteArgs struct {
Name string `json:"name"`
Node string `json:"node,omitempty"`
}
@@ -1,85 +0,0 @@
package contract
import (
"encoding/json"
"reflect"
"slices"
"strings"
"testing"
)
func limit(n int64) *int64 { return &n }
// A route is read strictly and whole (issue 774): what the contract names is admitted, what it does not, or a route
// it cannot serve, is refused naming why; and the published schema names exactly the fields the decoder reads.
func TestARouteIsReadStrictly(t *testing.T) {
for _, c := range []struct {
raw, refused string
want Request
}{
{raw: `{"name":"app.example.org","internal-name":"app.anchor.internal","label":"app","endpoint":"web","port":8080}`,
want: Request{Name: "app.example.org", InternalName: "app.anchor.internal", Label: "app", Endpoint: "web", Port: 8080}},
{raw: `{"internal-name":"app.anchor.internal","port":443,"scheme":"https","insecure":true,"max-request-body":1048576}`,
want: Request{InternalName: "app.anchor.internal", Port: 443, Scheme: "https", Insecure: true, MaxRequestBody: limit(1048576)}},
{raw: `{"name":"git.example.org","path":"/api/internal","deny":true,"priority":100000}`,
want: Request{Name: "git.example.org", Path: "/api/internal", Deny: true, Priority: 100000}},
{raw: `{"name":"old.example.org","redirect":"https://new.example.org"}`,
want: Request{Name: "old.example.org", Redirect: "https://new.example.org"}},
{raw: `{"name":"admin.example.org","port":80,"auth":"/run/secrets/admin"}`,
want: Request{Name: "admin.example.org", Port: 80, Auth: "/run/secrets/admin"}},
{raw: `{"label":"app","port":8080}`, refused: "names nothing"},
{raw: `{"name":"app.example.org"}`, refused: "gives no port"},
{raw: `{"name":"app.example.org","port":70000}`, refused: "not a port"},
{raw: `{"name":"app.example.org","port":80.5}`, refused: "not a route"},
{raw: `{"name":"app.example.org","port":"80"}`, refused: "not a route"},
{raw: `{"name":"app.example.org","port":80,"scheme":"ftp"}`, refused: "neither http nor https"},
{raw: `{"name":"app.example.org","port":80,"max-request-body":-1}`, refused: "max-request-body"},
{raw: `{"name":"app.example.org","port":80,"max-request-body":0}`, refused: "max-request-body"},
{raw: `{"name":"app.example.org","port":80,"auth":"admin:$2y$05$abc"}`, refused: "credential"},
{raw: `{"name":"app.example.org","port":80,"host":"x"}`, refused: "unknown field \"host\""},
{raw: `{"name":"app.example.org","port":80} {}`, refused: "more than one"},
} {
got, err := DecodeRequest([]byte(c.raw))
if c.refused != "" {
if err == nil || !strings.Contains(err.Error(), c.refused) {
t.Errorf("DecodeRequest(%s) = %+v, %v; want refused for %q", c.raw, got, err, c.refused)
}
continue
}
if err != nil || !reflect.DeepEqual(got, c.want) {
t.Errorf("DecodeRequest(%s) = %+v, %v; want %+v", c.raw, got, err, c.want)
continue
}
// What it encodes, it reads back the same.
if raw, err := got.Encode(); err != nil {
t.Errorf("Encode(%+v): %v", got, err)
} else if again, err := DecodeRequest(raw); err != nil || !reflect.DeepEqual(again, got) {
t.Errorf("Encode then DecodeRequest of %+v = %+v, %v", got, again, err)
}
}
if (Request{Path: "api"}).Prefix() != "/api" || (Request{}).Prefix() != "" || (Request{Scheme: " HTTPS"}).SchemeOrDefault() != "https" ||
(Request{}).SchemeOrDefault() != "http" || (Request{Redirect: "x"}).Forwards() {
t.Error("Prefix, SchemeOrDefault or Forwards says otherwise than the contract")
}
var schema struct {
Properties map[string]json.RawMessage `json:"properties"`
AdditionalProperties bool `json:"additionalProperties"`
}
if err := json.Unmarshal(Schema, &schema); err != nil || schema.AdditionalProperties {
t.Fatalf("the schema is not a closed object: %v", err)
}
var tags []string
for f := range reflect.TypeFor[Request]().Fields() {
tags = append(tags, strings.Split(f.Tag.Get("json"), ",")[0])
}
var said []string
for name := range schema.Properties {
said = append(said, name)
}
slices.Sort(tags)
slices.Sort(said)
if !slices.Equal(tags, said) {
t.Fatalf("the schema names %v, the decoder reads %v", said, tags)
}
}
-3
View File
@@ -1,3 +0,0 @@
module git.novox.be/novox/mesh-controller/modules/route-proxy/contract
go 1.26.0
-128
View File
@@ -1,128 +0,0 @@
package contract
import (
"bytes"
_ "embed"
"encoding/json"
"errors"
"fmt"
"strings"
)
// Request is one route a consumer asks for, as its manifest's `contributes.route` says it and the controller composes
// it: the object Schema describes. Every field is optional to the decoder; Validate says which a route needs.
type Request struct {
// Name is the public name, served to anyone and certified by the public ACME directory; InternalName the name on
// the private network, served inside it alone and certified by the mesh's own authority. The controller composes
// both from Label and the node's domains, as far as the endpoint reaches (novox/hq ADR 0138); a manifest writes a
// label, never a domain.
Name string `json:"name,omitempty"`
InternalName string `json:"internal-name,omitempty"`
Label string `json:"label,omitempty"`
// Endpoint names the manifest's endpoint the route serves; the controller fills Port from it.
Endpoint string `json:"endpoint,omitempty"`
Port int `json:"port,omitempty"`
// Path is the path prefix the rule is scoped to, every path when empty; Priority which rule wins where two match,
// higher first.
Path string `json:"path,omitempty"`
Priority int `json:"priority,omitempty"`
// Scheme is how the consumer is reached, http (the default) or https; Insecure skips verifying its certificate,
// for a backend that terminates TLS with its own.
Scheme string `json:"scheme,omitempty"`
Insecure bool `json:"insecure,omitempty"`
// MaxRequestBody is the largest body, in bytes, the route carries; no limit when absent. Said, it is whole and
// positive: a limit the proxy cannot keep is a route it does not serve.
MaxRequestBody *int64 `json:"max-request-body,omitempty"`
// The policy of the rule, a closed set (novox/hq ADR 0108): Deny refuses every request, Redirect answers with a
// permanent redirect to that address, and Auth names the file of the secret a request must present, never the
// secret itself.
Deny bool `json:"deny,omitempty"`
Redirect string `json:"redirect,omitempty"`
Auth string `json:"auth,omitempty"`
}
// Schema is Request's JSON schema, published with this package for what checks a manifest's `contributes.route`.
//
//go:embed route.schema.json
var Schema []byte
// DecodeRequest reads one contribution's object strictly: no field this contract does not name, one document, and
// a route Validate admits. What it refuses, it refuses whole, naming why: a route is never served half.
func DecodeRequest(raw []byte) (Request, error) {
var r Request
dec := json.NewDecoder(bytes.NewReader(raw))
dec.DisallowUnknownFields()
if err := dec.Decode(&r); err != nil {
return Request{}, fmt.Errorf("not a route this contract reads: %w", err)
}
if dec.More() {
return Request{}, errors.New("not a route this contract reads: more than one document")
}
if err := r.Validate(); err != nil {
return Request{}, err
}
return r, nil
}
// Encode is r as a contribution's object, refused where DecodeRequest would refuse it.
func (r Request) Encode() ([]byte, error) {
if err := r.Validate(); err != nil {
return nil, err
}
return json.Marshal(r)
}
// Validate says what makes r no route: no name of either kind; a port outside 1-65535, or none where the rule
// forwards (neither Deny nor Redirect); a scheme other than http or https; a body limit that is not positive; and a
// credential written where a secret's file is to be named (novox/hq ADR 0108: refused rather than tolerated).
func (r Request) Validate() error {
if strings.TrimSpace(r.Name) == "" && strings.TrimSpace(r.InternalName) == "" {
return errors.New("the route names nothing to be served under: neither name nor internal-name")
}
if r.Port < 0 || r.Port > 65535 {
return fmt.Errorf("the route's port %d is not a port", r.Port)
}
if r.Forwards() && r.Port == 0 {
return errors.New("the route forwards and gives no port")
}
switch strings.ToLower(strings.TrimSpace(r.Scheme)) {
case "", "http", "https":
default:
return fmt.Errorf("the route's scheme %q is neither http nor https", r.Scheme)
}
if r.MaxRequestBody != nil && *r.MaxRequestBody <= 0 {
return fmt.Errorf("the route's max-request-body %d is not a positive number of bytes", *r.MaxRequestBody)
}
if LooksLikeACredential(r.Auth) {
return errors.New("the route's auth holds a credential rather than the name of a secret's file (novox/hq ADR 0108)")
}
return nil
}
// Forwards is whether the rule sends a request on to the consumer, rather than answering it itself.
func (r Request) Forwards() bool { return !r.Deny && r.Redirect == "" }
// Prefix is the path prefix the rule is scoped to, beginning with a slash, or "" for every path.
func (r Request) Prefix() string {
p := strings.TrimSpace(r.Path)
if p != "" && !strings.HasPrefix(p, "/") {
p = "/" + p
}
return p
}
// SchemeOrDefault is how the consumer is reached: the scheme, lower-cased, http when none is said.
func (r Request) SchemeOrDefault() string {
if s := strings.ToLower(strings.TrimSpace(r.Scheme)); s != "" {
return s
}
return "http"
}
// LooksLikeACredential is the check that keeps a secret out of a declaration: a value holding a `:` (the htpasswd
// separator) or opening with a bcrypt identifier is a credential, not a path, and no path the mesh writes needs
// either. It errs towards refusing.
func LooksLikeACredential(v string) bool {
v = strings.TrimSpace(v)
return strings.Contains(v, ":") || strings.HasPrefix(v, "$2")
}
@@ -1,28 +0,0 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://git.novox.be/novox/mesh-controller/modules/route-proxy/contract/route.schema.json",
"title": "route",
"description": "One route a consumer asks route-proxy for, as a manifest's contributes.route says it (one, or several by local name) and the controller composes it. Version 1 of route-proxy's contract.",
"type": "object",
"additionalProperties": false,
"anyOf": [
{"required": ["name"]},
{"required": ["internal-name"]},
{"required": ["label"]}
],
"properties": {
"name": {"type": "string", "description": "the public name, served to anyone and certified by the public ACME directory; composed from label"},
"internal-name": {"type": "string", "description": "the name on the private network, served inside it alone and certified by the mesh's own authority; composed from label"},
"label": {"type": "string", "description": "the subdomain the module chooses; joined to the node's domains by the controller"},
"endpoint": {"type": "string", "description": "the manifest's endpoint the route serves; the controller fills port from it"},
"port": {"type": "integer", "minimum": 1, "maximum": 65535},
"path": {"type": "string", "description": "the path prefix the rule is scoped to; every path when absent"},
"priority": {"type": "integer", "description": "which rule wins where two match, higher first"},
"scheme": {"enum": ["http", "https"], "description": "how the consumer is reached; http when absent"},
"insecure": {"type": "boolean", "description": "skip verifying the consumer's certificate over https"},
"max-request-body": {"type": "integer", "minimum": 1, "description": "the largest body, in bytes, the route carries"},
"deny": {"type": "boolean", "description": "refuse every request matching the rule"},
"redirect": {"type": "string", "description": "answer with a permanent redirect to this address"},
"auth": {"type": "string", "description": "the file of the secret a request must present, never the secret itself"}
}
}
-152
View File
@@ -1,152 +0,0 @@
package contract
// What the web front keeps on the bus about the names it serves, and about the names it was asked for and does not
// serve (issue 774). The proxy writes it, on change, in its states `routes` and `unrouted`, announced readable; the
// module's tools, and any module that reads them, read it from there, so the question "is this site reachable, and
// is its certificate good" is answered from the bus rather than with curl.
//
// A key is `<node>.<name>`: the machine whose proxy answers, then the name as asked, lower-cased. A node's name holds
// no dot, so the first dot splits them.
import (
"net"
"regexp"
"strings"
"time"
)
// The states route-proxy keeps, by their local names (the buckets are route-proxy_<name>).
const (
// RoutesState holds one key per served name: where it forwards, its certificate, the last answer it got.
RoutesState = "routes"
// UnroutedState holds one key per name the proxy answered "no route for" with, and when it was last asked.
UnroutedState = "unrouted"
// BoundState is the provider helper's bucket of consumer entries (mesh-sdk go/provider).
BoundState = "bound"
)
// The words a route's reach and a certificate's authority are said in.
const (
ReachPublic = "public"
ReachInternal = "internal"
)
// The words a certificate's outcome is said in.
const (
// Issued: the authority issued it while this proxy ran, at RenewedAt.
Issued = "issued"
// Cached: this proxy served it from what it kept, and saw no issuance of it since it started.
Cached = "cached"
// Failed: the last attempt to have one failed, Error says why.
Failed = "failed"
)
// Target is one rule a name is served by: what a request matching it reaches, or what it is answered with.
type Target struct {
// Path is the path prefix the rule is scoped to; empty is every path.
Path string `json:"path,omitempty"`
// Module and Node are the consumer that contributed the route and the machine it runs on.
Module string `json:"module,omitempty"`
Node string `json:"node,omitempty"`
// Target is where the request is forwarded, scheme://address:port; empty for a rule that answers itself.
Target string `json:"target,omitempty"`
Deny bool `json:"deny,omitempty"`
Redirect string `json:"redirect,omitempty"`
// Auth says the rule asks for credentials; Sealed says why it cannot check them, and so refuses everything.
Auth bool `json:"auth,omitempty"`
Sealed string `json:"sealed,omitempty"`
}
// Certificate is what this proxy knows of a name's certificate.
type Certificate struct {
// Authority is which issuer certifies the name: public (the ACME directory the module names) or internal (the
// mesh's own).
Authority string `json:"authority"`
Issuer string `json:"issuer,omitempty"`
NotAfter *time.Time `json:"not_after,omitempty"`
// RenewedAt is when an authority last issued it to this proxy.
RenewedAt *time.Time `json:"renewed_at,omitempty"`
// Outcome is issued, cached or failed; empty until a handshake or an issuance said anything.
Outcome string `json:"outcome,omitempty"`
Error string `json:"error,omitempty"`
At *time.Time `json:"at,omitempty"`
}
// Answer is the last answer the proxy got from upstream for a name: the status it carried back, or the error that
// stood in for one (a 502).
type Answer struct {
Status int `json:"status"`
At time.Time `json:"at"`
Error string `json:"error,omitempty"`
Path string `json:"path,omitempty"`
}
// Route is one served name, the value of a key of RoutesState.
type Route struct {
Name string `json:"name"`
// Proxy is the machine whose proxy serves it.
Proxy string `json:"proxy"`
// Reach is public (a route's own name) or internal (only its internal-name, served inside the private network).
Reach string `json:"reach"`
Targets []Target `json:"targets"`
Certificate Certificate `json:"certificate"`
LastAnswer *Answer `json:"last_answer,omitempty"`
}
// Unrouted is one name the proxy answered "no route for", the value of a key of UnroutedState.
type Unrouted struct {
Name string `json:"name"`
Proxy string `json:"proxy"`
// AskedAt is when the name was last asked for. Who asked is not kept here: the journal keeps it, for the
// machine's intrusion prevention.
AskedAt time.Time `json:"asked_at"`
// Times is how often it was asked since the proxy last started.
Times int `json:"times"`
// InternalOnly says the name is served, but only inside the private network, and was asked from outside.
InternalOnly bool `json:"internal_only,omitempty"`
// Handshake says it was asked in a TLS handshake (the name a client sent), not in an HTTP request.
Handshake bool `json:"handshake,omitempty"`
}
// OtherName is the name of the one key per proxy that counts the refusals not kept by name: names under no domain
// the proxy serves (what a scan sends), and names past the limit of new names a minute. Its key is
// `<node>._other`; no host name begins with an underscore.
const OtherName = "_other"
// Other is the value of a proxy's OtherName key.
type Other struct {
Proxy string `json:"proxy"`
// Minute is the last whole minute counted, and LastMinute how many refusals it held.
Minute time.Time `json:"minute"`
LastMinute int `json:"last_minute"`
// SinceStart is every refusal not kept by name since the proxy started.
SinceStart int `json:"since_start"`
}
// Registrable is the domain a name is registered under, as its last two labels; "" for an address or a single
// label. Two labels is wrong for a public suffix of two (co.uk); it only decides which refused names are kept by name.
func Registrable(name string) string {
if net.ParseIP(strings.Trim(name, "[]")) != nil {
return ""
}
labels := strings.Split(name, ".")
if len(labels) < 2 {
return ""
}
return strings.Join(labels[len(labels)-2:], ".")
}
var keyable = regexp.MustCompile(`^[a-z0-9_]([a-z0-9_.-]{0,251}[a-z0-9_])?$`)
// Keyable says whether a name can be a key's name: what a host name or an address is. A name asked for in a request
// is whatever a client sent; one no key could hold is no site anybody runs.
func Keyable(name string) bool { return keyable.MatchString(name) && !strings.Contains(name, "..") }
// Key is the key of name served or asked at node's proxy.
func Key(node, name string) string { return node + "." + name }
// Split is a key's node and name.
func Split(key string) (node, name string) {
node, name, _ = strings.Cut(key, ".")
return node, name
}
+1 -2
View File
@@ -3,13 +3,12 @@ module route-proxy
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261031084708-acbf80ad910a
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261022190740-104533e8c0c9
github.com/nats-io/nats.go v1.54.0
golang.org/x/crypto v0.57.0
)
require (
git.novox.be/novox/mesh-controller/modules/route-proxy/contract v0.1.0
github.com/klauspost/compress v1.20.0 // indirect
github.com/nats-io/nkeys v0.4.16 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
+2 -4
View File
@@ -1,7 +1,5 @@
git.novox.be/novox/mesh-controller/modules/route-proxy/contract v0.1.0 h1:89jE1oBK+l7gFJjz6wx8mUD3KxrqSDa7pUTlQkzC1do=
git.novox.be/novox/mesh-controller/modules/route-proxy/contract v0.1.0/go.mod h1:Yr+Xp89kpMiKH6kA06EU+d9I2CyzrK9yMX7J3pSv0hE=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261031084708-acbf80ad910a h1:DcSvfEmdPBKqCg5OmXRNb269EkkPe7BsEp+RykEhkgE=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261031084708-acbf80ad910a/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261022190740-104533e8c0c9 h1:Ufmd2cYEXICQ817IGflccP9bexABE4WhIO49HPbaoSo=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261022190740-104533e8c0c9/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
+13 -30
View File
@@ -16,6 +16,9 @@
"serves": {
"route": {}
},
"receives": {
"route": "${dir:routes-dir}/mesh.json"
},
"requires": [
"internal-acme-ca"
],
@@ -70,6 +73,12 @@
"mode": "0700",
"place": "mesh"
},
{
"id": "routes-dir",
"type": "directory",
"path": "/var/lib/route-proxy/routes",
"mode": "0700"
},
{
"id": "acme-cache",
"type": "directory",
@@ -158,12 +167,14 @@
"${dir:state}/internal-acme.env"
],
"volumes": [
"${dir:routes-dir}:/routes:ro",
"${dir:acme-cache}:/acme",
"${dir:ca-dir}:/ca:ro",
"${dir:mesh-state}/broker:/run/secrets/broker:ro",
"${dir:state}/route-proxy:/opt/mesh:ro"
],
"env": {
"ROUTES": "/routes/mesh.json",
"LISTEN": ":80",
"TLS_LISTEN": ":443",
"ACME_CACHE": "/acme",
@@ -233,37 +244,9 @@
}
],
"state": [
"bound",
"routes",
{
"name": "unrouted",
"ttl-seconds": 604800
}
"bound"
],
"tools": [
"route_proxy_routes",
"route_proxy_route",
"route_proxy_unrouted"
],
"emits": [
"route.served",
"route.withdrawn",
"route.refused"
],
"replaces": {
"route_proxy_routes": [
"curl -I"
],
"route_proxy_route": [
"openssl s_client -connect"
],
"route_proxy_unrouted": [
"docker logs route-proxy"
]
},
"reads": [
"mesh-delivery-assigner.placement",
"mesh-provisioner.plans",
"mesh-provisioner.provisions"
"mesh-controller.assignments"
]
}
+1 -1
View File
@@ -9,7 +9,7 @@ a second place to clean besides the catalogue itself.
mesh-catalog f3a57dcc56fb6c198095cafd11d0decff9f720c4 modules/*/module.json, modules/*/files, modules/*/templates, predefined.json
mesh-host 9d9e3a46c4f5943cfffb413d962ce48b75e2fd39 examples/foundation-first-node.lock
mesh-sdk 2da0de75c3f0087cff9c2f5ef789ca6dc3c59045 conformance/events/module-event.json
mesh-sdk a37181261c24a15b8092c06e11cce63e0e5f8ade conformance/events/module-event.json
To move them, from this repository's root, with the two repositories checked out beside it: