A host's account reaches no part of the JetStream API — correctly, because the
controller is the only writer of consumer definitions — so the object a node
reads its declarations through has to be waiting before the host binds to it,
and nothing created one. Named after the node, because the node's own ack grant
is `$JS.ACK.NODES.<node>.>` and a consumer named anything else is one the host
cannot acknowledge a delivery from.
No max-deliver, and a five-minute ack wait: a declaration is settled only after
the node has applied it and reported, which is minutes on a machine pulling
images, and the stream holds exactly one message per node — so there is nothing
to dead-letter, only one message to redeliver for as long as that node is away.
Asserted on start as well as created at enrolment, for the reason the streams
are: a mesh raised from a restored backup has node records and no consumers, and
a node whose consumer is missing hears nothing while everything else about it
looks correct.
The test sets the consumer against the grant the node actually gets, because
each of the three ways of getting it wrong is silent: a wrong name cannot ack, a
wrong filter reads another node's declarations, and a pull consumer is one a host
has no authority to bind.
The other implementation behind the seam, so the store-window guarantee now has
both: one loop, one message at a time, the same window deciding. What differs is
where a held message lives, and that is the whole point of the move — the AMQP
side keeps an unacknowledged delivery in this process, bounded by the prefetch
and lost if the controller stops; this keeps eight bytes saying when the window
opened, and the message stays the server's.
Checked against a running server, seven claims that reasoning cannot answer: a
report is heard and leaves the work queue; one the store cannot take is naked
with a delay, stays in the stream, and is recorded when the store returns; one
about a superseded declaration is settled without being acted on; one the store
never takes is let go once the bound passes; a heartbeat is heard and nothing is
persisted; and the enrolment answer reaches the address the request carried in
its payload — the test design 25 §2 asks for, so the reason for that field
cannot quietly become folklore.
Three things the wiring forced into the open:
**The controller could not have consumed a module event.** Its permissions
granted no event subject to subscribe and no ack subject on the events stream,
so every announcement would have been redelivered for ever, refused by the list
it already had. Both narrow: each followed subject named, not `mesh.mod.*.>`.
**The controller's consumers are not derived.** It files no manifest, so its
authority cannot come from a declaration that does not exist; they sit beside the
mesh's own streams and are asserted the same way. No max-deliver on CONTROL —
the window's bound is the controller's, and a server that dead-lettered first
would discard the push the stream exists to protect.
**Channels, not callbacks.** The library would run a handler on its own
goroutine, and the window's bookkeeping is unlocked because the AMQP loop never
had two.
The outbound half went behind `Bus` and the transport stopped reaching its
callers; this is the other half, and the larger one. Every handler took
`amqp.Delivery`, so the serving loop could not move to another bus without
moving enrolment, reports, builds, upgrades and catch-up with it in one breath.
`Control` states one message in the mesh's words — took it, dropped it, or held
it for the store — and `Inbound` is where messages come from. The AMQP
implementation is today's loop moved rather than changed: same queues, same
prefetch, same holding, because the mesh is running on it and a bus nothing
speaks yet is no reason to alter the one every node is on.
The window (window.go) is now what decides, instead of the conditions that were
inlined in the loop. Two things that surfaced in the wiring:
**Supersession is asked before the store, not after.** A report about a
declaration the mesh has moved past would otherwise wait out a restarting store
to be written and then overwrite what the node is doing now.
**Half of a report is not about a declaration, and that half is never stale.**
What the machine *is* — the tunnel it took over, the ports its own bundle
holds, what an adopted node found, a node moving its overlay key — reaches the
mesh on a report and nowhere else. A rekey set aside as stale is a node whose
overlay key never moves, and no retry is coming, because the node said it once.
So staleness is asked only of a report that is purely an apply's account.
The one thing holding-in-memory can do that holding-in-the-server cannot is
named rather than hidden: `About` sets aside a held message when a newer one
about the same thing arrives, and the bus being built ignores it because the
digest answers the same question.
Design 25 §7. A person is not a module and holds no seat: nothing is
addressed to them, nothing is delivered to them, and they have no durable
consumer. What they have is permission to ask, as a list of tools or `*`
for an administrator.
Four properties the tests hold it to, each of which is a way of being
wrong that would not announce itself: a person reaches nothing but tools,
so one cannot claim a module said something; no ack subject, because
authority over a consumer that does not exist is authority nobody would
audit; no allow_responses, because a person who can answer a request is
impersonating a module on a bus where anyone may serve a tool; and two
people do not share an inbox.
Design 25 §2 builds enrolment around carrying the reply subject in the
payload, because a JetStream consumer claims the transport Reply field for
its own ack address. The whole handshake rests on it, so it is checked:
the caller asked for _INBOX.LCr3M83q... and the consumer saw
$JS.ACK.PROBE.probe_consumer... The design was right, and the workaround
is necessary rather than defensive.
Worth having as a test rather than a note: if a future server version
stopped doing this, enrolment would keep working and the reason for the
payload field would quietly become folklore.
Step 3.4. On the new bus there is no reply queue to declare and no
correlation to check: each account is granted one inbox prefix and no
other, so an answer cannot reach the wrong asker. That settles a cost
build.go records having paid — on a shared reply exchange every asker saw
every result, which is why the correlation was checked rather than assumed.
And a tool nobody serves says so at once rather than after the whole wait.
The difference between "that module is down" and "that tool is slow" is the
first thing a person asking wants, and both tests are against a real server
because both are claims about what the server does, not about this code.
RequestBuild stays as it is, and is a different shape on the new bus rather
than the same one: a build takes minutes, so it is work submitted to a
queue with the outcome returning to a reply subject the request carries —
the pattern design 25 §2 already sets for anything crossing a stream. It
touches the builder too, so it goes with that conversion.
introduces
Step 3.4, the consume side's hard part. The guarantee (ADR 0083) is that a
push the controller cannot record because its store is restarting is held
and retried — never dropped, never falsely acknowledged. Keeping the
delivery unacknowledged in memory becomes a nak with a delay: the server
holds it, the controller keeps no list of parked messages, and a controller
that restarts mid-window loses nothing it was holding.
That is a plain win and it introduces one problem. Holding in memory let
the controller drop an older report when a newer one for the same node
arrived, "because acting on it after the newer would undo the newer". A
naked message is the server's and comes back whatever happened meanwhile,
so the older report is redelivered after the newer was applied.
The answer was already in the message. A report carries `Declared`, the
digest of the declaration it is about, which exists because an earlier
attempt to order reports by time lost the race it invited. So supersession
stops being something the controller remembers and becomes something it
checks — the same shape as a node refusing a superseded declaration by
sequence (issue 107): ordering settled by what a message says, not by when
it arrived.
Pure, so the guarantee is testable without a bus, a store or a clock. Nine
tests, including that staleness is decided before the store is waited on —
a redelivery that lost its race must not hold a slot a current message
needs.
Comments framed the new bus by what it replaces — a comparison in almost
every explanation, which reads as though NATS were a variant of the old
thing rather than the mesh's nervous system. Removed throughout, and
OverAMQP becomes OverCurrent: the seam's two sides are the bus the mesh
runs on today and the one being built, not two protocols.
What remains is the client library's own package name, which is its name.
Step 3.4, first half. Every one of these took an *amqp.Channel, so the
transport reached every caller and swapping it meant touching all of them.
The seam turned out to be small — the controller sends exactly two kinds of
message that expect no answer — which is the same measurement that said
this bus could be replaced at all.
Bus is stated in the mesh's words, not a transport's: PublishEvent and
PublishDeclaration. Two implementations, both shipping, because steps 1 to
4 leave every node on AMQP and the NATS one is selected at the rollout.
Both ship is also what makes them comparable: one conformance fixture holds
both to the same envelope, and the NATS one is checked against a real
server reading back from the stream rather than from the code that wrote it.
Still on *amqp.Channel: RequestBuild and Ask, which carry reply-queue
machinery, and the whole consume side — the control loop, enrolment, serve.
Every required header set, each value in the pinned shape, and the subject
derived the same way. Read from the sdk's conformance directory by sibling
path, never copied.
Design 29 §1's load-bearing rule: a module names its events, tools and
seats locally and the mesh derives the subject, so reorganising the subject
space leaves every manifest correct. It held by construction, and a rule
held by construction is one a later field breaks quietly.
novox/hq ADR 0118: the prefix is the reservation rule, so a module
declaring any mesh-* name is refused and there is no reserved-names list to
drift. Ten seats renamed in the table, the manifests that claim them, the
controller's own shipped manifests, and the tests.
Not the migration 0118 expected: a holding is derived at resolution from
manifests and never stored, so nothing recorded points at an old name. A
kept rename table tells a manifest written against one what it became —
kept rather than retired, because a module lives in its own repository and
may be registered long after the catalogue stopped using it.
**A seat is not the interface it delivers.** The git seat became mesh-git
and the git provision did not; likewise the package registry. A blanket
replace renamed both, and the failure read "the package registry is served
on <nil>", which does not say "you renamed an interface". A test now pins
every seat against what it delivers, and that neither name is also the
other.
Task 3.9's other half and 1.4's missing client. The derivation is pure and
unit-tested; only "does the server accept this" needs one running, behind
MESH_TEST_NATS so the ordinary suite stays offline.
A seat's work queue is created at registration, not assignment, so work
queues until a holder appears — a stream created at assignment would make
"the holder is not here yet" mean "your messages are gone". Named after the
seat, because the holder can change and the queued work must not care.
A holder's worker uses a queue group even though the seat guarantees one
holder: the seat is authority, the queue group is delivery, and tying them
together means the day somebody allows two holders every message is
processed twice with nothing reporting it.
One consumer per module carrying every filter, because its ack permission is
derived from its name.
And a real bug the live server caught: a durable name may not contain a dot,
but an ack subject is $JS.ACK.<stream>.<consumer>, so the single string that
read correctly inside the permission was rejected as a consumer name. Split
in two, beside the permission that has to match. Unfixed, the symptom would
have been every message redelivered forever with a permission list that
looks right — which is the failure design 25 §4 warns about.
The manifest carries seats and uses; registration refuses a mesh-* name, a
duplicate declarer, an undeclared uses or claim, a seat with no protocol, a
scope mismatch, and a holder that does not answer what its seat promises.
The parser stops judging unknown claim names, because it cannot: another
module may declare that seat, and one manifest cannot tell. The test that
encoded the old rule is rewritten to assert the refusal at registration, and
a new one pins the case the parser could not have distinguished.
Tools are declared for the first time, under their own key — serves already
means a provision's facts.
NATS refuses overlapping streams rather than double-storing, which is the
opposite of what the Overlaps comment claimed. The check still earns its
place — it names both streams at composition rather than one at apply — and
the refusal is what rules out a shared stream beside per-module ones.
Step 2.3 of novox/hq ADR 0116. The refusal is the resolver's existing one;
these pin it for this seat, including that a different bus implementation is
refused for the same reason — the property that lets the bus be replaced.
Task 1.4. The foundation set only — a seat's streams come at registration
and a module's consumers at assignment, neither of which has happened at
genesis (ADR 0118).
Asserted rather than created: a stream that was deleted, or a mesh raised
from a backup, must converge rather than run without the guarantee its
messages assume.
Two things the definitions have to get right, both tested:
- CONTROL names its subjects instead of taking mesh.control.>, because
heartbeats live under that prefix and a stream of them competes for
retention with the messages that matter
- EVENTS filters on the event token, which is why that token exists; a
filter over a module's whole namespace would persist every tool call
Overlapping filters are refused where the set is written: NATS accepts two
streams matching one subject and stores the message twice under two
retentions, which nothing reports.
Adds nats.go as a dependency; it pulled golang.org/x/* forward. Full suite
green.
Task 1.3 of novox/hq ADR 0116. On AMQP an account was an HTTP call; on NATS
it is text the controller composes and the server reloads (ADR 0106). Pure,
so the mesh's whole authority model is testable as strings.
NATS closes a gap management.go recorded rather than hid: LavinMQ has no
topic permissions, so an emitter was granted the events exchange whole and
ADR 0042's origin reservation was "stamped by the sdk, not enforced here".
Per-subject permissions make it the server's refusal.
Two things found by composing a real file rather than reading the design:
- a scoped inbox leaves a responder unable to reply, because the answer goes
to the caller's inbox. allow_responses is the answer — one reply to the
subject of a message actually received — and only principals that serve
are granted it. Recorded in design 25 §4.
- composition must be deterministic: the module's entrypoint reloads on the
file's digest, so an order-dependent composer would reload the whole bus
on every controller restart. Covered by a test.
The golden fixture is the exact text `nats-server -t` accepts, so the syntax
is the server's rather than one we invented.
The bus is the only broker (novox/hq ADR 0117): messaging is subjects on it,
scoped by a module's own emits/consumes, not a server handed out as a
provision. So the seat joins mesh-controller and the-catalogue in delivering
no interface. Full suite green.
checkResources compared paths as written, so ${dir:state}/server.env —
the same characters in every module, a different directory in each —
refused the first two placed modules that met. Paths are placed before
they are compared, under the default root, which keeps every real
collision: distinct modules' places are distinct under any one root,
and a module stating another's placed root is caught because a pathless
directory now owns its placed path in the comparison too.
Slice two of ADR 0112. A pathless directory saying place "." is the
assignment's one directory, <root>/<module> — to-be 27's shape — and
place never reaches the host, which parses strictly. The maps naming
where bindings, credentials and contributions land (binds, secrets,
own-secrets, receives, grants) fill against the placed directories at
composition, into fresh maps and a fresh module slice, because one
resolution composes for many nodes. The five absolute-path checks on
those maps accept a placed reference — resolution makes it absolute
before anything reads it — while certificate, operator-keeps and
accesses paths stay absolute-only: those are the operator's or another
vocabulary's. unknownDirRefs scans the maps too, and validates place
itself: only on a directory, only ".", never beside a stated path.
Found by the foundation tests validating the sibling catalogue: the
first conversion's blanket replace turned /var/lib/gitea/database.json
into ${dir:data}base.json — which resolves to the right path by pure
string concatenation. Production was saved by a coincidence; the
catalogue cleanup that follows spells it ${dir:state}/database.json.
The first executable slice of ADR 0112 / to-be 27, sized to what the
operator settled tonight: a module definition names no host path for
its own data. A directory resource may omit path; composition resolves
it to <root>/<module>/<id>, the root a node's setting on Rendering with
/var/lib as the default — which reproduces exactly the layout novox
converged to by hand. ${dir:<id>} names the place from a resource's
path, content, mounts, environment and env-files, the same shape as
${bound:…}. A directory that states a path keeps it and still answers
by name — that is the adopted-data placement, mssql its live case.
Resolved in the controller at composition, so the wire format and the
host change not at all; a reference naming no directory refuses at the
manifest and again at composition; nested fills are rebuilt, never
written into the manifest's own maps, because one manifest composes
for many nodes.
Proxy configuration beside insecure, not a fifth policy — ADR 0108 closed that set at four, and both
of these tune how a request is carried rather than deciding what a name admits. A registry is the
case that needs it: image layers arrive as single requests of gigabytes and a proxy's own default
refuses them long before the workload is reached.
Absent is no limit, which is what every route already got. A limit that is not a whole positive
number of bytes takes the route with it, named in the log like a port that is not one — serving it
without the limit would carry exactly what the module said not to carry. Enforced on the declared
length where there is one, and while reading for a chunked body, which declares none: without the
second, a limit is advice.
The swap from a named volume to the host directory left the mount undeclared, which main's own
manifest check now refuses: a bind the module did not declare is created by the runtime as root, so
the module's owner and mode never reach it and ADR 0030's data rule does not cover it.
The directory is the broker's — lavinmq declares it as its own, mode 0700 — so from here it is an
access, read-only: a pre-existing path this module is granted use of and does not own.
autocert checks the host policy before the token and answers 403 — the
internal authority does this for every public name, so mail.novox.be's
challenge died on the internal manager's probe one commit after it
stopped dying on the public one's 404. Both shapes of refusal now fall
through to routing; a fifth test pins the 403 case with a refusing
policy.
autocert's HTTPHandler answers 404 itself for a token it does not hold
and never consults its fallback on the challenge path — the
predecessor's exact fault, rediscovered live when Mailu's renewal died
behind this proxy on cutover day. tokenOrRoute probes each authority
against a buffered writer and hands a token none of them holds to plain
routing, so a consumer's own ACME client answers its own challenge
through an ordinary path-scoped route. Four tests pin it, including the
cache-key shape a restart-surviving token actually has.
A private repository could not be built: the builder clones anonymously,
and had no way to say who it is. It already holds exactly one credential
to exactly the right place — the package-registry binding and its sealed
secret, one gitea user whose password answers npm and git alike — so a
clone now offers that, and nothing new is minted or carried.
Offered, never pushed: the credential is written as a git
credential-store file (0600, in the workspace, never argv) and named
with -c credential.helper, so git itself decides when it applies — only
on an authentication challenge, and only for the URL it was written
for, scheme, host and port included. A public repository clones exactly
as before; a repository on any other host is never shown it. The same
store rides along on an artifact's own context clone, so a private
module with a private context builds too.
Implements novox/hq ADR 0110 and 0111.
The seat set lives in internal/catalogue/seats.go: fourteen seats, each with a scope, what occupying
it delivers, and the record that made it one. A test asserts the count and a decision per entry, so
changing the set means finding the argument, as the host's vocabulary test does. The first set is
every seat already claimed — including the-private-network, which the network module claims from a
manifest composed in this repository's code, not from any module.json — plus npm-package-registry
(ADR 0109) and git (ADR 0111). A test parses every catalogue manifest and this repository's own and
fails on any refused claim, so closing the set refuses nothing in use.
ParseManifest now refuses a claim on a seat the mesh does not define, a seat claimed at another
scope, and a delivering seat claimed by a module that does not provide what it delivers. A
malformed claim is refused once, for being malformed.
Resolution: among several providers of a mesh provision, a pin still wins; then the holder of the
seat that delivers it; then the only provider; otherwise refused as before. ADR 0009's "never
guessed" holds — the seat is the choice made once, mesh-wide, rather than a pin per consumer node.
A provider now carries the module it came from, because a provider is a (node, module) pair and the
pair is what tells a holder from a neighbour on the same machine.
The planner's second pass is now given the first pass's holdings. Without them, a node consuming a
seat-delivered provision was refused there, and a refused node's own claims dropped out of what the
mesh holds — letting a second holder of one of its seats pass unrefused.
`seats [--json]` lists every seat, what it delivers, and each holder, derived from assignments
every time and never stored. Unheld seats are listed. A stored claim outside the set — possible
for a manifest registered before the set closed, since stored manifests are not re-validated — is
shown rather than hidden.
`build --self <owner>/<repo>` builds from a repository on the git seat's holder. The clone URL is
composed at build time from the holder's node and what it serves for git; the recorded source is the
path and the seat (migration 0032), never an address, so a moved forge changes nothing recorded.
Nobody holding the seat refuses self-hosted builds and says so; external URLs are unchanged. An
address passed with --self is refused rather than recorded as a path.
Replaces three foundation tests that defended the builder's carried package binding. The catalogue
removed that binding when the builder began requiring the registry through a real grant, so the
tests were already failing on main; they now assert the builder requires what the npm seat delivers
and carries no copy of its own, and that the forge holds the npm and git seats.
Verified: go vet clean; the whole suite passes against a throwaway Postgres (make postgres), the new
inventory tests included; gofmt clean apart from cmd/mesh-builder/stdout_test.go, which fails on
main too.
Internal aliases were served over plain HTTP only — correctly refused a
public certificate (no public CA can validate a private name), and then
left with nothing. The mesh has two authorities for its two name spaces
(08-connectivity §2), so the proxy now takes an optional internal ACME
directory and dispatches at the handshake by the same question HostPolicy
already answers: which authority may certify this name at all.
A route may also say its target speaks https, with insecure for a backend
whose own certificate nothing would trust — the shape Mailu's webmail
front needs, and the exception: everything else the mesh hands this proxy
stays plain http on the private network.
route-proxy's own Dockerfile documents the shape it has always needed and
never had: 'the proxy source is not vendored here... the build context is
the mesh-controller repository root, and this Dockerfile compiles
./examples/route-proxy from it.' Nothing in the mesh could do that — the
build command clones one repository and builds every artifact from
within it, so route-proxy has never once been built through the pipeline,
consistent with it never having been assigned anywhere. Found attempting
exactly that build tonight: 'stat go.mod: file does not exist', because
the context was mesh-catalog, which does not have one.
An image artifact may now carry a context: {repository, ref}, cloned
fresh alongside the module's own tree. The recipe (Dockerfile) is still
read from the module's own directory, at the module's own commit — only
docker build's own context argument moves. Packaging and source stay
exactly as separate as route-proxy's own comment already said they were,
now for real.
A route now consumed with two hosts when the mesh composed both — the
same host under internal-name reaches the same rule as its public name,
restoring the convenience a predecessor proxy gave for reaching a service
over the VPN without a public TLS round trip (the field composeName now
writes, feat/route-carries-internal-alias — this branch depends on that
one landing for internal-name to ever be populated; builds and tests
clean without it, just serves nothing extra).
Never certified: onlyWhatTheMeshSaid used routed(), which answered yes
for any host in the table regardless of how it got there. A new
eligibleForACME() checks a parallel 'public' set instead — every host
reached through a route's own name, never one reached only through its
internal-name — so an internal alias is proxied but never given its own
failing ACME order. routed() is unchanged and still used for the 404
message, which legitimately wants 'is this host served at all.'
jschoubben
jochen