Compare commits

...
Author SHA1 Message Date
jschoubben 878690697e A push raises what a module hears, not only a start
A module's declaration and its consumer are derived from the same records, and only one of them
followed a push: the consumers were raised when the control plane started serving, so a module that
gained a `consumes` was sent a declaration it could act on and a consumer that never delivered the
event — with nothing anywhere saying the two disagreed. Found on review: the catalogue's own replay
subscription was recorded, granted and never delivered.

Everything the raise does is idempotent, so a push may do it.
2026-09-28 16:46:42 +02:00
mesh-admin ad97297576 Merge pull request 'The seat declares the facts its holder states' (#131) from fix/the-seat-declares-the-facts-its-holder-states into main 2026-09-28 14:37:05 +00:00
jschoubben 683b1ed693 The seat declares the facts its holder states
The grant permitted the control plane to state what it applied and the seat said nothing about it, so
the check that every derived subscription has an owner found the catalogue subscribing to a subject
nothing publishes — which is exactly the fault that check exists for, pointed at me.

A seat carries the protocol of its role (novox/hq ADR 0129), so the facts are the mesh-controller
seat's `emits`. That is also what lets another module declare it consumes them. No accepts, so no work
queue is raised for the seat — only what its holder may say. The agreement test now holds all three
places to one another: the seat, the grant, and the words the mesh states them with.
2026-09-28 16:36:45 +02:00
mesh-admin 04f9f378b0 Merge pull request 'Two faults found on review' (#130) from fix/review-two-small-faults into main 2026-09-28 14:32:43 +00:00
jschoubben 1c3f44a526 Two faults found on review
A second Accept value would have overwritten the first, because the header was Set per value rather
than Added. One value is all any caller passes today, so nothing was wrong — but a helper that
quietly keeps only the last of what it was given is a trap for whoever passes two.

And a replayed announcement that could not be written was published as an empty body: a fact on the
mesh that says nothing, which the reader can only log and drop. It is now said and skipped, because a
body that cannot be marshalled is this program's fault rather than the bus's.
2026-09-28 16:32:41 +02:00
mesh-admin 89e152dfe2 Merge pull request 'The mesh says what it applied, and the replay has an address it may use' (#129) from feat/the-mesh-says-what-it-applied into main 2026-09-28 14:07:20 +00:00
jschoubben 1ebad3786c The mesh says what it applied, and the replay has an address it may use
The pipeline was observable from a merge to an artifact and went dark where it touched a machine: a
node's report is control traffic only the control plane reads, so nothing said which version a
machine runs, or that it refused to (novox/hq ADR 0134). The control plane now states both under the
seat it holds — a role's events belong to the role and keep their address when the holder is
replaced — and only when the report is news, because a machine reconciles every minute and a fact per
report would be a fact per minute per machine.

Whether a report is news is the store's answer: it holds the previous one, so the listener returns it
and the server states the fact. That also gives the catch-up replay a subject the controller may
publish: it was published as a module's event from a module called "control-plane", which does not
exist, so the controller's own account refused it and every catalogue that asked what it missed was
answered with nothing.
2026-09-28 16:07:18 +02:00
mesh-admin f2f526a60a Merge pull request 'A module's name may contain a dot, so the derived step adds none' (#128) from fix/a-modules-name-may-contain-a-dot into main 2026-09-28 13:44:21 +00:00
jschoubben 4b4c7e0e0d A module's name may contain a dot, so the derived step adds none
A resource's id is `<module>.<its own id>` and a module's name may itself contain a dot — novox.be is
one — so the owner of a resource is everything before the *last* dot. The preparation step's id used a
dot, which made its owner unreadable by that rule; it uses a hyphen, and the id says what it belongs
to whichever way a reader splits it.
2026-09-28 15:44:18 +02:00
mesh-admin cec792ce9d Merge pull request 'A manifest HEAD says what it accepts, or the registry answers 404' (#127) from fix/a-manifest-head-says-what-it-accepts into main 2026-09-28 11:02:10 +00:00
jschoubben 338d033632 A manifest HEAD says what it accepts, or the registry answers 404
The check that skips copying a base the mesh already holds asked with no Accept header, and a
registry answers a manifest only in a media type the caller named: the same digest answered 200 with
the manifest types and 404 without them. So the builder concluded it held nothing, copied every
vendor base again, and exhausted the public hub's pull limit a second time today.

The test could not have caught it, because the fake registry answered a manifest HEAD regardless of
Accept — more permissive than the thing it stands in for. It is now as strict as a real registry, and
fails without the fix.
2026-09-28 13:02:08 +02:00
mesh-admin 1be926cec4 Merge pull request 'The control plane prepares its own state, like any module' (#126) from feat/the-control-plane-prepares-its-own-state into main 2026-09-28 10:51:30 +00:00
jschoubben 2134768dfe The control plane prepares its own state, like any module
Now that every parser on the mesh knows the word, the control plane's manifest says it. Its schema
stops being a special case: the mesh derives the step from its own resource and gates its server on
it, which is the failure of novox/hq 04-ISSUES/133 closed by the mechanism rather than by a
hand-written step in one manifest.
2026-09-28 12:51:27 +02:00
mesh-admin ef825688ee Merge pull request 'The control plane learns 'prepares' one release before its manifest uses it' (#125) from fix/the-word-ships-before-the-manifest-uses-it into main 2026-09-28 10:49:36 +00:00
jschoubben 77a14360df The control plane learns 'prepares' one release before its manifest uses it
A manifest word has to reach every parser before a manifest carries it. The builder refused
mesh-controller's manifest with `unknown field "prepares"` until it was rebuilt; then the running
control plane could not read the manifest in the build result either, so the build was recorded with
no module and the version never moved. Strict parsing is deliberate (novox/hq 04-ISSUES/003), so the
word ships first and a manifest uses it next: this takes `prepares` back out of the control plane's
own manifest, leaving the code that understands it, and the manifest says it again once this is
running everywhere.
2026-09-28 12:49:33 +02:00
mesh-admin 9be2fb4750 Merge pull request 'A version prepares its state before it runs' (#124) from feat/a-version-prepares-its-state into main 2026-09-28 10:43:17 +00:00
jschoubben 5a963aec10 A version prepares its state before it runs
The mesh derives the preparation from the module's own resource instead of each module hand-writing
a step beside it (novox/hq ADR 0135). A manifest says one word — `prepares` — and the mesh runs that
module's own program in its preparation mode, in the module's own context: the same image, the same
environment, the same mounts, because it is the same code. A published port and a fixed address are
taken away rather than copied, since the version being replaced still holds them.

One word for every kind of module: a Go binary receives `prepare` as its argument, a bundle receives
it through the runtime whose entry takes the same word. The control plane answers it like anything
else — its own schema stops being a special case, and its hand-written step is gone.
2026-09-28 12:43:15 +02:00
mesh-admin 45d1c28a28 Merge pull request 'The control plane migrates before it serves' (#123) from fix/the-control-plane-migrates-before-it-serves into main 2026-09-28 08:27:44 +00:00
jschoubben a3e7683c63 The control plane migrates before it serves
The mesh replaced its own control plane with a build carrying a migration, applied none of it, and
then refused every build it recorded for three quarters of an hour while reporting itself healthy
(novox/hq 04-ISSUES/133). The module now declares the step ADR 0052 prescribes: a run-once
`migrate` before the server, re-run whenever the image moves because the image is part of a step's
digest, and gating — a migration that fails stops the new server from starting rather than letting
it serve against a schema it does not have.
2026-09-28 10:27:42 +02:00
mesh-admin da394b45e6 Merge pull request 'Work slower than the window says so, and one address is the bus's' (#122) from fix/work-longer-than-the-window-says-so into main 2026-09-28 07:51:52 +00:00
jschoubben 2f3bfda8c0 Work slower than the window says so, and one address is the bus's
Three faults the mesh's own logs showed this morning. A handler that outlives the acknowledgement
window was handed its message again while it was still working: acting on a merge builds modules,
minutes against a thirty-second window, so one merge ran the whole catalogue five times over. The
transport now says the work is in progress while it runs, which is where the window belongs.

Everything the mesh hands out — a token, a membership, a person's credential — took its address
from the enrolment setting, which on a mesh that has moved still names the broker it moved from:
the first person issued after the move was handed the retired broker's port. There is one bus, and
its address is the one the control plane is connected to.

And `operator issue` documented an argument order its parser refused.
2026-09-28 09:51:50 +02:00
mesh-admin 208388978a Merge pull request 'A merge rebuilds what it changed, and what packages it' (#121) from feat/a-merge-rebuilds-what-it-changed into main 2026-09-28 07:20:03 +00:00
jschoubben aa771616bb A merge rebuilds what it changed, and what packages it
Three faults in one path. A merge rebuilt every module built from the repository, so one change in
a repository holding twenty-six of them meant twenty-six builds. A merge into a repository a module
only *packages* source from rebuilt nothing — two modules are built from the control plane's own
repository and neither had ever been rebuilt when it moved — because the manifest the mesh keeps
carries no build section, so a build now says which repositories it read and the mesh keeps that
beside what it stood on. And a module handed over by hand could record a repository with no
directory inside it, which is a module nothing can ever rebuild (novox/hq 04-ISSUES/131, /132).

A change inside no module's own directory is a change to what they share, and everything built from
that repository is rebuilt: rebuilding too much is the safe direction, because the fault this whole
path exists for is a mesh that believes it is current and is not.
2026-09-28 09:20:01 +02:00
mesh-admin 1513bbaac9 Merge pull request 'An older merge does not move a source' (#120) from fix/an-older-merge-does-not-move-a-source into main 2026-09-28 03:12:40 +00:00
jschoubben 0014984116 An older merge does not move a source
The forge announces what it finds merged, and an old merge surfacing late moved the recorded head
backwards and rebuilt everything built from that repository, once per old merge. A merge made
before the source was last seen is history; one that says nothing about when is taken as news.
The catalogue now carries when each source was last seen. The bus-records test follows #116:
a module's tools are every one under its own name.
2026-09-28 05:12:37 +02:00
mesh-admin d6e49dbd68 Merge pull request 'A base the registry already holds is not pulled from upstream again' (#119) from fix/a-mirrored-base-is-not-pulled-twice into main 2026-09-28 02:48:35 +00:00
jschoubben 3756bb3460 A base the registry already holds is not pulled from upstream again
A base is named by digest, and a digest the mesh's registry holds under the module's repository
is the same bytes whatever upstream would say. Asked on every build, the public hub's anonymous
pull limit was reached on the first merge that rebuilt a whole catalogue, and every module whose
base lives there failed on a copy it did not need.
2026-09-28 04:48:32 +02:00
mesh-admin 60be9c5360 Merge pull request 'A module hears what it consumes: its consumer is raised with the bus, and it pulls it' (#118) from fix/a-module-hears-what-it-consumes into main 2026-09-28 02:29:34 +00:00
jschoubben da31bcb11e A module hears what it consumes: its consumer is raised with the bus, and it pulls it
Every module moved onto the bus by the rollout was issued on the old one, so none had a consumer
waiting; and the grant named a push delivery a runtime's client never binds, while the pull it
does make — asking about its consumer, asking it for messages — was refused. The consumers a
module's declarations imply are now raised whenever the bus is, and the grant is the pull.
2026-09-28 04:29:32 +02:00
mesh-admin f03e7b33c9 Merge pull request 'The controller may ask any module's tool' (#117) from fix/the-controller-may-ask-a-tool into main 2026-09-28 02:21:26 +00:00
jschoubben 0baf727f36 The controller may ask any module's tool
The control plane is the way in for tool calls (novox/hq ADR 0095): a person or an agent asks
through it, so it alone may publish to every module's tool subject. The first ask on the new bus
was refused the publish.
2026-09-28 04:21:25 +02:00
mesh-admin 94dd49a968 Merge pull request 'A module serves every tool under its own name, and may answer' (#116) from fix/a-module-serves-its-own-namespace into main 2026-09-28 02:14:52 +00:00
jschoubben 83a298e7e0 A module serves every tool under its own name, and may answer
Every module that served a tool was refused the subscription on the new bus: the grant listed
tools from a manifest field no module fills, because the tools a module serves are what its code
answers and a second copy of that list would be a second source of truth. The grant is now the
module's own tool namespace; nothing else may subscribe it, a caller is still granted per tool by
name, and a module may answer what it was asked.
2026-09-28 04:14:47 +02:00
mesh-admin 220b79f5cd Merge pull request 'rollout check dials the bus the way the mesh does' (#115) from fix/the-check-dials-as-the-mesh-does into main 2026-09-28 02:05:35 +00:00
jschoubben e62201e227 rollout check dials the bus the way the mesh does
The probe connected bare, and a bus that requires TLS and a user refused it at the handshake —
so the check reported the standing server as absent. It now dials with the controller's own
credential and pin, which is the one fact the check is there to report.
2026-09-28 04:05:32 +02:00
mesh-admin 0ab9b86f0a Merge pull request 'An edge is recorded by path, like the artifact it points at' (#114) from fix/an-edge-is-recorded-by-path into main 2026-09-28 01:52:10 +00:00
jschoubben c7aabd3037 An edge is recorded by path, like the artifact it points at
The test pinned what a build stood on to the address the builder pulled from; the edge names
another module's artifact and is kept the way that artifact is (novox/hq 04-ISSUES/102).
2026-09-28 03:52:08 +02:00
mesh-admin c74d990cee Merge pull request 'A build records the bases it was handed, and the mesh reads its edges from builds' (#113) from feat/build-edges-are-recorded into main 2026-09-28 01:51:16 +00:00
jschoubben 35252af665 A build records the bases it was handed, and the mesh reads its edges from builds
Bases reach a recipe as build arguments, so the digest was never in the file the builder read
edges from: no build on the mesh recorded what it stood on, and 'build --on', the bases-first
order and the merge follow-up all walked a graph with no edges (novox/hq 04-ISSUES/131). The
builder now reports every base it resolved; the controller records them by artifact path and
reads the newest build's edges from the store, since a recorded manifest carries no build.on.
2026-09-28 03:51:14 +02:00
mesh-admin aab6ded41b Merge pull request 'One bus: the AMQP transport is gone from the controller' (#112) from feat/one-bus into main 2026-09-28 01:36:27 +00:00
jschoubben aecac5bda2 One bus: the AMQP transport is gone from the controller
The mesh runs on the seat's bus alone (novox/hq ADR 0131, design 28 task 5.5). The old
transport's consume loop, build request, tool ask, management API and account scoping are
deleted, and the bus switch with them; the controller connects to the broker seat and to
nothing else. The store-window tests keep their assertions on a bus-less fake, and the tests
that only made sense for the old transport's in-memory holding go with it.
2026-09-28 03:36:16 +02:00
mesh-admin 30362118a1 Merge pull request 'The controller follows the subject it decodes' (#111) from fix/the-controller-follows-what-it-decodes into main 2026-09-28 01:15:18 +00:00
jschoubben 81e76fa485 The controller follows the subject it decodes
The decoder named the forge's merge subject as the fourth thing followed and the
list was three long: every message that fell through to that switch panicked the
control plane (2026-09-28). The entry was written and lost between two attempts
at the same edit. A test now walks the list; the composed grants and the genesis
template carry the subject.
2026-09-28 03:13:57 +02:00
mesh-admin 12ed35e87d Merge pull request 'A merge on the forge builds what it moved, bases first' (#110) from feat/a-merge-on-the-forge-builds-what-it-moved into main 2026-09-28 00:59:04 +00:00
jschoubben 525f10b858 A merge on the forge builds what it moved, bases first
The controller follows the forge's merges (novox/hq 04-ISSUES/131). For each
module recorded as built from that repository and branch it records the move to
the merge commit and builds it — bases first, because a module built before the
module it stands on is built against the old one and reports success, and a base
that fails stops what stands on it. Nothing is pushed here: what a finished build
does to the machines running the module stays the upgrade's decision.

Two more things the same ordering gives: `build --behind` builds bases first, and
`build --on <module>` rebuilds everything that stands on a module — the rebuild a
changed base needs, which "behind" does not see because their sources did not
move.
2026-09-28 02:59:02 +02:00
mesh-admin 0547316cf2 Merge pull request 'rollout hand: a machine's membership, minted afresh and handed to an operator once' (#109) from feat/rollout-hand into main 2026-09-28 00:40:25 +00:00
jschoubben 9fe9b5349c Merge pull request 'A store row keeps its seat's protocol, and a holder may take work from its queue' (#108) from fix/store-seats-keep-their-protocol into main 2026-09-28 02:21:24 +02:00
jschoubben d1e488efaf Merge pull request 'A store row keeps its seat's protocol, and a holder may take work from its queue' (#108) from fix/store-seats-keep-their-protocol into main 2026-09-28 00:09:28 +00:00
jschoubben c5dc7e732a A store row keeps its seat's protocol, and a holder may take work from its queue
The seat table has name, scope, delivers and decision, and the protocol ADR 0129
gave a seat lives only in the compiled defaults; loading the rows dropped it, so
no role's work queue was ever raised and the first build submitted over the new
bus met "no response from stream". Until the table gains the columns, a row with
no protocol keeps the compiled one of its name. And the holder of a seat is
granted what taking work from its queue needs — asking about the worker consumer
it binds, and acknowledging on it — which the first machine to try was refused.

The control plane's own seat placeholders no longer include the old bus's port,
which the switch removed with the variable.
2026-09-28 02:08:56 +02:00
jschoubben 7efcccd013 Merge pull request 'The build machine takes work on the bus its credential names, and the work queue has a taker' (#107) from feat/the-build-machine-takes-work-on-nats into main 2026-09-27 23:59:56 +00:00
jschoubben 964285f08c The build machine takes work on the bus its credential names, and the work queue has a taker
Two halves of one gap the first build over the new bus met. The machine decided
its bus from a variable its container never received, so the credential the mesh
sealed to it went unread; a credential for the new bus names the bus by scheme and
carries user, password and fingerprint beside the address, and that is enough to
dial it, pinned. And the roles' work queues were raised with no holders, so the
consumer a machine binds to take work was never created: the holders are read
from the catalogue and the handover record, as the resolver reads them.
2026-09-28 01:59:20 +02:00
jschoubben 5698dda11f Merge pull request 'A principal may hear what its consumer delivers' (#106) from fix/a-principal-may-hear-its-consumer into main 2026-09-27 23:47:29 +00:00
jschoubben 6005a8471f A principal may hear what its consumer delivers
A push consumer delivers on _DELIVER.<its name>, and a client bound to it
subscribes exactly that. No principal was granted it, and the server refused
every one the first time it bound a consumer: the control plane, each machine,
and a module would have been next. Each kind is granted its own consumers'
delivery subjects and no other's. The line announcing the raised bus printed the
URL with the credential in it; the address alone now.
2026-09-28 01:46:16 +02:00
jschoubben e2ee0dfe98 Merge pull request 'The bus account has JetStream, and the control plane's client has its own inbox' (#105) from fix/the-bus-account-has-jetstream into main 2026-09-27 23:40:38 +00:00
jschoubben 70341cfbc7 The bus account has JetStream, and the control plane's client has its own inbox
Two refusals the first live connections met. A user in the MESH account was told
"JetStream not enabled for account" the first time it bound a consumer: with
accounts defined, JetStream is enabled per account, not only globally — the
account's setting, which the mesh owns, not the server's block, which it does not.
And the control plane's client used a random inbox prefix where it is granted
exactly _INBOX.<its user>.>, so the server's first answer could not reach it. The
prefix now follows from the user in the URL, for every principal that dials so.
2026-09-28 01:40:10 +02:00
jschoubben 77643aa3f4 Merge pull request 'The control plane pins the bus's certificate, and keeps its password out of errors' (#104) from fix/the-controller-pins-the-bus-certificate into main 2026-09-27 23:35:56 +00:00
jschoubben 1fd6194ff8 The control plane pins the bus's certificate, and keeps its password out of errors
The bus presents the mesh's own certificate, which names nothing a public verifier
accepts; the client verified by name and failed against a bus that was answering
("certificate is not valid for any names", 2026-09-28). It now pins the leaf's
fingerprint from MESH_BROKER_CERTIFICATE, as every host does. And a connection
error named the whole URL, password included — the address alone now.
2026-09-28 01:35:20 +02:00
jschoubben c37018fdd2 Merge pull request 'The control plane serves and pushes on the bus it is told to' (#103) from feat/the-controller-serves-on-nats into main 2026-09-27 23:30:52 +00:00
jschoubben 3907ea0db0 The control plane serves and pushes on the bus it is told to
The seams were there and nothing chose a side: serve, push, ask and build all
opened the old bus's connection and declared over its channel, whatever
MESH_BUS_NATS said. So the switch moved every host and left the control plane
unable to follow — "this control plane has no MESH_BROKER_AMQP" with the new bus
named and standing (2026-09-28). That was task 4.3 of design 28, still open.

One place now decides: connectLink reads the switch, refuses both buses named at
once, raises the new bus's streams and this controller's consumers when it is
handed the inventory, and opens the link over whichever bus it is on. Every
caller that sent a declaration or asked a tool through the old channel goes
through the server's bus instead, which the new transport has and the channel is
not. OverNats is that outbound: a declaration is a JetStream publish into the
node's own subject, an event is announced on the subject its name derives to, a
tool is request and reply on the module's tool subject.
2026-09-28 01:29:44 +02:00
jschoubben 40f5e9a41c Merge pull request 'A machine may bind its consumer' (#102) from fix/a-node-may-bind-its-consumer into main 2026-09-27 23:18:20 +00:00
jschoubben 2c2eb51878 Only CONSUMER.INFO was missing from a machine's grants; the rest was already there 2026-09-28 01:17:40 +02:00
jschoubben aa2d0b51ea Golden: a machine's user may bind its consumer, ack, and hear its inbox 2026-09-28 01:17:15 +02:00
jschoubben 64d154d9d7 A machine may bind its consumer and hear the answer
Binding to a consumer asks the server about it and hears the answer on the
client's inbox; hearing a declaration acknowledges it. A machine's user was granted
none of that and was refused the first time one dialled a permissioned server:
"this node cannot read its declarations". Its inbox is its own prefix, which the
host now sets.
2026-09-28 01:16:46 +02:00
jschoubben ffa390f916 Merge pull request 'The mint leaves the control plane's old-bus secret alone' (#101) from fix/mint-leaves-the-control-planes-old-secret-alone into main 2026-09-27 23:08:35 +00:00
jschoubben 4d62e6caf1 The mint leaves the control plane's old-bus secret alone
The control plane is a module too, and its broker secret is the old bus's
credential it is still using while the mint runs. Writing the new bus's blob there
cut the mesh off from its own old bus mid-move. Its new-bus credential is the
controller principal's bus secret; the module principal is skipped.
2026-09-28 01:08:11 +02:00
jschoubben 386ae676ca Merge pull request 'The control plane mounts the bus secret it reads' (#100) from fix/the-controller-mounts-its-bus-secret into main 2026-09-27 23:03:42 +00:00
jschoubben f8a9c3d6bc The control plane mounts the bus secret it reads
MESH_BUS_NATS_FILE named /run/secrets/bus and nothing put a file there: the
manifest binds each secret explicitly, and the switch added the secret and the
variable but not the bind. Found live — the control plane came up on the new bus
and could not read its own credential.
2026-09-28 01:03:18 +02:00
jschoubben 83671fae5f Merge pull request 'The network map resolves each machine with the seat holders on record' (#99) from fix/the-network-map-knows-the-holders into main 2026-09-27 22:56:55 +00:00
jschoubben 9b715524a2 The network map resolves each machine with the seat holders on record
Without them, a machine running the next holder of a seat beside the current one
resolves as two holders, is refused, and drops out of the map — and with it the
address every other machine composes for what it offers. Found live: the control
node vanished from the private network the moment the new bus was assigned beside
the old one, and nothing on any machine could be composed.
2026-09-28 00:55:39 +02:00
jschoubben e06fc1ed16 Merge pull request 'The control plane speaks the new bus' (#98) from switch/the-controller-speaks-nats into main 2026-09-27 22:38:27 +00:00
jschoubben 13d7c5c5dd Merge pull request 'The mint names the bus by bare host, and can mint again' (#97) from fix/mint-bare-host-and-again into main 2026-09-27 22:37:36 +00:00
jschoubben 7b02feaebb The mint names the bus by bare host, and can mint again
BareAddress adds a scheme where none was, so the host it yielded carried one and
every URL built from it carried two. Caught before a push: the host is now taken
with no scheme and no port, and every URL adds its own. `rollout mint --again`
mints every credential afresh for exactly this case — a mint that was wrong before
anything received it.
2026-09-28 00:37:10 +02:00
jschoubben bd10e2c695 Merge pull request 'The mint finds the bus on the hub' (#96) from fix/mint-finds-the-bus-on-the-hub into main 2026-09-27 22:34:41 +00:00
jschoubben 4b209d944d The mint finds the bus on the hub
The map of who is where on the private network lists the machines placed around
the hub, not the hub — and the control node is the hub, and runs the new bus.
Found on the first live run: refused for having no address. The address every
machine already dials the current bus at is the same machine, so that host with
the new port is what they are told.
2026-09-28 00:34:17 +02:00
71 changed files with 2546 additions and 2036 deletions
+28 -77
View File
@@ -17,12 +17,7 @@ package main
import (
"context"
"crypto/sha256"
"crypto/tls"
"crypto/x509"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"net/url"
"os"
@@ -30,8 +25,6 @@ import (
"strings"
"syscall"
amqp "github.com/rabbitmq/amqp091-go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/builder"
"github.com/novox/mesh-controller/internal/link"
@@ -52,7 +45,6 @@ const usage = `mesh-builder — builds modules for the mesh
It consumes build requests and answers with what it made. Nothing is listened on and nothing
is dialled except the broker.
MESH_BROKER_AMQP where the broker is, with this builder's own credential
MESH_BROKER_FILE a file the mesh sealed to this machine holding the same
MESH_REGISTRY host:port to publish artifacts to, when the mesh has not said
MESH_BINDING a file the mesh wrote saying where the artifact store is
@@ -135,32 +127,17 @@ func run() error {
// machine told about both would take work from one and answer on the other, and every log line would
// say it was fine.
func takeWorkFrom(credential Credential, on string) (link.BuildMachine, error) {
address, onNATS, err := broker.OnNATS()
// **The credential names the bus, and there is one** (novox/hq ADR 0131, design 28 task 5.5).
// A credential for the mesh's bus carries user, password and fingerprint beside the address,
// and that is enough to dial it, pinned.
if !credential.onTheNewBus() {
return nil, fmt.Errorf("the credential at hand names %q, which is not the mesh's bus", credential.URL)
}
js, err := broker.DialPinned(credential.natsURL(), credential.Fingerprint)
if err != nil {
return nil, err
}
if err := broker.MustBeOneBus(credential.URL, address); err != nil {
return nil, err
}
if onNATS {
js, err := broker.Dial(address)
if err != nil {
return nil, fmt.Errorf("cannot reach the bus at %s: %w", address, err)
}
return link.MachineOverNATS(js, on), nil
}
conn, err := dial(credential)
if err != nil {
// Not quoted back: the URL carries this builder's broker password.
return nil, fmt.Errorf("cannot reach the broker: %w", err)
}
channel, err := conn.Channel()
if err != nil {
conn.Close()
return nil, err
}
return link.MachineOverCurrent(conn, channel, on), nil
return link.MachineOverNATS(js, on), nil
}
// answer does one build and says what happened, whichever way it went.
@@ -217,6 +194,9 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
})
}
result.Against = built.Against
for _, r := range built.Read {
result.Read = append(result.Read, link.ReadRepository{Repository: r.Repository, Ref: r.Ref})
}
fmt.Fprintf(os.Stderr, " built %s from %s\n", built.Manifest.Module, short(built.Commit))
}
}
@@ -442,13 +422,8 @@ func brokerFrom() (Credential, error) {
// broker is then verified against whatever this machine already trusts.
return Credential{URL: said}, nil
}
url := strings.TrimSpace(os.Getenv("MESH_BROKER_AMQP"))
if url == "" {
return Credential{}, fmt.Errorf(
"neither MESH_BROKER_FILE nor MESH_BROKER_AMQP: a builder with no broker has " +
"nothing to build")
}
return Credential{URL: url}, nil
return Credential{}, fmt.Errorf(
"no MESH_BROKER_FILE: a build machine with no credential for the bus has nothing to build")
}
// Credential is what a build machine is given so it can reach the broker.
@@ -460,48 +435,24 @@ func brokerFrom() (Credential, error) {
// **The same shape a node gets, for the same reason** (novox/hq ADR 0004): the fingerprint travels
// out of band — here, sealed with the credential — and the endpoint is verified once at connect.
type Credential struct {
URL string `json:"url"`
// Fingerprint is SHA-256 over the broker certificate's DER bytes, or empty to verify the
// ordinary way.
URL string `json:"url"`
Fingerprint string `json:"fingerprint,omitempty"`
// User and Password ride beside the address on the bus being built (design 25): a credential
// embedded in a URL leaks into every log line that prints a connection, so the mesh seals them
// as two fields and this machine joins them once, here, to dial.
User string `json:"user,omitempty"`
Password string `json:"password,omitempty"`
}
// dial opens the connection, pinning the broker's certificate when there is one to pin.
func dial(held Credential) (*amqp.Connection, error) {
if held.Fingerprint == "" {
return amqp.Dial(held.URL)
}
return amqp.DialTLS(held.URL, pinning(held.Fingerprint))
}
// onTheNewBus is whether a credential is for the bus being built: its address says so, and the
// mesh only ever seals such a credential with the user and password beside it.
func (c Credential) onTheNewBus() bool { return strings.HasPrefix(strings.TrimSpace(c.URL), "nats://") }
// pinning is a TLS configuration that trusts exactly one certificate.
//
// InsecureSkipVerify with a VerifyPeerCertificate is **pinning, not skipping**: the standard chain
// check is replaced, not removed, and what replaces it is stricter — one certificate is accepted
// rather than every certificate a public authority would sign.
//
// Its own function so a test can drive it against a real handshake. A pin check that is only ever
// exercised through a broker is a pin check nothing tests.
func pinning(fingerprint string) *tls.Config {
return &tls.Config{
InsecureSkipVerify: true,
VerifyPeerCertificate: func(raw [][]byte, _ [][]*x509.Certificate) error {
if len(raw) == 0 {
return errors.New("the broker presented no certificate")
}
// The leaf, and in the same spelling the mesh writes it — `sha256:` and 64 hex
// characters. Comparing a bare digest against a written fingerprint never matches,
// and the failure is indistinguishable from being pointed at the wrong broker.
sum := sha256.Sum256(raw[0])
got := "sha256:" + hex.EncodeToString(sum[:])
if got != fingerprint {
return fmt.Errorf(
"this is not the broker this builder was told about\n expected %s\n "+
"got %s\nEither this mesh's broker was replaced, or this builder is "+
"being pointed at something else. Retrying will not help",
fingerprint, got)
}
return nil
},
// natsURL is the address with this machine's credential in it, for the one dial that needs it.
func (c Credential) natsURL() string {
rest := strings.TrimPrefix(strings.TrimSpace(c.URL), "nats://")
if c.User == "" {
return "nats://" + rest
}
return "nats://" + c.User + ":" + c.Password + "@" + rest
}
+18 -8
View File
@@ -106,6 +106,9 @@ func buildOnce(ctx context.Context, args []string) error {
Manifest: built.Manifest,
Against: built.Against,
}
for _, r := range built.Read {
out.Read = append(out.Read, readRepository{Repository: r.Repository, Ref: r.Ref})
}
for _, made := range built.Built {
out.Made = append(out.Made, madeArtifact{Name: made.Name, Kind: made.Kind, Reference: made.Reference})
}
@@ -123,14 +126,21 @@ func buildOnce(ctx context.Context, args []string) error {
// The same fields the mesh records for a build, so a reader comparing a genesis build against an
// ordinary one is comparing the same thing said the same way.
type onceResult struct {
Module string `json:"module"`
Commit string `json:"commit"`
Repository string `json:"repository"`
Path string `json:"path,omitempty"`
Ref string `json:"ref,omitempty"`
Manifest any `json:"manifest"`
Made []madeArtifact `json:"made"`
Against []string `json:"against,omitempty"`
Module string `json:"module"`
Commit string `json:"commit"`
Repository string `json:"repository"`
Path string `json:"path,omitempty"`
Ref string `json:"ref,omitempty"`
Manifest any `json:"manifest"`
Made []madeArtifact `json:"made"`
Against []string `json:"against,omitempty"`
Read []readRepository `json:"read,omitempty"`
}
// readRepository is a repository this build read source from besides the module's own.
type readRepository struct {
Repository string `json:"repository"`
Ref string `json:"ref,omitempty"`
}
type madeArtifact struct {
+2 -2
View File
@@ -14,8 +14,8 @@ func TestBuilderDiagnosticsStayOffStdout(t *testing.T) {
allowed := map[string]bool{
"string(body)": true, // once.go: the result JSON, which IS stdout
"version)": true, // --version
`"stopping")`: true, // the loop.s shutdown line
"usage)": true, // --help text, for a human
`"stopping")`: true, // the loop.s shutdown line
"usage)": true, // --help text, for a human
}
for _, file := range []string{"once.go", "main.go"} {
src, err := os.ReadFile(file)
+4 -2
View File
@@ -15,6 +15,8 @@ import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/broker"
)
// Where a builder publishes.
@@ -193,9 +195,9 @@ func TestThePinIsComparedInTheSpellingTheMeshWritesIt(t *testing.T) {
}
}
// handshakeWith runs the builder's own pin check against an address.
// handshakeWith runs the pin check the builder dials with against an address.
func handshakeWith(address, pin string) error {
conn, err := tls.Dial("tcp", address, pinning(pin))
conn, err := tls.Dial("tcp", address, broker.PinnedToFingerprint(pin))
if err != nil {
return err
}
+7 -4
View File
@@ -17,8 +17,8 @@ import (
var aDigest = "sha256:" + strings.Repeat("e", 64)
// **A build is recorded by digest and path**, whatever address the builder pushed to — and only
// what the build made is rewritten: an image the module runs from elsewhere is left where it says.
// **A build is recorded by digest and path**, whatever address the builder pushed to — what it
// made and what it stood on both; an image the module runs from elsewhere is left where it says.
func TestABuildIsRecordedWithoutTheStoresAddress(t *testing.T) {
manifest, _ := json.Marshal(map[string]any{
"module": "gitea", "version": "1",
@@ -65,8 +65,11 @@ func TestABuildIsRecordedWithoutTheStoresAddress(t *testing.T) {
if strings.Contains(string(kept.Manifest), "anchor.internal:5100") {
t.Errorf("the recorded manifest still carries the store's address:\n%s", kept.Manifest)
}
if kept.Against[0] != "anchor.internal:5100/mesh-tools/runtime@"+aDigest {
t.Errorf("what the build stood on was rewritten: %v", kept.Against)
// What the build stood on is an edge to another module's artifact, and it is recorded the way
// that artifact is: by path in the store, so the edge still names the same thing when the
// store answers at another address.
if kept.Against[0] != catalogue.ArtifactStoreScheme+"mesh-tools/runtime@"+aDigest {
t.Errorf("what the build stood on was recorded by address: %v", kept.Against)
}
}
+1 -1
View File
@@ -88,7 +88,7 @@ func reportsReaching(t *testing.T, open *stores, reachable []link.Reach, held ..
if err := open.inventory.RecordSent(ctx, record.ID, digestOf(body)); err != nil {
t.Fatal(err)
}
if err := (link.Enrolment{Inventory: open.inventory}).Heard(ctx, link.Report{
if _, err := (link.Enrolment{Inventory: open.inventory}).Heard(ctx, link.Report{
Node: "anchor", Applied: []string{"hello-web.x"}, Declared: digestOf(body),
Firewall: "ufw", Held: held, Reachable: reachable,
}); err != nil {
+2 -2
View File
@@ -37,13 +37,13 @@ func askCommand(ctx context.Context, args []string) error {
arguments = json.RawMessage(positionals[2])
}
server, err := link.Connect(nil, nil)
server, err := connectLink(ctx, nil, nil, nil)
if err != nil {
return err
}
defer server.Close()
answer, err := link.Ask(ctx, server.Channel(), module, tool, arguments, *wait)
answer, err := link.Ask(ctx, server.Bus(), module, tool, arguments, *wait)
if err != nil {
return err
}
+109 -89
View File
@@ -2,8 +2,6 @@ package main
import (
"context"
"crypto/rand"
"encoding/base64"
"encoding/json"
"errors"
"flag"
@@ -31,6 +29,51 @@ import (
// control plane may send a machine is bounded by the declaration language. This is the shape the
// builder module will take when it is given work over the broker; today a person runs it, and the
// mesh records the result the same way either way.
// buildOn rebuilds every module the mesh holds that stands on the named module's artifacts — the
// rebuild a changed base needs, which nothing else asks for: their sources did not move, and
// "behind" does not see a base that did (novox/hq 04-ISSUES/131). Bases first among them too.
func buildOn(ctx context.Context, base string, wait time.Duration) error {
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
held, err := open.inventory.Catalogued(ctx)
if err != nil {
return err
}
against, err := open.inventory.BuiltAgainst(ctx)
if err != nil {
return err
}
var on []inventory.Entry
for _, e := range held {
if standsOnModule(e, base, against) {
on = append(on, e)
}
}
if len(on) == 0 {
fmt.Printf("nothing the mesh holds stands on %s\n", base)
return nil
}
on = orderByBases(on, against)
fmt.Printf("%d module(s) stand on %s:\n", len(on), base)
var failed []string
for _, e := range on {
fmt.Printf("--- %s\n", e.Manifest.Module)
source := buildSource{Repository: e.Source.Repository, Seat: e.Source.Seat}
if err := buildOne(ctx, source, e.Source.Path, e.Source.Ref, wait); err != nil {
fmt.Printf(" %v\n", err)
failed = append(failed, e.Manifest.Module)
}
}
if len(failed) > 0 {
return fmt.Errorf("%d of %d could not be built: %s", len(failed), len(on), strings.Join(failed, ", "))
}
fmt.Printf("\n%d module(s) rebuilt on %s. `push --behind` sends them on\n", len(on), base)
return nil
}
func buildCommand(ctx context.Context, args []string) error {
set := flag.NewFlagSet("build", flag.ContinueOnError)
ref := set.String("ref", "", "the branch, tag or commit to build")
@@ -46,6 +89,7 @@ func buildCommand(ctx context.Context, args []string) error {
// retype each repository is asking them to be the loop. Naming a repository and asking which
// ones need building are different requests, so they are not combined.
behind := set.Bool("behind", false, "every module the mesh holds older than its source has")
on := set.String("on", "", "rebuild every module that stands on this module's artifacts — the rebuild a changed base needs")
// A repository on the mesh's own forge, named by its path there (novox/hq ADR 0111). Without it
// the repository is external, cloned exactly as given — see source.go.
self := set.Bool("self", false, "the repository is a path on the forge holding the git seat")
@@ -53,6 +97,13 @@ func buildCommand(ctx context.Context, args []string) error {
if err != nil {
return err
}
if *on != "" {
if len(positionals) != 0 || *behind || *self {
return errors.New("build --on <module> names a base and nothing else")
}
return buildOn(ctx, *on, *wait)
}
if *behind {
if len(positionals) != 0 || *self {
return errors.New("build <repository> or build --behind, not both: one names a " +
@@ -61,7 +112,7 @@ func buildCommand(ctx context.Context, args []string) error {
return buildBehind(ctx, *wait)
}
if len(positionals) != 1 {
return errors.New("build <repository> [--self] [--path P] [--ref R] [--wait D] [--dry-run]")
return errors.New("build <repository> [--self] [--path P] [--ref R] [--wait D] [--dry-run] | build --behind | build --on <module>")
}
source := buildSource{Repository: positionals[0]}
if *self {
@@ -81,8 +132,9 @@ func buildCommand(ctx context.Context, args []string) error {
//
// **By digest and path, never by where it was pushed** (novox/hq 04-ISSUES/102). The builder
// says `<registry>:<port>/<module>/<artifact>@sha256:…`; the mesh records the artifact-store
// reference and composes the store's address back in where a reference is used. `against` is kept
// as announced: it is what the build stood on as the builder saw it, and the catalogue's edge.
// reference and composes the store's address back in where a reference is used. `against` — what
// the build stood on, the catalogue's edge — is recorded the same way, so an edge names a module's
// artifact and not the machine it was pulled from.
func buildFrom(result link.BuildResult) inventory.Build {
kept := inventory.Build{
ID: result.ID, Repository: result.Repository, Ref: result.Ref,
@@ -92,7 +144,13 @@ func buildFrom(result link.BuildResult) inventory.Build {
// edges, and it is not always listening when a build happens — on a fresh mesh it cannot
// be, for exactly the modules it needs most. Keeping them is what makes a replay able to
// rebuild the graph rather than a list of names.
Path: result.Path, Against: result.Against,
Path: result.Path,
}
for _, ref := range result.Against {
kept.Against = append(kept.Against, catalogue.Recorded(ref))
}
for _, r := range result.Read {
kept.Read = append(kept.Read, inventory.ReadRepository{Repository: r.Repository, Ref: r.Ref})
}
var announced []inventory.Artifact
for _, made := range result.Made {
@@ -206,85 +264,45 @@ func builderCommand(ctx context.Context, args []string) error {
}
name := positionals[1]
management, err := broker.ManagementFromEnvironment()
// **The build machine's credential is a module's credential** (novox/hq ADR 0131, design 28
// task 5.5): minted into the mesh's records and sealed to the machine as the builder module's
// broker secret, usable at the next push — the same act `module issue` performs, and the same
// account the composed user list carries. Nothing is created on a server; the bus reads the list.
open, err := openStores(ctx)
if err != nil {
return err
}
// The same shape of secret a token carries: enough entropy that guessing is not a strategy,
// and safe to put in a URL because that is where it goes.
raw := make([]byte, 32)
if _, err := rand.Read(raw); err != nil {
defer open.Close()
inv := open.inventory
shelf, err := inv.Catalogue(ctx)
if err != nil {
return err
}
password := base64.RawURLEncoding.EncodeToString(raw)
if err := management.CreateBuilderAccount(ctx, name, password); err != nil {
m, known := shelf[*module]
if !known {
return fmt.Errorf("%s is not in the catalogue; `module add` it first", *module)
}
fmt.Printf("build machine %s: ", name)
node := *forNode
if node == "" {
entries, err := inv.Catalogued(ctx)
if err != nil {
return err
}
for _, e := range entries {
if e.Manifest.Module == *module && len(e.On) > 0 {
node = e.On[0]
}
}
}
if node == "" {
return fmt.Errorf("%s is assigned nowhere; `assign <machine> %s` first, or say --node", *module, *module)
}
address, err := broker.BusAddress()
if err != nil {
return err
}
fmt.Printf("broker account %s created, scoped to the %s queue and the %s exchange\n\n",
name, link.BuildQueue, link.Exchange)
if *forNode != "" {
known, err := broker.FromEnvironment()
if err != nil {
return fmt.Errorf("cannot deliver a credential without knowing where the broker is: %w", err)
}
inv, err := openInventory(ctx)
if err != nil {
return err
}
defer inv.Close()
brokerAddr, err := brokerReachableAt(ctx, inv, known, *forNode)
if err != nil {
return err
}
// The URL and what verifies the broker, together. A mesh's broker presents a certificate
// of the mesh's own, which is in no public trust store — so a URL on its own reaches only
// a broker somebody else vouches for, and the connection fails at TLS with an error about
// an unknown authority rather than about a missing pin.
//
// **The same two facts a node's token carries** (novox/hq ADR 0004), delivered the same
// way: out of band relative to the broker, so what is trusted does not come from the thing
// being trusted.
held, err := json.Marshal(struct {
URL string `json:"url"`
Fingerprint string `json:"fingerprint,omitempty"`
}{
URL: fmt.Sprintf("amqps://%s:%s@%s/", name, password, brokerAddr),
Fingerprint: known.Fingerprint,
})
if err != nil {
return err
}
if err := inv.AcceptSecretForModule(ctx, *forNode, *module, "broker", string(held)); err != nil {
return err
}
// Not printed. It is sealed to that machine and the mesh cannot read it back, which is
// the whole point — printing it here would put the one copy that matters on a terminal.
fmt.Printf(" sealed to %s, for the %s module. It arrives with the next push.\n",
*forNode, *module)
fmt.Printf(" run `push %s` to send it\n", *forNode)
return nil
}
// The whole line only when the address is known. A URL with a placeholder where the host
// should be is a URL somebody pastes and then debugs, and the placeholder is the last thing
// they look at.
if known, err := broker.FromEnvironment(); err == nil {
fmt.Printf(" MESH_BROKER_AMQP=amqps://%s:%s@%s/\n\n", name, password, known.Address)
} else {
fmt.Printf(" the password is %s\n\n", password)
fmt.Printf(" This control plane has no %s, so it cannot say where the broker is.\n"+
" Put the password in MESH_BROKER_AMQP on the build machine.\n\n",
broker.AddressVar)
}
// Shown once, like a token, and for the same reason: what is stored is the broker's own hash
// of it, and a control plane that could show it back would be a control plane that holds it.
fmt.Println("This is the only time it is shown.")
return nil
return issueOnTheNewBus(ctx, inv, m, node, address)
}
// buildBehind builds every module the mesh holds older than its source has.
@@ -329,6 +347,14 @@ func buildBehind(ctx context.Context, wait time.Duration) error {
}
fmt.Println()
// Bases first: a module built before the module it stands on is built against the old one
// and reports success (novox/hq 04-ISSUES/131).
against, err := inv.BuiltAgainst(ctx)
if err != nil {
return err
}
stale = orderByBases(stale, against)
var failed []string
for _, e := range stale {
fmt.Printf("--- %s\n", e.Manifest.Module)
@@ -368,7 +394,7 @@ func buildOne(ctx context.Context, source buildSource, path, ref string, wait ti
}
defer ident.Close()
server, err := link.Connect(nil, nil)
server, err := connectLink(ctx, nil, nil, nil)
if err != nil {
return err
}
@@ -472,7 +498,7 @@ func buildAndShow(ctx context.Context, source buildSource, path, ref string, wai
return err
}
defer ident.Close()
server, err := link.Connect(nil, nil)
server, err := connectLink(ctx, nil, nil, nil)
if err != nil {
return err
}
@@ -575,16 +601,10 @@ func heldBy(ctx context.Context) map[string]string {
// **One place chooses**, as everywhere else the bus change went (novox/hq ADR 0116 step 5). On the bus
// the mesh runs on today this needs the controller's own connection, so it is handed one; on the bus
// being built it dials, because a build request is a one-shot and holds nothing else.
func askOver(server *link.Server) (link.Builders, error) {
address, onNATS, err := broker.OnNATS()
func askOver(_ *link.Server) (link.Builders, error) {
address, err := broker.BusAddress()
if err != nil {
return nil, err
}
if err := broker.MustBeOneBus(os.Getenv(broker.AMQPVarName), address); err != nil {
return nil, err
}
if onNATS {
return link.BuildsOverNATS(address)
}
return link.BuildsOverCurrent(server.Channel()), nil
return link.BuildsOverNATS(address)
}
+6 -1
View File
@@ -76,7 +76,10 @@ func run() error {
return pinCommand(ctx, args[1:], true)
case "unpin":
return pinCommand(ctx, args[1:], false)
case "migrate":
// `prepare` is how the mesh asks any module to bring its state to the shape this version needs
// (novox/hq ADR 0135), and the control plane answers it the same way as everything else — its
// own schema is not a special case. `migrate` remains the word a person types.
case "prepare", "migrate":
return migrate(ctx)
case "node":
return nodeCommand(ctx, args[1:])
@@ -138,6 +141,7 @@ func usage() {
fmt.Fprint(os.Stderr, `mesh-controller — the control plane
migrate bring each context's schema up to date
prepare the same, asked the way the mesh asks any module (ADR 0135)
node add <name> [--adopted] create a node record; --adopted: the machine is in use
node list the nodes this mesh knows about
node show <name> what one machine reported it can do, and why
@@ -184,6 +188,7 @@ func usage() {
operator key show the operator key, and what it can recover
build <repository> [--ref R] have a build machine build it, and record what came out
build --behind build every module the mesh holds older than its source
build --on <module> rebuild every module that stands on this module's artifacts, bases first
builds [<module>] what has been built lately, and what came of it
builder issue <name> a broker account for a build machine, scoped to build work,
delivered as the builder module's broker secret (module add it first)
+56 -81
View File
@@ -2,8 +2,6 @@ package main
import (
"context"
"crypto/rand"
"encoding/base64"
"encoding/json"
"errors"
"flag"
@@ -71,12 +69,20 @@ func moduleCommand(ctx context.Context, args []string) error {
repo := set.String("source", "", "where this module comes from")
ref := set.String("ref", "", "the branch followed there")
commit := set.String("commit", "", "the commit this manifest was read at")
// **Where inside the repository the module is** (novox/hq ADR 0069). A module is a
// repository *and* a directory, and a record that carries only the repository names a
// module.json at its root — so every later build of it looks in the wrong place and fails
// with "no module.json at its root". Nine modules on this mesh were registered that way
// and none of them could be rebuilt (2026-09-28).
path := set.String("path", "", "the module's directory inside that repository")
self := set.Bool("self", false, "the source is a path on the forge holding the git seat")
positionals, err := parseAround(set, args[1:])
if err != nil {
return err
}
if len(positionals) != 1 {
return errors.New("module add <manifest.json> [--source <repo> --ref <branch> --commit <sha>]")
return errors.New("module add <manifest.json> [--source <repo> [--self] [--path P] " +
"--ref <branch> --commit <sha>]")
}
raw, err := os.ReadFile(positionals[0])
if err != nil {
@@ -86,23 +92,24 @@ func moduleCommand(ctx context.Context, args []string) error {
if err != nil {
return err
}
// Provenance together or not at all. A source with no commit cannot be compared against
// anything, so it would record where the module came from and still never be able to say
// the mesh is behind it — which is the one thing recording it is for.
if (*repo == "") != (*commit == "") {
return errors.New("--source and --commit go together: a source with no commit " +
"cannot be compared against anything, and a commit with no source has nothing " +
"to be compared with")
from, err := whereItComesFrom(*repo, *ref, *commit, *path, *self)
if err != nil {
return err
}
if err := inv.RegisterModule(ctx, m, inventory.Source{
Repository: *repo, Ref: *ref, BuiltFrom: *commit,
}); err != nil {
if err := inv.RegisterModule(ctx, m, from); err != nil {
return err
}
fmt.Printf("%s registered", m.Module)
if *commit != "" {
fmt.Printf(" from %s", short(*commit))
}
if *repo != "" && *path == "" {
// Said, not refused: a module really at the root is the ordinary case for a repository
// of its own. But a repository holding many modules and a record naming none of them is
// a module nothing can rebuild, and the person adding it is the one who knows which.
fmt.Printf("\n no directory inside %s, so it is built from that repository's root — "+
"`--path` if the module lives in a directory there", *repo)
}
if len(m.Provides) > 0 {
fmt.Printf(", providing %s", describeOffers(m.Provides))
}
@@ -261,77 +268,11 @@ func moduleCommand(ctx context.Context, args []string) error {
// made in entirely different ways: on the bus the mesh runs on today an account is a
// management call, and on the bus being built it is a row the next composition writes into
// the server's user list (novox/hq design 25 §4).
busAddress, onNATS, err := broker.OnNATS()
busAddress, err := broker.BusAddress()
if err != nil {
return err
}
if err := broker.MustBeOneBus(os.Getenv(broker.AMQPVarName), busAddress); err != nil {
return err
}
if onNATS {
return issueOnTheNewBus(ctx, inv, m, *forNode, busAddress)
}
management, err := broker.ManagementFromEnvironment()
if err != nil {
return err
}
// The foundation owns the bus; make sure it exists before a module binds onto it.
if err := management.EnsureEventExchanges(ctx); err != nil {
return err
}
secret := make([]byte, 32)
if _, err := rand.Read(secret); err != nil {
return err
}
password := base64.RawURLEncoding.EncodeToString(secret)
account, err := management.CreateModuleAccount(ctx, *forNode, module, password, m.Emits, m.Consumes)
if err != nil {
return err
}
// A consumer's queue, with its dead-letter, is the foundation's to declare — its own account
// may not (ADR 0043). Made now, so it exists before the module binds onto it.
if len(m.Consumes) > 0 {
if err := management.EnsureModuleQueue(ctx, *forNode, module); err != nil {
return err
}
}
known, err := broker.FromEnvironment()
if err != nil {
return fmt.Errorf("cannot deliver a credential without knowing where the broker is: %w", err)
}
brokerAddr, err := brokerReachableAt(ctx, inv, known, *forNode)
if err != nil {
return err
}
// The URL and what verifies the broker, together — a mesh's broker presents its own
// certificate, in no public trust store, so a URL alone fails at TLS (as `builder issue`).
held, err := json.Marshal(struct {
URL string `json:"url"`
Fingerprint string `json:"fingerprint,omitempty"`
Node string `json:"node"`
Module string `json:"module"`
}{
URL: fmt.Sprintf("amqps://%s:%s@%s/", account, password, brokerAddr),
Fingerprint: known.Fingerprint,
// The node and module the account is for, so the runtime names its queue as the mesh
// scoped it (<node>.<module>.events) without a manifest having to interpolate a node.
Node: *forNode,
Module: module,
})
if err != nil {
return err
}
if err := inv.AcceptSecretForModule(ctx, *forNode, module, "broker", string(held)); err != nil {
return err
}
fmt.Printf("broker account %s created for %s, scoped to what it emits and consumes\n",
account, module)
fmt.Printf(" sealed to %s. It arrives with the next push — `push %s` to send it\n",
*forNode, *forNode)
return nil
return issueOnTheNewBus(ctx, inv, m, *forNode, busAddress)
default:
return fmt.Errorf("module has no %q; it has add, list, moved, forget and issue", args[0])
@@ -670,3 +611,37 @@ func issueWith(ctx context.Context, inv *inventory.Inventory, m catalogue.Manife
"machine holding mesh-broker\n")
return nil
}
// whereItComesFrom is the provenance a module handed over by hand records, and what a record must
// say to be worth anything later.
//
// **A module is a repository and a directory inside it** (novox/hq ADR 0069). A record carrying only
// the repository names a module.json at its root, so every later build of it looks in the wrong
// place — nine modules on this mesh were registered that way and none of them could be rebuilt
// (2026-09-28). The directory cannot be checked from here, because the control plane does not clone;
// what can be checked is that the record is whole.
func whereItComesFrom(repository, ref, commit, path string, self bool) (inventory.Source, error) {
// Provenance together or not at all. A source with no commit cannot be compared against
// anything, so it would record where the module came from and still never be able to say the
// mesh is behind it — which is the one thing recording it is for.
if (repository == "") != (commit == "") {
return inventory.Source{}, errors.New("--source and --commit go together: a source with " +
"no commit cannot be compared against anything, and a commit with no source has " +
"nothing to be compared with")
}
// A directory or a forge with no repository is half a location, and the half it keeps is the
// half nothing can be found with.
if repository == "" && (path != "" || self) {
return inventory.Source{}, errors.New("--path and --self say where inside a source and " +
"which forge holds it, so they need --source: without one there is nothing for them " +
"to be part of")
}
from := inventory.Source{Repository: repository, Ref: ref, BuiltFrom: commit, Path: path}
if self {
if err := onASeat(repository); err != nil {
return inventory.Source{}, err
}
from.Seat = gitSeat
}
return from, nil
}
+10 -1
View File
@@ -537,6 +537,15 @@ func onTheNetwork(ctx context.Context, inv *inventory.Inventory,
if err != nil {
return nil, err
}
// **With the seat holders on record**, or a machine running the next holder of a seat beside
// the current one resolves as two holders, is refused, and drops out of the map — taking the
// address every other machine composes for what it offers (novox/hq ADR 0131). Found live:
// the control node vanished from the private network the moment the new bus was assigned
// beside the old one.
holdings, err := inv.Holdings(ctx)
if err != nil {
return nil, err
}
var out []inventory.Overlay
for _, p := range places {
if p.Address == "" {
@@ -549,7 +558,7 @@ func onTheNetwork(ctx context.Context, inv *inventory.Inventory,
caps, _ := inv.ProfileOf(ctx, p.Name)
got, err := catalogue.Resolve(shelf, assigned,
catalogue.Node{Name: p.Name, Site: p.Site, Capabilities: caps},
catalogue.World{Unchecked: true})
catalogue.World{Unchecked: true, Holdings: holdings})
if err != nil {
continue
}
-10
View File
@@ -10,7 +10,6 @@ import (
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/token"
)
@@ -258,15 +257,6 @@ func tokenCommand(ctx context.Context, args []string) error {
// chicken-and-egg entirely: the mesh runs the broker, so a joining node's credentials can
// exist before it does. The one-time secret IS the password, so a node's first connection is
// already authenticated and enrolment is what happens over it.
if management, err := broker.ManagementFromEnvironment(); err == nil {
if err := management.CreateNodeAccount(ctx, issued.Node.Name, issued.Secret); err != nil {
return err
}
fmt.Printf("broker account %s created, scoped to %s and the %s exchange\n\n",
issued.Node.Name, link.QueueFor(issued.Node.Name), link.Exchange)
} else if !errors.Is(err, broker.ErrNotConfigured) {
return err
}
made := token.Token{Node: issued.Node.Name, Signer: key.Public, Secret: issued.Secret,
Adopted: issued.Node.Adopted}
+7 -3
View File
@@ -189,13 +189,17 @@ func readPrivateKey(path string) (string, error) {
func personIssue(ctx context.Context, args []string) error {
set := flag.NewFlagSet("operator issue", flag.ContinueOnError)
invokes := set.String("invokes", "", "the tools this person may call, comma-separated, or * for every one")
if err := set.Parse(args); err != nil {
// Flags on either side of the name, because the usage this command prints puts them after it —
// and the standard parser stops at the first thing that is not a flag, so the order the command
// documents was the one order it refused (2026-09-28).
positionals, err := parseAround(set, args)
if err != nil {
return err
}
if set.NArg() != 1 {
if len(positionals) != 1 {
return errors.New("operator issue <name> --invokes <tool,tool|*>")
}
name := set.Arg(0)
name := positionals[0]
if *invokes == "" {
return errors.New(
"say what this person may call: --invokes mesh-catalog.catalog_tools,gitea.repo_create, " +
+212
View File
@@ -0,0 +1,212 @@
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// entry is a module as the catalogue holds it: built, so its manifest carries no `build` any more.
func entry(module string, _ ...string) inventory.Entry {
return inventory.Entry{Manifest: catalogue.Manifest{Module: module}}
}
// stoodOn is what each module's newest build recorded it was handed.
func stoodOn(edges map[string][]string) map[string][]string {
out := map[string][]string{}
for module, bases := range edges {
for _, b := range bases {
out[module] = append(out[module], catalogue.ArtifactStoreScheme+b+"/runtime@sha256:"+strings.Repeat("0", 64))
}
}
return out
}
// A module built before the module it stands on is built against the old one and reports success
// (novox/hq 04-ISSUES/131). So bases come first, however the set arrived.
func TestBasesAreBuiltBeforeWhatStandsOnThem(t *testing.T) {
in := []inventory.Entry{entry("app"), entry("runtime"), entry("other"), entry("base")}
edges := stoodOn(map[string][]string{"app": {"runtime"}, "runtime": {"base"}})
got := orderByBases(in, edges)
pos := map[string]int{}
for i, e := range got {
pos[e.Manifest.Module] = i
}
if !(pos["base"] < pos["runtime"] && pos["runtime"] < pos["app"]) {
t.Fatalf("bases not first: %v", pos)
}
if len(got) != 4 {
t.Fatalf("an entry was lost or doubled: %d", len(got))
}
// A base outside the set is not waited for: it is not being rebuilt.
got = orderByBases([]inventory.Entry{entry("app")}, stoodOn(map[string][]string{"app": {"elsewhere"}}))
if len(got) != 1 {
t.Fatalf("a dependency outside the set changed the set: %v", got)
}
}
// A module registered from its manifest and never built still names its bases there; once built,
// the recorded edge is what says so. Both are read, and a module never stands on itself.
func TestWhatStandsOnAModuleIsReadFromItsBuildOrItsManifest(t *testing.T) {
built := entry("gitea")
edges := stoodOn(map[string][]string{"gitea": {"mesh-tools"}})
if !standsOnModule(built, "mesh-tools", edges) {
t.Fatal("a recorded edge was not read")
}
if standsOnModule(built, "gitea", edges) || standsOnModule(built, "postgres", edges) {
t.Fatal("an edge was invented")
}
fresh := inventory.Entry{Manifest: catalogue.Manifest{Module: "plex", Build: &catalogue.Build{
On: []catalogue.BuildsOn{{Arg: "RUNTIME_BASE", Module: "mesh-tools", Artifact: "runtime"}},
}}}
if !standsOnModule(fresh, "mesh-tools", nil) {
t.Fatal("a manifest's own base was not read")
}
}
// A merge names a repository the way the forge does; a source is recorded the way a build was
// asked for. The two meet on owner/repo and branch, whichever form the record took.
func TestAMergeMatchesTheSourcesBuiltFromIt(t *testing.T) {
m := link.SourceMoved{Owner: "novox", Repo: "mesh-controller", Base: "main",
CloneURL: "http://forge.internal:20000/novox/mesh-controller.git"}
for _, s := range []inventory.Source{
{Repository: "http://forge.internal:20000/novox/mesh-controller.git", Ref: "main"},
{Repository: "novox/mesh-controller", Seat: "git", Ref: ""},
{Repository: "https://elsewhere.example/novox/mesh-controller", Ref: "main"},
} {
if !sourceIs(s, m) {
t.Errorf("%+v was not matched by the merge", s)
}
}
for _, s := range []inventory.Source{
{Repository: "novox/mesh-host", Seat: "git"},
{Repository: "http://forge.internal:20000/novox/mesh-controller.git", Ref: "release"},
} {
if sourceIs(s, m) {
t.Errorf("%+v was matched by a merge that is not its", s)
}
}
}
// A merge made before the source was last seen is history: it does not move the source, and a
// merge that says nothing about when it was made is taken as news.
func TestAMergeOlderThanTheLastLookIsHistory(t *testing.T) {
seen := time.Date(2026, 9, 28, 3, 0, 0, 0, time.UTC)
if !isHistory("2026-09-28T02:00:00Z", seen) {
t.Fatal("an older merge was taken as news")
}
if isHistory("2026-09-28T04:00:00Z", seen) {
t.Fatal("a newer merge was taken as history")
}
if isHistory("", seen) || isHistory("2026-09-28T02:00:00Z", time.Time{}) {
t.Fatal("a merge or a source with no time on it was refused")
}
}
// A module as the catalogue holds it: built from a repository, at a directory inside it.
func fromRepo(module, repository, path string) inventory.Entry {
return inventory.Entry{
Manifest: catalogue.Manifest{Module: module},
Source: inventory.Source{Repository: repository, Path: path, Ref: "main"},
}
}
// A merge rebuilds the modules whose own directories it changed, and everything when what it changed
// is shared. One repository holding many modules is the ordinary case here, and rebuilding all of
// them for a change to one is what exhausted a registry's pull limit the first night this ran.
func TestAMergeRebuildsTheModulesItChanged(t *testing.T) {
const repo = "http://forge.internal:20000/novox/mesh-catalog.git"
gitea := fromRepo("gitea", repo, "modules/gitea")
keycloak := fromRepo("keycloak", repo, "modules/keycloak")
known := []inventory.Entry{gitea, keycloak, fromRepo("plex", repo, "modules/plex")}
candidates := []inventory.Entry{gitea, keycloak}
merge := func(paths []string, truncated bool) link.SourceMoved {
return link.SourceMoved{Owner: "novox", Repo: "mesh-catalog", Base: "main",
Paths: paths, PathsTruncated: truncated}
}
named := func(entries []inventory.Entry) string {
var names []string
for _, e := range entries {
names = append(names, e.Manifest.Module)
}
return strings.Join(names, ",")
}
for _, c := range []struct {
what string
m link.SourceMoved
want string
}{
{"one module's own files", merge([]string{"modules/gitea/index.ts", "modules/gitea/client.ts"}, false), "gitea"},
{"two modules' files", merge([]string{"modules/gitea/index.ts", "modules/keycloak/module.json"}, false), "gitea,keycloak"},
{"a file they share", merge([]string{"tsconfig.json"}, false), "gitea,keycloak"},
{"a module the mesh does not hold", merge([]string{"modules/plex/index.ts"}, false), ""},
{"nothing said about the files", merge(nil, false), "gitea,keycloak"},
{"more files than were listed", merge([]string{"modules/gitea/index.ts"}, true), "gitea,keycloak"},
} {
if got := named(whatTheMergeTouched(candidates, known, c.m)); got != c.want {
t.Errorf("%s: rebuilt %q, wanted %q", c.what, got, c.want)
}
}
}
// A module whose recipe packages source from another repository is affected when that repository
// moves — the manifest the mesh keeps says nothing about it, so the record of what the build read is
// the only thing that can say so.
func TestAModuleIsAffectedByTheRepositoryItPackages(t *testing.T) {
m := link.SourceMoved{Owner: "novox", Repo: "mesh-controller", Base: "main",
CloneURL: "http://forge.internal:20000/novox/mesh-controller.git"}
for _, read := range [][]inventory.ReadRepository{
{{Repository: "http://forge.internal:20000/novox/mesh-controller.git", Ref: "main"}},
{{Repository: "novox/mesh-controller"}},
{{Repository: "https://elsewhere.example/novox/other"}, {Repository: "novox/mesh-controller.git", Ref: "main"}},
} {
if !readsFrom(read, m) {
t.Errorf("%+v was not matched by the merge", read)
}
}
for _, read := range [][]inventory.ReadRepository{
nil,
{{Repository: "novox/mesh-host", Ref: "main"}},
{{Repository: "novox/mesh-controller", Ref: "release"}},
} {
if readsFrom(read, m) {
t.Errorf("%+v was matched by a merge that is not its", read)
}
}
}
// What a module handed over by hand records about where it came from, and what is refused.
func TestWhatAHandedOverModuleRecordsAboutItsSource(t *testing.T) {
// The whole location: a repository on the mesh's own forge, the directory inside it, the branch
// and the commit the manifest was read at.
from, err := whereItComesFrom("novox/mesh-catalog", "main", "c0ffee", "modules/gitea", true)
if err != nil {
t.Fatal(err)
}
if from.Path != "modules/gitea" || from.Seat != "git" || from.Repository != "novox/mesh-catalog" {
t.Fatalf("the source records as %+v", from)
}
// A manifest with no provenance at all is legitimate: fixing something in a hurry.
if from, err := whereItComesFrom("", "", "", "", false); err != nil || from != (inventory.Source{}) {
t.Fatalf("a manifest handed over with no provenance was refused: %+v, %v", from, err)
}
for _, c := range []struct {
what string
repository, ref, commit, path string
self bool
}{
{what: "a source with no commit", repository: "novox/mesh-catalog", commit: ""},
{what: "a commit with no source", commit: "c0ffee"},
{what: "a directory inside nothing", path: "modules/gitea"},
{what: "a forge holding nothing", self: true},
{what: "an address given as a path on the forge", repository: "http://forge.internal:20000/novox/x.git", commit: "c0ffee", self: true},
} {
if _, err := whereItComesFrom(c.repository, c.ref, c.commit, c.path, c.self); err == nil {
t.Errorf("%s was recorded as a source", c.what)
}
}
}
+106 -31
View File
@@ -38,6 +38,27 @@ func reportUnhostable(node string, plan catalogue.Resolution) {
// nothing in it was wrong, and no one edit was the one that should have been a new file.
// serve is the control plane running: one connection to the broker, one queue, one consumer.
// connectLink opens the controller's link over whichever bus this process is on (design 25: one
// variable moves it). The streams and this controller's consumers are raised first on the new bus,
// so nothing served here finds them missing.
func connectLink(ctx context.Context, inv *inventory.Inventory, enroller link.Enroller, listener link.Listener) (*link.Server, error) {
busAddress, err := broker.BusAddress()
if err != nil {
return nil, err
}
if inv != nil {
if err := raiseTheBus(ctx, inv, busAddress); err != nil {
return nil, err
}
}
js, err := broker.Dial(busAddress)
if err != nil {
return nil, fmt.Errorf("the mesh is on the bus at %s and this control plane cannot reach it: %w",
broker.BareAddress(busAddress), err)
}
return link.ConnectNats(js, enroller, listener), nil
}
func serve(ctx context.Context) error {
open, err := openStores(ctx)
if err != nil {
@@ -61,11 +82,6 @@ func serve(ctx context.Context) error {
}
fmt.Printf("signing as %s\n", key.Fingerprint()[:16])
management, err := broker.ManagementFromEnvironment()
if err != nil && !errors.Is(err, broker.ErrNotConfigured) {
return err
}
// Where the broker is and what to expect there, so a node can be told how to come back
// without a person and a new token.
known, err := broker.FromEnvironment()
@@ -80,17 +96,10 @@ func serve(ctx context.Context) error {
// **Which bus this mesh is on, read once** (novox/hq ADR 0116 step 5). Both clients ship; both
// being live is refused, because a mesh half on each is one where a declaration goes out on one
// and the report comes back on the other, and every component logs success while it happens.
busAddress, onNATS, err := broker.OnNATS()
if err != nil {
return err
}
if err := broker.MustBeOneBus(os.Getenv(broker.AMQPVarName), busAddress); err != nil {
return err
}
work := link.Enrolment{Inventory: inv, Identity: ident, Management: management, Broker: known,
OnNATS: onNATS}
server, err := link.Connect(work, work)
work := link.Enrolment{Inventory: inv, Identity: ident, Broker: known,
OnNATS: true}
server, err := connectLink(ctx, inv, work, work)
if err != nil {
return err
}
@@ -100,11 +109,6 @@ func serve(ctx context.Context) error {
// somebody deleted, a mesh raised from a restored backup, or a bus whose data directory was
// replaced all have records and no objects — and a node whose consumer is missing hears nothing
// while everything else about it looks correct.
if onNATS {
if err := raiseTheBus(ctx, inv, busAddress); err != nil {
return err
}
}
// And build results nobody was waiting for. A build triggered any other way than `build`
// would otherwise be reported into the void, which is the same as not reporting it.
server.Records(builds{inv})
@@ -158,13 +162,19 @@ func declare(ctx context.Context, args []string) error {
return err
}
server, err := link.Connect(nil, nil)
// **With the inventory, so the bus is raised** (novox/hq ADR 0134, design 30). A module's
// declaration and how it hears what it consumes move together: its consumer is derived from the
// same records this declaration is composed from. Raised only when the control plane started
// serving, a module that gained a `consumes` was sent a declaration it could act on and a
// consumer that never delivered the event — and nothing anywhere said the two disagreed
// (found on review, 2026-09-28). Everything the raise does is idempotent.
server, err := connectLink(ctx, inv, nil, nil)
if err != nil {
return err
}
defer server.Close()
if err := link.Declare(ctx, link.OverCurrent{Channel: server.Channel()}, ident, node, raw, 15*time.Second); err != nil {
if err := link.Declare(ctx, server.Bus(), ident, node, raw, 15*time.Second); err != nil {
return err
}
fmt.Printf("sent %s a signed declaration (%d bytes)\n", node, len(raw))
@@ -271,7 +281,7 @@ func pushCommand(ctx context.Context, args []string) error {
return err
}
server, err := link.Connect(nil, nil)
server, err := connectLink(ctx, nil, nil, nil)
if err != nil {
return err
}
@@ -332,7 +342,7 @@ func pushCommand(ctx context.Context, args []string) error {
if err != nil {
return err
}
if err := link.Declare(ctx, link.OverCurrent{Channel: server.Channel()}, ident, s.node, body, 15*time.Second); err != nil {
if err := link.Declare(ctx, server.Bus(), ident, s.node, body, 15*time.Second); err != nil {
return err
}
// After it is away, not before. A digest recorded for something that failed to send would
@@ -415,7 +425,7 @@ func pushCommand(ctx context.Context, args []string) error {
return declarationWith(held, open, node, plan, settings, gens, Allocating)
},
func(s readyNode, body []byte) error {
if err := link.Declare(ctx, link.OverCurrent{Channel: server.Channel()}, ident, s.node, body,
if err := link.Declare(ctx, server.Bus(), ident, s.node, body,
15*time.Second); err != nil {
return err
}
@@ -628,7 +638,7 @@ func sendTo(ctx context.Context, open *stores, names []string) error {
len(refusals), strings.Join(refusals, "\n\n"))
}
server, err := link.Connect(nil, nil)
server, err := connectLink(ctx, nil, nil, nil)
if err != nil {
return err
}
@@ -639,7 +649,7 @@ func sendTo(ctx context.Context, open *stores, names []string) error {
if err != nil {
return err
}
if err := link.Declare(ctx, link.OverCurrent{Channel: server.Channel()}, ident, s.node, body, 15*time.Second); err != nil {
if err := link.Declare(ctx, server.Bus(), ident, s.node, body, 15*time.Second); err != nil {
return err
}
record, err := inv.NodeByName(ctx, s.node)
@@ -704,7 +714,7 @@ func raiseTheBus(ctx context.Context, inv *inventory.Inventory, address string)
js, err := broker.Dial(address)
if err != nil {
return fmt.Errorf("the mesh is on the bus at %s and this control plane cannot reach it: %w",
address, err)
broker.BareAddress(address), err)
}
defer js.Close()
@@ -739,10 +749,75 @@ func raiseTheBus(ctx context.Context, inv *inventory.Inventory, address string)
// The work queues of the mesh's own roles (novox/hq ADR 0121). The queue before the holder,
// deliberately: work queues until somebody arrives to do it, so assigning a build machine a week
// after something started asking for builds flushes the backlog instead of having lost it.
if err := broker.RaiseSeats(js, inventory.MeshSeats(), nil); err != nil {
// With the seats' holders, so each role's work queue gets the consumer its holder takes
// work from. Passed as nil until the first live raise, which left the build machine bound to a
// consumer nothing had created (2026-09-28).
holders, err := seatHolders(ctx, inv)
if err != nil {
return err
}
fmt.Printf("the bus at %s has its streams, and %d machine(s) can hear a declaration\n",
address, len(names))
if err := broker.RaiseSeats(js, inventory.MeshSeats(), holders); err != nil {
return err
}
// And how every module hears what it consumes. Derived from the same records the user list is
// composed from, so a module the mesh grants a consumer's subjects has that consumer waiting.
// Done on every raise, not only when a credential is issued: every module moved onto this bus
// by the rollout was issued on the old one, and came up with nothing to bind to (2026-09-28).
records, err := inv.BusRecords(ctx)
if err != nil {
return err
}
users, err := broker.Users(records)
if err != nil {
return err
}
hearing := 0
for _, p := range users {
consumer, needed := broker.ConsumerFor(p)
if !needed {
continue
}
if err := js.EnsureConsumer(consumer); err != nil {
return fmt.Errorf("how %s on %s hears what it consumes: %w", p.Module, p.Node, err)
}
hearing++
}
fmt.Printf("the bus at %s has its streams, %d machine(s) can hear a declaration, and %d module(s) "+
"can hear what they consume\n", broker.BareAddress(address), len(names), hearing)
return nil
}
// seatHolders is who holds each of the mesh's seats, by seat name: the record where a handover
// wrote one, and the assigned module claiming the seat otherwise — the same derivation the
// resolver makes, read from the catalogue rather than re-resolved.
func seatHolders(ctx context.Context, inv *inventory.Inventory) (map[string]broker.Holder, error) {
out := map[string]broker.Holder{}
entries, err := inv.Catalogued(ctx)
if err != nil {
return nil, err
}
for _, e := range entries {
if len(e.On) == 0 {
continue
}
for _, c := range e.Manifest.Claims {
seat, known := catalogue.SeatNamed(c.Name)
if !known {
continue
}
if _, taken := out[seat.Name]; !taken {
out[seat.Name] = broker.Holder{Node: e.On[0], Module: e.Manifest.Module}
}
}
}
recorded, err := inv.Holdings(ctx)
if err != nil {
return nil, err
}
for _, h := range recorded {
if seat, known := catalogue.SeatNamed(h.Claim); known {
out[seat.Name] = broker.Holder{Node: h.Node, Module: h.Module}
}
}
return out, nil
}
+111 -10
View File
@@ -5,6 +5,7 @@ import (
"encoding/json"
"errors"
"fmt"
"os"
"strings"
"time"
@@ -33,14 +34,21 @@ import (
// ability to change things, not the services its modules are serving — measured on 2026-09-27, when
// a seat emptied mid-change and the control plane looped for two hours while every service stayed up.
const rolloutUsage = "rollout check | rollout mint | rollout --confirm"
const rolloutUsage = "rollout check | rollout mint [--again] | rollout hand <node> | rollout --confirm"
func rolloutCommand(ctx context.Context, args []string) error {
switch {
case len(args) == 1 && args[0] == "check":
return rolloutCheck(ctx)
case len(args) == 1 && args[0] == "mint":
return rolloutMint(ctx)
return rolloutMint(ctx, false)
case len(args) == 2 && args[0] == "hand":
return rolloutHand(ctx, args[1])
case len(args) == 2 && args[0] == "mint" && args[1] == "--again":
// Every credential minted afresh, whether or not one exists — for a mint that was wrong
// before anything was pushed. Afterwards nothing that received the old one still works,
// which is fine exactly when nothing received it.
return rolloutMint(ctx, true)
case len(args) == 1 && args[0] == "--confirm":
return errors.New(
"the rollout itself is not built yet: `rollout check` answers whether it could run, and " +
@@ -119,9 +127,13 @@ func readinessOf(ctx context.Context, inv *inventory.Inventory) (broker.Readines
if address != "" {
// One dial, briefly. "Is it answering" is the one fact records cannot hold, and a mesh about
// to move onto a server that is not there should hear it here rather than afterwards.
if conn, err := nats.Connect(broker.BareAddress(address), nats.Timeout(5*time.Second)); err == nil {
//
// **Dialled the way the mesh dials it** — credential and pin — because a bare connect to a
// bus that requires TLS and a user fails at the handshake, and the check then reported a
// standing server as absent (seen live, 2026-09-28).
if js, err := broker.Dial(address, nats.Timeout(5*time.Second)); err == nil {
state.ServerStanding = true
conn.Close()
js.Close()
}
}
@@ -205,7 +217,7 @@ func notReadyOf(state broker.Readiness) []string { return broker.NotReady(state)
// why `rollout check` refuses until this has run. The bus's address is worked out here, from where
// the module that provides it is assigned, rather than read from this process's environment: this
// process is still on the old bus when this runs, and must be.
func rolloutMint(ctx context.Context) error {
func rolloutMint(ctx context.Context, again bool) error {
open, err := openStores(ctx)
if err != nil {
return err
@@ -243,11 +255,33 @@ func rolloutMint(ctx context.Context) error {
if err != nil {
return err
}
if onNetwork[busNode] == "" {
return fmt.Errorf("%s runs the new bus and has no address on the private network, so no machine "+
"could be told where it is", busNode)
busHost := onNetwork[busNode]
if busHost == "" {
// **The hub is not in that map.** The machine that took over the tunnel is where the current
// bus already answers, and every machine dials it at the address the mesh handed them — so
// when the new bus runs on the same machine, that address is the one to tell them, with the
// new port. Found live: the control node is the hub, and the map lists the machines placed
// around it.
// The host alone: no scheme (BareAddress adds one where none was, which is the wrong
// direction here — every URL built below adds its own) and no port.
_, _, host := broker.CredentialIn(known.Address)
if host == "" {
host = known.Address
}
if _, after, hasScheme := strings.Cut(host, "://"); hasScheme {
host = after
}
host = strings.TrimSpace(host)
if i := strings.LastIndex(host, ":"); i > 0 && !strings.Contains(host[i:], "]") {
host = host[:i]
}
if host == "" {
return fmt.Errorf("%s runs the new bus and has no address on the private network, and the "+
"current bus's address is unknown too, so no machine could be told where it is", busNode)
}
busHost = host
}
busAddress := onNetwork[busNode] + ":4222"
busAddress := busHost + ":4222"
records, err := inv.BusRecords(ctx)
if err != nil {
@@ -273,7 +307,7 @@ func rolloutMint(ctx context.Context) error {
var machines, modules, skipped int
for _, p := range users {
if !wanted[p.Username()] {
if !again && !wanted[p.Username()] {
continue
}
switch p.Kind {
@@ -313,6 +347,14 @@ func rolloutMint(ctx context.Context) error {
machines++
case broker.KindModule:
if p.Module == "mesh-controller" {
// The control plane is a module too, and its `broker` secret is the old bus's
// credential it is still using while this runs. Writing the new bus's blob there
// cut the mesh off from its own old bus mid-move (2026-09-28). Its new-bus credential
// is the controller principal's `bus` secret above; nothing else is needed here.
skipped++
continue
}
m, inShelf := shelf[p.Module]
if !inShelf {
skipped++
@@ -352,3 +394,62 @@ func providesBus(m catalogue.Manifest) bool {
}
return false
}
// rolloutHand mints a machine its credential for the new bus afresh and prints its membership
// once, for an operator to carry by hand — the rescue for a machine that cannot be reached over
// any bus: rotated while it still held the old password, or reachable only by ssh. The plaintext
// exists on this terminal and then only where it is written; the store keeps the hash, and the
// sealed copy in the machine's declaration is replaced too, so the next push says the same.
func rolloutHand(ctx context.Context, node string) error {
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
inv := open.inventory
known, err := broker.FromEnvironment()
if err != nil {
return fmt.Errorf("the bus's certificate is not known to this process: %w", err)
}
busAddress, _, err := broker.OnNATS()
if err != nil {
return err
}
if busAddress == "" {
return errors.New("this control plane is not on the new bus, so there is no membership to hand out")
}
_, _, bare := broker.CredentialIn(busAddress)
if _, after, has := strings.Cut(bare, "://"); has {
bare = after
}
if _, err := inv.NodeByName(ctx, node); err != nil {
return err
}
p := broker.Principal{Kind: broker.KindNode, Node: node}
password, err := inv.MintBusPassword(ctx, inventory.BusUser{Username: p.Username(), Kind: inventory.BusNode, Node: node})
if err != nil {
return err
}
membership, _ := json.Marshal(map[string]string{
"broker": bare, "fingerprint": known.Fingerprint, "password": password, "transport": "nats",
})
key, err := inv.SealingKeyOf(ctx, node)
if err != nil {
return err
}
sealed, err := secrets.Seal(key, membership)
if err != nil {
return err
}
if err := inv.PutBusMembership(ctx, node, sealed); err != nil {
return err
}
// The one line of output is the membership itself, so it can be piped to the machine without
// being read on the way. Everything else goes to stderr.
fmt.Fprintf(os.Stderr, "%s's credential is minted afresh. Write this to %s on it and restart its host; "+
"then push the machine running the bus so the user list carries the new hash.\n",
node, catalogue.BusMembershipPath)
fmt.Println(string(membership))
return nil
}
+304
View File
@@ -6,7 +6,9 @@ import (
"flag"
"fmt"
"strings"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
@@ -218,3 +220,305 @@ func notNow(err error) error {
}
return err
}
// SourceMoved is the forge announcing a merge: every module recorded as built from that
// repository and branch is marked as moved to the merge commit, and built — bases first, so a
// module that stands on another's artifact is built after it and not against the old one
// (novox/hq 04-ISSUES/131). Nothing is pushed here: what a finished build does to the machines
// running the module is the upgrade's decision, taken when the catalogue announces it.
func (f following) SourceMoved(ctx context.Context, m link.SourceMoved) error {
inv := f.open.inventory
entries, err := inv.Catalogued(ctx)
if err != nil {
return notNow(err)
}
read, err := inv.ReadRepositories(ctx)
if err != nil {
return notNow(err)
}
// Two kinds of module are affected by one merge, and they are affected differently.
//
// A module **built from** this repository and branch has moved: the mesh records the new commit
// as what its source now has, and only what the merge actually changed is rebuilt. A module that
// only **packages source from** it has not moved — its own source is somewhere else, at the
// commit it already records — so it is rebuilt and its record left alone. Writing this commit as
// its source would make it permanently behind a repository its manifest does not come from.
var from, packaging []inventory.Entry
already := 0
for _, e := range entries {
switch {
case sourceIs(e.Source, m):
if e.Source.BuiltFrom == m.Commit {
already++
continue
}
// **A merge older than the last look at the source is history, not a move.** The forge
// announces what it finds merged, and an old merge surfacing late would otherwise move
// the recorded head backwards and rebuild everything built from that repository, once
// per old merge (2026-09-28).
if isHistory(m.MergedAt, e.Source.Seen) {
continue
}
from = append(from, e)
case readsFrom(read[e.Manifest.Module], m):
packaging = append(packaging, e)
}
}
if len(from) == 0 && len(packaging) == 0 {
// "Already built from it" and "nothing reads it" are different facts, and reading the first
// as the second sends somebody looking for a broken trigger when the mesh is up to date.
if already > 0 {
fmt.Printf("%s/%s merged into %s (%.8s); %d module(s) the mesh holds are already built "+
"from it\n", m.Owner, m.Repo, m.Base, m.Commit, already)
return nil
}
fmt.Printf("%s/%s merged into %s (%.8s); nothing the mesh holds reads it\n",
m.Owner, m.Repo, m.Base, m.Commit)
return nil
}
// The same judgement for the packaging kind, against the newest look at that repository by
// anything built from it: they keep no record of it themselves, and a replayed old merge should
// not rebuild them either.
if isHistory(m.MergedAt, lastLookAt(entries, m)) {
packaging = nil
}
touched := whatTheMergeTouched(from, entries, m)
for _, e := range touched {
if err := inv.SourceMoved(ctx, e.Manifest.Module, m.Commit); err != nil {
return notNow(err)
}
}
moved := append(append([]inventory.Entry{}, touched...), packaging...)
if len(moved) == 0 {
fmt.Printf("%s/%s merged into %s (%.8s); it changed nothing any module the mesh holds is "+
"built from\n", m.Owner, m.Repo, m.Base, m.Commit)
return nil
}
against, err := inv.BuiltAgainst(ctx)
if err != nil {
return notNow(err)
}
ordered := orderByBases(moved, against)
names := make([]string, 0, len(ordered))
for _, e := range ordered {
names = append(names, e.Manifest.Module)
}
fmt.Printf("%s/%s merged into %s (%.8s); building %s\n",
m.Owner, m.Repo, m.Base, m.Commit, strings.Join(names, ", "))
if len(packaging) > 0 {
var also []string
for _, e := range packaging {
also = append(also, e.Manifest.Module)
}
fmt.Printf(" %s package source from it, so they are rebuilt and their own source record "+
"is left where it is\n", strings.Join(also, ", "))
}
var failed []string
for _, e := range ordered {
source := buildSource{Repository: e.Source.Repository, Seat: e.Source.Seat}
if err := buildOne(ctx, source, e.Source.Path, e.Source.Ref, 20*time.Minute); err != nil {
fmt.Printf(" %s: %v\n", e.Manifest.Module, err)
failed = append(failed, e.Manifest.Module)
// A base that failed is a reason to stop: what stands on it would be built against
// the old one, and report success (novox/hq 04-ISSUES/131).
if standsOn(ordered, e.Manifest.Module, against) {
fmt.Printf(" stopping: %s is a base of what was still to build\n", e.Manifest.Module)
break
}
}
}
if len(failed) > 0 {
fmt.Printf("%d of %d not built: %s\n", len(failed), len(ordered), strings.Join(failed, ", "))
}
return nil
}
// sourceIs is whether a recorded source is the repository and branch a merge announced. A source on
// the git seat is recorded as its path on the forge; one elsewhere as the URL it was cloned from.
// An empty recorded ref is the repository's default branch, which is what a merge into the base
// branch of the forge's default means.
func sourceIs(s inventory.Source, m link.SourceMoved) bool {
if !sameRepository(s.Repository, m) {
return false
}
return s.Ref == "" || s.Ref == m.Base
}
// sameRepository is whether a recorded repository is the one a merge names, in either spelling it
// may have been recorded in: a path on the git seat, or the URL it was cloned from.
func sameRepository(repository string, m link.SourceMoved) bool {
want := strings.ToLower(m.Owner + "/" + m.Repo)
repo := strings.ToLower(strings.TrimSuffix(repository, ".git"))
return repo == want || strings.HasSuffix(repo, "/"+want) ||
(m.CloneURL != "" && repo == strings.ToLower(strings.TrimSuffix(m.CloneURL, ".git")))
}
// readsFrom is whether a module's build read the repository a merge names: the second repository its
// recipe packages source from. Its ref must be the branch that moved, or unset — the same rule a
// module's own source follows.
func readsFrom(read []inventory.ReadRepository, m link.SourceMoved) bool {
for _, r := range read {
if sameRepository(r.Repository, m) && (r.Ref == "" || r.Ref == m.Base) {
return true
}
}
return false
}
// lastLookAt is the most recent look at this repository by anything built from it.
func lastLookAt(entries []inventory.Entry, m link.SourceMoved) time.Time {
var newest time.Time
for _, e := range entries {
if sameRepository(e.Source.Repository, m) && e.Source.Seen.After(newest) {
newest = e.Source.Seen
}
}
return newest
}
// whatTheMergeTouched narrows the modules built from a repository to the ones the merge changed.
//
// **A change inside no module's own directory is a change to what they share.** The forge lists the
// files a merge changed; a module is affected when one of them is inside its own directory, when it
// is built from the repository's root — everything there is its source — or when some changed file
// belongs to no module's directory at all, which is how a shared file, a build recipe or a
// dependency at the root rebuilds everything built from that repository.
//
// A change inside *another* module's directory is that module's business and not this one's, even
// when the mesh does not hold that module: `known` is every module this repository is known to hold,
// whatever branch it was registered from. That is also the limit of this — a repository whose shared
// code sits inside a directory the mesh has never seen a module in reads as shared, and everything
// is rebuilt. Rebuilding too much is the safe direction: the fault this whole path exists for is a
// mesh that believes it is current and is not (novox/hq 04-ISSUES/131).
func whatTheMergeTouched(candidates, known []inventory.Entry, m link.SourceMoved) []inventory.Entry {
// Nothing said about the files, or not all of them said: everything built from it is affected.
if len(m.Paths) == 0 || m.PathsTruncated {
return candidates
}
var dirs []string
for _, e := range known {
if e.Source.Path != "" && sameRepository(e.Source.Repository, m) {
dirs = append(dirs, e.Source.Path)
}
}
for _, p := range m.Paths {
if !insideAny(p, dirs) {
return candidates
}
}
var out []inventory.Entry
for _, e := range candidates {
if e.Source.Path == "" || anyInside(m.Paths, e.Source.Path) {
out = append(out, e)
}
}
return out
}
// inside is whether a changed file is in a directory: that directory itself, or under it.
func inside(path, dir string) bool {
dir = strings.Trim(dir, "/")
path = strings.TrimPrefix(path, "/")
return path == dir || strings.HasPrefix(path, dir+"/")
}
// insideAny is whether a changed file is in any of these directories.
func insideAny(path string, dirs []string) bool {
for _, dir := range dirs {
if inside(path, dir) {
return true
}
}
return false
}
// anyInside is whether any of these changed files is in a directory.
func anyInside(paths []string, dir string) bool {
for _, p := range paths {
if inside(p, dir) {
return true
}
}
return false
}
// orderByBases is the entries with every base before what stands on it: a module whose build stood
// on another's artifact comes after that module. Entries outside the set are not waited for — they
// are not being rebuilt. Stable for what has no order between it.
//
// `against` is what each module's newest build stood on (inventory.BuiltAgainst): the edges are
// derived from builds, not declared, because a recorded manifest no longer carries `build.on`.
func orderByBases(entries []inventory.Entry, against map[string][]string) []inventory.Entry {
inSet := map[string]bool{}
for _, e := range entries {
inSet[e.Manifest.Module] = true
}
var out []inventory.Entry
placed := map[string]bool{}
var place func(e inventory.Entry, seen map[string]bool)
place = func(e inventory.Entry, seen map[string]bool) {
name := e.Manifest.Module
if placed[name] || seen[name] {
return
}
seen[name] = true
for _, base := range entries {
if base.Manifest.Module != name && inSet[base.Manifest.Module] && standsOnModule(e, base.Manifest.Module, against) {
place(base, seen)
}
}
placed[name] = true
out = append(out, e)
}
for _, e := range entries {
place(e, map[string]bool{})
}
return out
}
// standsOn is whether anything in the set is built on the named module's artifacts.
func standsOn(entries []inventory.Entry, module string, against map[string][]string) bool {
for _, e := range entries {
if standsOnModule(e, module, against) {
return true
}
}
return false
}
// standsOnModule is whether an entry's build stood on the named module: by what its newest build
// recorded it was handed (`artifact-store://<module>/<artifact>@…`, the module's own artifact), or
// — for a module registered from a manifest and not yet built — by the base its manifest names.
func standsOnModule(e inventory.Entry, module string, against map[string][]string) bool {
if e.Manifest.Module == module {
return false
}
if e.Manifest.Build != nil {
for _, on := range e.Manifest.Build.On {
if on.Module == module {
return true
}
}
}
prefix := catalogue.ArtifactStoreScheme + module + "/"
for _, ref := range against[e.Manifest.Module] {
if strings.HasPrefix(ref, prefix) {
return true
}
}
return false
}
// isHistory is whether a merge made at mergedAt predates the last time the source was seen. A merge
// with no time on it is taken as news: refusing it would silence a forge that says less.
func isHistory(mergedAt string, seen time.Time) bool {
if mergedAt == "" || seen.IsZero() {
return false
}
at, err := time.Parse(time.RFC3339, mergedAt)
if err != nil {
return false
}
return at.Before(seen)
}
+1 -2
View File
@@ -4,7 +4,7 @@ go 1.26.0
require (
github.com/jackc/pgx/v5 v5.10.0
github.com/rabbitmq/amqp091-go v1.14.0
github.com/nats-io/nats.go v1.54.0
golang.org/x/crypto v0.57.0
)
@@ -13,7 +13,6 @@ require (
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/klauspost/compress v1.20.0 // indirect
github.com/nats-io/nats.go v1.54.0 // indirect
github.com/nats-io/nkeys v0.4.16 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
golang.org/x/net v0.58.0 // indirect
-4
View File
@@ -19,15 +19,11 @@ github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw=
github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/rabbitmq/amqp091-go v1.14.0 h1:RSaT7aOKt/OrkVUyswPDW29lnRz9psuGmfZFBmLqLek=
github.com/rabbitmq/amqp091-go v1.14.0/go.mod h1:Hy4jKW5kQART1u+JkDTF9YYOQUHXqMuhrgxOEeS7G4o=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/net v0.58.0 h1:ynWG7rqYi4ccpTEuPZ2QGWHktVEM9DMCj9yzDE0Q7To=
-67
View File
@@ -1,67 +0,0 @@
package broker_test
import (
"regexp"
"testing"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/link"
)
// The names are written twice, so a test keeps them agreeing.
//
// `link` imports `broker`, so `broker` cannot import `link` — the queue and exchange names
// therefore exist in both. A scoped account naming a queue nothing publishes to produces a
// builder that takes no work and says nothing about why, which is the worst kind of silence.
//
// An external test package, because it may import both without either importing the other.
func TestTheNamesTheBrokerScopesAreTheNamesTheLinkUses(t *testing.T) {
for _, agreed := range []struct {
what string
scoped string
actually string
}{
{"the build queue", broker.BuildQueueName, link.BuildQueue},
{"the exchange", broker.ExchangeName, link.Exchange},
{"a node's queue", broker.QueueFor("somewhere"), link.QueueFor("somewhere")},
} {
if agreed.scoped != agreed.actually {
t.Errorf("%s: the broker scopes %q and the link uses %q",
agreed.what, agreed.scoped, agreed.actually)
}
}
}
func TestABuilderMayWriteToAReplyQueueAndReadNoNodesDeclarations(t *testing.T) {
// The scoping, checked as patterns rather than by connecting: what it may write must include
// the queue an asker actually waits on, and what it may read must not include any node's.
//
// This exists because the first version scoped writes to `amq.gen-*` — the name one broker
// happens to generate — and the builder built, could not answer, and the connection simply
// closed saying only "not allowed to publish to exchange \'\'". Answering through the
// exchange is what removed the need for any of that.
write := regexp.MustCompile("^" + regexp.QuoteMeta(broker.ExchangeName) + "$")
if !write.MatchString(broker.ExchangeName) {
t.Error("a builder may not write to the exchange, so it can take work and never answer")
}
// And not the default exchange, where permission is per exchange rather than per queue — a
// builder allowed to use it could publish into any node's queue.
//
// Confirmed against a real broker as well, and worth recording how that nearly went wrong:
// an unconfirmed publish is asynchronous, so a refusal arrives as a channel close afterwards
// and a naive check reports success. With publisher confirms the broker's refusal is
// immediate. **A negative security assertion made against an asynchronous call is not an
// assertion.**
if write.MatchString("") {
t.Error("a builder may publish to the default exchange, and so into any node's queue")
}
read := regexp.MustCompile("^" + regexp.QuoteMeta(broker.BuildQueueName) + "$")
if !read.MatchString(broker.BuildQueueName) {
t.Error("a builder may not read the build queue")
}
if read.MatchString(link.QueueFor("someone-else")) {
// A build machine is not a node, and a node's queue carries its declarations.
t.Error("a builder may read another machine's declarations")
}
}
+13
View File
@@ -66,6 +66,19 @@ func FromEnvironment() (Broker, error) {
if err != nil {
return Broker{}, err
}
// **One bus, one address** (novox/hq ADR 0131). Everything the mesh hands out — a token, a
// machine's membership, a person's credential — must name the bus the control plane itself is
// connected to; the setting above predates the move and, on a mesh that has moved, still names
// the broker it moved from. The first person issued after the move was handed the retired
// broker's port and could not connect to anything (2026-09-28).
//
// Read from the credential rather than from a second setting somebody keeps in step: the
// control plane cannot be wrong about where it is connected.
if bus, on, err := OnNATS(); err == nil && on {
if where := strings.TrimPrefix(BareAddress(bus), "nats://"); where != "" {
address = where
}
}
return Broker{Address: address, Fingerprint: fingerprint}, nil
}
-13
View File
@@ -194,16 +194,3 @@ func TestTheAddressPortFollowsThePortTwin(t *testing.T) {
t.Fatalf("the address is %q; the node put the bus on 5679", b.Address)
}
}
func TestTheManagementPortFollowsThePortTwin(t *testing.T) {
t.Setenv(ManagementVar, "http://guest:guest@127.0.0.1:15672")
t.Setenv(ManagementVar+"_FILE", "")
t.Setenv(ManagementVar+"_PORT", "15673")
m, err := ManagementFromEnvironment()
if err != nil {
t.Fatal(err)
}
if m.base.Host != "127.0.0.1:15673" {
t.Fatalf("the management API is at %q; the node put it on 15673", m.base.Host)
}
}
+67 -4
View File
@@ -1,8 +1,14 @@
package broker
import (
"crypto/sha256"
"crypto/tls"
"crypto/x509"
"encoding/hex"
"errors"
"fmt"
"os"
"strings"
"time"
"github.com/nats-io/nats.go"
@@ -24,21 +30,78 @@ type JetStream struct {
// Dial connects and returns the controller's JetStream handle.
func Dial(url string, opts ...nats.Option) (*JetStream, error) {
// A name, because a connection nobody can identify in the server's own monitoring is one
// nobody can attribute a problem to.
opts = append(opts, nats.Name("mesh-controller"), nats.Timeout(10*time.Second))
// **Pinned, not named.** The bus presents the mesh's own certificate, which names nothing a
// public verifier would accept (design 25 §4: a host pins the server's exact certificate and
// checks nothing else, and so does this). Without this, the first connection failed with
// "certificate is not valid for any names" against a bus that was answering (2026-09-28).
if path := strings.TrimSpace(os.Getenv(CertificateVar)); path != "" {
pinned, err := pinnedTo(path)
if err != nil {
return nil, err
}
opts = append(opts, nats.Secure(pinned))
}
// **Its own inbox, and nothing wider.** Every principal is granted `_INBOX.<its user>.>` and
// no other inbox; the client's default prefix is random, and the server refused the first
// subscription to it (2026-09-28). The user is in the URL, so the prefix follows from it.
if user, _, _ := CredentialIn(url); user != "" {
opts = append(opts, nats.CustomInboxPrefix("_INBOX."+user))
}
// The address in an error is the address alone. The URL carries this controller's password,
// and an error here is written on the assumption it will be logged.
where := BareAddress(url)
conn, err := nats.Connect(url, opts...)
if err != nil {
return nil, fmt.Errorf("connecting to the bus at %s: %w", url, err)
return nil, fmt.Errorf("connecting to the bus at %s: %w", where, err)
}
js, err := conn.JetStream()
if err != nil {
conn.Close()
return nil, fmt.Errorf("the bus at %s has no JetStream: %w", url, err)
return nil, fmt.Errorf("the bus at %s has no JetStream: %w", where, err)
}
return &JetStream{conn: conn, js: js}, nil
}
// pinnedTo is a TLS configuration that accepts exactly the certificate in the file and no other:
// the leaf's SHA-256, compared on every handshake, with the name and the chain deliberately not
// consulted — a self-signed certificate with no names is the ordinary case for a mesh's bus.
func pinnedTo(path string) (*tls.Config, error) {
want, err := FingerprintOf(path)
if err != nil {
return nil, err
}
return PinnedToFingerprint(want), nil
}
// DialPinned is Dial with the server's certificate pinned by a fingerprint the caller already holds
// — a module or a build machine that was handed one beside its credential, and has no file.
func DialPinned(url, fingerprint string, opts ...nats.Option) (*JetStream, error) {
if strings.TrimSpace(fingerprint) != "" {
opts = append(opts, nats.Secure(PinnedToFingerprint(fingerprint)))
}
return Dial(url, opts...)
}
// PinnedToFingerprint accepts exactly the certificate with this SHA-256 and no other.
func PinnedToFingerprint(want string) *tls.Config {
return &tls.Config{
InsecureSkipVerify: true, //nolint:gosec // replaced by the pin below, which is stricter
MinVersion: tls.VersionTLS12,
VerifyPeerCertificate: func(rawCerts [][]byte, _ [][]*x509.Certificate) error {
if len(rawCerts) == 0 {
return errors.New("the bus presented no certificate")
}
sum := sha256.Sum256(rawCerts[0])
got := "sha256:" + hex.EncodeToString(sum[:])
if got != want {
return fmt.Errorf("the bus presented a certificate this mesh does not know (%s…), expected %s…", got[:23], want[:23])
}
return nil
},
}
}
// Conn is the connection itself, for what the mesh keeps off JetStream on purpose — a heartbeat,
// a tool call — where a lost message is answered by the next one or by a timeout the caller
// already handles (design 25 §3).
-366
View File
@@ -1,366 +0,0 @@
package broker
import (
"bytes"
"context"
"encoding/json"
"fmt"
"github.com/novox/mesh-controller/internal/envfile"
"io"
"net/http"
"net/url"
"regexp"
"strings"
"time"
)
// The mesh runs the broker, so there is no chicken-and-egg in a node needing an account before it
// can connect: the account is created when the token is issued, and the one-time secret in that
// token IS the password. A node's first connection is already authenticated, and enrolment is
// what happens over it.
//
// novox/hq ADR 0004's *a node holds its own identity and nothing else* is why the account is per
// node rather than shared. A shared enrolment account would let any node consume another's queue,
// which is the shared-credential fault that record exists to remove, reappearing at the transport.
// ManagementVar holds the broker's management API, credentials included.
const ManagementVar = "MESH_BROKER_MANAGEMENT"
// safeName is what a node may be called at the broker.
//
// The name goes into a URL path and into permission patterns, which are regular expressions. A
// name carrying a `.` or a `*` would silently widen what that node may reach — so it is
// constrained here rather than escaped later, because an escape that is forgotten once is a node
// reading everybody's queues.
var safeName = regexp.MustCompile(`^[a-z0-9][a-z0-9-]{0,62}$`)
// Management is the broker's administrative interface.
type Management struct {
base *url.URL
client *http.Client
}
// ManagementFromEnvironment reads where the management API is, if it is configured.
//
// On the port MESH_BROKER_MANAGEMENT_PORT names when the node moved it (novox/hq 04-ISSUES/102);
// the URL's own port otherwise.
func ManagementFromEnvironment() (*Management, error) {
raw, err := envfile.Placed(ManagementVar)
if err != nil {
return nil, err
}
if raw == "" {
return nil, ErrNotConfigured
}
base, err := url.Parse(raw)
if err != nil || base.Host == "" {
// The value carries a password, so it is not quoted back.
return nil, fmt.Errorf("%s is not a usable URL", ManagementVar)
}
return &Management{base: base, client: &http.Client{Timeout: 15 * time.Second}}, nil
}
// QueueFor is the queue a node consumes from. One per node, named after it.
func QueueFor(node string) string { return "node." + node }
// ExchangeName is where nodes publish what they have to say. One exchange, and the control plane
// is the only consumer behind it (novox/hq ADR 0006 — one consumer, so two cannot silently split
// the traffic between them).
const ExchangeName = "mesh"
// BuildQueueName is where build work waits. Duplicated from `link` rather than imported, for the
// same reason QueueFor above is: this package must not depend on the one that uses it, and a
// constant that differed would be caught by the test that asserts they agree.
const BuildQueueName = "builds"
// The events bus (novox/hq ADR 0042): one topic exchange every event rides, a second for tool
// RPC kept apart, and a dead-letter home for a poison event. The foundation owns these — a module's
// account cannot declare them, only bind its own queue to the events one.
const (
EventsExchangeName = "mesh.events"
RPCExchangeName = "mesh.rpc"
DeadExchangeName = "mesh.events.dead"
)
// ModuleQueueFor is the durable queue a module consumes its events from — one per module per node
// (novox/hq ADR 0042), named so the account that may read it is exactly this module's.
func ModuleQueueFor(node, module string) string { return node + "." + module + ".events" }
// modulePermissions is a module's authority on the bus, derived from its manifest (novox/hq
// ADR 0043): what it consumes and what it emits, and nothing else. Pure, so the scope is tested as
// patterns without a broker — the way a builder's is.
//
// A note on the limit: the broker's write permission is per exchange, not per routing key (LavinMQ
// has no topic permissions), so an emitting module is granted the events exchange whole. ADR 0042's
// origin reservation — a module publishes only under `module.<self>.*` — is stamped by the sdk, not
// enforced here; that gap is the broker's, and is recorded rather than hidden. A pure consumer like
// the audit logger is unaffected: it is granted no write to the exchange at all.
func modulePermissions(node, module string, emits, consumes []string) (configure, write, read string) {
queue := regexp.QuoteMeta(ModuleQueueFor(node, module))
events := regexp.QuoteMeta(EventsExchangeName)
rpc := regexp.QuoteMeta(RPCExchangeName)
// A module serves each of its tools on its own queue, namespaced by the module (novox/hq
// ADR 0047) — serve.<module>.<tool> — so the account may declare, bind and read exactly its own,
// and no other module's.
serve := "serve\\." + regexp.QuoteMeta(module) + "\\..*"
// Declare its own events queue and its own tool serve queues.
configure = "^(" + queue + "|" + serve + ")$"
// Write to bind its queue and serve queues (binding is a write on the queue), and to the RPC
// exchange to publish replies (ADR 0047: replies ride mesh.rpc, never the default exchange, which
// would let it publish into any queue). To the events exchange only if it emits.
writes := []string{queue, serve, rpc}
if len(emits) > 0 {
writes = append(writes, events)
}
write = "^(" + strings.Join(writes, "|") + ")$"
// Read its own queue and serve queues to consume them, and the RPC exchange to bind its serve
// queues onto. The events exchange to bind onto only if it consumes.
reads := []string{queue, serve, rpc}
if len(consumes) > 0 {
reads = append(reads, events)
}
read = "^(" + strings.Join(reads, "|") + ")$"
return configure, write, read
}
// CreateModuleAccount gives an assigned module its own broker account, scoped by what it emits and
// consumes (novox/hq ADR 0043). The account name carries the node so the same module on two machines
// holds two accounts, each sealed to its own; the permissions carry the module so one module cannot
// read another's queue. Generic — the builder is one instance of this rule, not a separate kind.
func (m *Management) CreateModuleAccount(ctx context.Context, node, module, password string, emits, consumes []string) (string, error) {
if !safeName.MatchString(node) {
return "", fmt.Errorf("%q cannot be part of a broker account: it is a permission pattern", node)
}
if !safeName.MatchString(module) {
return "", fmt.Errorf("%q cannot be part of a broker account: it is a permission pattern", module)
}
account := node + "-" + module
if !safeName.MatchString(account) {
return "", fmt.Errorf("%q is not a usable broker account name", account)
}
if err := m.put(ctx, "/api/users/"+url.PathEscape(account),
map[string]string{"password": password, "tags": ""}); err != nil {
return "", fmt.Errorf("cannot create the broker account for %s on %s: %w", module, node, err)
}
configure, write, read := modulePermissions(node, module, emits, consumes)
if err := m.put(ctx, "/api/permissions/%2f/"+url.PathEscape(account), map[string]string{
"configure": configure, "write": write, "read": read,
}); err != nil {
return "", fmt.Errorf("cannot scope the broker account for %s on %s: %w", module, node, err)
}
return account, nil
}
// EnsureModuleQueue declares a consuming module's queue with its dead-letter exchange, idempotently.
// The foundation declares it because a scoped module account may not: the broker refuses a queue with
// a dead-letter exchange to a non-administrator (novox/hq ADR 0043), so a consumer passively checks
// the queue the mesh made rather than declaring its own.
func (m *Management) EnsureModuleQueue(ctx context.Context, node, module string) error {
queue := ModuleQueueFor(node, module)
if err := m.put(ctx, "/api/queues/%2f/"+url.PathEscape(queue), map[string]any{
"durable": true,
"arguments": map[string]any{"x-dead-letter-exchange": DeadExchangeName},
}); err != nil {
return fmt.Errorf("cannot declare the queue for %s on %s: %w", module, node, err)
}
return nil
}
// EnsureEventExchanges declares the bus's exchanges and the dead-letter home, idempotently. The
// foundation owns them (a module's account may not declare an exchange), and a dead-letter exchange
// with no queue behind it drops what it receives — so a durable queue bound to `#` retains a poison
// event for inspection, which is the whole reason the trail exists.
func (m *Management) EnsureEventExchanges(ctx context.Context) error {
for _, exchange := range []string{EventsExchangeName, RPCExchangeName, DeadExchangeName} {
if err := m.put(ctx, "/api/exchanges/%2f/"+url.PathEscape(exchange),
map[string]any{"type": "topic", "durable": true}); err != nil {
return fmt.Errorf("cannot declare the %s exchange: %w", exchange, err)
}
}
if err := m.put(ctx, "/api/queues/%2f/"+url.PathEscape(DeadExchangeName),
map[string]any{"durable": true}); err != nil {
return fmt.Errorf("cannot declare the dead-letter queue: %w", err)
}
if err := m.post(ctx, "/api/bindings/%2f/e/"+url.PathEscape(DeadExchangeName)+
"/q/"+url.PathEscape(DeadExchangeName), map[string]string{"routing_key": "#"}); err != nil {
return fmt.Errorf("cannot bind the dead-letter queue: %w", err)
}
return nil
}
// CreateNodeAccount gives a node its own broker account, with the token's secret as the password.
//
// Scoped so a node can reach its own queue and the one exchange, and nothing else. The patterns
// are anchored: a node called `laptop` must not be able to read `laptop-of-somebody-else`.
func (m *Management) CreateNodeAccount(ctx context.Context, node, password string) error {
if !safeName.MatchString(node) {
return fmt.Errorf(
"%q cannot be a broker account name: it becomes part of a permission pattern, so it "+
"is lower-case letters, digits and dashes", node)
}
if err := m.put(ctx, "/api/users/"+url.PathEscape(node),
map[string]string{"password": password, "tags": ""}); err != nil {
return fmt.Errorf("cannot create the broker account for %s: %w", node, err)
}
queue := regexp.QuoteMeta(QueueFor(node))
if err := m.put(ctx, "/api/permissions/%2f/"+url.PathEscape(node), map[string]string{
"configure": "^" + queue + "$",
"write": "^(" + regexp.QuoteMeta(ExchangeName) + "|" + queue + ")$",
"read": "^" + queue + "$",
}); err != nil {
return fmt.Errorf("cannot scope the broker account for %s: %w", node, err)
}
return nil
}
// CreateBuilderAccount scopes an account to taking build work and answering it.
//
// **A build machine is not a node**, and giving it a node's account would let it read another
// machine's declarations. What it needs is narrower and different: read the build queue, and
// write to the exchange and to whatever temporary queue an asker is waiting on.
//
// The reply queues are the reason `write` is not simply the exchange. `RequestBuild` declares an
// exclusive queue with a generated name and waits on it, so a builder that could not write to it
// could take work and never answer — which is the failure that looks like a builder that is not
// running.
func (m *Management) CreateBuilderAccount(ctx context.Context, name, password string) error {
if !safeName.MatchString(name) {
return fmt.Errorf(
"%q cannot be a broker account name: it becomes part of a permission pattern, so it "+
"is lower-case letters, digits and dashes", name)
}
if err := m.put(ctx, "/api/users/"+url.PathEscape(name),
map[string]string{"password": password, "tags": ""}); err != nil {
return fmt.Errorf("cannot create the broker account for %s: %w", name, err)
}
builds := regexp.QuoteMeta(BuildQueueName)
if err := m.put(ctx, "/api/permissions/%2f/"+url.PathEscape(name), map[string]string{
// It declares the build queue, because whichever builder starts first must be able to —
// and a queue nobody may declare is a queue that exists only if the control plane has
// already run, which makes the order they start in matter.
"configure": "^" + builds + "$",
// Two exchanges, and nothing else. **Not the default exchange**: permission there is
// granted per exchange rather than per queue, so a builder allowed to use it could
// publish into any node's queue — the privilege a build machine most obviously should
// not have. Answers go through the node exchange; announcing what was built goes through
// the events exchange, which is a different act with a different audience (novox/hq
// ADR 0072). A builder that could answer and not announce would leave the module graph
// knowing less than the registry does.
"write": "^(" + regexp.QuoteMeta(ExchangeName) + "|" + regexp.QuoteMeta(EventsExchangeName) + ")$",
// The build queue and nothing else. Not another machine's declarations.
"read": "^" + builds + "$",
}); err != nil {
return fmt.Errorf("cannot scope the broker account for %s: %w", name, err)
}
return nil
}
// RemoveNodeAccount withdraws a node's access.
func (m *Management) RemoveNodeAccount(ctx context.Context, node string) error {
if !safeName.MatchString(node) {
return fmt.Errorf("%q is not a broker account name", node)
}
return m.do(ctx, http.MethodDelete, "/api/users/"+url.PathEscape(node), nil)
}
// Accounts lists the broker's users, so a picture can be read from the system rather than assumed
// (novox/hq ADR 0018).
func (m *Management) Accounts(ctx context.Context) ([]string, error) {
body, err := m.get(ctx, "/api/users")
if err != nil {
return nil, err
}
var users []struct {
Name string `json:"name"`
}
if err := json.Unmarshal(body, &users); err != nil {
return nil, err
}
names := make([]string, 0, len(users))
for _, u := range users {
names = append(names, u.Name)
}
return names, nil
}
func (m *Management) put(ctx context.Context, path string, body any) error {
return m.do(ctx, http.MethodPut, path, body)
}
func (m *Management) post(ctx context.Context, path string, body any) error {
return m.do(ctx, http.MethodPost, path, body)
}
func (m *Management) get(ctx context.Context, path string) ([]byte, error) {
request, err := m.request(ctx, http.MethodGet, path, nil)
if err != nil {
return nil, err
}
response, err := m.client.Do(request)
if err != nil {
return nil, err
}
defer response.Body.Close()
if response.StatusCode >= 300 {
return nil, fmt.Errorf("the broker's management API answered %s to GET %s",
response.Status, path)
}
return io.ReadAll(io.LimitReader(response.Body, 1<<20))
}
func (m *Management) do(ctx context.Context, method, path string, body any) error {
request, err := m.request(ctx, method, path, body)
if err != nil {
return err
}
response, err := m.client.Do(request)
if err != nil {
return err
}
defer response.Body.Close()
if response.StatusCode >= 300 {
detail, _ := io.ReadAll(io.LimitReader(response.Body, 4096))
return fmt.Errorf("the broker's management API answered %s to %s %s: %s",
response.Status, method, path, strings.TrimSpace(string(detail)))
}
return nil
}
func (m *Management) request(ctx context.Context, method, path string, body any) (*http.Request, error) {
var payload io.Reader
if body != nil {
raw, err := json.Marshal(body)
if err != nil {
return nil, err
}
payload = bytes.NewReader(raw)
}
// Path joined by hand rather than through url.Parse: %2f is the default vhost and must reach
// the broker still encoded. Parsing would decode it to a slash and address a different route.
target := strings.TrimSuffix(m.base.String(), "/")
if user := m.base.User; user != nil {
target = strings.TrimSuffix(m.base.Scheme+"://"+m.base.Host, "/")
}
request, err := http.NewRequestWithContext(ctx, method, target+path, payload)
if err != nil {
return nil, err
}
if user := m.base.User; user != nil {
password, _ := user.Password()
request.SetBasicAuth(user.Username(), password)
}
if body != nil {
request.Header.Set("Content-Type", "application/json")
}
return request, nil
}
-96
View File
@@ -1,96 +0,0 @@
package broker_test
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"regexp"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/broker"
)
// The audit logger consumes everything and emits nothing. Its account must let it declare and read
// its own queue and read the events exchange to bind onto — and must not reach another module's
// queue, nor grant any write to the events exchange (novox/hq ADR 0043).
func TestAConsumerReadsItsOwnQueueAndTheEventsExchangeAndNoOthers(t *testing.T) {
// Rebuilt through CreateModuleAccount's own path by asking for the same scope it would apply.
// The queue this module reads:
mine := broker.ModuleQueueFor("anchor", "audit-logger")
other := broker.ModuleQueueFor("anchor", "plex")
read := scope(t, "read", "anchor", "audit-logger", nil, []string{"#"})
if !read.MatchString(mine) {
t.Error("the audit logger may not read its own queue, so it consumes nothing")
}
if !read.MatchString(broker.EventsExchangeName) {
t.Error("the audit logger may not read the events exchange, so it cannot bind onto it")
}
if read.MatchString(other) {
t.Error("the audit logger may read another module's queue")
}
// It emits nothing, so it is granted no write to the events exchange — only its own queue, to bind.
write := scope(t, "write", "anchor", "audit-logger", nil, []string{"#"})
if write.MatchString(broker.EventsExchangeName) {
t.Error("a pure consumer was granted write to the events exchange")
}
if !write.MatchString(mine) {
t.Error("the audit logger may not write to its own queue, so it cannot bind it")
}
}
// An emitter is granted the events exchange to write; a consumer is not.
func TestAnEmitterMayWriteTheEventsExchangeAndAConsumerMayNot(t *testing.T) {
emitter := scope(t, "write", "anchor", "umami", []string{"module.umami.site.created"}, nil)
if !emitter.MatchString(broker.EventsExchangeName) {
t.Error("an emitting module may not write the events exchange, so it cannot emit")
}
}
// scope reconstructs one of the three permission patterns CreateModuleAccount would apply, by
// reading it back from a captured request against a stub management API.
func scope(t *testing.T, which, node, module string, emits, consumes []string) *regexp.Regexp {
t.Helper()
pat := capturePermission(t, which, node, module, emits, consumes)
re, err := regexp.Compile(pat)
if err != nil {
t.Fatalf("the %s pattern does not compile: %v", which, err)
}
return re
}
// capturePermission runs CreateModuleAccount against a stub management API and returns the pattern
// it set for `which` ("configure"/"write"/"read"). The scope is tested where it is applied, not
// reconstructed by the test — so a change to the mapping cannot pass a test that hard-codes the old
// one.
func capturePermission(t *testing.T, which, node, module string, emits, consumes []string) string {
t.Helper()
var captured map[string]string
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if strings.HasPrefix(r.URL.Path, "/api/permissions/") {
_ = json.NewDecoder(r.Body).Decode(&captured)
}
w.WriteHeader(http.StatusNoContent)
}))
defer server.Close()
t.Setenv(broker.ManagementVar, server.URL)
m, err := broker.ManagementFromEnvironment()
if err != nil {
t.Fatalf("stub management not usable: %v", err)
}
if _, err := m.CreateModuleAccount(context.Background(), node, module, "pw", emits, consumes); err != nil {
t.Fatalf("CreateModuleAccount: %v", err)
}
if captured == nil {
t.Fatal("no permissions were set")
}
pattern, ok := captured[which]
if !ok {
t.Fatalf("no %s permission was set; got %v", which, captured)
}
return pattern
}
+60 -10
View File
@@ -169,7 +169,11 @@ func PermissionsFor(p Principal) (Permissions, error) {
// The controller owns the mesh's own traffic and the streams. It is the only writer of
// stream definitions (design 25 §3), so it alone reaches the JetStream API.
pub = []string{"mesh.control.>", "mesh.node.>", "$JS.API.>"}
sub = []string{"mesh.control.>", "$JS.API.>"}
// **And where its consumers deliver.** A push consumer delivers on `_DELIVER.<its name>`,
// and a client bound to it subscribes exactly that; the server refused it for every
// principal the first time one bound a consumer (2026-09-28). Each kind below is granted
// its own consumers' delivery subjects and no other's.
sub = []string{"mesh.control.>", "$JS.API.>", "_DELIVER." + ControllerName, "_DELIVER." + ControllerName + ".>"}
// Work the mesh's own flows submit to a role, and the outcomes they wait on (ADR 0121). A
// build is the one today: the controller asks, and reads the answer from the seat's event
@@ -177,6 +181,19 @@ func PermissionsFor(p Principal) (Permissions, error) {
for _, seat := range meshSeatsTheControllerUses {
pub = append(pub, "mesh.seat."+seat+".accept.>")
}
// **And what the mesh says it did** (novox/hq ADR 0134). The control plane states its own
// facts under the seat it holds, because a role's events belong to the role and keep their
// address while the holder is replaced. Named one by one rather than as a whole namespace:
// least authority, and a fact nothing states is authority nobody uses.
for _, event := range ControllerStates {
pub = append(pub, seatEventSubject(ControllerSeat, event))
}
// Every module's tools: **the control plane is the way in** (novox/hq ADR 0095). A person
// or an agent asks through it and every question passes one process where an audit
// belongs — so it, alone among principals, may call any tool by name. The first `ask` on
// the new bus was refused the publish (2026-09-28).
pub = append(pub, "mesh.mod.*.tool.>")
// The two events it reacts to, and its ack subject on the stream they arrive from
// (streams.go). **Each named, not a pattern**: `mesh.mod.*.event.>` would make the
@@ -244,8 +261,15 @@ func PermissionsFor(p Principal) (Permissions, error) {
case KindNode:
// A host publishes its own node's control traffic and subscribes its own declaration —
// and nothing of any other node's.
pub = []string{"mesh.control." + p.Node + ".>"}
sub = []string{"mesh.node." + p.Node + ".declare"}
// And binding to its consumer, which asks the server about it (CONSUMER.INFO) — the one
// thing the host does that nothing granted. Found the first time a machine dialled a
// permissioned server: "this node cannot read its declarations" (2026-09-28). The ack and
// the inbox are granted below with every principal's.
pub = []string{
"mesh.control." + p.Node + ".>",
"$JS.API.CONSUMER.INFO.NODES." + p.Node,
}
sub = []string{"mesh.node." + p.Node + ".declare", "_DELIVER." + p.Node}
case KindModule:
// 1. Its own namespace: it publishes its events there and serves its tools there. Nothing
@@ -255,9 +279,13 @@ func PermissionsFor(p Principal) (Permissions, error) {
for _, e := range p.Emits {
pub = append(pub, own+".event."+e)
}
for _, t := range p.Serves {
sub = append(sub, own+".tool."+t)
}
// Every tool under its own name, not a list: the tools a module serves are what its code
// answers, and a second copy of that list in the manifest would be a second source of
// truth for the mesh to keep in step (2026-09-28: every module that served a tool was
// refused the subscription, because none had written the list twice). Nothing is given
// away — no other principal may subscribe this namespace, and a caller's authority is
// still granted per tool, by name, on the publish side.
sub = append(sub, own+".tool.>")
// 2. What it consumes, by the emitter's own subject — an event is addressed to its
// emitter, because the emitter's identity is the meaning (ADR 0118).
@@ -277,8 +305,26 @@ func PermissionsFor(p Principal) (Permissions, error) {
}
}
// 2c. Its own consumer, which it **pulls**: the runtime asks for the next message and is
// answered on its own inbox, so what it needs is to ask about the consumer and to ask it
// for messages — its own consumer's name, and no other's. Pulled rather than pushed
// because that is the one shape a runtime's client binds without creating anything; the
// controller and the hosts are pushed to. Named here rather than through ConsumerFor,
// which asks for these permissions to build the consumer and would ask forever. A
// subject for a consumer that turns out not to exist grants nothing anybody can use.
pub = append(pub,
"$JS.API.CONSUMER.INFO."+consumerStream(p)+"."+consumerDurable(p),
"$JS.API.CONSUMER.MSG.NEXT."+consumerStream(p)+"."+consumerDurable(p))
// 3. Seats it holds: full participation.
for _, s := range p.Holds {
// Taking work from the role's queue: the worker consumer it binds (asked about,
// delivered on, acknowledged), each on the seat's own stream. The first machine to
// take work over the new bus was refused the asking (2026-09-28).
worker := "SEAT_" + upperSnake(s.Name) + "_worker"
stream := seatStreamName(s.Name)
sub = append(sub, "_DELIVER."+worker)
pub = append(pub, "$JS.API.CONSUMER.INFO."+stream+"."+worker, "$JS.ACK."+stream+"."+worker+".>")
for _, a := range s.Accepts {
sub = append(sub, seatSubject(s, "accept", a))
}
@@ -326,9 +372,10 @@ func PermissionsFor(p Principal) (Permissions, error) {
return Permissions{
Publish: pub,
Subscribe: sub,
// Only something that serves is ever answering. A pure consumer is granted nothing here.
AllowResponses: p.Kind == KindModule && (len(p.Serves) > 0 || len(p.Holds) > 0) ||
p.Kind == KindController,
// A module answers what it was asked — a tool call reaches it on its own namespace, so the
// authority is bounded by having been asked — and so does the controller. A node and a
// person are never asked anything, and are granted nothing here.
AllowResponses: p.Kind == KindModule || p.Kind == KindController,
}, nil
}
@@ -514,7 +561,10 @@ func ComposeAccounts(principals []Principal) (string, error) {
// One account for the mesh: accounts in NATS isolate subject spaces entirely, and the mesh is
// one space (design 25 §4). The cost of that — that permissions are the only isolation — is
// paid in the scoping of every inbox and every ack subject.
b.WriteString("accounts {\n MESH {\n users = [\n")
// JetStream is enabled per account once accounts exist at all: with only the global block set,
// a user in MESH is told "JetStream not enabled for account" the first time it binds a
// consumer, which is the first thing every host does (2026-09-28).
b.WriteString("accounts {\n MESH {\n jetstream: enabled\n users = [\n")
for _, p := range sorted {
perms, err := PermissionsFor(p)
if err != nil {
+45 -10
View File
@@ -1,6 +1,7 @@
package broker
import (
"slices"
"strings"
"testing"
)
@@ -89,17 +90,30 @@ func TestAnInboxIsScopedToItsOwner(t *testing.T) {
}
// A responder answers on the caller's inbox, which it has no permission for. allow_responses is
// what makes a scoped inbox workable at all — the authority is bounded by having been asked.
func TestOnlySomethingThatServesMayAnswer(t *testing.T) {
serving, _ := PermissionsFor(Principal{Kind: KindModule, Node: "one", Module: "billing",
Serves: []string{"status"}, PasswordHash: "x"})
if !serving.AllowResponses {
t.Fatal("a module serving a tool cannot answer the caller's inbox")
}
consumer, _ := PermissionsFor(Principal{Kind: KindModule, Node: "one", Module: "audit",
// what makes a scoped inbox workable at all — the authority is bounded by having been asked. A
// module is asked on its own namespace and may answer; a node and a person are never asked.
func TestOnlyWhatCanBeAskedMayAnswer(t *testing.T) {
module, _ := PermissionsFor(Principal{Kind: KindModule, Node: "one", Module: "audit",
Consumes: []string{"shop.order.placed"}, PasswordHash: "x"})
if consumer.AllowResponses {
t.Fatal("a pure consumer was granted the right to answer, which nothing asked it to do")
if !module.AllowResponses {
t.Fatal("a module cannot answer a tool call on its own namespace")
}
node, _ := PermissionsFor(Principal{Kind: KindNode, Node: "one", PasswordHash: "x"})
if node.AllowResponses {
t.Fatal("a node was granted the right to answer, and nothing asks a node anything")
}
}
// A module serves every tool under its own name, and no other module's.
func TestAModuleServesItsOwnNamespaceAndNoOthers(t *testing.T) {
p, _ := PermissionsFor(Principal{Kind: KindModule, Node: "one", Module: "gitea", PasswordHash: "x"})
if !slices.Contains(p.Subscribe, "mesh.mod.gitea.tool.>") {
t.Fatalf("a module may not serve its own tools: %v", p.Subscribe)
}
for _, s := range p.Subscribe {
if strings.HasPrefix(s, "mesh.mod.") && !strings.HasPrefix(s, "mesh.mod.gitea.") {
t.Fatalf("a module may subscribe another's namespace: %s", s)
}
}
}
@@ -324,3 +338,24 @@ func admits(pattern, subject []string) bool {
}
return len(pattern) == len(subject)
}
// A module pulls its own consumer — asks about it, asks it for messages — and no other module's.
func TestAModulePullsItsOwnConsumerAndNoOthers(t *testing.T) {
p, _ := PermissionsFor(Principal{Kind: KindModule, Node: "one", Module: "audit",
Consumes: []string{"shop.order.placed"}, PasswordHash: "x"})
for _, want := range []string{"$JS.API.CONSUMER.INFO.EVENTS.one_audit", "$JS.API.CONSUMER.MSG.NEXT.EVENTS.one_audit"} {
if !slices.Contains(p.Publish, want) {
t.Errorf("a module cannot bind its own consumer: %v lacks %s", p.Publish, want)
}
}
for _, s := range p.Publish {
if strings.Contains(s, "CONSUMER.") && !strings.HasSuffix(s, ".one_audit") {
t.Errorf("a module may reach another consumer: %s", s)
}
}
for _, s := range p.Subscribe {
if strings.HasPrefix(s, "_DELIVER.") {
t.Errorf("a module is granted a push delivery it never binds: %s", s)
}
}
}
+11 -19
View File
@@ -68,24 +68,16 @@ func BareAddress(address string) string {
return "nats://" + bare
}
// MustBeOneBus refuses a configuration that names both buses for the mesh's own traffic.
//
// **Both clients ship and that is the point; both being live is not.** The rollout moves every node
// at once (ADR 0116 step 5): a mesh half on each is one where a declaration goes out on one bus and
// the report comes back on the other, and nothing anywhere says so — every component would log
// success. Refused at start, where it can be said in one sentence.
func MustBeOneBus(amqp, nats string) error {
if strings.TrimSpace(amqp) != "" && strings.TrimSpace(nats) != "" {
return fmt.Errorf(
"this control plane is told about both buses (%s and %s) and can only be on one. A mesh "+
"half on each is one where a declaration goes out on one and the report comes back "+
"on the other, and every component reports success while it happens. The rollout "+
"moves every node at once: unset %s to stay, or unset %s to move",
AMQPVarName, NATSVar, NATSVar, AMQPVarName)
// BusAddress is where the mesh's bus is, with this process's credential, and refuses to be empty:
// there is one bus, and a control plane without it can hold records and answer nothing (novox/hq
// ADR 0131, design 28 task 5.5).
func BusAddress() (string, error) {
address, _, err := OnNATS()
if err != nil {
return "", err
}
return nil
if address == "" {
return "", fmt.Errorf("this control plane has no %s, so it cannot reach the mesh's bus", NATSVar)
}
return address, nil
}
// AMQPVarName is the variable naming the bus the mesh runs on today. Named here rather than
// imported from the link package, for the one direction of dependency.
const AMQPVarName = "MESH_BROKER_AMQP"
-32
View File
@@ -1,43 +1,11 @@
package broker
import (
"strings"
"testing"
)
// Which bus the mesh is on is one fact, and being told about both is refused.
//
// **Not a warning.** A mesh half on each bus is one where a declaration goes out on one and the
// report comes back on the other, and every component reports success while it happens — which is
// the exact failure ADR 0074 exists to catch, arriving through configuration instead of through code.
func TestBeingToldAboutBothBusesIsRefused(t *testing.T) {
err := MustBeOneBus("amqps://broker:5671/", "nats://bus:4222")
if err == nil {
t.Fatal("a control plane told about both buses was allowed to start")
}
// The remedy is in the words, because whoever reads this has to choose one and the wrong choice
// is a rollout half done.
for _, want := range []string{AMQPVarName, NATSVar, "unset"} {
if !strings.Contains(err.Error(), want) {
t.Errorf("the refusal does not mention %s: %v", want, err)
}
}
}
// One bus, or none, is ordinary. None is a control plane that publishes nothing and holds records,
// which several of its own commands are.
func TestOneBusOrNeitherIsAllowed(t *testing.T) {
for _, c := range []struct{ what, amqp, nats string }{
{"the bus the mesh runs on today", "amqps://broker:5671/", ""},
{"the bus being built", "", "nats://bus:4222"},
{"neither", "", ""},
{"neither, with whitespace for an address", " ", "\t"},
} {
if err := MustBeOneBus(c.amqp, c.nats); err != nil {
t.Errorf("%s was refused: %v", c.what, err)
}
}
}
// The controller's own credential arrives in its address, and has to be readable out of it — its user
// is created by the installer at a bootstrap password, before the controller exists to mint one.
+44
View File
@@ -0,0 +1,44 @@
package broker_test
import (
"slices"
"testing"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/link"
)
// The facts the control plane states are named twice — in the grant that permits them and in the code
// that states them — because `link` imports `broker` and the dependency cannot go the other way. So a
// test keeps them agreeing: a subject the grant omits is refused at the moment the mesh has something
// to say, and one the grant adds that nothing states is authority nobody uses.
//
// An external test package, because it may import both while neither imports the other.
func TestTheFactsTheGrantPermitsAreTheFactsTheMeshStates(t *testing.T) {
if broker.ControllerSeat != link.MeshControllerSeat {
t.Fatalf("the grant is written for the %q seat and the mesh states its facts under %q",
broker.ControllerSeat, link.MeshControllerSeat)
}
for _, event := range []string{link.KeyApplied, link.KeyRefused, link.KeyBuiltBefore} {
if !slices.Contains(broker.ControllerStates, event) {
t.Errorf("the mesh states %q and its account may not publish it", event)
}
}
if len(broker.ControllerStates) != 3 {
t.Errorf("the grant permits %v, which is more than the mesh states", broker.ControllerStates)
}
// **And the seat says it.** A seat carries the protocol of its role (novox/hq ADR 0129), so the
// facts the control plane states are the seat's `emits` — which is what lets anything else declare
// that it consumes them, and what the subject-agreement check reads to know they have an owner.
var declared []string
for _, seat := range catalogue.SeatsWithAProtocol() {
if seat.Name == broker.ControllerSeat {
declared = seat.Emits
}
}
if !slices.Equal(declared, broker.ControllerStates) {
t.Errorf("the %s seat emits %v and the grant permits %v", broker.ControllerSeat,
declared, broker.ControllerStates)
}
}
+14
View File
@@ -160,6 +160,17 @@ func Overlaps() []string {
// ack subject is derived from (nats.go: `$JS.ACK.<stream>.controller.>`).
const ControllerName = "controller"
// ControllerSeat is the role the control plane holds, and ControllerStates are the facts it states
// under it (novox/hq ADR 0134).
//
// **Written here as well as in `link`, and a test keeps them agreeing.** `link` imports `broker`, so
// `broker` cannot import `link`; a grant naming a subject the controller never publishes is authority
// nobody uses, and a controller publishing one the grant omits is refused at the moment it has
// something to say.
const ControllerSeat = "mesh-controller"
var ControllerStates = []string{"applied", "refused", "built-before"}
// ControllerFollows are the events the controller reacts to: the catalogue saying a module's
// current version moved, and a catalogue that has just started saying it may have missed builds.
//
@@ -175,6 +186,9 @@ var ControllerFollows = []string{
// message on the control branch. Same three audiences, one publish: whoever asked, this, and the
// catalogue.
seatEventSubject("mesh-build-machine", "built"),
// The forge's merges: what moved a source, so the mesh builds what that source produces
// without anybody telling it (novox/hq 04-ISSUES/131). Appended, because the index is a name.
moduleEventSubject("gitea", "pull.merged"),
}
// moduleEventSubject is where one module's event lands. The same derivation PermissionsFor uses, so
+13 -10
View File
@@ -21,10 +21,11 @@ jetstream {
accounts {
MESH {
jetstream: enabled
users = [
{ user: "controller", password: "$2a$11$cccccccccccccccccccccc", permissions: {
publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "_INBOX.enrol.>", "mesh.control.>", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>"] }
subscribe: { allow: ["$JS.API.>", "_INBOX.controller.>", "mesh.control.>", "mesh.mod.mesh-catalog.event.catching-up", "mesh.mod.mesh-catalog.event.upgraded", "mesh.seat.mesh-build-machine.event.built"] }
publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "_INBOX.enrol.>", "mesh.control.>", "mesh.mod.*.tool.>", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>", "mesh.seat.mesh-controller.event.applied", "mesh.seat.mesh-controller.event.built-before", "mesh.seat.mesh-controller.event.refused"] }
subscribe: { allow: ["$JS.API.>", "_DELIVER.controller", "_DELIVER.controller.>", "_INBOX.controller.>", "mesh.control.>", "mesh.mod.gitea.event.pull.merged", "mesh.mod.mesh-catalog.event.catching-up", "mesh.mod.mesh-catalog.event.upgraded", "mesh.seat.mesh-build-machine.event.built"] }
allow_responses: { max: 1, ttl: "1m" }
} }
{ user: "enrol.one", password: "$2a$11$eeeeeeeeeeeeeeeeeeeeee", permissions: {
@@ -32,21 +33,23 @@ accounts {
subscribe: { allow: ["_INBOX.enrol.one.>"] }
} }
{ user: "node.one", password: "$2a$11$nnnnnnnnnnnnnnnnnnnnnn", permissions: {
publish: { allow: ["$JS.ACK.NODES.one.>", "mesh.control.one.>"] }
subscribe: { allow: ["_INBOX.node.one.>", "mesh.node.one.declare"] }
publish: { allow: ["$JS.ACK.NODES.one.>", "$JS.API.CONSUMER.INFO.NODES.one", "mesh.control.one.>"] }
subscribe: { allow: ["_DELIVER.one", "_INBOX.node.one.>", "mesh.node.one.declare"] }
} }
{ user: "one.telegram", password: "$2a$11$tttttttttttttttttttttt", permissions: {
publish: { allow: ["$JS.ACK.EVENTS.one_telegram.>", "mesh.seat.telegram-sender.event.delivered", "mesh.seat.telegram-sender.event.failed"] }
subscribe: { allow: ["_INBOX.one.telegram.>", "mesh.mod.telegram.tool.status", "mesh.seat.telegram-sender.accept.send"] }
publish: { allow: ["$JS.ACK.EVENTS.one_telegram.>", "$JS.ACK.SEAT_TELEGRAM_SENDER.SEAT_TELEGRAM_SENDER_worker.>", "$JS.API.CONSUMER.INFO.EVENTS.one_telegram", "$JS.API.CONSUMER.INFO.SEAT_TELEGRAM_SENDER.SEAT_TELEGRAM_SENDER_worker", "$JS.API.CONSUMER.MSG.NEXT.EVENTS.one_telegram", "mesh.seat.telegram-sender.event.delivered", "mesh.seat.telegram-sender.event.failed"] }
subscribe: { allow: ["_DELIVER.SEAT_TELEGRAM_SENDER_worker", "_INBOX.one.telegram.>", "mesh.mod.telegram.tool.>", "mesh.seat.telegram-sender.accept.send"] }
allow_responses: { max: 1, ttl: "1m" }
} }
{ user: "two.audit", password: "$2a$11$aaaaaaaaaaaaaaaaaaaaaa", permissions: {
publish: { allow: ["$JS.ACK.EVENTS.two_audit.>"] }
subscribe: { allow: ["_INBOX.two.audit.>", "mesh.mod.shop.event.order.placed"] }
publish: { allow: ["$JS.ACK.EVENTS.two_audit.>", "$JS.API.CONSUMER.INFO.EVENTS.two_audit", "$JS.API.CONSUMER.MSG.NEXT.EVENTS.two_audit"] }
subscribe: { allow: ["_INBOX.two.audit.>", "mesh.mod.audit.tool.>", "mesh.mod.shop.event.order.placed"] }
allow_responses: { max: 1, ttl: "1m" }
} }
{ user: "two.shop", password: "$2a$11$ssssssssssssssssssssss", permissions: {
publish: { allow: ["$JS.ACK.EVENTS.two_shop.>", "mesh.mod.shop.event.order.placed", "mesh.seat.telegram-sender.accept.send"] }
subscribe: { allow: ["_INBOX.two.shop.>"] }
publish: { allow: ["$JS.ACK.EVENTS.two_shop.>", "$JS.API.CONSUMER.INFO.EVENTS.two_shop", "$JS.API.CONSUMER.MSG.NEXT.EVENTS.two_shop", "mesh.mod.shop.event.order.placed", "mesh.seat.telegram-sender.accept.send"] }
subscribe: { allow: ["_INBOX.two.shop.>", "mesh.mod.shop.tool.>"] }
allow_responses: { max: 1, ttl: "1m" }
} }
]
}
+7 -1
View File
@@ -186,7 +186,13 @@ func TestWhatTheMeshWritesIsUsersAndNothingAboutTheServer(t *testing.T) {
}
// None of the server's own settings. Each of these in the mesh's file is a value the controller
// would then own, and the module could no longer change its own image without the mesh agreeing.
for _, absent := range []string{"port:", "http:", "jetstream", "tls {", "store_dir", "cert_file"} {
// `jetstream {` is the server's block (its store, its limits); `jetstream: enabled` inside the
// account is the account's, and the mesh owns the account — a user in it is told "JetStream
// not enabled for account" without it (2026-09-28).
if !strings.Contains(got, "jetstream: enabled") {
t.Errorf("the account does not enable JetStream, so no user in it can bind a consumer")
}
for _, absent := range []string{"port:", "http:", "jetstream {", "tls {", "store_dir", "cert_file"} {
if strings.Contains(got, absent) {
t.Errorf("the accounts file contains %q, which belongs to the module that raises the "+
"server, not to the mesh", absent)
+68 -15
View File
@@ -61,6 +61,12 @@ type Result struct {
Commit string
// Built is each artifact, for reporting.
Built []catalogue.Built
// Read is every repository this build read source from besides the module's own — the second
// repository an artifact's recipe names (ArtifactContext). Reported because the manifest the
// mesh keeps carries no build section, so nothing else could say that a merge there is a
// change to this module (novox/hq 04-ISSUES/131).
Read []catalogue.ArtifactContext
}
// GitCredential is the forge credential a clone may present when the server asks for one.
@@ -169,6 +175,8 @@ func Build(ctx context.Context, run Runner, publish Publisher,
}
var built []catalogue.Built
// stoodOn is every base the build was handed, as resolved — the edges the catalogue derives.
var stoodOn []string
if manifest.Build != nil {
// What this module said it stands on, answered with what this mesh actually holds. Done
// before anything is built, so a missing base is refused in front of the person who can
@@ -186,11 +194,12 @@ func Build(ctx context.Context, run Runner, publish Publisher,
}
return from, nil
}
args, err := standingOn(ctx, manifest, held, mirror)
args, bases, err := standingOn(ctx, manifest, held, mirror)
if err != nil {
say("bases", "UNMET: %v", err)
return Result{}, err
}
stoodOn = bases
if len(args) > 0 {
say("bases", "%d resolved from what the mesh holds", len(args)/2)
}
@@ -217,7 +226,7 @@ func Build(ctx context.Context, run Runner, publish Publisher,
}
say("done", "%s at %s — %d artifact(s) pinned", manifest.Module, short(commit), len(built))
return Result{Manifest: resolved, Commit: commit, Built: built,
Against: against(within, manifest)}, nil
Against: against(within, manifest, stoodOn), Read: readBy(manifest)}, nil
}
// Log is where a build says what it is doing, step by step. Nil is silent — the tests pass none,
@@ -339,14 +348,27 @@ func describe(path string) string {
// allowed to name — a tag is something somebody else can move under you.
var pinnedImage = regexp.MustCompile(`[A-Za-z0-9][A-Za-z0-9._/:-]*@sha256:[0-9a-f]{64}`)
// against reads what this module's image artifacts are built on top of, out of the files that
// build them. Nothing is guessed: a reference that is not written down is not reported.
func against(within string, manifest catalogue.Manifest) []string {
// against is what this module's image artifacts are built on top of: every base the mesh resolved
// and handed the recipe as a build argument (`build.on`), and any image a recipe pins by digest
// itself. Nothing is guessed: a reference that was neither resolved nor written down is not
// reported.
//
// **The resolved bases are the edges.** A recipe reads its base from an argument (`FROM
// ${RUNTIME_BASE}`), so the digest is never in the file, and a derivation that read files alone
// recorded no edge for any module on the mesh — which is why nothing knew what a changed base
// meant to rebuild (novox/hq 04-ISSUES/131).
func against(within string, manifest catalogue.Manifest, resolved []string) []string {
if manifest.Build == nil {
return nil
}
seen := map[string]bool{}
var out []string
for _, r := range resolved {
if r != "" && !seen[r] {
seen[r] = true
out = append(out, r)
}
}
for _, a := range manifest.Build.Artifacts {
if a.Kind != catalogue.ArtifactImage || a.From == "" {
continue
@@ -704,40 +726,42 @@ var _ io.Writer = (*stringWriter)(nil)
// built cannot be built here yet, and the useful sentence names which module is missing — not the
// one a container runtime produces when a recipe's first line refers to an image nobody has.
//
// The order is fixed so two builds of one commit invoke the same command.
// The order is fixed so two builds of one commit invoke the same command. Returned alongside the
// arguments is every reference they resolved to, which is what the build stood on.
func standingOn(ctx context.Context, manifest catalogue.Manifest, held map[string]string,
mirror func(ctx context.Context, from, repository string) (string, error)) ([]string, error) {
mirror func(ctx context.Context, from, repository string) (string, error)) ([]string, []string, error) {
if manifest.Build == nil || len(manifest.Build.On) == 0 {
return nil, nil
return nil, nil, nil
}
on := append([]catalogue.BuildsOn{}, manifest.Build.On...)
sort.Slice(on, func(i, j int) bool { return on[i].Arg < on[j].Arg })
var args []string
var args, resolved []string
for _, base := range on {
if base.Image != "" {
// A vendor's image, declared (novox/hq 04-ISSUES/064, ADR 0097). Pinned, because a tag
// is what somebody else can move; copied into the mesh's registry, because a build
// that reaches a public registry on its own is a build that works sometimes.
if base.Arg == "" || base.Module != "" || base.Artifact != "" {
return nil, fmt.Errorf(
return nil, nil, fmt.Errorf(
"%s stands on the image %s, and a base is either a module's artifact or an "+
"image — never both — read from one build argument", manifest.Module, base.Image)
}
if !strings.Contains(base.Image, "@sha256:") {
return nil, fmt.Errorf(
return nil, nil, fmt.Errorf(
"%s stands on the image %q, which is not pinned by digest. A tag is what "+
"somebody else can move; name it as <image>@sha256:…", manifest.Module, base.Image)
}
reference, err := mirror(ctx, base.Image, manifest.Module+"/on-"+strings.ToLower(base.Arg))
if err != nil {
return nil, fmt.Errorf("%s stands on %s: %w", manifest.Module, base.Image, err)
return nil, nil, fmt.Errorf("%s stands on %s: %w", manifest.Module, base.Image, err)
}
args = append(args, "--build-arg", base.Arg+"="+reference)
resolved = append(resolved, reference)
continue
}
if base.Arg == "" || base.Module == "" || base.Artifact == "" {
return nil, fmt.Errorf(
return nil, nil, fmt.Errorf(
"%s says its build stands on something, and does not say all of what: a base "+
"needs the module, the artifact, and the build argument the recipe reads it "+
"from", manifest.Module)
@@ -745,14 +769,15 @@ func standingOn(ctx context.Context, manifest catalogue.Manifest, held map[strin
key := base.Module + "/" + base.Artifact
reference, has := held[key]
if !has {
return nil, fmt.Errorf(
return nil, nil, fmt.Errorf(
"%s builds on %s, and this mesh has not built it. Build %s first — every module "+
"in this toolchain stands on it, so it is the thing to have before anything "+
"else", manifest.Module, key, base.Module)
}
args = append(args, "--build-arg", base.Arg+"="+reference)
resolved = append(resolved, reference)
}
return args, nil
return args, resolved, nil
}
// compile runs a module's own code through its toolchain, and says where the result is.
@@ -997,3 +1022,31 @@ func instructions(recipe string) []string {
flush()
return out
}
// readBy is every repository other than the module's own that this build's recipes read source from,
// each once and in a fixed order, so two builds of one commit report the same thing the same way.
func readBy(manifest catalogue.Manifest) []catalogue.ArtifactContext {
if manifest.Build == nil {
return nil
}
seen := map[string]bool{}
var out []catalogue.ArtifactContext
for _, a := range manifest.Build.Artifacts {
if a.Context == nil || a.Context.Repository == "" {
continue
}
key := a.Context.Repository + "#" + a.Context.Ref
if seen[key] {
continue
}
seen[key] = true
out = append(out, *a.Context)
}
sort.Slice(out, func(i, j int) bool {
if out[i].Repository != out[j].Repository {
return out[i].Repository < out[j].Repository
}
return out[i].Ref < out[j].Ref
})
return out
}
+14
View File
@@ -180,6 +180,20 @@ func (r Registry) MirrorImage(ctx context.Context, from, repository string) (str
if err != nil {
return "", err
}
// **Already held is already mirrored.** A base is named by digest, and a digest this registry
// holds under the module's repository is the same bytes whatever upstream would say — so
// upstream is not asked. Asked every build, the public hub's anonymous pull limit was reached
// on the first merge that rebuilt a whole catalogue (2026-09-28), and every module whose base
// lives there failed on a copy it did not need.
if strings.HasPrefix(where.reference, "sha256:") {
held, err := r.has(ctx, "http://"+r.Address+"/v2/"+repository+"/manifests/"+where.reference, manifestAccept)
if err != nil {
return "", fmt.Errorf("asking %s whether it holds %s: %w", r.Address, from, err)
}
if held {
return r.Address + "/" + repository + "@" + where.reference, nil
}
}
src := &source{client: r.client()}
digest, err := r.copyManifest(ctx, src, where, where.reference, repository)
if err != nil {
+38
View File
@@ -103,6 +103,20 @@ func (m *theMeshsRegistry) handler() http.Handler {
m.mu.Lock()
defer m.mu.Unlock()
switch {
case r.Method == http.MethodHead && strings.Contains(r.URL.Path, "/manifests/"):
// **As strictly as a real registry.** A manifest is answered only in a media type the
// caller named; a request with no Accept is answered as if nothing were there. The fake
// used to answer regardless, which is why it could not catch a check that asked without
// one — and the mesh copied every base again (2026-09-28).
if !strings.Contains(r.Header.Get("Accept"), "manifest") && !strings.Contains(r.Header.Get("Accept"), "index") {
w.WriteHeader(http.StatusNotFound)
return
}
if _, ok := m.manifests[r.URL.Path[strings.LastIndex(r.URL.Path, "/")+1:]]; ok {
w.WriteHeader(http.StatusOK)
} else {
w.WriteHeader(http.StatusNotFound)
}
case r.Method == http.MethodHead && strings.Contains(r.URL.Path, "/blobs/"):
if _, ok := m.blobs[r.URL.Path[strings.LastIndex(r.URL.Path, "/")+1:]]; ok {
w.WriteHeader(http.StatusOK)
@@ -219,3 +233,27 @@ func TestATagBeforeTheDigestIsNotPartOfTheRepository(t *testing.T) {
t.Fatalf("got %+v", got)
}
}
// A base this registry already holds by digest is not asked of upstream at all: the public hub
// limits anonymous pulls, and a catalogue rebuilt on one merge asked it once per module.
func TestABaseAlreadyHeldIsNotAskedOfUpstream(t *testing.T) {
src, indexDigest, _ := anUpstreamRegistry(t)
dst := &theMeshsRegistry{blobs: map[string][]byte{}, manifests: map[string][]byte{}}
dstServer := httptest.NewServer(dst.handler())
defer dstServer.Close()
address := strings.TrimPrefix(dstServer.URL, "http://")
r := Registry{Address: address, HTTP: src.Client()}
host := strings.TrimPrefix(src.URL, "http://")
if _, err := r.MirrorImage(context.Background(), host+"/library/thing:latest", "hello-web/server"); err != nil {
t.Fatal(err)
}
// Upstream gone: the pinned base is answered from what the mesh holds.
src.Close()
reference, err := r.MirrorImage(context.Background(), host+"/library/thing@"+indexDigest, "hello-web/server")
if err != nil {
t.Fatalf("a base the registry holds was asked of an upstream that is gone: %v", err)
}
if reference != address+"/hello-web/server@"+indexDigest {
t.Fatalf("pinned as %q", reference)
}
}
+13 -1
View File
@@ -122,11 +122,23 @@ func (r Registry) PublishArchive(ctx context.Context, repository string, body []
return final, nil
}
func (r Registry) has(ctx context.Context, url string) (bool, error) {
// has is whether this registry already holds what is at that URL.
//
// **A manifest HEAD must say what it accepts.** A registry answers a manifest request only in a media
// type the caller named, and a bare HEAD — no Accept at all — is answered 404 for a manifest it holds
// perfectly well. Measured against the mesh's own registry (2026-09-28): the same digest answered 200
// with the manifest media types and 404 without them, so a check written without them concluded the
// registry held nothing, copied every base again, and exhausted the public hub's pull limit. A blob
// needs no Accept, which is why this went unnoticed: the same helper was right for blobs and wrong
// for manifests.
func (r Registry) has(ctx context.Context, url string, accept ...string) (bool, error) {
request, err := http.NewRequestWithContext(ctx, http.MethodHead, url, nil)
if err != nil {
return false, err
}
for _, media := range accept {
request.Header.Add("Accept", media)
}
response, err := r.client().Do(request)
if err != nil {
return false, fmt.Errorf("cannot reach the registry at %s: %w", r.Address, err)
+55 -6
View File
@@ -22,7 +22,7 @@ func TestABaseTheMeshHasNotBuiltIsRefused(t *testing.T) {
On: []catalogue.BuildsOn{{Arg: "RUNTIME_BASE", Module: "mesh-tools", Artifact: "runtime"}},
},
}
_, err := standingOn(context.Background(), manifest, map[string]string{}, noMirror)
_, _, err := standingOn(context.Background(), manifest, map[string]string{}, noMirror)
if err == nil {
t.Fatal("a base nothing has built was accepted; the build would have failed on its first line")
}
@@ -42,7 +42,7 @@ func TestABaseTheMeshHoldsBecomesABuildArgument(t *testing.T) {
},
}
held := map[string]string{"mesh-tools/runtime": "127.0.0.1:5000/mesh-tools/runtime@sha256:" + strings.Repeat("a", 64)}
args, err := standingOn(context.Background(), manifest, held, noMirror)
args, _, err := standingOn(context.Background(), manifest, held, noMirror)
if err != nil {
t.Fatalf("a base this mesh holds was refused: %v", err)
}
@@ -54,7 +54,7 @@ func TestABaseTheMeshHoldsBecomesABuildArgument(t *testing.T) {
// A module naming no base asks for nothing, which is most modules.
func TestAModuleNamingNoBaseAddsNoArguments(t *testing.T) {
args, err := standingOn(context.Background(), catalogue.Manifest{Module: "hello-web", Build: &catalogue.Build{}}, nil, noMirror)
args, _, err := standingOn(context.Background(), catalogue.Manifest{Module: "hello-web", Build: &catalogue.Build{}}, nil, noMirror)
if err != nil || args != nil {
t.Fatalf("a module naming no base produced %v, %v", args, err)
}
@@ -66,7 +66,7 @@ func TestAnIncompleteBaseIsRefused(t *testing.T) {
Module: "postgres",
Build: &catalogue.Build{On: []catalogue.BuildsOn{{Module: "mesh-tools", Artifact: "runtime"}}},
}
if _, err := standingOn(context.Background(), manifest, map[string]string{"mesh-tools/runtime": "x"}, noMirror); err == nil {
if _, _, err := standingOn(context.Background(), manifest, map[string]string{"mesh-tools/runtime": "x"}, noMirror); err == nil {
t.Fatal("a base with no build argument was accepted; nothing would have read it")
}
}
@@ -86,7 +86,7 @@ func TestADeclaredVendorImageIsCopiedInAndHandedToTheRecipe(t *testing.T) {
},
}
var asked []string
args, err := standingOn(context.Background(), manifest, nil, func(_ context.Context, from, repository string) (string, error) {
args, _, err := standingOn(context.Background(), manifest, nil, func(_ context.Context, from, repository string) (string, error) {
asked = append(asked, from+" -> "+repository)
return "127.0.0.1:5000/" + repository + "@sha256:" + strings.Repeat("d", 64), nil
})
@@ -101,7 +101,7 @@ func TestADeclaredVendorImageIsCopiedInAndHandedToTheRecipe(t *testing.T) {
}
// Unpinned, it is refused: a tag is what somebody else can move.
manifest.Build.On[0].Image = "quay.io/minio/mc:latest"
if _, err := standingOn(context.Background(), manifest, nil, noMirror); err == nil || !strings.Contains(err.Error(), "not pinned") {
if _, _, err := standingOn(context.Background(), manifest, nil, noMirror); err == nil || !strings.Contains(err.Error(), "not pinned") {
t.Fatalf("an unpinned vendor image was accepted: %v", err)
}
}
@@ -151,3 +151,52 @@ func TestARecipeIsReadAsInstructions(t *testing.T) {
t.Fatalf("a heredoc line or a continued stage was read as a base: %v", bases)
}
}
// What a build was handed as its bases is what it stood on — recorded, so a changed base knows what
// to rebuild (novox/hq 04-ISSUES/131). A recipe reads the base from an argument, so nothing else
// could know.
func TestTheBasesABuildWasHandedAreWhatItStoodOn(t *testing.T) {
manifest := catalogue.Manifest{
Module: "gitea",
Build: &catalogue.Build{
On: []catalogue.BuildsOn{
{Arg: "RUNTIME_BASE", Module: "mesh-tools", Artifact: "runtime"},
{Arg: "BUILD_BASE", Module: "mesh-tools", Artifact: "build"},
},
Artifacts: []catalogue.Artifact{{Name: "runtime", Kind: catalogue.ArtifactImage, From: "Dockerfile"}},
},
}
held := map[string]string{
"mesh-tools/runtime": "127.0.0.1:5000/mesh-tools/runtime@sha256:" + strings.Repeat("a", 64),
"mesh-tools/build": "127.0.0.1:5000/mesh-tools/build@sha256:" + strings.Repeat("b", 64),
}
_, resolved, err := standingOn(context.Background(), manifest, held, noMirror)
if err != nil {
t.Fatal(err)
}
got := against(t.TempDir(), manifest, resolved)
if len(got) != 2 || got[0] != held["mesh-tools/build"] || got[1] != held["mesh-tools/runtime"] {
t.Fatalf("the bases the build was handed were not what it stood on: %v", got)
}
}
// What a build read besides its module's own repository is the second repository its recipes name,
// each once: a module that packages source living elsewhere is affected when that source moves.
func TestWhatABuildReadIsTheRepositoriesItsRecipesName(t *testing.T) {
elsewhere := catalogue.ArtifactContext{Repository: "http://forge.internal:20000/novox/mesh-controller.git", Ref: "main"}
manifest := catalogue.Manifest{
Module: "builder",
Build: &catalogue.Build{Artifacts: []catalogue.Artifact{
{Name: "server", Kind: catalogue.ArtifactImage, From: "Dockerfile", Context: &elsewhere},
{Name: "tools", Kind: catalogue.ArtifactImage, From: "Dockerfile", Context: &elsewhere},
{Name: "config", Kind: catalogue.ArtifactArchive, From: "etc"},
}},
}
read := readBy(manifest)
if len(read) != 1 || read[0] != elsewhere {
t.Fatalf("the repositories this build read are %+v", read)
}
if readBy(catalogue.Manifest{Module: "gitea", Build: &catalogue.Build{}}) != nil {
t.Fatal("a module whose recipes name no other repository read one")
}
}
+88
View File
@@ -623,6 +623,9 @@ func (r Resolution) compose(with Rendering, owner map[string]string) ([]map[stri
// one of them as its environment without saying so (ADR 0086, issue 041).
secretFiles := secretFilesOf(resources)
// Which of this module's resources its preparation runs before, if it prepares anything.
prepareBefore := preparationTarget(m)
for _, unsettled := range resources {
resource, err := ApplySettings(unsettled, with.Settings[m.Module])
if err != nil {
@@ -708,6 +711,18 @@ func (r Resolution) compose(with Rendering, owner map[string]string) ([]map[stri
if renamed := reflectsRenamed(m.Module, resource["reload-on"]); renamed != nil {
copied["reload-on"] = renamed
}
// **A version prepares its state before it runs** (novox/hq ADR 0135). Derived from the
// module's own resource rather than declared beside it: what prepares the state is the
// module's own code, so what it is given has to be what that code is given — and a
// second resource written by hand is a second copy to drift from the first. Placed
// immediately before it, because a run-once step stops everything the declaration
// places after it (ADR 0052), which is how a version whose preparation failed does not
// serve.
if prepareBefore != "" && fmt.Sprint(resource["id"]) == prepareBefore {
step := prepared(copied)
owner[fmt.Sprint(step["id"])] = m.Module
out = append(out, step)
}
owner[fmt.Sprint(copied["id"])] = m.Module
out = append(out, copied)
}
@@ -1610,3 +1625,76 @@ func atMachinePort(serves map[string]any, module string, ports map[string]map[in
func AtPublishedPort(values map[string]any, module string, published map[int]int) map[string]any {
return atMachinePort(values, module, map[string]map[int]int{module: published})
}
// PreparationArgument is how the mesh asks a module to prepare its state: one word, to the module's
// own program, whatever that program is (novox/hq ADR 0135).
//
// **One word for every kind of module.** A module built as a Go binary receives it as its argument;
// one built as a bundle receives it through the runtime, whose entry takes the same word. So the
// mesh has one way of asking and a module has one way of answering, and neither learns the other's
// shape.
const PreparationArgument = "prepare"
// preparationTarget is the resource a module's preparation runs before: its own workload.
//
// The first container carrying an artifact this module built, and not itself a step — that is the
// thing that runs the module's code, and therefore the thing whose state must be ready. Empty when
// the module prepares nothing, or when nothing it declares could run its code.
//
// **A module with two own workloads gates the first of them.** Five modules in the catalogue declare
// more than one container of their own, none of them preparing anything today. If one ever does and
// its second workload shares the state, the gate is in front of the first — stated here because the
// alternative is a field asking an author to restate what the mesh can see.
func preparationTarget(m Manifest) string {
if !m.Prepares {
return ""
}
for _, r := range m.Resources {
if fmt.Sprint(r["type"]) != "container" || !ownArtifact(r, m.Module) {
continue
}
if once, _ := r["run-once"].(bool); once {
continue
}
return fmt.Sprint(r["id"])
}
return ""
}
// ownArtifact is whether a resource runs something this module built, in either spelling a manifest
// may be in: naming the artifact, before a build resolved it, or carrying the reference a build
// recorded — this mesh's own store, under this module's name.
func ownArtifact(resource map[string]any, module string) bool {
if named, _ := resource["artifact"].(string); named != "" {
return true
}
image, _ := resource["image"].(string)
return strings.HasPrefix(image, ArtifactStoreScheme+module+"/")
}
// prepared is the module's own resource as the step that prepares its state: the same image, the same
// context, run to completion with the mesh's preparation argument.
//
// Three things are taken away rather than copied, each because the step runs while the version it
// prepares for is still running. A published port cannot be bound twice, and a step that tried would
// fail for a reason that has nothing to do with the state. A fixed address cannot be held twice, for
// the same reason. And a cadence is what a step is the opposite of: a container runs once and gates,
// or on a schedule, or stays up, never two (ADR 0053).
func prepared(from map[string]any) map[string]any {
step := map[string]any{}
for k, v := range from {
step[k] = v
}
// **A hyphen, not a dot.** A resource's id is `<module>.<its own id>`, and a module's name may
// itself contain a dot (`novox.be`), so the module is everything before the *last* dot — which
// only works if what the mesh derives adds no dot of its own.
step["id"] = fmt.Sprint(from["id"]) + "-prepare"
step["name"] = fmt.Sprint(from["name"]) + "-prepare"
step["run-once"] = true
step["args"] = []any{PreparationArgument}
delete(step, "ports")
delete(step, "ip")
delete(step, "schedule")
delete(step, "reload-on")
return step
}
+15
View File
@@ -143,3 +143,18 @@ func TestAMembershipForTheNewBusIsComposedAsASealedFile(t *testing.T) {
}
}
}
// The store's seat rows have no protocol columns yet; loading them must not drop the protocol the
// bus is derived from, or no role's work queue is ever raised (found live, 2026-09-28).
func TestAStoreRowWithoutAProtocolKeepsTheCompiledOne(t *testing.T) {
was := Seats()
t.Cleanup(func() { UseSeats(was) })
UseSeats([]Seat{{Name: "mesh-build-machine", Scope: ScopeMesh, Decision: "row"}})
got, ok := SeatNamed("mesh-build-machine")
if !ok || len(got.Accepts) == 0 {
t.Fatalf("the build machine's seat lost what it accepts when loaded from the store: %+v", got)
}
if got.Decision != "row" {
t.Fatalf("the store's own columns were not kept: %+v", got)
}
}
+24
View File
@@ -225,6 +225,19 @@ type Manifest struct {
// subscription to the queue it writes to (design 29 §2).
Uses []string `json:"uses,omitempty"`
// Prepares says this module has state that must be brought to the shape this version needs
// before this version runs, and that the module's own code does it (novox/hq ADR 0135).
//
// **A word, not an arrangement.** The mesh runs the module's own program in its preparation
// mode, in the module's own context — every binding, credential and setting its code receives,
// because it *is* its code. Nothing here names a container, a command, a mount or a variable:
// the module already said all of that once, and a second copy is a second thing to drift.
//
// **Declared, never inferred.** The control plane cannot read what is inside an artifact, so a
// module that ships a migration and does not say this breaks on its first upgrade. That is
// stated in the record rather than guarded here, because nothing mechanical can guard it.
Prepares bool `json:"prepares,omitempty"`
// Tools are the tools this module answers — request and reply, awaited.
//
// **New, and not `serves`**, which this manifest already uses for the facts a consumer needs
@@ -1276,6 +1289,17 @@ func ParseManifest(raw []byte) (Manifest, error) {
"program that reads what the mesh delivered and reconciles",
m.Module, r["id"]))
}
// **A module that prepares its state must have code the mesh can run** (novox/hq ADR 0135). The
// preparation is the module's own program in its preparation mode, so it is derived from the
// resource that runs that program — and a module declaring none has asked for something the mesh
// cannot compose. Said here, where the manifest is read, rather than by a declaration that
// quietly prepares nothing.
if m.Prepares && preparationTarget(m) == "" {
problems = append(problems, fmt.Sprintf(
"%s says it prepares its state, and declares no container running an artifact it built — "+
"the preparation is this module's own program, so there has to be one for the mesh to "+
"run it in", m.Module))
}
// **A run-once container is a step the host runs to completion** (novox/hq ADR 0052). It is a
// boolean modifier on the container shape — the host runs the container, requires it to exit 0,
// and starts whatever the declaration places after it only once it has. A value that is not a
+141
View File
@@ -0,0 +1,141 @@
package catalogue
import (
"encoding/json"
"fmt"
"strings"
"testing"
)
// A module version prepares its state before it runs (novox/hq ADR 0135).
//
// What the mesh derives is the module's own resource, run once with one word, placed immediately in
// front of the thing it prepares for. What matters in these tests is that the derivation is a copy
// rather than a second description: the failure it replaces was a hand-written step repeating six
// fields of the resource it preceded, each free to drift from it.
func aPreparingModule() Manifest {
return Manifest{
Module: "gitea",
Prepares: true,
Resources: []map[string]any{
{"id": "state", "type": "directory", "path": "/var/lib/gitea", "mode": "0700"},
{"id": "server", "type": "container", "name": "mesh-gitea-server",
"image": "gitea/gitea@" + digest, "ports": []any{"3000:3000"}},
{"id": "runtime", "type": "container", "name": "mesh-gitea",
"image": ArtifactStoreScheme + "gitea/runtime@" + digest, "network": "host",
"env": map[string]any{"MESH_GITEA_STATE_DIR": "/run/state"},
"volumes": []any{"/var/lib/gitea:/run/state:ro"},
"ports": []any{"9000:9000"}},
},
}
}
func declaredFor(t *testing.T, m Manifest) []map[string]any {
t.Helper()
// The manifest as the mesh holds it: a build resolved the module's own artifact into the
// reference it recorded, which is also how the composition knows whose code a resource runs.
out, err := Resolution{Node: "anchor", Modules: []Manifest{m}}.Declaration(
Rendering{ArtifactStore: "anchor.internal:5100"})
if err != nil {
t.Fatal(err)
}
return out
}
func idsOf(resources []map[string]any) []string {
var ids []string
for _, r := range resources {
ids = append(ids, fmt.Sprint(r["id"]))
}
return ids
}
// The step runs the module's own code, and comes immediately before it — not before the upstream
// server the module packages, which may be the very thing the state lives in.
func TestThePreparationRunsTheModulesOwnCodeAndComesRightBeforeIt(t *testing.T) {
out := declaredFor(t, aPreparingModule())
ids := idsOf(out)
at := -1
for i, id := range ids {
if id == "gitea.runtime-prepare" {
at = i
}
}
if at < 0 {
t.Fatalf("nothing prepares this module's state: %v", ids)
}
// A module's name may contain a dot, so a resource's module is everything before the last one —
// which the derived id must not add to, or a machine reads the wrong owner from it.
if strings.Count("gitea.runtime-prepare", ".") != 1 {
t.Fatal("the derived id adds a dot, so what owns it cannot be read from it")
}
if ids[at+1] != "gitea.runtime" {
t.Fatalf("the preparation is not immediately before the module's own code: %v", ids)
}
for _, id := range ids[:at] {
if id == "gitea.runtime" {
t.Fatalf("the module's own code runs before its state is prepared: %v", ids)
}
}
}
// It is given exactly what the module's own code is given. Asserted field by field against the
// resource it was derived from, because writing it twice is the fault this replaces.
func TestThePreparationIsGivenWhatTheModuleIsGiven(t *testing.T) {
out := declaredFor(t, aPreparingModule())
declared := byID(out)
step, workload := declared["gitea.runtime-prepare"], declared["gitea.runtime"]
if step == nil || workload == nil {
t.Fatalf("expected both, got %v", idsOf(out))
}
for _, field := range []string{"image", "network", "env", "volumes", "type"} {
if fmt.Sprint(step[field]) != fmt.Sprint(workload[field]) {
t.Errorf("the preparation's %s is %v and the module's is %v", field, step[field], workload[field])
}
}
if once, _ := step["run-once"].(bool); !once {
t.Error("the preparation is not a step, so nothing waits for it and nothing is gated by it")
}
if fmt.Sprint(step["args"]) != fmt.Sprint([]any{PreparationArgument}) {
t.Errorf("the preparation is asked for as %v", step["args"])
}
if fmt.Sprint(step["name"]) == fmt.Sprint(workload["name"]) {
t.Error("the preparation and the workload have one name, so one removes the other")
}
// A published port cannot be bound twice, and the version being replaced is still running.
if _, published := step["ports"]; published {
t.Errorf("the preparation publishes a port the running version holds: %v", step["ports"])
}
}
// A module that says nothing about preparing gets nothing, which is most modules.
func TestAModuleThatPreparesNothingGetsNoStep(t *testing.T) {
m := aPreparingModule()
m.Prepares = false
for _, id := range idsOf(declaredFor(t, m)) {
if id == "gitea.runtime-prepare" {
t.Fatal("a module that prepares nothing was given a preparation")
}
}
}
// A module whose own code the mesh cannot find has nothing to ask, and saying so where the manifest
// is read beats a declaration that quietly prepares nothing.
func TestAModuleThatPreparesAndRunsNoneOfItsOwnCodeIsRefused(t *testing.T) {
m := Manifest{
Module: "gitea",
Prepares: true,
Resources: []map[string]any{
// Only the upstream server it packages: nothing here runs gitea's own code.
{"id": "server", "type": "container", "name": "mesh-gitea-server", "image": "gitea/gitea@" + digest},
},
}
raw, err := json.Marshal(m)
if err != nil {
t.Fatal(err)
}
if _, err := ParseManifest(raw); err == nil {
t.Fatal("a module that prepares its state with nothing of its own to run was accepted")
}
}
+1 -5
View File
@@ -13,7 +13,6 @@ func TestASeatPlaceholderAnswersWhereThisMachinePutTheHolder(t *testing.T) {
"type": "container", "id": "server", "name": "mesh-controller",
"env": map[string]any{
"MESH_STORE_INVENTORY_PORT": "${seat:mesh-store:5432}",
"MESH_BROKER_AMQP_PORT": "${seat:mesh-broker:5672}",
"MESH_BROKER_ADDRESS_PORT": "${seat:mesh-broker:5671}",
"MESH_STORE_INVENTORY_FILE": "/run/secrets/inventory",
},
@@ -27,7 +26,6 @@ func TestASeatPlaceholderAnswersWhereThisMachinePutTheHolder(t *testing.T) {
env := control["env"].(map[string]any)
for key, want := range map[string]string{
"MESH_STORE_INVENTORY_PORT": "6852",
"MESH_BROKER_AMQP_PORT": "5679",
"MESH_BROKER_ADDRESS_PORT": "5671",
"MESH_STORE_INVENTORY_FILE": "/run/secrets/inventory",
} {
@@ -146,7 +144,6 @@ func TestTheControlPlanesOwnAddressesFollowTheNodesPorts(t *testing.T) {
"MESH_STORE_INVENTORY_PORT": "6852",
"MESH_STORE_IDENTITY_PORT": "6852",
"MESH_STORE_LICENCES_PORT": "6852",
"MESH_BROKER_AMQP_PORT": "5679",
"MESH_BROKER_MANAGEMENT_PORT": "15673",
"MESH_BROKER_ADDRESS_PORT": "5671",
} {
@@ -164,7 +161,7 @@ func TestTheControlPlanesOwnAddressesFollowTheNodesPorts(t *testing.T) {
t.Fatal(err)
}
env, _ = fileNamed(out, "mesh-controller.server")["env"].(map[string]any)
if env["MESH_STORE_INVENTORY_PORT"] != "" || env["MESH_BROKER_AMQP_PORT"] != "" {
if env["MESH_STORE_INVENTORY_PORT"] != "" {
t.Errorf("with no settings, the control plane is told %v", env)
}
}
@@ -175,7 +172,6 @@ var SeatPorts = map[string]string{
"MESH_STORE_INVENTORY_PORT": "${seat:mesh-store:5432}",
"MESH_STORE_IDENTITY_PORT": "${seat:mesh-store:5432}",
"MESH_STORE_LICENCES_PORT": "${seat:mesh-store:5432}",
"MESH_BROKER_AMQP_PORT": "${seat:mesh-broker:5672}",
"MESH_BROKER_MANAGEMENT_PORT": "${seat:mesh-broker:15672}",
"MESH_BROKER_ADDRESS_PORT": "${seat:mesh-broker:5671}",
}
+28 -3
View File
@@ -48,7 +48,11 @@ type Seat struct {
//
// In the order a person reads it: the mesh's own, then a node's.
var defaultSeats = []Seat{
{Name: "mesh-controller", Scope: ScopeMesh, Decision: "novox/hq ADR 0079"},
// The control plane states what it did under the seat it holds (novox/hq ADR 0134): a role's
// events belong to the role, so they keep their address while the holder is replaced. No accepts,
// so no work queue is raised for it — only what its holder may say.
{Name: "mesh-controller", Scope: ScopeMesh, Decision: "novox/hq ADR 0079",
Emits: []string{"applied", "refused", "built-before"}},
{Name: "mesh-store", Scope: ScopeMesh, Delivers: "postgres-database", Decision: "novox/hq ADR 0079"},
// **Delivers the mesh's own bus, not `amqp`.** Those were the same word until
// ADR 0127 separated them: `amqp` is a backing service a module may require, and this seat is
@@ -109,9 +113,30 @@ func DefaultSeats() []Seat { return append([]Seat(nil), defaultSeats...) }
// than running on the set the binary shipped with. So the store can only ever *replace* the set with
// a non-empty one, never erase it.
func UseSeats(s []Seat) {
if len(s) > 0 {
seats = s
if len(s) == 0 {
return
}
// **The store's rows carry no protocol yet, and the protocol is what the bus is derived
// from.** ADR 0129 gives a seat what it accepts, emits and serves; ADR 0122 moved the set into
// a table that has name, scope, delivers and decision and nothing else, and the columns for
// the rest are not there yet. So a row replacing a compiled entry would silently drop the
// protocol, and the roles' work queues would never be raised — found live as "no response
// from stream" the first time a build was submitted over the new bus (2026-09-28). Until the
// table gains the columns, a row without a protocol keeps the compiled one of the same name.
byName := map[string]Seat{}
for _, d := range defaultSeats {
byName[d.Name] = d
}
merged := make([]Seat, 0, len(s))
for _, row := range s {
if len(row.Accepts)+len(row.Emits)+len(row.Serves) == 0 {
if d, known := byName[row.Name]; known {
row.Accepts, row.Emits, row.Serves = d.Accepts, d.Emits, d.Serves
}
}
merged = append(merged, row)
}
seats = merged
}
// aliases maps a seat's former names to its current canonical name (novox/hq ADR 0122). Loaded from
+89 -3
View File
@@ -36,12 +36,21 @@ type Build struct {
// Against is every artifact this build stood on, as references rather than module names —
// what makes a build edge derived rather than declared (ADR 0009).
Against []string
// Read is every repository this build read source from besides the module's own (novox/hq
// 04-ISSUES/131), at the ref it read.
Read []ReadRepository
// Failed is the builder's own words, empty when it worked.
Failed string
Made []Artifact
At time.Time
}
// ReadRepository is a repository a build read source from besides the module's own.
type ReadRepository struct {
Repository string `json:"repository"`
Ref string `json:"ref,omitempty"`
}
// Artifact is one thing a build published.
type Artifact struct {
Name string `json:"name"`
@@ -66,17 +75,21 @@ func (i *Inventory) RecordBuild(ctx context.Context, b Build) error {
if err != nil {
return err
}
read, err := json.Marshal(b.Read)
if err != nil {
return err
}
var module *string
if b.Module != "" {
module = &b.Module
}
_, err = i.store.Pool().Exec(ctx,
`insert into build (id, repository, ref, module, commit_hash, built_on, failed, made,
source_path, manifest, built_against)
values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11)
source_path, manifest, built_against, built_contexts)
values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12)
on conflict (id) do nothing`,
b.ID, b.Repository, b.Ref, module, b.Commit, b.On, b.Failed, made,
b.Path, manifestOrNil(b.Manifest), against)
b.Path, manifestOrNil(b.Manifest), against, read)
return err
}
@@ -164,6 +177,79 @@ func (i *Inventory) Held(ctx context.Context) (map[string]string, error) {
return held, rows.Err()
}
// BuiltAgainst is what each module's newest successful build stood on, as recorded — the build
// edges (ADR 0009). A module whose last build recorded no bases is absent, which is also what a
// module standing on nothing looks like: an edge the mesh has not derived is not an edge.
func (i *Inventory) BuiltAgainst(ctx context.Context) (map[string][]string, error) {
rows, err := i.store.Pool().Query(ctx,
`select distinct on (module) module, built_against
from build
where module is not null and module <> '' and failed = ''
order by module, at desc`)
if err != nil {
return nil, err
}
defer rows.Close()
against := map[string][]string{}
for rows.Next() {
var module string
var raw []byte
if err := rows.Scan(&module, &raw); err != nil {
return nil, err
}
if len(raw) == 0 {
continue
}
var refs []string
if err := json.Unmarshal(raw, &refs); err != nil {
continue
}
if len(refs) > 0 {
against[module] = refs
}
}
return against, rows.Err()
}
// ReadRepositories is what each module's newest successful build read source from besides its own
// repository, by module name.
//
// The mirror of BuiltAgainst, and derived the same way and for the same reason: a merge into a
// repository a module only packages is a change to that module, and the manifest the mesh keeps
// carries nothing that would say so (novox/hq 04-ISSUES/131).
func (i *Inventory) ReadRepositories(ctx context.Context) (map[string][]ReadRepository, error) {
rows, err := i.store.Pool().Query(ctx,
`select distinct on (module) module, built_contexts
from build
where module is not null and module <> '' and failed = ''
order by module, at desc`)
if err != nil {
return nil, err
}
defer rows.Close()
read := map[string][]ReadRepository{}
for rows.Next() {
var module string
var raw []byte
if err := rows.Scan(&module, &raw); err != nil {
return nil, err
}
if len(raw) == 0 {
continue
}
var of []ReadRepository
if err := json.Unmarshal(raw, &of); err != nil {
continue
}
if len(of) > 0 {
read[module] = of
}
}
return read, rows.Err()
}
// manifestOrNil keeps the difference between "declared nothing" and "predates this being kept".
//
// A build recorded before the mesh kept manifests has no manifest, and that is not the same as one
+33
View File
@@ -136,3 +136,36 @@ func ids(builds []Build) []string {
}
return out
}
// What a build read besides its module's own repository comes back for the newest build of each
// module, and only for builds that worked. Nothing recorded is absent rather than empty, which is how
// a build made before the mesh kept this is told from one that read nothing (novox/hq 04-ISSUES/131).
func TestWhatABuildReadComesBackForTheNewestBuildOfEachModule(t *testing.T) {
inv := fresh(t)
ctx := context.Background()
older := aBuild("older", "builder", "")
older.Read = []ReadRepository{{Repository: "novox/mesh-controller", Ref: "release"}}
newer := aBuild("newer", "builder", "")
newer.Read = []ReadRepository{{Repository: "novox/mesh-controller", Ref: "main"}}
plain := aBuild("plain", "gitea", "")
failed := aBuild("failed", "route-proxy", "cannot clone")
failed.Read = []ReadRepository{{Repository: "novox/mesh-controller", Ref: "main"}}
for _, b := range []Build{older, newer, plain, failed} {
if err := inv.RecordBuild(ctx, b); err != nil {
t.Fatal(err)
}
}
read, err := inv.ReadRepositories(ctx)
if err != nil {
t.Fatal(err)
}
if len(read["builder"]) != 1 || read["builder"][0].Ref != "main" {
t.Fatalf("the newest build's reading is %+v", read["builder"])
}
if _, has := read["gitea"]; has {
t.Fatalf("a build that read nothing but its own repository reads as %+v", read["gitea"])
}
if _, has := read["route-proxy"]; has {
t.Fatal("a failed build's reading was kept as what that module reads")
}
}
+3 -1
View File
@@ -86,9 +86,11 @@ func TestAnAssignedModuleBecomesAUserWithWhatItDeclared(t *testing.T) {
if err != nil {
t.Fatal(err)
}
// Its tools are every one under its own name — the list in the manifest is a person's
// vocabulary for asking, not the module's permission to answer.
if !granted(perms.Publish, "mesh.mod.shop.event.order.placed") ||
!granted(perms.Publish, "mesh.seat.telegram-sender.accept.send") ||
!granted(perms.Subscribe, "mesh.mod.shop.tool.price") {
!granted(perms.Subscribe, "mesh.mod.shop.tool.>") {
t.Fatalf("one.shop's authority is not what it declared: %+v", perms)
}
}
+10 -2
View File
@@ -8,6 +8,7 @@ import (
"sort"
"strconv"
"strings"
"time"
"github.com/jackc/pgx/v5"
"github.com/novox/mesh-controller/internal/catalogue"
@@ -38,6 +39,9 @@ type Source struct {
BuiltFrom string
// Head is the newest commit the source is known to have.
Head string
// Seen is when the source was last looked at — by a build, by hand, or by the forge saying it
// moved. What a late report of an older move is judged against.
Seen time.Time
}
// Current reports whether what the mesh holds is what the source last had.
@@ -936,12 +940,13 @@ func (i *Inventory) Catalogued(ctx context.Context) ([]Entry, error) {
`select m.name, m.manifest,
coalesce(m.source, ''), m.source_path, m.source_seat, coalesce(m.ref, ''),
coalesce(m.built_from, ''), coalesce(m.source_head, ''),
coalesce(m.source_seen, to_timestamp(0)),
coalesce(array_agg(n.name order by n.name) filter (where n.name is not null), '{}')
from module m
left join assignment a on a.module = m.name
left join node n on n.id = a.node
group by m.name, m.manifest, m.source, m.source_path, m.source_seat, m.ref, m.built_from,
m.source_head
m.source_head, m.source_seen
order by m.name`)
if err != nil {
return nil, err
@@ -955,9 +960,12 @@ func (i *Inventory) Catalogued(ctx context.Context) ([]Entry, error) {
var source Source
var on []string
if err := rows.Scan(&name, &raw, &source.Repository, &source.Path, &source.Seat, &source.Ref,
&source.BuiltFrom, &source.Head, &on); err != nil {
&source.BuiltFrom, &source.Head, &source.Seen, &on); err != nil {
return nil, err
}
if source.Seen.Unix() == 0 {
source.Seen = time.Time{}
}
var m catalogue.Manifest
if err := json.Unmarshal(raw, &m); err != nil {
return nil, err
+12 -12
View File
@@ -30,12 +30,12 @@ func TestRefusedAndFailedAreDifferentSituations(t *testing.T) {
refuser := nodeNamed(t, inv, "refuser")
failer := nodeNamed(t, inv, "failer")
if err := inv.RecordDoing(ctx, refuser, Doing{
if _, err := inv.RecordDoing(ctx, refuser, Doing{
Outcome: OutcomeRefused, Refused: "resource \"x\": a file needs a path",
}); err != nil {
t.Fatal(err)
}
if err := inv.RecordDoing(ctx, failer, Doing{
if _, err := inv.RecordDoing(ctx, failer, Doing{
Outcome: OutcomeFailed,
Failed: []FailedResource{{ID: "svc", Error: "unit not found"}},
Applied: 4,
@@ -70,7 +70,7 @@ func TestAMachineDoingWhatItWasToldIsNotOnTheList(t *testing.T) {
inv := fresh(t)
ctx := context.Background()
id := nodeNamed(t, inv, "fine")
if err := inv.RecordDoing(ctx, id, Doing{Outcome: OutcomeApplied, Applied: 6}); err != nil {
if _, err := inv.RecordDoing(ctx, id, Doing{Outcome: OutcomeApplied, Applied: 6}); err != nil {
t.Fatal(err)
}
wrong, err := inv.NotDoingWhatTheyWereTold(ctx)
@@ -97,12 +97,12 @@ func TestTheLastReportReplacesTheOneBefore(t *testing.T) {
inv := fresh(t)
ctx := context.Background()
id := nodeNamed(t, inv, "recovered")
if err := inv.RecordDoing(ctx, id, Doing{
if _, err := inv.RecordDoing(ctx, id, Doing{
Outcome: OutcomeFailed, Failed: []FailedResource{{ID: "a", Error: "no"}},
}); err != nil {
t.Fatal(err)
}
if err := inv.RecordDoing(ctx, id, Doing{Outcome: OutcomeApplied, Applied: 3}); err != nil {
if _, err := inv.RecordDoing(ctx, id, Doing{Outcome: OutcomeApplied, Applied: 3}); err != nil {
t.Fatal(err)
}
wrong, err := inv.NotDoingWhatTheyWereTold(ctx)
@@ -141,7 +141,7 @@ func TestWhatANodeSaidGoesWhenTheNodeDoes(t *testing.T) {
inv := fresh(t)
ctx := context.Background()
id := nodeNamed(t, inv, "leaving")
if err := inv.RecordDoing(ctx, id, Doing{Outcome: OutcomeFailed}); err != nil {
if _, err := inv.RecordDoing(ctx, id, Doing{Outcome: OutcomeFailed}); err != nil {
t.Fatal(err)
}
if _, err := inv.store.Pool().Exec(ctx, `delete from node where name = 'leaving'`); err != nil {
@@ -257,7 +257,7 @@ func TestTheSameFailureReportedAgainIsCountedNotRestarted(t *testing.T) {
id := nodeNamed(t, inv, "looping")
same := Doing{Outcome: OutcomeFailed, Failed: []FailedResource{{ID: "img", Error: "no such image"}}}
if err := inv.RecordDoing(ctx, id, same); err != nil {
if _, err := inv.RecordDoing(ctx, id, same); err != nil {
t.Fatal(err)
}
first, _, err := inv.DoingOf(ctx, "looping")
@@ -269,7 +269,7 @@ func TestTheSameFailureReportedAgainIsCountedNotRestarted(t *testing.T) {
}
for range StuckAfter - 1 {
if err := inv.RecordDoing(ctx, id, same); err != nil {
if _, err := inv.RecordDoing(ctx, id, same); err != nil {
t.Fatal(err)
}
}
@@ -287,7 +287,7 @@ func TestTheSameFailureReportedAgainIsCountedNotRestarted(t *testing.T) {
// The same resource failing with different words — a duration, a counter — is still the same
// failure: it is the resource that loops, not the sentence.
reworded := Doing{Outcome: OutcomeFailed, Failed: []FailedResource{{ID: "img", Error: "no such image (after 31s)"}}}
if err := inv.RecordDoing(ctx, id, reworded); err != nil {
if _, err := inv.RecordDoing(ctx, id, reworded); err != nil {
t.Fatal(err)
}
still, _, err := inv.DoingOf(ctx, "looping")
@@ -300,7 +300,7 @@ func TestTheSameFailureReportedAgainIsCountedNotRestarted(t *testing.T) {
// A different failure is a new situation, not a longer one.
other := Doing{Outcome: OutcomeFailed, Failed: []FailedResource{{ID: "svc", Error: "unit not found"}}}
if err := inv.RecordDoing(ctx, id, other); err != nil {
if _, err := inv.RecordDoing(ctx, id, other); err != nil {
t.Fatal(err)
}
changed, _, err := inv.DoingOf(ctx, "looping")
@@ -312,7 +312,7 @@ func TestTheSameFailureReportedAgainIsCountedNotRestarted(t *testing.T) {
}
// And a clean apply clears it: the machine is doing what it was told, since nothing.
if err := inv.RecordDoing(ctx, id, Doing{Outcome: OutcomeApplied, Applied: 2}); err != nil {
if _, err := inv.RecordDoing(ctx, id, Doing{Outcome: OutcomeApplied, Applied: 2}); err != nil {
t.Fatal(err)
}
fine, _, err := inv.DoingOf(ctx, "looping")
@@ -324,7 +324,7 @@ func TestTheSameFailureReportedAgainIsCountedNotRestarted(t *testing.T) {
}
// The list of what is wrong carries the count, so `status` can say it.
if err := inv.RecordDoing(ctx, id, same); err != nil {
if _, err := inv.RecordDoing(ctx, id, same); err != nil {
t.Fatal(err)
}
wrong, err := inv.NotDoingWhatTheyWereTold(ctx)
@@ -0,0 +1,13 @@
-- A build says which repositories it read, so a merge can find everything it affects.
--
-- A module is built from the one repository the mesh records — where its module.json lives — and some
-- modules' recipes reach into a second for the source they package: the packaging and the source are
-- allowed to live apart (catalogue's ArtifactContext). That second repository is named in the manifest
-- the build read, and the manifest the mesh *keeps* carries no build section, so nothing on the mesh
-- could say that a merge into the other repository is a change to this module at all. Two modules are
-- built from the control plane's own repository, and neither had ever been rebuilt when it moved
-- (novox/hq 04-ISSUES/131).
--
-- Nullable, like the two derived columns beside it: null is a build recorded before the mesh kept
-- this, which is not the same as a build that read nothing but its module's own repository.
alter table build add column built_contexts jsonb;
+29 -18
View File
@@ -670,31 +670,39 @@ func sameFailure(a, b Doing) bool {
// a clean apply clears both (novox/hq 04-ISSUES/065). The previous row is read first and the
// comparison made here, so "the same" is a rule this package states rather than a jsonb equality
// that would restart the count on a changed word in an error.
func (i *Inventory) RecordDoing(ctx context.Context, node string, d Doing) error {
// **And whether this report was news**, which is what makes a fact about it worth stating (novox/hq
// ADR 0134). A machine reconciles continuously and reports each time; the same outcome about the same
// declaration is the same state said again, and a fact per report would be a fact per minute per
// machine that tells nobody anything. Read here because the previous row is read here anyway.
func (i *Inventory) RecordDoing(ctx context.Context, node string, d Doing) (news bool, err error) {
failed, err := json.Marshal(d.Failed)
if err != nil {
return err
return false, err
}
var before Doing
var beforeFailed []byte
found := i.store.Pool().QueryRow(ctx,
`select outcome, refused, failed, failing_since, failures, coalesce(declared,'')
from node_report where node = $1`,
node).Scan(&before.Outcome, &before.Refused, &beforeFailed, &before.Since, &before.Times,
&before.Declared)
switch {
case errors.Is(found, pgx.ErrNoRows):
news = true
case found != nil:
return false, found
default:
if err := json.Unmarshal(beforeFailed, &before.Failed); err != nil {
return false, err
}
news = before.Outcome != d.Outcome || before.Declared != d.Declared || !sameFailure(before, d)
}
var since *time.Time
times := 0
if d.Outcome != OutcomeApplied {
var before Doing
var beforeFailed []byte
err := i.store.Pool().QueryRow(ctx,
`select outcome, refused, failed, failing_since, failures from node_report where node = $1`,
node).Scan(&before.Outcome, &before.Refused, &beforeFailed, &before.Since, &before.Times)
switch {
case errors.Is(err, pgx.ErrNoRows):
case err != nil:
return err
default:
if err := json.Unmarshal(beforeFailed, &before.Failed); err != nil {
return err
}
}
now := time.Now()
since, times = &now, 1
if err == nil && sameFailure(before, d) && before.Since != nil {
if found == nil && sameFailure(before, d) && before.Since != nil {
since, times = before.Since, before.Times+1
}
}
@@ -707,7 +715,10 @@ func (i *Inventory) RecordDoing(ctx context.Context, node string, d Doing) error
declared = excluded.declared,
failing_since = excluded.failing_since, failures = excluded.failures`,
node, d.Outcome, d.Refused, failed, d.Applied, d.Declared, since, times)
return err
if err != nil {
return false, err
}
return news, nil
}
// NotDoingWhatTheyWereTold is every machine whose last report was not a clean apply.
+22 -60
View File
@@ -3,16 +3,13 @@ package link
import (
"context"
"encoding/json"
"errors"
"fmt"
"time"
amqp "github.com/rabbitmq/amqp091-go"
"github.com/nats-io/nats.go"
)
// RPCExchange is where a module's tools are asked over the broker, keyed `<module>.<tool>`, and
// where the answer comes back, keyed by the asker's reply queue (novox/hq ADR 0047).
const RPCExchange = "mesh.rpc"
// Answer is what a module's tool replies: one of the two, never both.
type Answer struct {
Result json.RawMessage `json:"result,omitempty"`
@@ -27,70 +24,35 @@ type Answer struct {
// grants, and should not. The control plane already holds a connection that may, so a person or
// an agent asks through it, and every question passes one process where an audit belongs.
//
// The reply queue is the caller's own, server-named and exclusive, bound to the RPC exchange under
// its own name: a serving module answers through that exchange and never the default one, whose
// permission is per exchange rather than per queue. The correlation is checked rather than
// assumed, as every RPC here is.
func Ask(ctx context.Context, channel *amqp.Channel, module, tool string, args json.RawMessage,
// The answer comes back on the asker's own inbox, which only the asker may read; the serving
// module answers there and nowhere else. The bus refuses a request nothing serves at once, so a
// module that is down or a tool that does not exist is said now rather than after the whole wait.
func Ask(ctx context.Context, bus Bus, module, tool string, args json.RawMessage,
timeout time.Duration) (Answer, error) {
if len(args) == 0 {
args = json.RawMessage(`{}`)
}
replies, err := channel.QueueDeclare("", false, true, true, false, nil)
if err != nil {
return Answer{}, err
}
if err := channel.QueueBind(replies.Name, replies.Name, RPCExchange, false, nil); err != nil {
return Answer{}, fmt.Errorf("cannot bind a reply queue to %s: %w", RPCExchange, err)
}
answers, err := channel.ConsumeWithContext(ctx, replies.Name, "", true, true, false, false, nil)
if err != nil {
return Answer{}, err
}
// Mandatory, so a request nothing consumes comes straight back: a module that is down, or a
// tool that does not exist, is said at once rather than after the whole wait.
returned := channel.NotifyReturn(make(chan amqp.Return, 1))
id := fmt.Sprintf("ask-%d", time.Now().UnixNano())
key := module + "." + tool
if err := channel.PublishWithContext(ctx, RPCExchange, key, true, false, amqp.Publishing{
ContentType: "application/json",
CorrelationId: id,
ReplyTo: replies.Name,
Body: args,
}); err != nil {
return Answer{}, fmt.Errorf("cannot ask %s: %w", key, err)
}
waiting, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
for {
select {
case back := <-returned:
if back.CorrelationId == id {
return Answer{}, fmt.Errorf(
"nothing serves %s: no runtime has bound %q on the broker. The module is not "+
"assigned, its runtime is not up, or it serves no such tool — `status` "+
"says whether the machine carrying it has applied", module, key)
}
case <-waiting.Done():
reply, err := bus.AskTool(ctx, module, tool, args, timeout)
if err != nil {
if errors.Is(err, nats.ErrNoResponders) {
return Answer{}, fmt.Errorf(
"nothing serves %s: no runtime has bound %q on the bus. The module is not "+
"assigned, its runtime is not up, or it serves no such tool — `status` says "+
"whether the machine carrying it has applied", module, key)
}
if errors.Is(err, context.DeadlineExceeded) || errors.Is(err, nats.ErrTimeout) {
return Answer{}, fmt.Errorf(
"%s did not answer within %s. Its runtime serves %q when it is up and has bound "+
"the broker — `status` says whether the machine carrying it has applied",
"the bus — `status` says whether the machine carrying it has applied",
module, timeout, key)
case delivery, ok := <-answers:
if !ok {
return Answer{}, fmt.Errorf("the connection closed while waiting for %s", key)
}
if delivery.CorrelationId != id {
continue
}
var answer Answer
if err := json.Unmarshal(delivery.Body, &answer); err != nil {
return Answer{}, fmt.Errorf("%s answered with something unreadable: %w", key, err)
}
return answer, nil
}
return Answer{}, fmt.Errorf("cannot ask %s: %w", key, err)
}
var answer Answer
if err := json.Unmarshal(reply, &answer); err != nil {
return Answer{}, fmt.Errorf("%s answered with something unreadable: %w", key, err)
}
return answer, nil
}
+13 -94
View File
@@ -1,12 +1,7 @@
package link
import (
"context"
"encoding/json"
"fmt"
"time"
amqp "github.com/rabbitmq/amqp091-go"
)
// Asking a machine to build a module, and hearing what came out.
@@ -22,21 +17,6 @@ import (
// holds no opinion about what they contain, and a host that also built things would be a host
// with a container runtime requirement and a git dependency (novox/hq ADR 0005).
// BuildQueue is where build requests wait. One queue, so several build machines can share the
// work and each request is done exactly once — which is what a queue is for and what a
// per-machine routing key would not give.
const BuildQueue = "builds"
// KeyBuilt is what a builder publishes when it has finished, successfully or not.
const KeyBuilt = "built"
// ReplyQueue is where the answer to one request goes.
//
// **Named here rather than left to the broker**, so it can be scoped and reasoned about. A broker
// generates its own name for an unnamed queue, and a builder permitted to write to whatever that
// convention happens to produce works on one broker and silently cannot answer on another.
func ReplyQueue(id string) string { return BuildQueue + ".reply." + id }
// BuildRequest is one module to build.
type BuildRequest struct {
// ID correlates the answer with the asking. Not the module name: two builds of one module can
@@ -108,87 +88,26 @@ type BuildResult struct {
// (novox/hq ADR 0009). The catalogue turns these into edges; nothing else need care.
Against []string `json:"against,omitempty"`
// Read is every repository this build read source from besides the module's own. A module whose
// recipe packages source that lives elsewhere is affected when that repository moves, and the
// manifest the mesh keeps says nothing about it (novox/hq 04-ISSUES/131).
Read []ReadRepository `json:"read,omitempty"`
// Failed is why, when it did.
Failed string `json:"failed,omitempty"`
}
// ReadRepository is a repository a build read source from besides the module's own, at the branch,
// tag or commit it read. Spelled here as well as in the catalogue and the inventory, for the reason
// MadeArtifact is: one direction of dependency.
type ReadRepository struct {
Repository string `json:"repository"`
Ref string `json:"ref,omitempty"`
}
// MadeArtifact is one thing a build produced, as a person would want it reported.
type MadeArtifact struct {
Name string `json:"name"`
Kind string `json:"kind"`
Reference string `json:"reference"`
}
// RequestBuild asks for a module to be built and waits for the answer.
//
// Waiting rather than returning immediately, because the thing a person wants after asking for a
// build is to know whether it worked. A build that is dispatched and forgotten needs somewhere to
// look afterwards, and there is nowhere yet.
func RequestBuild(ctx context.Context, channel *amqp.Channel, request BuildRequest,
timeout time.Duration) (BuildResult, error) {
// Its own queue for the answer, declared before the ask. Consuming from the shared exchange
// would mean competing with the control plane's own consumer for a message meant for this
// caller — which is the fault this package's own doc comment records having had.
replies, err := channel.QueueDeclare(ReplyQueue(request.ID), false, true, true, false, nil)
if err != nil {
return BuildResult{}, err
}
// Bound to the exchange, and the answer comes back through it.
//
// **A builder never publishes to the default exchange**, because permission there is per
// exchange and not per queue — a builder allowed to use it could publish into any node's
// queue, which is the privilege a build machine most obviously should not have. Found by
// running it: the builder built, could not answer, and the connection closed saying only
// "not allowed to publish to exchange ''".
//
// The cost is that every asker sees every result, which is why the correlation is checked
// below rather than assumed.
if err := channel.QueueBind(replies.Name, KeyBuilt, Exchange, false, nil); err != nil {
return BuildResult{}, err
}
answers, err := channel.ConsumeWithContext(ctx, replies.Name, "", true, true, false, false, nil)
if err != nil {
return BuildResult{}, err
}
body, err := json.Marshal(request)
if err != nil {
return BuildResult{}, err
}
if err := channel.PublishWithContext(ctx, "", BuildQueue, false, false, amqp.Publishing{
ContentType: "application/json",
DeliveryMode: amqp.Persistent,
CorrelationId: request.ID,
ReplyTo: replies.Name,
Body: body,
}); err != nil {
return BuildResult{}, err
}
waiting, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
for {
select {
case <-waiting.Done():
return BuildResult{}, fmt.Errorf(
"no builder answered within %s. Something must be consuming %q, and nothing is "+
"— or it is building something that takes longer than this",
timeout, BuildQueue)
case delivery, ok := <-answers:
if !ok {
return BuildResult{}, fmt.Errorf("the connection closed while waiting for a build")
}
var result BuildResult
if err := json.Unmarshal(delivery.Body, &result); err != nil {
return BuildResult{}, fmt.Errorf("a builder answered with something unreadable: %w", err)
}
if result.ID != request.ID {
// Somebody else's answer on this queue. Ignored rather than returned, because
// returning it would attribute one build's outcome to another's.
continue
}
return result, nil
}
}
}
-202
View File
@@ -1,202 +0,0 @@
package link
import (
"context"
"encoding/json"
"errors"
"fmt"
"time"
amqp "github.com/rabbitmq/amqp091-go"
)
// The build flow on the bus the mesh runs on today.
//
// Moved behind the seam rather than changed. The queue, the reply binding and the correlation are
// what they were, because the mesh is running on this.
// currentBuilds asks for builds over a channel.
type currentBuilds struct{ channel *amqp.Channel }
// BuildsOverCurrent is the asking side on the bus the mesh has.
func BuildsOverCurrent(channel *amqp.Channel) Builders { return currentBuilds{channel: channel} }
func (b currentBuilds) Close() {}
func (b currentBuilds) Submit(ctx context.Context, request BuildRequest,
wait time.Duration) (BuildResult, error) {
// Its own queue for the answer, declared before the ask. Consuming from the shared exchange
// would mean competing with the controller's own consumer for a message meant for this caller.
replies, err := b.channel.QueueDeclare(ReplyQueue(request.ID), false, true, true, false, nil)
if err != nil {
return BuildResult{}, err
}
// **A builder never publishes to the default exchange**, because permission there is per
// exchange and not per queue — a builder allowed to use it could publish into any node's queue,
// which is the privilege a build machine most obviously should not have. The cost is that every
// asker sees every result, which is why the correlation is checked below rather than assumed.
if err := b.channel.QueueBind(replies.Name, KeyBuilt, Exchange, false, nil); err != nil {
return BuildResult{}, err
}
answers, err := b.channel.ConsumeWithContext(ctx, replies.Name, "", true, true, false, false, nil)
if err != nil {
return BuildResult{}, err
}
body, err := json.Marshal(request)
if err != nil {
return BuildResult{}, err
}
if err := b.channel.PublishWithContext(ctx, "", BuildQueue, false, false, amqp.Publishing{
ContentType: "application/json",
DeliveryMode: amqp.Persistent,
CorrelationId: request.ID,
ReplyTo: replies.Name,
Body: body,
}); err != nil {
return BuildResult{}, err
}
waiting, cancel := context.WithTimeout(ctx, wait)
defer cancel()
for {
select {
case <-waiting.Done():
return BuildResult{}, waitingFor(wait)
case delivery, ok := <-answers:
if !ok {
return BuildResult{}, errors.New("the connection closed while waiting for a build")
}
result, mine, err := theOutcomeOf(delivery.Body, request.ID)
if err != nil {
return BuildResult{}, err
}
if mine {
return result, nil
}
}
}
}
// --- the machine's side ---------------------------------------------------------------------
type currentMachine struct {
conn *amqp.Connection
channel *amqp.Channel
on string
}
// MachineOverCurrent takes build work over a channel.
func MachineOverCurrent(conn *amqp.Connection, channel *amqp.Channel, on string) BuildMachine {
return &currentMachine{conn: conn, channel: channel, on: on}
}
func (m *currentMachine) Close() {}
func (m *currentMachine) Take(ctx context.Context, do func(context.Context, Build)) error {
if _, err := m.channel.QueueDeclare(BuildQueue, true, false, false, false, nil); err != nil {
return err
}
// One at a time. A machine that took five requests at once would run five container builds
// against one runtime and finish all of them slower than it would have finished the first — and
// the queue is what shares work between machines, so nothing is lost by it.
if err := m.channel.Qos(1, 0, false); err != nil {
return err
}
// Not auto-acknowledged: a request acknowledged on arrival is a build that vanishes if this
// process dies mid-way, with nobody waiting on it ever hearing why.
requests, err := m.channel.ConsumeWithContext(ctx, BuildQueue, "mesh-builder",
false, false, false, false, nil)
if err != nil {
return err
}
for {
select {
case <-ctx.Done():
return nil
case delivery, ok := <-requests:
if !ok {
return errors.New("the broker closed the connection")
}
var request BuildRequest
if err := json.Unmarshal(delivery.Body, &request); err != nil {
// Unreadable: rejected rather than retried, because the next attempt reads the same
// bytes. Nobody waiting hears an answer, which is correct — there was no request.
_ = delivery.Reject(false)
continue
}
do(ctx, &currentBuild{request: request, delivery: delivery, on: m.on, channel: m.channel})
}
}
}
type currentBuild struct {
request BuildRequest
delivery amqp.Delivery
on string
channel *amqp.Channel
}
func (b *currentBuild) Request() BuildRequest { return b.request }
// Announce answers and announces, which on this bus are two publishes to two exchanges.
//
// The reply goes to whoever asked, correlated to their request; the announcement says to the whole
// mesh that a module now exists at a commit (novox/hq ADR 0072). Only a successful build is
// announced: a failed one produced no module version, and announcing one would put something in the
// graph that was never made.
func (b *currentBuild) Announce(ctx context.Context, result BuildResult) error {
body, err := json.Marshal(result)
if err != nil {
return err
}
if err := b.channel.PublishWithContext(ctx, Exchange, KeyBuilt, false, false, amqp.Publishing{
ContentType: "application/json",
CorrelationId: result.ID,
Body: body,
}); err != nil {
return fmt.Errorf("cannot answer a build request: %w", err)
}
if result.Failed != "" || result.Commit == "" {
return nil
}
// **Announced under both names on this bus, for exactly as long as this bus lives.**
//
// A build's outcome belongs to the role now (novox/hq ADR 0121), so a catalogue built from the
// current manifests listens for the role's name. A catalogue that is *already running* listens for
// the module's, because that is what it was told when it was installed. A rename on a live bus
// needs the publisher and the subscriber to change together, and a merge cannot promise that: one
// of them is deployed first, and in that window the graph silently stops being updated — which is
// the failure this whole change was cleaning up after.
//
// So both, and the order stops mattering. The module's own name goes with the bus, in step 5's
// retirement list; nothing has ever run on the bus being built, so there is no legacy name there
// and this doubling has no counterpart.
announced := announcementOf(result)
if err := EmitEvent(ctx, OverCurrent{Channel: b.channel}, KeyModuleBuilt, "builder", b.on,
announced); err != nil {
return err
}
return EmitEvent(ctx, OverCurrent{Channel: b.channel}, KeyRoleBuilt, TheBuildMachine, b.on,
announced)
}
func (b *currentBuild) Done() error { return b.delivery.Ack(false) }
func (b *currentBuild) Hold(time.Duration) error {
// No delayed redelivery on this bus: handed back at once, which is what it has always done.
return b.delivery.Nack(false, true)
}
// announcementOf is what the mesh is told about a finished build. One function, so the two
// transports cannot describe the same build differently.
func announcementOf(result BuildResult) map[string]any {
return map[string]any{
"module": ModuleOf(result.Manifest), "commit": result.Commit,
"repository": result.Repository, "path": result.Path, "ref": result.Ref,
"manifest": json.RawMessage(result.Manifest), "against": result.Against,
"made": result.Made,
}
}
+33 -46
View File
@@ -7,7 +7,6 @@ import (
"time"
"github.com/nats-io/nats.go"
amqp "github.com/rabbitmq/amqp091-go"
)
// Bus is what the controller needs of the mesh's bus, **in the mesh's own words rather than a
@@ -31,6 +30,11 @@ type Bus interface {
// (design 29 §4, the *state* shape).
PublishDeclaration(ctx context.Context, node string, body []byte) error
// PublishSeatEvent states a fact under a role's own name, for the holder of that role. A
// module's event is addressed to the module; a role's is addressed to the role, so it keeps
// meaning when the holder changes (novox/hq ADR 0121, ADR 0129).
PublishSeatEvent(ctx context.Context, seat, event string, body []byte) error
// AskTool sends one question to a module's tool and awaits one answer. A tool nobody serves
// must say so **at once** rather than after the whole wait: the difference between "that
// module is down" and "that tool is slow" is the first thing a person asking wants.
@@ -39,51 +43,6 @@ type Bus interface {
// --- The bus the mesh runs on today -----------------------------------------------------
// OverCurrent is the bus the mesh runs on today, until the rollout.
type OverCurrent struct{ Channel *amqp.Channel }
func (b OverCurrent) PublishEvent(ctx context.Context, key, source, node string, body []byte) error {
id, err := eventID()
if err != nil {
return err
}
return b.Channel.PublishWithContext(ctx, EventsExchange, key, false, false, amqp.Publishing{
ContentType: "application/json",
DeliveryMode: amqp.Persistent,
MessageId: id,
Timestamp: time.Now().UTC(),
Body: body,
Headers: amqp.Table{
"x-event-id": id,
"x-source": source,
"x-node": node,
"x-time": time.Now().UTC().Format(time.RFC3339),
"content-type": "application/json",
},
})
}
// AskTool is implemented over the existing reply-queue machinery in ask.go; this seam does not
// change how it works today.
func (b OverCurrent) AskTool(ctx context.Context, module, tool string, args []byte, timeout time.Duration) ([]byte, error) {
answer, err := Ask(ctx, b.Channel, module, tool, args, timeout)
if err != nil {
return nil, err
}
return answer.Result, nil
}
func (b OverCurrent) PublishDeclaration(ctx context.Context, node string, body []byte) error {
// To the queue directly rather than through an exchange: a declaration is for one node, and
// routing it by name through a shared exchange would mean a binding per node that nothing
// removes when a node is retired.
return b.Channel.PublishWithContext(ctx, "", QueueFor(node), false, false, amqp.Publishing{
ContentType: "application/json",
DeliveryMode: amqp.Persistent,
Body: body,
})
}
// --- NATS, the bus being built ----------------------------------------------------------------
// OverNATS is the bus as a JetStream context.
@@ -127,6 +86,13 @@ func EventSubject(source, key string) string {
return "mesh.mod." + source + ".event." + key
}
// SeatEventSubject is where a role's own event lands. Derived from the role, never from its holder:
// a fact about the build machine or about the control plane keeps its address when the module holding
// that role is replaced (novox/hq ADR 0121, ADR 0129).
func SeatEventSubject(seat, event string) string {
return "mesh.seat." + seat + ".event." + event
}
// DeclareSubject is where one node's declaration lands. Last-per-subject on the NODES stream, so
// a node that was away gets exactly the current one and a replayed older one is refused by
// sequence — the wire-level answer to novox/hq issue 107.
@@ -158,6 +124,27 @@ func (b OverNATS) PublishEvent(ctx context.Context, key, source, node string, bo
return nil
}
// PublishSeatEvent states a role's own fact. Same envelope as a module's event and a different
// address: the source header is the role, because that is what the fact is about.
func (b OverNATS) PublishSeatEvent(ctx context.Context, seat, event string, body []byte) error {
id, err := eventID()
if err != nil {
return err
}
h := nats.Header{}
h.Set("x-event-id", id)
h.Set("x-source", seat)
_, err = b.JS.PublishMsg(&nats.Msg{
Subject: SeatEventSubject(seat, event),
Header: h,
Data: body,
}, nats.MsgId(id), nats.Context(ctx))
if err != nil {
return fmt.Errorf("stating %s of the %s seat: %w", event, seat, err)
}
return nil
}
func (b OverNATS) PublishDeclaration(ctx context.Context, node string, body []byte) error {
_, err := b.JS.Publish(DeclareSubject(node), body, nats.Context(ctx))
if err != nil {
+23
View File
@@ -0,0 +1,23 @@
package link
import (
"github.com/novox/mesh-controller/internal/broker"
)
// ConnectNats is Connect for the bus being built: the controller's inbound and outbound over one
// JetStream connection the caller has already raised the streams on. Nothing is declared here —
// the streams and the controller's consumers are asserted by Raise, before anything is served.
func ConnectNats(js *broker.JetStream, enroller Enroller, listener Listener) *Server {
return &Server{
inbound: Nats(js),
bus: OverNATS{JS: js.Context(), Conn: js.Conn()},
js: js,
enroller: enroller,
listener: listener,
log: newLog(),
}
}
// Bus is the controller's outbound, whichever transport it connected over. Callers that send a
// declaration or ask a tool use this rather than the channel, which one transport does not have.
func (s *Server) Bus() Bus { return s.bus }
+19 -31
View File
@@ -24,10 +24,9 @@ import (
// — it holds both grants and asks each for its part, which is what the process running them is
// for.
type Enrolment struct {
Inventory *inventory.Inventory
Identity *identity.Identity
Management *broker.Management
Broker broker.Broker
Inventory *inventory.Inventory
Identity *identity.Identity
Broker broker.Broker
// OnNATS says the mesh's own traffic is on the bus being built, so a node's credential is
// minted into the mesh's records and composed into the bus's user list rather than pushed
@@ -194,17 +193,6 @@ func (e Enrolment) Enrol(ctx context.Context, request EnrolRequest) (reply Enrol
reply.Password = password
}
case e.Management != nil:
password, err := freshPassword()
if err != nil {
return EnrolReply{}, err
}
if err := e.Management.CreateNodeAccount(ctx, node.Name, password); err != nil {
log.Printf("%s is enrolled and its broker password could not be replaced, so it keeps "+
"the token's secret as its password: %v", node.Name, err)
} else {
reply.Password = password
}
}
if profile != nil {
@@ -261,9 +249,6 @@ func freshPassword() (string, error) {
var _ Enroller = Enrolment{}
// ErrNoBrokerManagement is returned when an account cannot be made because nothing was configured.
var ErrNoBrokerManagement = errors.New("no broker management configured")
// Heard records what a node reported about itself.
//
// A node states; the owning context writes (novox/hq ADR 0006). What a node says it applied is
@@ -280,7 +265,7 @@ func (e Enrolment) Outstanding(ctx context.Context, node string) (string, error)
return e.Inventory.Outstanding(ctx, node)
}
func (e Enrolment) Heard(ctx context.Context, report Report) (err error) {
func (e Enrolment) Heard(ctx context.Context, report Report) (news bool, err error) {
// A store that could not be asked right now is said as such, so the report is kept for
// another attempt rather than acknowledged and lost (novox/hq issue 082).
defer func() {
@@ -289,11 +274,11 @@ func (e Enrolment) Heard(ctx context.Context, report Report) (err error) {
}
}()
if report.Node == "" {
return errors.New("a report named no node")
return false, errors.New("a report named no node")
}
node, err := e.Inventory.NodeByName(ctx, report.Node)
if err != nil {
return err
return false, err
}
// What an adopted node holds, which firewall it found, and what is reachable on it (novox/hq
@@ -313,7 +298,7 @@ func (e Enrolment) Heard(ctx context.Context, report Report) (err error) {
Port: r.Port, By: r.By, Published: r.Published, ContainerPort: r.ContainerPort})
}
if err := e.Inventory.RecordAdoption(ctx, node.ID, held, report.Firewall, reachable); err != nil {
return err
return false, err
}
}
// What it says about the tunnel it carried (novox/hq ADR 0105), whenever it says it.
@@ -323,7 +308,7 @@ func (e Enrolment) Heard(ctx context.Context, report Report) (err error) {
Peers: report.Tunnel.Peers, State: report.Tunnel.State, Note: report.Tunnel.Note,
Kept: report.Tunnel.Kept,
}); err != nil {
return err
return false, err
}
}
// A node taking a found tunnel's key after enrolment (novox/hq ADR 0105). Verified against the
@@ -332,9 +317,9 @@ func (e Enrolment) Heard(ctx context.Context, report Report) (err error) {
// does not verify or is stale — a refusal, not "not now", so the node hears why.
if report.Rekey != nil {
if err := e.rekey(ctx, node, *report.Rekey); err != nil {
return err
return false, err
}
return e.Inventory.Seen(ctx, node.ID)
return false, e.Inventory.Seen(ctx, node.ID)
}
// A bare word that a node is there is not an account of what the machine did or holds: it
@@ -349,7 +334,7 @@ func (e Enrolment) Heard(ctx context.Context, report Report) (err error) {
if report.Superseded != "" {
log.Printf("%s set aside declaration %s for the newer %s", report.Node, report.Declared, report.Superseded)
}
return e.Inventory.Seen(ctx, node.ID)
return false, e.Inventory.Seen(ctx, node.ID)
}
// What it did is kept whichever way it went. Until this, a refusal or a failure moved
// last_seen and the reason went to a log line, so "which machine is not doing what it was
@@ -376,17 +361,20 @@ func (e Enrolment) Heard(ctx context.Context, report Report) (err error) {
// on top of it (novox/hq ADR 0038). Kept even when the declaration was refused: what the
// machine carries is true regardless of what it thought of the last thing it was sent.
if err := e.Inventory.RecordCarried(ctx, report.Node, report.Carried); err != nil {
return err
return false, err
}
if err := e.Inventory.RecordDoing(ctx, node.ID, doing); err != nil {
return err
// **Whether this is news** is the store's answer: it holds the previous report, and a machine
// that reconciles every minute says the same thing until something changes (novox/hq ADR 0134).
news, err = e.Inventory.RecordDoing(ctx, node.ID, doing)
if err != nil {
return false, err
}
// A refusal, a failure, or a bare word that the node is there — none of them is an account of
// what the machine holds, so each moves last_seen and nothing else. Recording a partial list
// as though it were the whole would tell a rebuilding node to remove what it still has.
if report.Refused != "" || len(report.Failed) > 0 || report.Applied == nil {
return e.Inventory.Seen(ctx, node.ID)
return news, e.Inventory.Seen(ctx, node.ID)
}
return e.Inventory.RecordOwned(ctx, node.ID, report.Applied)
return news, e.Inventory.RecordOwned(ctx, node.ID, report.Applied)
}
+58
View File
@@ -49,6 +49,39 @@ func eventID() (string, error) {
return hex.EncodeToString(raw), nil
}
// MeshControllerSeat is the role the control plane holds, and therefore where its own facts live: a
// role's events belong to the role, not to whichever container is holding it today (novox/hq ADR 0121,
// ADR 0129). It is what makes them addressable while the control plane itself is being replaced.
const MeshControllerSeat = "mesh-controller"
// The facts the mesh states about its own work (novox/hq ADR 0134).
const (
// KeyApplied: a machine now runs what it was sent.
KeyApplied = "applied"
// KeyRefused: a machine did not take what it was sent, and why.
KeyRefused = "refused"
// KeyBuiltBefore: a build the mesh already held, for a catalogue that asked what it missed. Not
// `built` — that is the build machine's, said as it happens, and a replay is neither.
KeyBuiltBefore = "built-before"
)
// Applied is what a machine now runs, as the mesh states it.
type Applied struct {
Node string `json:"node"`
Declared string `json:"declared,omitempty"`
// Resources is how many the machine applied, not which: the list is the machine's own account
// of itself and belongs in the records, not in a fact every listener has to read past.
Resources int `json:"resources"`
}
// Refused is a machine that would not take what it was sent.
type Refused struct {
Node string `json:"node"`
Declared string `json:"declared,omitempty"`
Refused string `json:"refused,omitempty"`
Failed map[string]string `json:"failed,omitempty"`
}
// KeyModuleBuilt is what the builder announces when it has built something. The catalogue places
// it in the module graph; nothing else need care.
const KeyModuleBuilt = "module.builder.built"
@@ -117,6 +150,31 @@ type Announcement struct {
}
// Upgraded is what the catalogue says when a module's current version moves.
// SourceMoved is what the forge announces when a pull request is merged: which repository, into
// which branch, producing which commit. The mesh matches it against every module's recorded
// source and builds what moved, bases first.
type SourceMoved struct {
Owner string `json:"owner"`
Repo string `json:"repo"`
Base string `json:"base"`
Head string `json:"head"`
Commit string `json:"merge_commit_sha"`
CloneURL string `json:"clone_url"`
HTMLURL string `json:"html_url"`
// MergedAt is when the forge merged it, RFC 3339. What decides whether this is news.
MergedAt string `json:"merged_at"`
// Paths are the files the merge changed, from the repository's root. Empty means the forge said
// nothing about them, and every module built from the repository is treated as affected.
Paths []string `json:"paths,omitempty"`
// PathsTruncated says the merge changed more files than the forge was asked to list, so Paths is
// a beginning rather than the whole change — and again, everything is treated as affected. Said
// rather than inferred from a round number, because "this is all of it" and "this is as much as
// I asked for" are the difference between rebuilding a module and leaving it stale.
PathsTruncated bool `json:"paths_truncated,omitempty"`
}
type Upgraded struct {
Module string `json:"module"`
Commit string `json:"commit"`
+6 -6
View File
@@ -22,7 +22,7 @@ func heardFrom(t *testing.T, report link.Report) (*inventory.Inventory, inventor
if _, err := inv.AddNode(ctx, report.Node); err != nil {
t.Fatal(err)
}
if err := (link.Enrolment{Inventory: inv}).Heard(ctx, report); err != nil {
if _, err := (link.Enrolment{Inventory: inv}).Heard(ctx, report); err != nil {
t.Fatal(err)
}
doing, said, err := inv.DoingOf(ctx, report.Node)
@@ -103,7 +103,7 @@ func TestABareAliveDoesNotWipeTheDeclarationThatSaysANodeIsCurrent(t *testing.T)
if err := inv.RecordSent(ctx, node.ID, digest); err != nil {
t.Fatal(err)
}
if err := (link.Enrolment{Inventory: inv}).Heard(ctx, link.Report{
if _, err := (link.Enrolment{Inventory: inv}).Heard(ctx, link.Report{
Node: "anchor", Applied: []string{"a", "b"}, Declared: digest, Carried: []int{5432},
}); err != nil {
t.Fatal(err)
@@ -126,7 +126,7 @@ func TestABareAliveDoesNotWipeTheDeclarationThatSaysANodeIsCurrent(t *testing.T)
}
// Now the node says only that it is there, as it does every minute.
if err := (link.Enrolment{Inventory: inv}).Heard(ctx, link.Report{Node: "anchor"}); err != nil {
if _, err := (link.Enrolment{Inventory: inv}).Heard(ctx, link.Report{Node: "anchor"}); err != nil {
t.Fatal(err)
}
if !currentOf("anchor") {
@@ -162,7 +162,7 @@ func TestAFailureDoesNotBecomeTheAccountOfWhatTheMachineHolds(t *testing.T) {
if err := inv.RecordOwned(ctx, node.ID, []string{"one", "two", "three"}); err != nil {
t.Fatal(err)
}
if err := (link.Enrolment{Inventory: inv}).Heard(ctx, link.Report{
if _, err := (link.Enrolment{Inventory: inv}).Heard(ctx, link.Report{
Node: "workstation", Applied: []string{"one"}, Failed: map[string]string{"two": "no"},
}); err != nil {
t.Fatal(err)
@@ -200,13 +200,13 @@ func TestWhatAnAdoptedNodeHoldsIsKeptAndAnAliveWordDoesNotWipeIt(t *testing.T) {
}
check("after the report")
if err := (link.Enrolment{Inventory: inv}).Heard(ctx, link.Report{Node: "anchor"}); err != nil {
if _, err := (link.Enrolment{Inventory: inv}).Heard(ctx, link.Report{Node: "anchor"}); err != nil {
t.Fatal(err)
}
check("after an alive word")
// A reconcile report carrying only adoption is recorded, though it applied nothing.
if err := (link.Enrolment{Inventory: inv}).Heard(ctx, link.Report{Node: "anchor",
if _, err := (link.Enrolment{Inventory: inv}).Heard(ctx, link.Report{Node: "anchor",
Firewall: "ufw"}); err != nil {
t.Fatal(err)
}
+3 -13
View File
@@ -30,6 +30,9 @@ const (
KindHeartbeat = "heartbeat"
KindBuilt = "built"
KindModuleMoved = "module-moved"
// KindSourceMoved is the forge announcing a merge: a source moved, and what it produces is
// built without anybody telling the mesh (novox/hq 04-ISSUES/131).
KindSourceMoved = "source-moved"
KindCatchUp = "catch-up"
)
@@ -75,19 +78,6 @@ type Control interface {
// Took settles the message: acted on, or understood and needing no action.
Took() error
// About names what this message is about — a node's report, one module's move, one build's
// outcome — and is said before the store is asked.
//
// A transport that holds messages **in memory** uses it to set aside anything older it is
// holding about the same thing: the older is the past, and letting it come back after the
// newer was acted on would undo the newer.
//
// **This is the one thing holding-in-memory can do that holding-in-the-server cannot**, and
// naming it here rather than hiding it is deliberate. On the bus being built the message
// belongs to the server and comes back whatever happened meanwhile, so this is ignored and the
// digest a report carries answers the same question instead (window.go, design 25 §3).
About(what string)
// Hold keeps the message and asks for it again after the delay — the store window.
Hold(after time.Duration) error
-317
View File
@@ -1,317 +0,0 @@
package link
import (
"context"
"errors"
"fmt"
"time"
amqp "github.com/rabbitmq/amqp091-go"
)
// The consume side on the bus the mesh runs on today.
//
// Everything here was the serving loop's until the seam went in: the queues, the binds, the
// prefetch, and the list of messages the store could not take yet. It moved rather than changed —
// the behaviour this transport has is the behaviour it had, because the mesh is running on it and
// a bus nothing speaks yet is no reason to alter the one every node is on (ADR 0116).
// Prefetch is how many messages the bus hands the controller before it has settled them.
//
// More than one because a message the store could not take is held, unsettled, while the loop
// goes on answering others — an enrolment above all, which a host is waiting on (novox/hq issue
// 083). Bounded, because what is held is also what the bus has not kept on its own disk as
// pending.
const Prefetch = 64
// PrefetchHeadroom is how much of the prefetch is never held, so the loop always has messages to
// answer — an enrolment above all — while others wait for the store.
const PrefetchHeadroom = 8
// TryAgainAfter is how often the held are looked at. A store comes back in seconds, and a report a
// few seconds late is still current.
const TryAgainAfter = 2 * time.Second
// currentInbound consumes what nodes say over the bus the mesh has.
type currentInbound struct {
conn *amqp.Connection
channel *amqp.Channel
// upgrades and catchups are bound only when something is listening (Also).
upgrades bool
catchups bool
// held is every message the store could not take, by the bus's own delivery tag. Kept here
// rather than in the serving loop because holding a delivery unacknowledged is this
// transport's way of keeping it, and the other's is to hand it back to the server.
held map[uint64]*holding
// again is how often the held are looked at; zero means TryAgainAfter. Set by tests.
again time.Duration
}
// holding is one message kept for the store, and when to try it again.
type holding struct {
message *currentControl
due time.Time
about string
}
// Current is the consume side of the bus the mesh runs on today.
func Current(conn *amqp.Connection, channel *amqp.Channel) Inbound {
return &currentInbound{conn: conn, channel: channel, held: map[uint64]*holding{}}
}
// Also binds the queue one more kind arrives on.
//
// The kinds nodes publish all share one queue and are bound at Connect, because a node may
// publish any of them and binding one while forgetting another is a message the bus accepts, finds
// no queue for, and drops — the publisher sees success and the consumer sees nothing. The two that
// are events get their own queue each, and only when something is listening.
func (c *currentInbound) Also(kind string) error {
switch kind {
case KindModuleMoved:
if err := c.bindEvent(UpgradeQueue, KeyModuleUpgraded); err != nil {
return err
}
c.upgrades = true
case KindCatchUp:
if err := c.bindEvent(CatchUpQueue, KeyCatchingUp); err != nil {
return err
}
c.catchups = true
default:
return fmt.Errorf("nothing binds a queue for %s on this bus", kind)
}
return nil
}
func (c *currentInbound) bindEvent(queue, key string) error {
if _, err := c.channel.QueueDeclare(queue, true, false, false, false, nil); err != nil {
return fmt.Errorf("cannot declare the %s queue: %w", queue, err)
}
if err := c.channel.QueueBind(queue, key, EventsExchange, false, nil); err != nil {
return fmt.Errorf("cannot bind %s to %s/%s: %w", queue, EventsExchange, key, err)
}
return nil
}
func (c *currentInbound) Close() {}
// Receive consumes until the context ends.
//
// One consumer per queue, deliberately: with two on one queue the bus would round-robin between
// them and each would receive half of what it expects — a fault this project has already had,
// between a module's daemon and its capability server.
func (c *currentInbound) Receive(ctx context.Context, act func(context.Context, Control)) error {
// A bounded prefetch rather than one. The loop still takes messages one at a time; what the
// prefetch buys is that a message the store could not take can be held while the loop goes on
// to the next, instead of every enrolment waiting behind it (novox/hq issue 083). Anything
// held goes back to the bus if the controller stops, because nothing held is acknowledged.
if err := c.channel.Qos(Prefetch, 0, false); err != nil {
return err
}
deliveries, err := c.channel.ConsumeWithContext(ctx, ControlQueue, "control-plane",
false, false, false, false, nil)
if err != nil {
return err
}
// Its own queue and its own consumer for each event, for the reason above: two consumers on
// one queue split its messages between them, and an upgrade or a catch-up request going to
// whichever half was not listening is a gap that looks like a working mesh.
var upgrades, catchups <-chan amqp.Delivery
if c.upgrades {
upgrades, err = c.channel.ConsumeWithContext(ctx, UpgradeQueue, "control-plane-upgrades",
false, false, false, false, nil)
if err != nil {
return err
}
}
if c.catchups {
catchups, err = c.channel.ConsumeWithContext(ctx, CatchUpQueue, "control-plane-catchup",
false, false, false, false, nil)
if err != nil {
return err
}
}
closed := c.conn.NotifyClose(make(chan *amqp.Error, 1))
again := c.again
if again == 0 {
again = TryAgainAfter
}
ticker := time.NewTicker(again)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return nil
case <-ticker.C:
if ctx.Err() != nil {
return nil
}
c.retryHeld(ctx, act)
case delivery, ok := <-catchups:
if !ok {
if catchups != nil {
return errors.New("the bus stopped delivering catch-up requests")
}
continue
}
act(ctx, c.wrap(KindCatchUp, delivery))
case delivery, ok := <-upgrades:
// A nil channel blocks for ever, so this case simply never fires when nothing is
// listening for upgrades. Closed is different, and means the bus stopped.
if !ok {
if upgrades != nil {
return errors.New("the bus stopped delivering upgrades")
}
continue
}
act(ctx, c.wrap(KindModuleMoved, delivery))
case reason := <-closed:
// Said rather than returned quietly. A controller whose bus connection dropped is a
// mesh where nothing can be told anything, and the reason is the first thing anybody
// will want.
return fmt.Errorf("the bus connection closed: %v", reason)
case delivery, ok := <-deliveries:
if !ok {
return errors.New("the bus stopped delivering")
}
kind, known := kindOfKey[delivery.RoutingKey]
if !known {
// Rejected without requeue: a message nothing understands will not be understood
// on the next attempt either, and requeuing it would spin.
_ = delivery.Reject(false)
continue
}
act(ctx, c.wrap(kind, delivery))
}
}
}
// kindOfKey is how this transport's addressing becomes what the mesh calls a message.
var kindOfKey = map[string]string{
KeyEnrol: KindEnrolment,
KeyReport: KindReport,
KeyAlive: KindHeartbeat,
KeyBuilt: KindBuilt,
KeyModuleUpgraded: KindModuleMoved,
KeyCatchingUp: KindCatchUp,
}
func (c *currentInbound) wrap(kind string, delivery amqp.Delivery) *currentControl {
return &currentControl{kind: kind, delivery: delivery, on: c}
}
// retryHeld hands every message whose delay has passed back to the loop. Each handler holds it
// again, settles it, or lets it go past the bound.
func (c *currentInbound) retryHeld(ctx context.Context, act func(context.Context, Control)) {
now := time.Now()
due := make([]*currentControl, 0, len(c.held))
for _, h := range c.held {
if !h.due.After(now) {
due = append(due, h.message)
}
}
for _, m := range due {
if ctx.Err() != nil {
return
}
act(ctx, m)
}
}
// currentControl is one delivery from the bus the mesh has, as the controller reads it.
type currentControl struct {
kind string
delivery amqp.Delivery
on *currentInbound
// about is what this message is about, as the handler named it; empty until it does.
about string
// first is when this message was first held for the store; zero while it has not been.
first time.Time
}
func (m *currentControl) Kind() string { return m.kind }
func (m *currentControl) Body() []byte { return m.delivery.Body }
func (m *currentControl) Redelivered() bool { return m.delivery.Redelivered }
func (m *currentControl) HeldFor() time.Duration {
if m.first.IsZero() {
return 0
}
return time.Since(m.first)
}
// Answer publishes to the reply queue the request named.
func (m *currentControl) Answer(ctx context.Context, body []byte) error {
if m.delivery.ReplyTo == "" {
return errors.New("that request named no reply queue, so nothing can be told the answer")
}
return m.on.channel.PublishWithContext(ctx, "", m.delivery.ReplyTo, false, false,
amqp.Publishing{
ContentType: "application/json",
CorrelationId: m.delivery.CorrelationId,
Body: body,
})
}
func (m *currentControl) Took() error {
m.forget()
return m.delivery.Ack(false)
}
// Drop rejects without requeue: on this bus that is what "understood, and not worth another
// attempt" is spelled as, and it is what feeds a dead-letter queue where one is configured.
func (m *currentControl) Drop() error {
m.forget()
return m.delivery.Reject(false)
}
// About names what this message is about, and lets go of whatever is held about the same thing:
// the held one is the past, and acting on it after this one would undo this one. Acknowledged
// rather than left to come back, because a held message nothing will act on is a place in the
// prefetch nothing gets back.
func (m *currentControl) About(what string) {
m.about = what
if what == "" {
return
}
for tag, h := range m.on.held {
if h.about != what || tag == m.delivery.DeliveryTag {
continue
}
delete(m.on.held, tag)
_ = h.message.delivery.Ack(false)
}
}
// Hold keeps the message unacknowledged and sets it aside to be handed back after the delay.
//
// Held no further than the prefetch leaves room: past that the bus would hand the loop nothing
// new — enrolments included — until something held was let go. A message that cannot be held says
// so, and the handler settles it its own way.
func (m *currentControl) Hold(after time.Duration) error {
if m.on.held == nil {
m.on.held = map[uint64]*holding{}
}
if _, already := m.on.held[m.delivery.DeliveryTag]; !already {
if len(m.on.held) >= Prefetch-PrefetchHeadroom {
return fmt.Errorf("%d messages are already held for the store, and holding more "+
"would stop the queue", len(m.on.held))
}
m.first = time.Now()
}
m.on.held[m.delivery.DeliveryTag] = &holding{
message: m, due: time.Now().Add(after), about: m.about,
}
return nil
}
func (m *currentControl) forget() {
if m.on != nil {
delete(m.on.held, m.delivery.DeliveryTag)
}
}
-71
View File
@@ -1,71 +0,0 @@
package link
import (
"context"
"encoding/json"
"io"
"log"
"testing"
"time"
amqp "github.com/rabbitmq/amqp091-go"
)
// The harness for the consume side on the bus the mesh runs on today.
//
// Messages arrive through the seam, so what these tests exercise is the controller's decision
// about a message and this transport's way of keeping one — which is what the seam separated. A
// fake acknowledger stands in for the bus, because what is asserted is how a message was settled
// and that needs no server.
// settled is how the bus was told to settle one message.
type settled struct{ acked, nacked, requeued, rejected bool }
func (a *settled) Ack(uint64, bool) error { a.acked = true; return nil }
func (a *settled) Nack(_ uint64, _ bool, requeue bool) error {
a.nacked, a.requeued = true, requeue
return nil
}
func (a *settled) Reject(uint64, bool) error { a.rejected = true; return nil }
// unsettled is a message the controller has neither taken nor let go: it is held, and the bus will
// hand it to whatever consumes next if the controller stops.
func (a *settled) unsettled() bool { return !a.acked && !a.nacked && !a.rejected }
var tag uint64
func quiet() *log.Logger { return log.New(io.Discard, "", 0) }
// serving is a controller with nothing but a way of receiving, ready for a listener, a recorder,
// an upgrader or a replayer to be set on it.
func serving() (*Server, *currentInbound) {
in := &currentInbound{held: map[uint64]*holding{}}
return &Server{inbound: in, bus: OverCurrent{}, log: quiet()}, in
}
// sends is one message arriving over this transport, as the controller reads it.
func (c *currentInbound) sends(t *testing.T, to *settled, kind string, v any) Control {
t.Helper()
body, err := json.Marshal(v)
if err != nil {
t.Fatal(err)
}
tag++
return &currentControl{kind: kind, on: c, delivery: amqp.Delivery{
Acknowledger: to, Body: body, DeliveryTag: tag,
}}
}
// dueNow brings every held message forward, so a test need not wait out the backoff a real store
// restart would be given (RedeliverAfter).
func (c *currentInbound) dueNow() {
for _, h := range c.held {
h.due = time.Now().Add(-time.Second)
}
}
// retries hands every held message back to the controller, the way the ticker does.
func (c *currentInbound) retries(ctx context.Context, s *Server) {
c.dueNow()
c.retryHeld(ctx, s.act)
}
+94
View File
@@ -0,0 +1,94 @@
package link
import (
"context"
"encoding/json"
"testing"
"time"
)
// A harness for the controller's decisions about a message, with no bus behind it.
//
// What these tests exercise is the server's verdict — taken, dropped, held for the store, let go
// once the store has been away too long — and the transport's part in that is only to keep a held
// message and hand it back. A fake that does exactly that stands in for the bus, so what is
// asserted is how a message was settled and the decision needs no server.
// settled is how the bus was told to settle one message.
type settled struct{ acked, nacked, requeued, rejected bool }
// unsettled is a message the controller has neither taken nor let go: it is held, and the bus will
// hand it to whatever consumes next if the controller stops.
func (a *settled) unsettled() bool { return !a.acked && !a.nacked && !a.rejected }
// fakeInbound keeps the messages the controller held, the way a stream would.
type fakeInbound struct{ held map[uint64]*fakeControl }
var tag uint64
// serving is a controller with nothing but a way of receiving, ready for a listener, a recorder,
// an upgrader or a replayer to be set on it.
func serving() (*Server, *fakeInbound) {
in := &fakeInbound{held: map[uint64]*fakeControl{}}
return &Server{inbound: in, log: quiet()}, in
}
func (c *fakeInbound) Also(string) error { return nil }
func (c *fakeInbound) Close() {}
func (c *fakeInbound) Receive(context.Context, func(context.Context, Control)) error {
return nil
}
// sends is one message arriving, as the controller reads it.
func (c *fakeInbound) sends(t *testing.T, to *settled, kind string, v any) Control {
t.Helper()
body, err := json.Marshal(v)
if err != nil {
t.Fatal(err)
}
tag++
return &fakeControl{kind: kind, body: body, tag: tag, to: to, on: c}
}
// retries hands every held message back to the controller, the way a stream redelivers.
func (c *fakeInbound) retries(ctx context.Context, s *Server) {
for tag, m := range c.held {
delete(c.held, tag)
m.redelivered = true
s.act(ctx, m)
}
}
type fakeControl struct {
kind string
body []byte
tag uint64
to *settled
on *fakeInbound
redelivered bool
since time.Time
}
func (m *fakeControl) Kind() string { return m.kind }
func (m *fakeControl) Body() []byte { return m.body }
func (m *fakeControl) Redelivered() bool { return m.redelivered }
func (m *fakeControl) About(string) {}
func (m *fakeControl) Answer(context.Context, []byte) error { return nil }
func (m *fakeControl) Took() error { m.to.acked = true; return nil }
func (m *fakeControl) Drop() error { m.to.rejected = true; return nil }
// HeldFor is how long the store has been waited on for this message — zero on a first delivery.
func (m *fakeControl) HeldFor() time.Duration {
if m.since.IsZero() {
return 0
}
return time.Since(m.since)
}
func (m *fakeControl) Hold(time.Duration) error {
if m.since.IsZero() {
m.since = time.Now()
}
m.on.held[m.tag] = m
return nil
}
+44 -9
View File
@@ -23,6 +23,10 @@ import (
// it first could not take it, and a controller that restarts mid-window has nothing to lose.
// natsInbound consumes what nodes and modules say over NATS.
// Prefetch is how many controls the controller holds unacknowledged at once: the store window
// (ADR 0083) is the server's, and this is what it may hand this process ahead of its acting.
const Prefetch = 64
type natsInbound struct {
js *broker.JetStream
// follows is the kinds asked for beyond what nodes say (Also). The events those are are the
@@ -48,7 +52,7 @@ func Nats(js *broker.JetStream) Inbound {
// whatever was asked for — and not at all when nothing was.
func (n *natsInbound) Also(kind string) error {
switch kind {
case KindModuleMoved, KindCatchUp:
case KindModuleMoved, KindCatchUp, KindSourceMoved:
n.follows[kind] = true
return nil
default:
@@ -150,10 +154,47 @@ func (n *natsInbound) deliver(ctx context.Context, act func(context.Context, Con
m.seq = meta.Sequence.Stream
m.delivered = meta.NumDelivered
}
// **Work that outlives the acknowledgement window says so while it runs.**
//
// The bus waits a fixed time to be told a message was taken, and then hands it to whoever
// consumes next — which is right for a consumer that died and wrong for one that is busy.
// Acting on a merge builds every module the merge changed: minutes of work against a
// thirty-second window. So the same merge was handed over again while the first build was
// still running, and again after that — on 2026-09-28 one merge ran the mesh's whole
// catalogue five times over and exhausted a public registry's pull limit.
//
// Here rather than in each handler, because the window belongs to the transport and every
// handler would otherwise have to remember it. It changes nothing about a handler that
// dies: a message is kept alive only while this goroutine is, so a controller that stops
// stops saying so, and the bus redelivers exactly as it should.
working := make(chan struct{})
defer close(working)
go stillWorking(msg, working)
}
act(ctx, m)
}
// heartbeatWhileWorking is how often a handler still running tells the bus so — comfortably inside
// the shortest acknowledgement window the mesh gives any of its consumers.
const heartbeatWhileWorking = 10 * time.Second
// stillWorking keeps one message alive until the work on it returns.
//
// An error is not worth reporting: what the bus does when it is not told is redeliver, which is
// exactly what happens if this fails, and the handler's own outcome is the thing worth logging.
func stillWorking(msg *nats.Msg, done <-chan struct{}) {
tick := time.NewTicker(heartbeatWhileWorking)
defer tick.Stop()
for {
select {
case <-done:
return
case <-tick.C:
_ = msg.InProgress()
}
}
}
// kindOfSubject is how this transport's addressing becomes what the mesh calls a message.
//
// By subject, which is the only thing the server enforces: a body claiming to be a report does not
@@ -180,6 +221,8 @@ func kindOfSubject(subject string) (string, bool) {
return KindModuleMoved, true
case broker.ControllerFollows[1]:
return KindCatchUp, true
case broker.ControllerFollows[3]:
return KindSourceMoved, true
case BuildOutcome():
// A build's outcome is the role's event now, so it arrives on the events stream rather than
// the control branch — and is acted on by the same handler, because what the controller does
@@ -220,14 +263,6 @@ func (m *natsControl) HeldFor() time.Duration {
return time.Since(first)
}
// About is nothing here, and that is the point.
//
// Setting a held message aside when a newer one about the same thing arrives is what a controller
// holding deliveries in memory can do. A naked message belongs to the server and comes back
// whatever happened meanwhile, so the question "is this the past?" is answered by what the message
// says instead — the digest of the declaration a report is about (window.go, design 25 §3).
func (m *natsControl) About(string) {}
// Answer publishes to the reply subject the request carries **in its payload**.
//
// Not `Respond`, and not the message's reply field: a message a JetStream consumer delivers has had
+188 -5
View File
@@ -4,6 +4,8 @@ import (
"context"
"encoding/json"
"errors"
"io"
"log"
"os"
"sync"
"testing"
@@ -84,14 +86,15 @@ type counted struct {
heard []Report
}
func (c *counted) Heard(_ context.Context, r Report) error {
func (c *counted) Heard(_ context.Context, r Report) (bool, error) {
c.mu.Lock()
defer c.mu.Unlock()
if c.err != nil {
return c.err
return false, c.err
}
c.heard = append(c.heard, r)
return nil
// News, so what the mesh states about a report is exercised wherever a report is.
return true, nil
}
func (c *counted) refusing(err error) {
@@ -120,6 +123,8 @@ func eventually(t *testing.T, what string, is func() bool) {
// A report published by a node reaches the controller, is recorded, and is acknowledged — so the
// stream does not hold it. A work queue is the check: what is acknowledged leaves it.
func quiet() *log.Logger { return log.New(io.Discard, "", 0) }
func TestNatsAReportIsHeardAndLeavesTheStream(t *testing.T) {
js := aBus(t)
store := &counted{}
@@ -203,11 +208,11 @@ type sentAndHeardSafely struct {
heard []Report
}
func (s *sentAndHeardSafely) Heard(_ context.Context, r Report) error {
func (s *sentAndHeardSafely) Heard(_ context.Context, r Report) (bool, error) {
s.mu.Lock()
defer s.mu.Unlock()
s.heard = append(s.heard, r)
return nil
return true, nil
}
func (s *sentAndHeardSafely) Outstanding(context.Context, string) (string, error) {
@@ -374,3 +379,181 @@ func TestNatsAReportIsLetGoOnceTheStoreHasBeenGoneTooLong(t *testing.T) {
t.Fatalf("a report was recorded by a store that never came back")
}
}
// A merge announcement is not what these tests are about; taken and forgotten.
func (t *toldAbout) SourceMoved(context.Context, SourceMoved) error { return nil }
// Every subject the controller follows decodes to a kind, and decoding never reaches past the
// list: the day the list was three long and the decoder named a fourth, every message panicked
// the control plane (2026-09-28).
func TestEverySubjectTheControllerFollowsDecodesToAKind(t *testing.T) {
for _, subject := range broker.ControllerFollows {
if _, ok := kindOfSubject(subject); !ok {
t.Errorf("%s is followed and decodes to nothing", subject)
}
}
if _, ok := kindOfSubject("mesh.mod.nobody.event.nothing"); ok {
t.Error("a subject nobody follows decoded to a kind")
}
}
// **A handler slower than the acknowledgement window is not handed its message again.**
//
// The bus waits a fixed time to be told a message was taken and then redelivers, which is right for
// a consumer that died and wrong for one that is busy. Acting on a merge builds modules — minutes
// against a thirty-second window — and the same merge was handed over five times while the first
// build was still running (2026-09-28). Here the window is two seconds and the work takes six.
func TestNatsWorkSlowerThanTheWindowIsNotHandedOverAgain(t *testing.T) {
js := aBus(t)
// The controller's own consumer, with a window short enough to outlive in a test.
if err := js.EnsureConsumer(broker.Consumer{
Name: broker.ControllerName, Stream: "CONTROL", Push: true, AckWaitSeconds: 2,
Why: "a window short enough to outlive in a test",
}); err != nil {
t.Fatal(err)
}
var mu sync.Mutex
handled := 0
slow := make(chan struct{})
s, stop := servingOn(t, js, nil)
defer stop()
s.listener = slowly{func() {
mu.Lock()
handled++
first := handled == 1
mu.Unlock()
if first {
time.Sleep(6 * time.Second)
close(slow)
}
}}
body, err := json.Marshal(Report{Node: "anchor", Declared: "d1", Applied: []string{"store"}})
if err != nil {
t.Fatal(err)
}
if _, err := js.Context().Publish(ReportSubject("anchor"), body); err != nil {
t.Fatal(err)
}
select {
case <-slow:
case <-time.After(30 * time.Second):
t.Fatal("the slow work never finished")
}
// A moment for a redelivery to arrive, if the bus were going to send one.
time.Sleep(3 * time.Second)
mu.Lock()
defer mu.Unlock()
if handled != 1 {
t.Fatalf("one report was handled %d times, so slow work is run again while it is running", handled)
}
}
// slowly is a listener that runs whatever it was given.
type slowly struct{ work func() }
func (s slowly) Heard(context.Context, Report) (bool, error) { s.work(); return true, nil }
// **The mesh says what it applied** (novox/hq ADR 0134), under the seat the control plane holds — and
// says nothing when a report is the same state said again, which is what a machine reconciling every
// minute sends.
func TestNatsTheMeshSaysWhatAMachineApplied(t *testing.T) {
js := aBus(t)
heard := make(chan *nats.Msg, 4)
sub, err := js.Conn().Subscribe(SeatEventSubject(MeshControllerSeat, ">"), func(m *nats.Msg) {
heard <- m
})
if err != nil {
t.Fatal(err)
}
defer sub.Unsubscribe() //nolint:errcheck // the subscription dies with the connection
_, stop := servingOn(t, js, &counted{})
defer stop()
// A report that changed something: the store says it was news.
body, err := json.Marshal(Report{Node: "anchor", Declared: "d1", Applied: []string{"store", "broker"}})
if err != nil {
t.Fatal(err)
}
if _, err := js.Context().Publish(ReportSubject("anchor"), body); err != nil {
t.Fatal(err)
}
select {
case m := <-heard:
if m.Subject != SeatEventSubject(MeshControllerSeat, KeyApplied) {
t.Fatalf("the mesh stated %q", m.Subject)
}
var said Applied
if err := json.Unmarshal(m.Data, &said); err != nil {
t.Fatal(err)
}
if said.Node != "anchor" || said.Declared != "d1" || said.Resources != 2 {
t.Fatalf("it said %+v", said)
}
case <-time.After(10 * time.Second):
t.Fatal("the mesh said nothing about a machine that now runs something else")
}
// A refusal is its own fact, with the reason in it rather than only in a log.
refusal, err := json.Marshal(Report{Node: "anchor", Declared: "d2",
Failed: map[string]string{"gitea.server": "no such image"}})
if err != nil {
t.Fatal(err)
}
if _, err := js.Context().Publish(ReportSubject("anchor"), refusal); err != nil {
t.Fatal(err)
}
select {
case m := <-heard:
if m.Subject != SeatEventSubject(MeshControllerSeat, KeyRefused) {
t.Fatalf("a refusal was stated as %q", m.Subject)
}
var said Refused
if err := json.Unmarshal(m.Data, &said); err != nil {
t.Fatal(err)
}
if said.Failed["gitea.server"] == "" {
t.Fatalf("the refusal does not say which resource or why: %+v", said)
}
case <-time.After(10 * time.Second):
t.Fatal("the mesh said nothing about a machine that refused what it was sent")
}
}
// And a report that is not news is not a fact. A machine reconciles every minute; a fact per report
// would be a fact per minute per machine, which is a stream nobody reads.
func TestNatsAReportThatIsNotNewsIsNotStated(t *testing.T) {
js := aBus(t)
heard := make(chan *nats.Msg, 4)
sub, err := js.Conn().Subscribe(SeatEventSubject(MeshControllerSeat, ">"), func(m *nats.Msg) {
heard <- m
})
if err != nil {
t.Fatal(err)
}
defer sub.Unsubscribe() //nolint:errcheck // the subscription dies with the connection
// A store that records the report and says it was nothing new — which is what the mesh's own
// store says about a machine repeating itself.
_, stop := servingOn(t, js, sameAgain{})
defer stop()
body, err := json.Marshal(Report{Node: "anchor", Declared: "d1", Applied: []string{"store"}})
if err != nil {
t.Fatal(err)
}
if _, err := js.Context().Publish(ReportSubject("anchor"), body); err != nil {
t.Fatal(err)
}
select {
case m := <-heard:
t.Fatalf("the mesh stated %q about a machine that changed nothing", m.Subject)
case <-time.After(3 * time.Second):
}
}
// sameAgain records a report and says it was the same state said again.
type sameAgain struct{}
func (sameAgain) Heard(context.Context, Report) (bool, error) { return false, nil }
+4 -4
View File
@@ -60,7 +60,7 @@ func TestASignedRekeyMovesTheHubOntoItsTunnel(t *testing.T) {
rekey := &link.Rekey{Previous: ownKey, OverlayKey: tunnelKey, Tunnel: theTunnel()}
rekey.Proof = ed25519.Sign(private, link.RekeyProof("anchor", ownKey, tunnelKey, theTunnel()))
if err := e.Heard(ctx, link.Report{Node: "anchor", Rekey: rekey}); err != nil {
if _, err := e.Heard(ctx, link.Report{Node: "anchor", Rekey: rekey}); err != nil {
t.Fatal(err)
}
placed, err := e.Inventory.Overlays(ctx)
@@ -77,7 +77,7 @@ func TestASignedRekeyMovesTheHubOntoItsTunnel(t *testing.T) {
_ = hub
// Replayed, it is stale: the previous key it names is no longer the node's.
err = e.Heard(ctx, link.Report{Node: "anchor", Rekey: rekey})
_, err = e.Heard(ctx, link.Report{Node: "anchor", Rekey: rekey})
if err == nil || !strings.Contains(err.Error(), "previous overlay key") {
t.Fatalf("a replayed rekey was accepted: %v", err)
}
@@ -93,7 +93,7 @@ func TestARekeySignedByAnotherKeyIsRefusedAndChangesNothing(t *testing.T) {
rekey := &link.Rekey{Previous: ownKey, OverlayKey: tunnelKey, Tunnel: theTunnel()}
rekey.Proof = ed25519.Sign(stranger, link.RekeyProof("anchor", ownKey, tunnelKey, theTunnel()))
err = e.Heard(ctx, link.Report{Node: "anchor", Rekey: rekey})
_, err = e.Heard(ctx, link.Report{Node: "anchor", Rekey: rekey})
if err == nil || !strings.Contains(err.Error(), "not signed by anchor's identity key") {
t.Fatalf("a rekey signed by a stranger was accepted: %v", err)
}
@@ -111,7 +111,7 @@ func TestARekeySignedByAnotherKeyIsRefusedAndChangesNothing(t *testing.T) {
other := theTunnel()
other.Port = 51820
moved.Proof = ed25519.Sign(mustPrivate(t, e, "anchor"), link.RekeyProof("anchor", ownKey, tunnelKey, other))
if err := e.Heard(ctx, link.Report{Node: "anchor", Rekey: moved}); err == nil {
if _, err := e.Heard(ctx, link.Report{Node: "anchor", Rekey: moved}); err == nil {
t.Fatal("a proof over another tunnel was accepted")
}
}
+2 -20
View File
@@ -9,12 +9,12 @@ import (
type heardWith struct{ err error }
func (h heardWith) Heard(context.Context, Report) error { return h.err }
func (h heardWith) Heard(context.Context, Report) (bool, error) { return h.err == nil, h.err }
// switchable answers with whatever it is set to — the store away, then back.
type switchable struct{ err error }
func (h *switchable) Heard(context.Context, Report) error { return h.err }
func (h *switchable) Heard(context.Context, Report) (bool, error) { return h.err == nil, h.err }
func aReport(node, declared string) Report {
return Report{Node: node, Declared: declared, Applied: []string{"store"}}
@@ -46,24 +46,6 @@ func TestAReportTheStoreCouldNotTakeIsHeldAndOneItRefusedIsNot(t *testing.T) {
}
}
// A newer report from the same node supersedes one of its reports still held: recorded after the
// newer, the older would overwrite what the node is doing now.
func TestANewerReportSupersedesAHeldOneFromTheSameNode(t *testing.T) {
s, in := serving()
s.listener = heardWith{err: errors.Join(ErrTryAgain, errors.New("starting up"))}
older, newer, other := &settled{}, &settled{}, &settled{}
s.act(context.Background(), in.sends(t, older, KindReport, aReport("anchor", "d1")))
s.act(context.Background(), in.sends(t, other, KindReport, aReport("laptop", "d7")))
s.act(context.Background(), in.sends(t, newer, KindReport, aReport("anchor", "d2")))
if !older.acked {
t.Fatalf("the older report was not set aside by the newer: %+v", older)
}
if !newer.unsettled() || !other.unsettled() || len(in.held) != 2 {
t.Fatalf("the newer report and another node's were not both held: newer %+v other %+v, %d held",
newer, other, len(in.held))
}
}
// A store that has not come back within the bound is not restarting: the report is let go, loudly,
// rather than held for ever.
func TestAReportIsLetGoOnceTheStoreHasBeenGoneTooLong(t *testing.T) {
+98 -85
View File
@@ -7,13 +7,10 @@ import (
"encoding/json"
"errors"
"fmt"
"github.com/novox/mesh-controller/internal/broker"
"log"
"os"
"time"
amqp "github.com/rabbitmq/amqp091-go"
"github.com/novox/mesh-controller/internal/envfile"
)
// AMQPVar is the controller's own connection to the bus the mesh runs on today.
@@ -32,7 +29,12 @@ type Enroller interface {
// Listener is what the controller does with a report. Separate from Enroller so the two can be
// given independently, and so a server that only sends declarations needs neither.
type Listener interface {
Heard(ctx context.Context, report Report) error
// Heard records what a node said, and says whether it was **news** — a machine that now runs
// something else, or refuses something it did not refuse before. A machine reconciles
// continuously and reports each time, so what is news is the store's answer rather than the
// bus's: only this side has the previous report to compare with. What the mesh states about it
// is the server's (novox/hq ADR 0134).
Heard(ctx context.Context, report Report) (news bool, err error)
}
// Recorder keeps what builders say.
@@ -57,6 +59,8 @@ type Upgrader interface {
// stop every upgrade behind it — except the store unreachable for the moment, which is asked
// again for a bounded time (novox/hq issue 083).
Upgraded(ctx context.Context, u Upgraded) error
// SourceMoved is a merge on the forge: build what that source produces, bases first.
SourceMoved(ctx context.Context, m SourceMoved) error
}
// Server acts on what nodes and modules say.
@@ -68,8 +72,7 @@ type Upgrader interface {
type Server struct {
inbound Inbound
bus Bus
conn *amqp.Connection
channel *amqp.Channel
js *broker.JetStream
enroller Enroller
listener Listener
@@ -99,6 +102,9 @@ func (s *Server) Records(r Recorder) { s.recorder = r }
// Follows says what to do about upgrades, and asks for them to be delivered.
func (s *Server) Follows(u Upgrader) error {
if err := s.inbound.Also(KindSourceMoved); err != nil {
return err
}
if err := s.inbound.Also(KindModuleMoved); err != nil {
return err
}
@@ -120,82 +126,20 @@ func (s *Server) Answers(r Replayer) error {
// On the port MESH_BROKER_AMQP_PORT names when the node's settings moved the broker (novox/hq
// 04-ISSUES/102) — the URL is genesis's, sealed, and its port is the one thing in it the node may
// have moved since.
func Connect(enroller Enroller, listener Listener) (*Server, error) {
url, err := envfile.Placed(AMQPVar)
if err != nil {
return nil, err
}
if url == "" {
return nil, fmt.Errorf(
"this control plane has no %s, so it cannot reach its broker. Nodes talk to it over "+
"the broker and nowhere else, so without this it can hold records and answer "+
"nothing", AMQPVar)
}
conn, err := amqp.Dial(url)
if err != nil {
// Not quoted back: the URL carries the controller's own bus password.
return nil, fmt.Errorf("cannot reach the broker named in %s: %w", AMQPVar, err)
}
channel, err := conn.Channel()
if err != nil {
conn.Close()
return nil, err
}
// Declared here rather than assumed. The controller is the only thing that may create them — a
// node's account can write to this exchange and read its own queue, and configure nothing
// else, so a node arriving before the controller has ever run finds nothing and says so,
// rather than quietly creating a topology nobody designed.
if err := channel.ExchangeDeclare(Exchange, "direct", true, false, false, false, nil); err != nil {
conn.Close()
return nil, fmt.Errorf("cannot declare the %s exchange: %w", Exchange, err)
}
// The events exchange too. The controller is not the only publisher on it — modules announce
// onto it with their own accounts — but it is the only thing permitted to create it, for the
// same reason it is the only thing permitted to create the direct one.
if err := channel.ExchangeDeclare(EventsExchange, "topic", true, false, false, false, nil); err != nil {
conn.Close()
return nil, fmt.Errorf("cannot declare the %s exchange: %w", EventsExchange, err)
}
if _, err := channel.QueueDeclare(ControlQueue, true, false, false, false, nil); err != nil {
conn.Close()
return nil, fmt.Errorf("cannot declare the %s queue: %w", ControlQueue, err)
}
// Every key a node may publish. Binding one and forgetting another is a message the bus
// accepts, finds no queue for, and drops — the publisher sees success and the consumer sees
// nothing. That is exactly what happened to reports: `report` was left unbound while `enrol`
// worked, so nodes announced what they had applied into a void for an afternoon.
for _, key := range []string{KeyEnrol, KeyReport, KeyAlive, KeyBuilt} {
if err := channel.QueueBind(ControlQueue, key, Exchange, false, nil); err != nil {
conn.Close()
return nil, fmt.Errorf("cannot bind %s to %s/%s: %w", ControlQueue, Exchange, key, err)
}
}
return &Server{
inbound: Current(conn, channel),
bus: OverCurrent{Channel: channel},
conn: conn,
channel: channel,
enroller: enroller,
listener: listener,
log: log.New(os.Stdout, "", log.LstdFlags),
}, nil
// Connect is ConnectNats: the mesh has one bus (novox/hq ADR 0131, design 28 task 5.5). Kept as
// the name callers know; the transport it opened before is gone with the bus it spoke to.
func Connect(js *broker.JetStream, enroller Enroller, listener Listener) *Server {
return ConnectNats(js, enroller, listener)
}
// Channel is the controller's channel, for the command line's own publishing.
func (s *Server) Channel() *amqp.Channel { return s.channel }
func newLog() *log.Logger { return log.New(os.Stdout, "", log.LstdFlags) }
func (s *Server) Close() {
if s.inbound != nil {
s.inbound.Close()
}
if s.channel != nil {
_ = s.channel.Close()
}
if s.conn != nil {
_ = s.conn.Close()
if s.js != nil {
s.js.Close()
}
}
@@ -225,6 +169,8 @@ func (s *Server) act(ctx context.Context, m Control) {
s.wasBuilt(ctx, m)
case KindModuleMoved:
s.moved(ctx, m)
case KindSourceMoved:
s.sourceMoved(ctx, m)
case KindCatchUp:
s.catchingUp(ctx, m)
default:
@@ -316,7 +262,7 @@ func (s *Server) heartbeat(m Control) {
return
}
if s.listener != nil {
if err := s.listener.Heard(context.Background(), Report{Node: alive.Node}); err != nil {
if _, err := s.listener.Heard(context.Background(), Report{Node: alive.Node}); err != nil {
s.log.Printf("could not record that %s is here: %v", alive.Node, err)
}
}
@@ -335,7 +281,6 @@ func (s *Server) reported(ctx context.Context, m Control) {
_ = m.Drop()
return
}
m.About("report " + report.Node)
what := fmt.Sprintf("%s's report of declaration %s", report.Node, report.Declared)
if s.listener != nil {
@@ -351,7 +296,7 @@ func (s *Server) reported(ctx context.Context, m Control) {
return
}
err := s.listener.Heard(context.Background(), report)
news, err := s.listener.Heard(context.Background(), report)
switch s.decide(ctx, m, what, declaredIn, outstanding, err) {
case Hold:
// Held, not settled, while the store cannot take it: the node reports an apply once,
@@ -367,6 +312,13 @@ func (s *Server) reported(ctx context.Context, m Control) {
// node whose recovery copy is silently older than it looks.
s.log.Printf("could not record %s's report: %v", report.Node, err)
}
// **And the mesh says what it did** (novox/hq ADR 0134). Only when the report was news: a
// machine reports every convergence, and a fact per report would be a fact per minute per
// machine saying nothing. Stated after it is recorded, so nothing is announced that the
// mesh does not hold.
if err == nil && news {
s.saysWhatItDid(ctx, report)
}
}
switch {
@@ -468,9 +420,6 @@ func (s *Server) wasBuilt(ctx context.Context, m Control) {
_ = m.Drop()
return
}
// Each build result its own subject: none supersedes another, and recording one twice is
// harmless — the build is kept by its id.
m.About("build " + digest(m.Body()))
err := s.recorder.Built(ctx, result)
switch s.decide(ctx, m, fmt.Sprintf("a build result from %s", result.On), "", "", err) {
case Hold:
@@ -502,7 +451,6 @@ func (s *Server) wasBuilt(ctx context.Context, m Control) {
// the catalogue next restarts (issue 083). One request stands for all: a newer one supersedes one
// still held.
func (s *Server) catchingUp(ctx context.Context, m Control) {
m.About("catch-up")
if s.replayer == nil {
s.log.Printf("a catalogue asked to catch up and this control plane has nothing to replay")
_ = m.Took()
@@ -524,7 +472,19 @@ func (s *Server) catchingUp(ctx context.Context, m Control) {
sent := 0
for _, a := range announcements {
a.Replay = true
if err := EmitEvent(ctx, s.bus, KeyModuleBuilt, "control-plane", "", a); err != nil {
// Under the control plane's own seat (novox/hq ADR 0134). It used to be published as a
// module's event from a module called "control-plane", which does not exist — so the
// controller's own account refused it, every catalogue that asked what it missed was
// answered with nothing, and its graph kept the gap (found 2026-09-28).
body, err := json.Marshal(a)
if err != nil {
// A body that cannot be written is this program's fault, not the bus's, and publishing
// an empty one would put a fact on the mesh that says nothing.
s.log.Printf("cannot re-announce %s at %s: %v", a.Module, short(a.Commit), err)
_ = m.Took()
return
}
if err := s.bus.PublishSeatEvent(ctx, MeshControllerSeat, KeyBuiltBefore, body); err != nil {
// Said and abandoned rather than retried: the catalogue asks again every time it
// starts, and half a graph delivered twice is no better than half delivered once.
s.log.Printf("replaying %s at %s failed, and the rest is abandoned: %v",
@@ -555,7 +515,6 @@ func (s *Server) moved(ctx context.Context, m Control) {
_ = m.Took()
return
}
m.About("upgrade " + u.Module)
if u.Module == "" {
s.log.Printf("an upgrade announcement named no module; ignored")
_ = m.Took()
@@ -596,3 +555,57 @@ func short(commit string) string {
}
return commit
}
// sourceMoved acts on the forge's announcement of a merge. Taken whatever happens: a build that
// fails is reported by the build itself, and re-delivering the merge would only re-fail it.
func (s *Server) sourceMoved(ctx context.Context, m Control) {
var moved SourceMoved
if err := json.Unmarshal(m.Body(), &moved); err != nil {
s.log.Printf("a merge announcement could not be read: %v", err)
_ = m.Took()
return
}
if moved.Commit == "" || moved.Repo == "" {
s.log.Printf("a merge announcement named no repository or no commit; ignored")
_ = m.Took()
return
}
if err := s.upgrader.SourceMoved(ctx, moved); err != nil {
s.log.Printf("%s/%s moved to %.8s and the mesh could not act on it: %v", moved.Owner, moved.Repo, moved.Commit, err)
}
_ = m.Took()
}
// saysWhatItDid states what a machine now runs, or what it would not take, as a fact on the bus
// (novox/hq ADR 0134).
//
// **The control plane speaks, as the holder of its seat.** A node's report is control traffic only
// this process may read, so the chain from a merge to a machine went dark exactly where it touched
// one: nothing said which version a machine runs, or that it refused to. The facts are second-hand
// on purpose — one emitter, one ordering — and a machine that cannot reach the bus produces none, so
// absence is not health.
//
// A failure to state a fact is logged and nothing else: the report is recorded, which is the part
// that must not be lost, and the next change says the same thing again.
func (s *Server) saysWhatItDid(ctx context.Context, report Report) {
if s.bus == nil {
return
}
event, body := KeyApplied, any(Applied{
Node: report.Node, Declared: report.Declared, Resources: len(report.Applied),
})
if report.Refused != "" || len(report.Failed) > 0 {
event, body = KeyRefused, Refused{
Node: report.Node, Declared: report.Declared,
Refused: report.Refused, Failed: report.Failed,
}
}
raw, err := json.Marshal(body)
if err != nil {
s.log.Printf("could not say what %s did: %v", report.Node, err)
return
}
if err := s.bus.PublishSeatEvent(ctx, MeshControllerSeat, event, raw); err != nil {
s.log.Printf("could not say that %s %s: %v", report.Node, event, err)
}
}
+3 -3
View File
@@ -23,12 +23,12 @@ type sentAndHeard struct {
err error
}
func (s *sentAndHeard) Heard(_ context.Context, r Report) error {
func (s *sentAndHeard) Heard(_ context.Context, r Report) (bool, error) {
if s.err != nil {
return s.err
return false, s.err
}
s.heard = append(s.heard, r)
return nil
return true, nil
}
func (s *sentAndHeard) Outstanding(context.Context, string) (string, error) { return s.sent, nil }
+2 -48
View File
@@ -3,7 +3,6 @@ package link
import (
"context"
"errors"
"fmt"
"testing"
"github.com/jackc/pgx/v5/pgconn"
@@ -18,7 +17,8 @@ func (r recordsWith) Built(context.Context, BuildResult) error { return r.err }
type upgradesWith struct{ err error }
func (u upgradesWith) Upgraded(context.Context, Upgraded) error { return u.err }
func (u upgradesWith) Upgraded(context.Context, Upgraded) error { return u.err }
func (u upgradesWith) SourceMoved(context.Context, SourceMoved) error { return u.err }
type replaysWith struct{ err error }
@@ -106,49 +106,3 @@ func TestAMessageHandledDuringShutdownIsLeftForTheBus(t *testing.T) {
t.Fatalf("a build result handled during shutdown was settled, and so lost: %+v", *to)
}
}
// Two identical build results: the newer sets the older aside rather than leaving it unsettled
// for ever, holding a place in the prefetch.
func TestAnIdenticalBuildResultSetsTheHeldOneAside(t *testing.T) {
s, in := serving()
s.recorder = recordsWith{err: restarting}
built := BuildResult{On: "anchor", Repository: "/r", Commit: "abc"}
first, second := &settled{}, &settled{}
s.act(context.Background(), in.sends(t, first, KindBuilt, built))
s.act(context.Background(), in.sends(t, second, KindBuilt, built))
if !first.acked || !second.unsettled() || len(in.held) != 1 {
t.Fatalf("an identical build result did not set the held one aside: first %+v second %+v, %d held",
*first, *second, len(in.held))
}
}
// What is held stops short of the prefetch, so the loop always has room to answer an enrolment.
func TestWhatIsHeldLeavesRoomInThePrefetch(t *testing.T) {
s, in := serving()
s.recorder = recordsWith{err: restarting}
var last *settled
for i := 0; i < Prefetch; i++ {
last = &settled{}
s.act(context.Background(), in.sends(t, last, KindBuilt, BuildResult{On: "anchor", Commit: fmt.Sprint(i)}))
}
if len(in.held) != Prefetch-PrefetchHeadroom {
t.Fatalf("%d messages were held; the ceiling is %d", len(in.held), Prefetch-PrefetchHeadroom)
}
if last.unsettled() {
t.Fatalf("a message past the ceiling was held: %+v", *last)
}
}
// An upgrade handled during shutdown is left for the bus too — the upgrader's error is the
// cancelled context, which is no answer about the announcement.
func TestAnUpgradeHandledDuringShutdownIsLeftForTheBus(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
cancel()
s, in := serving()
s.upgrader = upgradesWith{err: context.Canceled}
to := &settled{}
s.act(ctx, in.sends(t, to, KindModuleMoved, Upgraded{Module: "gitea", Commit: "abcdef0123"}))
if !to.unsettled() {
t.Fatalf("an upgrade was settled during shutdown, and so lost: %+v", *to)
}
}
+2
View File
@@ -27,6 +27,7 @@
"bus": "/var/lib/mesh/mesh-controller/bus"
},
"secrets-owner": "65534:65534",
"prepares": true,
"resources": [
{
"id": "mesh-state",
@@ -62,6 +63,7 @@
"/var/lib/mesh/mesh-controller/identity:/run/secrets/identity:ro",
"/var/lib/mesh/mesh-controller/licences:/run/secrets/licences:ro",
"/var/lib/mesh/mesh-controller/broker:/run/secrets/broker:ro",
"/var/lib/mesh/mesh-controller/bus:/run/secrets/bus:ro",
"/var/lib/mesh/mesh-controller/broker-management:/run/secrets/broker-management:ro",
"/var/lib/mesh/mesh-controller/broker-address:/run/secrets/broker-address:ro"
],