Building the bus: the decisions the work needed, and what it taught back #150
+16
-2
@@ -31,8 +31,22 @@ another — and a mesh you cannot name precisely is a mesh two people describe d
|
|||||||
- **store** — the one postgres server. It holds the controller's own context databases
|
- **store** — the one postgres server. It holds the controller's own context databases
|
||||||
(`inventory`, `identity`, `licences` — a context owns its store, [ADR 0008](../02-DECISIONS/0008-a-context-owns-its-store.md))
|
(`inventory`, `identity`, `licences` — a context owns its store, [ADR 0008](../02-DECISIONS/0008-a-context-owns-its-store.md))
|
||||||
and every module's own database. One server, many databases — never one shared "mesh database".
|
and every module's own database. One server, many databases — never one shared "mesh database".
|
||||||
- **broker** — the one lavinmq message bus. It carries the mesh bus on the `/` vhost and a vhost per
|
- **bus** — the mesh's own nervous system: NATS, one per mesh, carrying every link the mesh has —
|
||||||
consumer that requires `amqp`.
|
control, declarations, builds, events, tool calls
|
||||||
|
([ADR 0106](../02-DECISIONS/0106-the-bus-is-nats.md)). A module reaches it by requiring
|
||||||
|
`mesh-bus` ([ADR 0120](../02-DECISIONS/0120-the-mesh-bus-is-required-not-ambient.md)); one that
|
||||||
|
does not require it has no account on it. Held by the `mesh-broker` seat, which is named after
|
||||||
|
the *role* rather than the server, so the server can change without the seat doing so.
|
||||||
|
- **the deprecated broker** — the lavinmq module. It was the mesh's bus and is not any more. It
|
||||||
|
keeps running as an **ordinary provider** of the `amqp` provision, for modules that need a
|
||||||
|
message broker of their own the way something needs a database
|
||||||
|
([ADR 0119](../02-DECISIONS/0119-amqp-is-a-provision-not-the-bus.md)) — no seat, not foundation,
|
||||||
|
never raised at genesis, and a mesh that never installs it is complete.
|
||||||
|
|
||||||
|
Say *the deprecated broker*, not "the compatibility broker" (it serves the mesh's own modules,
|
||||||
|
not only the predecessor's) and not "the AMQP broker" (naming it after a protocol invites
|
||||||
|
describing the bus by contrast with it, which is backwards: the bus is the mesh's nervous
|
||||||
|
system and this is a module).
|
||||||
|
|
||||||
## What the mesh stores and serves
|
## What the mesh stores and serves
|
||||||
|
|
||||||
|
|||||||
@@ -30,19 +30,37 @@ Prose and diagrams only; no configuration is pasted.
|
|||||||
|
|
||||||
## 1. What the bus is for
|
## 1. What the bus is for
|
||||||
|
|
||||||
The bus carries five kinds of traffic today, and this design keeps the five, renaming nothing a
|
**The bus is where the mesh happens.** Not a transport the mesh sends things over — the place a
|
||||||
module can see:
|
module is reachable at all, where a role is addressed without knowing who holds it, where the
|
||||||
|
mesh's own state lives, and where what a module may say is decided by what it declared.
|
||||||
|
|
||||||
| Traffic | Today | Guarantee it needs |
|
| Traffic | Shape | Guarantee it needs |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| **control** — a node's report, its heartbeat, a build's outcome, an enrolment | queues `control`, `.upgrades`, `.catchup` | nothing lost while the store restarts; retried; in order per node |
|
| **control** — a node's report, a build's outcome, an enrolment | job | nothing lost while the store restarts; retried; in order per node |
|
||||||
| **declarations** — the controller tells a node what to be | queue `node.<name>` | the node gets the newest; a stale one is never applied |
|
| **heartbeat** — a node saying it is alive | fire and forget | none; a lost one is the next one |
|
||||||
| **builds** — the controller asks the build machine to build | queue `builds` | at least once, one builder at a time |
|
| **declarations** — the controller tells a node what to be | state | the node gets the newest; a stale one is never applied |
|
||||||
| **events** — a module says something happened | topic exchange `mesh.events`, keys `<module>.<event>` | delivered to every consumer that declared it; dead-lettered when it cannot be |
|
| **builds** — work for the build machine | job | at least once, one worker at a time |
|
||||||
| **tools** — one module or person asks another's tool a question | exchange `mesh.rpc`, per-tool service queues `serve.<module>.<tool>` | one answer, from one server, or a timeout |
|
| **events** — a module says something happened | 1:many | delivered to every consumer that declared it; dead-lettered when it cannot be |
|
||||||
|
| **tools** — a module or a person asks another's tool | request/reply | one answer, from one server, or a timeout |
|
||||||
|
| **work to a role** — a module submits to a capability without knowing who provides it | job | exactly one holder does it; it queues while nobody does |
|
||||||
|
|
||||||
The sdk's contract — `request`, `handle`, `publish`, `subscribe`, `close` — is the whole surface a
|
The last two rows are the ones worth dwelling on, because they are not messaging in the sense of
|
||||||
module sees, and it does not change ([ADR 0039](../../02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md)).
|
carrying bytes from A to B. **A role is addressable**, so a caller names the capability and never
|
||||||
|
the module or the node — and the implementation can be replaced under it without a caller
|
||||||
|
changing ([ADR 0118](../../02-DECISIONS/0118-a-module-declares-its-own-seats.md)). That is a
|
||||||
|
property of the mesh's architecture that happens to be expressed in subjects.
|
||||||
|
|
||||||
|
And more of the mesh lands here as it is built: conditions and observed state in key-value
|
||||||
|
buckets that anything may watch, the server's own advisories becoming observations like any other
|
||||||
|
([research 017](../../01-RESEARCH/017-a-mesh-that-heals-itself/00-overview.md)), and a person's
|
||||||
|
client speaking the bus directly rather than through a surface built over it (§7). None of that
|
||||||
|
is a message being moved; all of it is the bus being the mesh's centre.
|
||||||
|
|
||||||
|
**What a module sees of it is small and derived.** It declares what it emits, consumes, serves
|
||||||
|
and uses, and the subjects, streams, consumers and permissions all follow from that
|
||||||
|
([design 29](29-what-a-module-declares.md)). The sdk's contract — `request`, `handle`, `publish`,
|
||||||
|
`subscribe`, `close` — is the whole surface, and it does not change
|
||||||
|
([ADR 0039](../../02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md)).
|
||||||
|
|
||||||
## 2. Subjects
|
## 2. Subjects
|
||||||
|
|
||||||
@@ -212,11 +230,11 @@ same place `modules/gitea/token.ts` keeps its own state rather than asking the h
|
|||||||
The host's only job is what it already does for any directory resource: keep the file's content
|
The host's only job is what it already does for any directory resource: keep the file's content
|
||||||
current. Nothing is declared as `reload-on` or `restart-on` for this resource at all.
|
current. Nothing is declared as `reload-on` or `restart-on` for this resource at all.
|
||||||
|
|
||||||
Its guard is the same rule as the AMQP broker's: the monitoring port is refused from anything but
|
Its guard is the same rule as the deprecated broker's: the monitoring port is refused from anything but
|
||||||
the private network. It is raised at genesis like the store, adopted as a module in the same
|
the private network. It is raised at genesis like the store, adopted as a module in the same
|
||||||
phase.
|
phase.
|
||||||
|
|
||||||
**The AMQP broker is an ordinary module, not a compatibility layer.** Revision, second review
|
**The deprecated broker is an ordinary module, not a compatibility layer.** Revision, second review
|
||||||
([ADR 0119](../../02-DECISIONS/0119-amqp-is-a-provision-not-the-bus.md)): earlier text here, and
|
([ADR 0119](../../02-DECISIONS/0119-amqp-is-a-provision-not-the-bus.md)): earlier text here, and
|
||||||
ADR 0106 before it, called it `lavinmq-compat` — one purpose, the predecessor's clients, and a
|
ADR 0106 before it, called it `lavinmq-compat` — one purpose, the predecessor's clients, and a
|
||||||
retirement condition of no client connected for a period the operator sets. It is none of those.
|
retirement condition of no client connected for a period the operator sets. It is none of those.
|
||||||
@@ -231,17 +249,6 @@ that remains is about direction rather than software: *inter-module communicatio
|
|||||||
bus.* A module may hold a broker, a database or a cache for itself; it may not use one as a
|
bus.* A module may hold a broker, a database or a cache for itself; it may not use one as a
|
||||||
channel to another module.
|
channel to another module.
|
||||||
|
|
||||||
**And with one purpose, which it did not have when this was written.** Revision, second review
|
|
||||||
([ADR 0119](../../02-DECISIONS/0119-amqp-is-a-provision-not-the-bus.md) (superseding [ADR 0117](../../02-DECISIONS/0117-the-bus-is-the-only-broker.md))): two modules of the new mesh
|
|
||||||
required a broker *of their own* from it — a private vhost per consumer, the analog of
|
|
||||||
database-per-login, which is a different thing from the mesh's bus and was never examined here.
|
|
||||||
**There is one bus, and no module is handed a broker as a resource.** A module's messaging is
|
|
||||||
subjects on the bus under its own account, scoped by its `emits` and `consumes`; a module that
|
|
||||||
wants a queue of its own has a subject nothing else may publish to and a durable consumer, both
|
|
||||||
from its declaration. What it does not get is a server of its own. The `amqp` interface retires
|
|
||||||
with the compatibility broker instead of gaining a successor, and the two modules move to the bus
|
|
||||||
in step 4.
|
|
||||||
|
|
||||||
## 6. Joining: the enrolment handshake
|
## 6. Joining: the enrolment handshake
|
||||||
|
|
||||||
Unchanged in shape, changed in transport. A node that has a token connects to the bus over TLS
|
Unchanged in shape, changed in transport. A node that has a token connects to the bus over TLS
|
||||||
@@ -323,7 +330,7 @@ wrong until there is a second mesh. *Ends at: the genesis bed.*
|
|||||||
**Step 2 — adoption puts the broker in its seat.** A mesh already running does not get a foundation
|
**Step 2 — adoption puts the broker in its seat.** A mesh already running does not get a foundation
|
||||||
module by being raised again; it adopts one in place
|
module by being raised again; it adopts one in place
|
||||||
([ADR 0100](../../02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md)). The
|
([ADR 0100](../../02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md)). The
|
||||||
server is raised beside the AMQP broker on its own ports, carrying no mesh traffic yet, and the
|
server is raised beside the deprecated broker on its own ports, carrying no mesh traffic yet, and the
|
||||||
`nats` module is adopted onto it. The seat it claims is **`mesh-broker`**, unchanged — the
|
`nats` module is adopted onto it. The seat it claims is **`mesh-broker`**, unchanged — the
|
||||||
foundation seats are named after the server's role rather than the product
|
foundation seats are named after the server's role rather than the product
|
||||||
([ADR 0079](../../02-DECISIONS/0079-the-foundation-seats-are-named-after-their-servers.md)) for
|
([ADR 0079](../../02-DECISIONS/0079-the-foundation-seats-are-named-after-their-servers.md)) for
|
||||||
@@ -384,7 +391,7 @@ itself enforces and a plain client can therefore check:
|
|||||||
watcher-poll interval, without a restart.
|
watcher-poll interval, without a restart.
|
||||||
|
|
||||||
**Step 2 — the adoption bed**, a mesh already running that has never had this server:
|
**Step 2 — the adoption bed**, a mesh already running that has never had this server:
|
||||||
- the server is raised beside the AMQP broker on its own ports and the `nats` module is adopted
|
- the server is raised beside the deprecated broker on its own ports and the `nats` module is adopted
|
||||||
onto it in place, holding the data and the configuration it was raised with;
|
onto it in place, holding the data and the configuration it was raised with;
|
||||||
- the seat it claims is `mesh-broker`, and a second assignment of it anywhere in the mesh is
|
- the seat it claims is `mesh-broker`, and a second assignment of it anywhere in the mesh is
|
||||||
refused at resolution — *one per mesh*, as ADR 0079 requires;
|
refused at resolution — *one per mesh*, as ADR 0079 requires;
|
||||||
|
|||||||
@@ -148,7 +148,7 @@ paper is wrong until there is a second mesh to find out.
|
|||||||
server's role, not the product. **Already true of the controller and needed no change**: it
|
server's role, not the product. **Already true of the controller and needed no change**: it
|
||||||
resolves the broker by seat ("that is where the broker is, whatever else the topology says")
|
resolves the broker by seat ("that is where the broker is, whatever else the topology says")
|
||||||
and names no broker module anywhere in its source. What remains is naming `nats` instead of
|
and names no broker module anywhere in its source. What remains is naming `nats` instead of
|
||||||
the AMQP broker where a genesis module set is declared, which is scenario and installer
|
the deprecated broker where a genesis module set is declared, which is scenario and installer
|
||||||
configuration — carried with 1.6 rather than before it.
|
configuration — carried with 1.6 rather than before it.
|
||||||
- [ ] 1.6 the genesis-broker bed — **deferred**: beds are run once, at the end, rather than per
|
- [ ] 1.6 the genesis-broker bed — **deferred**: beds are run once, at the end, rather than per
|
||||||
step (novox/hq design 22's rule, and the operator's instruction). Every claim step 1 makes
|
step (novox/hq design 22's rule, and the operator's instruction). Every claim step 1 makes
|
||||||
@@ -188,7 +188,7 @@ a foundation module by being raised again; it adopts one in place
|
|||||||
true and now proved**: the refusal is generic to any mesh-scoped seat, and three tests pin
|
true and now proved**: the refusal is generic to any mesh-scoped seat, and three tests pin
|
||||||
what matters for this one — a second bus anywhere is refused naming the seat, a *different*
|
what matters for this one — a second bus anywhere is refused naming the seat, a *different*
|
||||||
bus implementation is refused for the same reason (which is what lets the bus be replaced
|
bus implementation is refused for the same reason (which is what lets the bus be replaced
|
||||||
at all), and the AMQP broker no longer contends for it, so both run on one mesh
|
at all), and the deprecated broker no longer contends for it, so both run on one mesh
|
||||||
- [ ] 2.4 the adoption bed — deferred with the other beds
|
- [ ] 2.4 the adoption bed — deferred with the other beds
|
||||||
|
|
||||||
> **Adoption here is not a no-op, and pretending it would be is the trap.** The host keeps an
|
> **Adoption here is not a no-op, and pretending it would be is the trap.** The host keeps an
|
||||||
@@ -344,13 +344,13 @@ reserves them for after the move, and a flow built ahead of its design would be
|
|||||||
|
|
||||||
**Why here.** It is the only step that moves a node's bus, and it moves every node's at once.
|
**Why here.** It is the only step that moves a node's bus, and it moves every node's at once.
|
||||||
|
|
||||||
- [ ] 5.1 the cutover bed: a mesh on AMQP with a predecessor stand-in on the compatibility broker
|
- [ ] 5.1 the cutover bed: a mesh on AMQP with a predecessor stand-in on the deprecated broker
|
||||||
moves its bus in one rollout, every node reporting on NATS afterwards, the stand-in's own
|
moves its bus in one rollout, every node reporting on NATS afterwards, the stand-in's own
|
||||||
client still connected throughout
|
client still connected throughout
|
||||||
- [ ] 5.2 the rollout: accounts composed, then the controller, every host and every runtime
|
- [ ] 5.2 the rollout: accounts composed, then the controller, every host and every runtime
|
||||||
together; every node confirmed heard before AMQP stops
|
together; every node confirmed heard before AMQP stops
|
||||||
- [ ] 5.3 the mesh's accounts removed from the compatibility broker, leaving the predecessor's users
|
- [ ] 5.3 the mesh's accounts removed from the deprecated broker, leaving the predecessor's users
|
||||||
- [ ] 5.4 the compatibility broker retires when its condition holds — no client connected for the
|
- [ ] 5.4 the deprecated broker retires when its condition holds — no client connected for the
|
||||||
period the operator sets
|
period the operator sets
|
||||||
|
|
||||||
**Done when.** Every node reports on NATS and the predecessor's clients never noticed.
|
**Done when.** Every node reports on NATS and the predecessor's clients never noticed.
|
||||||
|
|||||||
@@ -302,7 +302,7 @@ The mesh's own bus is **`mesh-bus`**, delivered by the `mesh-broker` seat and an
|
|||||||
controller — because the bus's accounts are configuration rather than something a provisioner
|
controller — because the bus's accounts are configuration rather than something a provisioner
|
||||||
creates, so there is no provisioner process in the path and nothing waiting on a bus account in
|
creates, so there is no provisioner process in the path and nothing waiting on a bus account in
|
||||||
order to make bus accounts. A module that runs a NATS server of its own and offers it as a
|
order to make bus accounts. A module that runs a NATS server of its own and offers it as a
|
||||||
backing service provides **`nats`**, exactly as the AMQP broker provides `amqp`
|
backing service provides **`nats`**, exactly as the deprecated broker provides `amqp`
|
||||||
([ADR 0119](../../02-DECISIONS/0119-amqp-is-a-provision-not-the-bus.md)).
|
([ADR 0119](../../02-DECISIONS/0119-amqp-is-a-provision-not-the-bus.md)).
|
||||||
|
|
||||||
They are never the same name. A manifest saying `nats` could otherwise mean either the mesh's
|
They are never the same name. A manifest saying `nats` could otherwise mean either the mesh's
|
||||||
|
|||||||
Reference in New Issue
Block a user