93 lines
5.1 KiB
Markdown
93 lines
5.1 KiB
Markdown
---
|
|
status: resolved
|
|
opened: 2026-10-03
|
|
located-in:
|
|
- mesh-controller
|
|
- mesh-tools
|
|
fixed-by:
|
|
- mesh-controller#248
|
|
- mesh-tools#45
|
|
- mesh-tools#46
|
|
amended-design:
|
|
---
|
|
|
|
# 218 — A seat held once for the mesh is answered by a module on a machine that does not hold it
|
|
|
|
## What was observed
|
|
|
|
2026-10-03. Asked which databases the mesh's store holds, `mesh-store.databases` answered from the
|
|
postgres on one machine with that machine's application databases; the controller's own database
|
|
lives on the postgres of the other machine, which the controller's records name as the seat's one
|
|
holder:
|
|
|
|
```
|
|
mesh-store scope: mesh delivers: postgres-database holders: [ {node: <the control machine>, module: postgres} ]
|
|
```
|
|
|
|
The discovery console, which reads what answers on the bus ([ADR 0197](../../02-DECISIONS/0197-every-tool-announces-itself-on-the-bus-in-the-nats-services-protocol.md)),
|
|
shows the same seat announced from **both** machines running postgres.
|
|
|
|
## Why it matters beyond this instance
|
|
|
|
A seat held once for the mesh promises one answerer: the role's holder. A module that implements a
|
|
seat's verbs on every machine it runs on, and is let serve them on each, turns "the mesh's store" into
|
|
"whichever postgres replied first" — a read against the wrong database that looks like a right one,
|
|
and a write would be worse. Every mesh-scoped seat whose implementing module runs on more than one
|
|
machine has this shape.
|
|
|
|
## Where to look
|
|
|
|
Whether the runtime serves a seat's verbs where its module merely *claims* the seat rather than where
|
|
the mesh made it the holder ([ADR 0159](../../02-DECISIONS/0159-a-tool-call-names-the-machine-and-a-holder-serves-its-seats-verbs.md),
|
|
[ADR 0160](../../02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md)):
|
|
the membership the controller issues each assignment, and what the runtime admits from it. A check:
|
|
a mesh-scoped seat's verbs are served by exactly the holder the records name, on every machine.
|
|
|
|
## Root cause
|
|
|
|
The controller composed each assignment's held seats from what its module *claims*, once per module
|
|
and not once per machine. Every machine running postgres was therefore given the store seat's grants
|
|
and issued its subjects, and each runtime served the seat's verbs because it serves what it is issued
|
|
([ADR 0160](../../02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md)).
|
|
The runtime behaved as designed. The fault was in what it was issued.
|
|
|
|
The seat's verbs are not the module's tools. The store's `databases` and `query` are a separate
|
|
implementation registered under the seat's name ([ADR 0159](../../02-DECISIONS/0159-a-tool-call-names-the-machine-and-a-holder-serves-its-seats-verbs.md)).
|
|
Only that implementation should be withdrawn where the module does not hold the seat. postgres's own
|
|
tools stay served on every machine it runs on.
|
|
|
|
## Fix
|
|
|
|
The controller now reads the recorded seat holdings when it composes grants and memberships. A seat
|
|
held once for the mesh is issued only to the machine and module the records name as its holder. A
|
|
seat held once per machine, and a mesh seat with no holder on record, are issued as before. Grants
|
|
and memberships come from the same list, so they cannot disagree.
|
|
|
|
**How it is checked.** A controller test asserts that a claimant on another machine keeps its node
|
|
seats and loses the recorded mesh seat. Live, the discovery console's overview must show each
|
|
mesh-scoped seat announced from exactly the holder the records name. Status moves to `resolved` once
|
|
that holds after the fix is rolled out.
|
|
|
|
## A second cause, and what the rollout broke (2026-10-04)
|
|
|
|
With the grants corrected, calls to the store reached only the holder, yet the console still showed
|
|
the seat announced from both machines. The module's runtime added every seat its start-up credential
|
|
claims, even after the mesh issued a membership that left the seat out. Once a membership exists,
|
|
it now alone decides which seat verbs a runtime serves (mesh-tools#45).
|
|
|
|
The rollout then exposed a third fault. The module on the machine that does not hold the seat was
|
|
still running an image built before #45, so it subscribed to the seat's subject. The corrected grants
|
|
refused that subscription, and the refusal ended the process. Its runtime crash-looped until the
|
|
module was rebuilt on the new runtime image. The database itself kept running. A refused tool
|
|
subscription is now logged and costs only that subject (mesh-tools#46), as a refused announcement
|
|
already did ([issue 217](../217-a-refused-announcement-took-down-every-containers-runtime/00-report.md)).
|
|
|
|
The module was not rebuilt by the plan that rebuilt the runtime image. This is the ordering gap of
|
|
[issue 211](../211-a-bundle-is-built-before-the-toolchain-it-is-compiled-in/00-report.md) seen
|
|
from a container module.
|
|
|
|
**Proven 2026-10-04.** The discovery console's overview shows the store seat announced from the
|
|
recorded holder only. Repeated calls to the store are answered by that machine, and the answers
|
|
include the controller's own database. The non-holder still answers its own module tools. No
|
|
container restarts on any machine.
|