Author SHA1 Message Date
mesh-admin 7aa98e64ce Merge pull request 'Grant the bus's own module the snapshot API and nothing else (hq ADR 0235)' (#90) from feat/bus-snapshot into main 2026-10-06 16:24:53 +00:00
jochen 48581af35c Grant the bus's own module the snapshot API and nothing else (hq ADR 0235)
The night's backup of the bus takes each stream through JetStream's snapshot
API, run by the nats module under its own account. The module holding
mesh-broker is composed that account: stream names and info, the snapshot
request, its flow-control acks, its own inbox — no write, which the writers
table checks. A bus module declaring anything else to say on the bus is
refused by module check rather than silently granted nothing. The genesis
user list is unchanged: the controller's grants are.
2026-10-06 18:20:57 +02:00
mesh-admin 4d05385819 Merge pull request 'A machine waiting for its push is waiting, not uncomposable (hq issue 275)' (#89) from fix/d1-a-push-not-made-yet-is-pending into main 2026-10-06 16:07:17 +00:00
jochen dcee8cb5bf Say a machine waiting for its push as waiting, not uncomposable (hq issue 275)
Between assign and push a module's own secrets are not made yet; D1 composed
without making them and raised an urgent 'nothing can be sent' that the next
push resolved silently. D1 now composes as the push would (Foreseeing): a
secret the push makes gets a stand-in and is named, one the push is refused on
is refused with the push's words. Waiting is said only past 30 minutes, as a
warning. D3 and D13 expect a holder only once its machine was sent it and
reported or had ten minutes to.
2026-10-06 18:05:38 +02:00
mesh-admin 2b5060789f Merge pull request 'A module declares the data it holds; protection and D13 derived from it (hq ADR 0233)' (#88) from feat/a-module-declares-the-data-it-holds into main 2026-10-06 15:00:53 +00:00
jochen abf9125689 Measure each item its own way; never compare a partial size (hq ADR 0233)
A walk over a large library every hour loads the array that protects it. An item now says how it
is measured — a bounded daily walk, a dataset's counters, or its top level only — and a size that
is a lower bound is kept as such and never read as a shrink.
2026-10-06 17:00:22 +02:00
jochen 52af210e47 Derive data protection from a module's declared data (hq ADR 0233)
A module's data section says what it keeps and how precious it is; the backup holder's lines,
binding stickiness, retirement on unassign and D13's conditions follow from it, so issue 273's
empty replacement is said and an unassigned module's data is remembered, not forgotten.
2026-10-06 16:47:49 +02:00
mesh-admin 4b25af2aa3 Merge pull request 'Grant a provider only the consumers bound to it (hq issue 274)' (#87) from fix/a-grant-follows-the-binding into main 2026-10-06 14:09:28 +00:00
jochen fc65215c25 Grant a provider only the consumers bound to it (hq issue 274)
grantsFor granted every consumer a pair credential from the provider was
ever made for, so a consumer pinned back to its own store was still asked
of the store it left, which then never retired it. A credential whose
consumer's resolution binds it elsewhere is now withdrawn like one nobody
asks for, kept on record for the login the provider keeps, and said on
plan and push.
2026-10-06 16:07:12 +02:00
mesh-admin c988d6d7be Merge pull request 'Keep a consumer bound where its data is; only a pin moves it (hq ADR 0232, issue 273)' (#86) from fix/a-stateful-binding-moves-only-by-a-person into main 2026-10-06 13:22:42 +00:00
jochen 8bfaf1523e Keep a consumer bound where its data is; only a pin moves it (hq ADR 0232, issue 273)
Issue 258's fix let a mesh seat's holder elsewhere answer before this machine's own provider. Right
for the resolver, which any provider answers alike; for the store's seat it re-bound every database
consumer on a machine running its own store to the holder on another, each was given a fresh, empty
database there, and nothing said so for twenty hours.

- An offer says whether it keeps its consumers' data (`keeps-consumer-data`); unsaid, a provider
  that grants each consumer a credential does. For such a provision the seat's holder no longer
  overrules a provider beside the consumer; a pin still does.
- Where each such consumer was sent is recorded (migration 0071). A resolution that would bind it
  elsewhere keeps the recorded provider and says the move; one whose provider is gone is refused,
  never answered by another.
- A push says a kept move and raises it as an urgent condition at once; the self-check's D12 raises
  it every run, with a pinned move not yet sent as a warning and any unasked move as urgent.
2026-10-06 15:19:23 +02:00
mesh-admin b037c73fd5 Merge pull request 'Retire the networking bundle and what the control plane stops shipping (hq ADR 0226)' (#77) from feat/retire-the-networking-bundle into main 2026-10-06 13:12:52 +00:00
jochen 18da8b37e9 Retire the networking bundle and what the control plane stops shipping (hq ADR 0226)
networking required mesh-wireguard and nothing else; machines are assigned the network directly.
module forget refuses a provided module, so a retired one is removed at start once no machine has it.
Guard route-proxy's public account directory against a reissue.
2026-10-06 14:59:28 +02:00
mesh-admin 4f4d365360 Merge pull request 'Retire, approve, reject, cleanup: the controller's half of hq ADR 0230' (#84) from feat/retired-consumers into main 2026-10-06 12:46:33 +00:00
jochen 08e11ad761 Keep a person's retirement decisions out of healer-wanted
Approving a retirement and deleting what was retired are a person's act by
design (hq ADR 0230); counted by S15 they would ask for a healer that must not
exist.
2026-10-06 14:45:57 +02:00
jochen 3b2adda1c8 Say a mark-only provider's retirement as mark only
A provider that cannot disable keeps the consumer reachable until a person
deletes it (hq ADR 0230); the listing and the approval say so instead of
claiming access was disabled.
2026-10-06 14:41:15 +02:00
jochen 68009b16fe Hear what providers retire, and let a person approve, reject and delete (hq ADR 0230)
A provider now waits for a person before retiring more than three consumers
or half of what it holds, and deletes only when asked. The controller is that
person's way in: it keeps waiting and rejected sets as conditions, answers them
with retire approve|reject, lists and deletes retired consumers through the
provider's own tools on its machine, records each act in the hand-act log, and
probes for anything retired longer than thirty days (D11).
2026-10-06 14:41:15 +02:00
mesh-admin 298daa6fbe Merge pull request 'Heal what is known, under a brake, and say every repair (hq to-be 45 Phase 3)' (#85) from feat/a-core-that-cannot-fail-silently-phase-3 into main 2026-10-06 12:29:15 +00:00
jochen 751e39186c Heal what is known, under a brake, and say every repair (hq to-be 45 Phase 3)
Research 031 counted the repairs people made by hand: a push to unstick a
plan waiting on a report, a controller restarted to make an object again, a
plan closed, a consumer re-made from now. Each was the ordinary path taken
again by someone who noticed. The healer registry makes each a registered
response to one condition kind, with a budget, a settle and its event:

- H1 sent-not-reported: ask the machine's node-engine to report again
  (mesh.node.<n>.ask.report); if it does not report what it was sent, send
  it again, never moving a build a policy or a plan holds back
- H2 stalled: close a plan whose wait is superseded or finished
- H3 holder-silent / consumer-lost: the send's own assertion of the bus's
  objects (issue 208's note)
- H4 consumer-behind: consumer-reset, only for a consumer the stream table
  marks resettable (the controller's own events consumer)
- H5 is the identity provider's own repair (ADR 0224 §5), registered only

Success is the observation clearing the condition, never the healer; a spent
budget hands the condition to the operator, urgent, with what was tried, and
no healer touches it again. Every act is begun in the store before it is made
(migration 0070), kept in the condition's tried as "healer Hn" and said as
the seat event healer-acted; a heal is never a hand act. More than twelve acts
in an hour stop every healer until an hour after the last, said urgently.
Only the lease holder heals.

S15 is live: a cause repaired by hand twice in a fortnight raises
healer-wanted, naming the healer that was not enough where one exists. D6's
far-behind finding has its own kind, consumer-behind. Nodes are granted the
question; the controller's grant gains healer-acted (genesis lock in
mesh-host). `healers` lists the registry, the acts and the brake; status
counts the week's heals.
2026-10-06 14:26:26 +02:00
mesh-admin 20c147ffdb Merge pull request 'Act under a lease, keep accounts by order, one writer at composition (hq to-be 45 Phase 2)' (#83) from feat/a-core-that-cannot-fail-silently-phase-2 into main 2026-10-06 10:30:29 +00:00
jochen 2eb9a22c24 Act under a lease, keep accounts by order, one writer at composition (hq to-be 45 Phase 2)
Two controllers could both act (issue 204), a reconcile's report could
overtake the apply after it and the digest decided (issue 267), and a grant
could make a second writer of a machine's report.

- The lease (internal/lease, ADR 0229): mesh-controller_lease key `holder`,
  15 s age, renewed every 5 s by compare-and-set; the epoch is the revision
  it was taken at. The gate is the clock (stops 3 s before expiry); a refused
  renewal is a loss and the process exits; a holder that stops gives it back.
  serve takes it before asserting the bus. Epochs kept in the store
  (migration 0068 controller_epoch) as a floor: a bucket raised from nothing
  is compacted past it. Unleased (no epoch, S12 urgent) only when nobody
  holds it and the bus will not let it be written. A shell command acts
  under the holder's epoch, or its own lease when none.
- Declarations carry `epoch` inside the signed envelope, only to a machine
  whose latest account carried a report_sequence (mesh-host #35); would-send
  is composed with the epoch last sent. Allot and the send both pass the gate.
- Reports: contract in internal/link/order.go (epoch, sequence,
  report_sequence, older_than, refused_older). Accounts kept by epoch, then
  sequence, then report sequence; older refused, counted; unordered reports
  keep the digest rule. Plans by compare-and-set on a revision, with epoch.
  Conditions and calls carry the epoch and are not written off the lease.
- S12 and S13 (naming the writer by epoch) watched, D5 run; reset of the
  bucket said. Writers table compiled in and enforced in PermissionsFor; the
  controller no longer publishes mesh.control.>. A contract per consumed
  kind, and the empty-on-error lint over the repository.
- mesh-host pinned to its main with the epoch in the validator (D1 validates
  the envelope as sent).

Needs mesh-host's genesis lock with the lease grant (mesh-host PR) for
TestTheInstallersFirstUserListIsWhatTheControllerWouldCompose.
2026-10-06 12:29:18 +02:00
mesh-admin 070ecafc07 Merge pull request 'Replace a value given by hand like one the mesh made (hq ADR 0228)' (#82) from feat/a-given-secret-lives-until-the-first-good-start into main 2026-10-06 10:16:30 +00:00
jochen e51c6a2cb9 Replace a value given by hand like one the mesh made (hq ADR 0228)
A given own secret the module reads at start is held by nobody but that
module, so the mesh need not read it to replace it: secret rotate now
works on it, and a value given through secret accept is replaced on its
own after the module's first good start under the mesh. Only a value an
outside party issues (own-secrets "issued-by": "outside") or one the
module applies stays as given, refused with the reason.
2026-10-06 12:13:48 +02:00
mesh-admin 722682f1c4 Merge pull request 'Assert the bus's objects on every send, not only at start (hq issue 208)' (#81) from fix/issue-208-bus-objects-on-send into main 2026-10-06 09:47:01 +00:00
jochen b853439792 Assert the bus's objects on every send, not only at start (hq issue 208)
A module or seat holder assigned after the controller started was sent
its declaration and found nothing to bind: messenger on novox
("consumer novox_messenger not found", 2026-10-06) and every first
build-agent holder (2026-10-03). assertBusObjects ran only in the start
raise; a push ensured a module's consumer only when a bus credential was
minted, which a module carried by the runtime never is.

The send's grant now runs the same derivation (assertOnSend) before the
memberships, on every push, cascade, plan send and rotation. A failure
is said in the send's output and raised as bus.objects.unasserted, which
the next send that asserts everything clears; the send itself goes on,
because the objects are the mesh's and holding every machine back for
one would turn one fault into all. Module consumers are each tried and
every failure named. The start raise stays as it was.
2026-10-06 11:45:51 +02:00
mesh-admin 9cf47f4429 Merge pull request 'Grant the self-check its ban-list question, say a refusal at once, judge the engine by its delivered version (hq to-be 45 Phase 1)' (#80) from fix/phase-1-d8-grant-and-d10-version into main 2026-10-06 08:45:16 +00:00
jochen 8ddc019cd2 Grant the self-check its ban-list question, say a refusal at once, judge the engine by its delivered version (hq to-be 45 Phase 1)
Live on 2026-10-06, two of the first self-check's findings were its own:

- D8 asked every machine's node-intrusion-prevention.banned, and the
  controller's grant did not name the subject: the bus refused it 24 times
  and D8 timed out after thirty seconds instead of saying so. The verbs the
  self-check asks are named in broker.VerbsTheSelfCheckAsks and granted
  (mesh.seat.<seat>.tool.<verb>.*); each probe declares the seat verbs it
  calls, askSeatTool refuses an undeclared one, and a test over the
  registry fails a probe whose question the controller is not granted.
  AskSeatTool now returns a refused publish at once ("the bus refused…")
  instead of waiting out its timeout; D8 asks the machines in parallel.
- D10 read every machine as behind right after a push: a node-engine says
  its version as the directory it is delivered into, the archive's digest
  (31045596c83a, catalogue versionOf), and D10 compared that with the
  build's commit (1545b00a). It now compares with the versions the
  registered build is delivered as, and a hand-placed engine's commit.
2026-10-06 10:44:38 +02:00
mesh-admin fab6b0059e Merge pull request 'Say when the mesh is wrong: conditions, watchdogs, the bus's advisories, doctor (hq to-be 45 Phase 1)' (#79) from feat/a-core-that-cannot-fail-silently-phase-1 into main 2026-10-06 08:29:31 +00:00
jochen e1f5d4fdf0 Vendor every dependency, so no build fetches the host's validator (hq to-be 45 D1)
The controller imports mesh-host/validate through a replace onto the forge
that holds it, and every build — the build agent's go build in a fresh
toolchain container, the Dockerfile's go mod download — would have fetched
it through the public proxy and checksum database at build time: a merge
breaking main on the network, the class Phase 1 removes. vendor/ is
committed; go builds from it with nothing fetched, and refuses to build
when it and go.mod disagree, so a pin moved without go mod vendor fails at
once. The Dockerfile copies vendor/ and builds with GOPROXY=off.
2026-10-06 10:29:10 +02:00
jochen bb1607e424 Say when the mesh is wrong: conditions, watchdogs, the bus's advisories, doctor (hq to-be 45 Phase 1)
Every one of the 48 core failures of research 031 was found by a person
looking; the mesh's answers carried the fact for whoever asked and told
nobody.

- The condition store (to-be 45 §2): mesh-controller_conditions, one key
  per open condition, written by compare-and-set so a person's silence
  and the watchdogs never lose each other's word; every transition kept
  ninety days in mesh-controller_condition-history and said as the
  seat's events condition-raised / condition-changed / condition-cleared
  (the condition at the top level, with event, at, change, why, show),
  offered again while the bus is away. Raised and cleared by observation
  only; a clearing reopened within ten minutes is the same condition with
  its count up, its silence kept. Verbs: conditions, conditions show,
  conditions silence (a hand act, at most a week), conditions history.
- ADR 0224's provider standing is the first kind, provider-failing, held
  by the provider's events; the provider_standing table is no longer read
  or written (left in place: dropping it is the operator's word).
- status leads with the open conditions, urgent first, and says all well
  only with none open; conditions it cannot read are said and not well.
- The signals table compiled in, one watchdog loop over it every 30s: S1
  heartbeat (3 intervals, asleep machines excepted, control node urgent
  after 30 min), S2 report after a send, S3 plan tier, S4 event loop deaf,
  S5 merge not acted, S6 ask lost, S7 call hung, S8 provider silent, S9
  advisories, S10 self-check silent, S11 node tools silent, S13 stale
  refusals; S12, S14, S15 deferred with their reasons. A row that cannot
  see raises probe-failed and clears nothing. A test generated from the
  table suppresses each signal inside and past its bound.
- The bus's advisories (maximum deliveries, a mesh consumer deleted) and
  the controller's own slow consumer and refused subjects, said in the
  mesh's words.
- doctor: the probe registry D1-D10 (D5 deferred) and DW, every five
  minutes, each in thirty seconds; a probe that cannot run is never a
  pass. D1 validates with mesh-host's own validator. Every run ends with
  the doctor-heartbeat event mesh-watcher listens for.
- The controller is granted its new buckets, events, the two advisories
  and $SRV.INFO; the node tools their tools-alive heartbeat. The streams
  and consumers the controller asserts and the ones D6/D7 expect are one
  derivation.
2026-10-06 10:21:11 +02:00
mesh-admin cf4834a36c Merge pull request 'Keep calls and hand acts on the bus, answer status at once, record durations (hq to-be 45 Phase 0)' (#78) from feat/a-core-that-cannot-fail-silently-phase-0 into main 2026-10-06 07:13:39 +00:00
jochen 9d8cbe7b81 Grant the controller its work queues' cancelled sets (hq issue 269)
A cancel writes the ask's id into the seat's cancelled set before deleting
the ask, and a write is a publish to the bucket's subject, which the
controller was not granted: against a server holding exactly the
controller's composed list, every cancel timed out.
2026-10-06 03:01:19 +02:00
jochen e74c32ed50 Keep calls and hand acts on the bus, answer status at once, record durations (hq to-be 45 Phase 0)
A controller restart lost every call's outcome, `status` composed the mesh
while its caller waited (18.6s live on 2026-10-06, past the 10s window), a
repair by hand left no trace, and the core's bounds had nothing measured to
be set from.

- calls: kept in the controller's bucket mesh-controller_calls (last 1000 or
  14 days, answers bounded to 64 KiB), read by id across a restart; a
  controller starting marks a stopped one's running calls abandoned; each
  call names its caller from the inbox its answer goes to.
- status: the serving controller composes it at start, after news from a
  machine, a build or an acting verb, and every minute; the verb answers the
  last composition at once with when and how long it took. Composing resolves
  each machine once instead of twice.
- hand-act log in mesh-controller_hand-acts: push (required through the seat),
  plans stop/close, broker consumer-reset and the new hand-act record take
  --why/--cause/--condition; `hand-acts` lists them and repeated causes;
  status counts the week's.
- durations (migration 0066): apply (send to first report), heartbeat gap,
  plan tier and build, recorded as heard; `durations` summarises them.
- the controller's seat row takes this binary's definition of its own verbs,
  so the console no longer judges calls against an older build's schema.
- the controller is granted its two buckets' subjects.
2026-10-06 02:59:36 +02:00
mesh-admin 146c48fd96 Merge pull request 'Bound a consumer's identity by the provision it requires (hq issue 263, ADR 0225)' (#76) from fix/263-identity-bound-per-provision into main 2026-10-06 00:28:47 +00:00
mesh-admin 1f3abd3e0e Merge pull request 'rotate: narrow a pair credential to one consuming module (hq issue 268)' (#75) from feat/rotate-one-consuming-module into main 2026-10-06 00:25:31 +00:00
jochen 6d620f77c3 Bound a consumer's identity by the provision it requires (hq issue 263)
The one global 20-character bound made every consumer pay an object
store's key length, even for provisions that keep no name, and a single
overflow refused the provider's whole declaration. An offer now states
its own bound (identity: {max, in} or false); unsaid, a provider told its
consumers keeps 20 and one told nothing keeps none. module check judges
every identity on the longest machine name before merge, and a provider
leaves an overflowing consumer out of its grants and composes, with the
consumer named by push, plan and status (ADR 0225).
2026-10-06 02:16:20 +02:00
jochen f8286c063d rotate: narrow a pair credential to one consuming module (hq issue 268)
A machine runs many consumers of one provision, each with its own
credential. When one module leaks its credential, `rotate <provision>
--consumer <machine>` was the narrowest act and replaced every module's
on that machine, restarting all of them. --module (and the verb's
module argument beside provision) rotates only that module's.
2026-10-06 02:13:48 +02:00
mesh-admin e096b4595a Merge pull request 'Keep the account of the sent declaration over an older one (hq issue 267)' (#74) from fix/stale-report-overwrites into main 2026-10-05 23:47:10 +00:00
jochen 09c0c6b367 Keep the account of the sent declaration over an older one (hq issue 267)
The last report stored per node decides whether a release plan moves on,
and it was whichever arrived last. A report about a declaration the mesh
has moved past now records what it says about the machine but leaves the
account of the apply alone, so arrival order cannot undo the newer.
2026-10-06 01:45:35 +02:00
mesh-admin d1fc25f682 Merge pull request 'Catch up on merges the bus announced and never handed over (hq issue 266)' (#73) from fix/missed-merges-are-caught-up into main 2026-10-05 23:33:18 +00:00
mesh-admin eda457f415 Merge pull request 'Answer every seat call within ten seconds and keep what came of it (hq issue 265)' (#72) from fix/a-verb-answers-before-its-caller-gives-up into main 2026-10-05 23:33:11 +00:00
jochen 59f4d486b1 Catch up on merges the bus announced and never handed over (hq issue 266)
The controller acted only on what its events consumer handed it, so a merge
the bus skipped left modules behind with nothing said. The stream is now read
back every five minutes on a single-filter consumer, and any merge that would
still move a module after ten minutes is said and acted on.
2026-10-06 01:29:21 +02:00
jochen 801552c0eb Answer every seat call within ten seconds and keep what came of it (hq issue 265)
A push outlasted the console's 30s wait and, when it sent the bus its
changed user list, the broker's reload forgot the reply it may send:
the push happened and its caller was told it did not answer. Calls now
answer in full or as running with an id, a push answers before it
sends, refused answers are recorded on their call, and 'calls' reads
them back.
2026-10-06 01:14:58 +02:00
mesh-admin ede9bce6ef Merge pull request 'Refuse a verb argument the seat would pass over; a push without a machine says it is the whole mesh (hq issue 244)' (#71) from fix/verb-schemas into main 2026-10-05 22:43:07 +00:00
jochen 0f0028785c Refuse a verb argument the seat would pass over, and say a push is of the whole mesh
A push naming one machine reached the verb without it and pushed every
machine behind (hq issue 244). The controller now refuses any argument a
verb does not declare, any it composed its command line without, and a
switch that is not true or false; a push that names no machine says first
that it is the whole mesh. Tests walk every served verb: no argument is
ever ignored, and every flag of a verb's command, read from the source, is
in its schema or accounted for. plan gains files, push behind, builds and
plans limit.
2026-10-06 00:37:14 +02:00
mesh-admin 6fdcfad8d3 Merge pull request 'Report a provider that keeps failing a consumer in status (hq ADR 0224)' (#70) from feat/a-provider-failing-a-consumer-is-reported into main 2026-10-05 22:20:45 +00:00
jochen 8d9d33ae85 Report a provider that keeps failing a consumer in status (hq ADR 0224)
The identity provider failed every consumer for a day and status called the
mesh well (hq issue 179). The controller now follows every provider's
provisioner.failing/recovered, keeps the newest failing word per provider,
machine and consumer (migration 0065), and status, its JSON and node show
name it until it recovers. Every module that receives contributions is
granted the two events, so no manifest can forget them.
2026-10-06 00:13:23 +02:00
mesh-admin cc25baa563 Merge pull request 'Rename node-hosts-file to node-hostname, and refuse one seat claimed under two names (hq ADR 0223 part 3)' (#69) from hostname-module into main 2026-10-05 22:11:41 +00:00
mesh-admin 68a2ebdcc3 Merge pull request 'Retire node-resolver-config and the seat need only it used (hq ADR 0223 part 2, step 2 of 2)' (#68) from retire-resolv-conf into main 2026-10-05 22:08:03 +00:00
jochen 69b99eec68 Number the hostname seat migration 0064: it merges after the resolver-config one 2026-10-06 00:07:57 +02:00
jochen 09bd0eec4f Number the resolver-config migration 0063: it merges first 2026-10-06 00:07:54 +02:00
mesh-admin 222a38e050 Merge pull request 'Test resolv.conf as the uplink's, and refuse a second writer of a fact's path (hq ADR 0223 part 2, step 1 of 2)' (#67) from resolv-conf-to-uplink into main 2026-10-05 21:57:03 +00:00
jochen ee99a24f77 Rename node-hosts-file to node-hostname, and refuse one seat claimed under two names (hq ADR 0223)
The seat now covers /etc/hostname too. The migration keeps the old name as
an alias so hosts, still assigned while machines move, holds the same seat.
Claims were compared by spelling, so the old and new module would both have
held it on one machine; they are now compared by the seat they resolve to.
2026-10-05 23:43:15 +02:00
jochen f68521da28 Retire node-resolver-config and the seat need it alone used (hq ADR 0223)
The uplink's holder writes /etc/resolv.conf, so the seat that wrote it and
ADR 0220's dependency of it on the uplink have nothing left to say. The
migration deletes the store's row; nothing holds it once resolv-conf is
unassigned everywhere.
2026-10-05 23:40:39 +02:00
jochen 296064c799 Test the resolver file as the uplink's, and refuse a second writer of a fact's path (hq ADR 0223)
The catalogue moves /etc/resolv.conf from resolv-conf to the three uplink
modules. A rendered fact was not compared with other modules' paths, so two
modules could each write the resolver file on one machine, the last winning
every apply; a fact's path now counts as its module's.
2026-10-05 23:39:15 +02:00
mesh-admin df9231c734 Merge pull request 'A mesh seat may be replicated: the resolver held on two machines (hq ADR 0223)' (#65) from feat/the-mesh-has-two-resolvers into main 2026-10-05 20:48:23 +00:00
jochen 843b709b59 The registry-trust test reads the runtime's module, which now writes the trust (ADR 0222) 2026-10-05 22:47:43 +02:00
jochen f506fb34ec Let the mesh's resolver seat have several holders on record
musl takes the first reply from any listed nameserver, so a public fallback
beside the mesh's resolver answered NXDOMAIN for mesh names in every Alpine
container (hq ADR 0223). The fix is two mesh resolvers and no public one, which
needs mesh-dns-resolver held on two machines: a seat can now be replicated,
each holder recorded by 'seat <name> --add', checkClaims accepts every holder
on record and still refuses a second holder of any other mesh seat, a holder
answers its own requirement, and a roster fact gives each replicated seat's
holders, this machine first, so resolv-conf can list them. Migration 0062 keys
a holding by seat and assignment.
2026-10-05 22:42:53 +02:00
mesh-admin c34b937dd3 Merge pull request 'The private network writes nothing into the runtime's file; generated resources are collision-checked (hq ADR 0222, issue 190 — 3 of 3)' (#63) from fix/190-the-overlay-writes-no-runtime-file into main 2026-10-05 20:42:32 +00:00
mesh-admin d84c9699b3 Merge pull request 'Tell a module where a mesh seat's holder is reached: ${seat:<seat>:reach} (hq ADR 0222, issue 190 — 1 of 3)' (#62) from fix/190-seat-reach into main 2026-10-05 20:31:32 +00:00
mesh-admin 002d5e578c Merge pull request 'A named push sends no build a policy or a plan holds back (hq issue 259, ADR 0221)' (#64) from fix/259-a-named-push-sends-no-held-build into main 2026-10-05 20:26:50 +00:00
jochen 2421b82ad2 Keep a named push from sending builds a policy or a plan holds back
A named push flushed every other machine whose declaration differed from
what it was last sent (hq ADR 0083). Under an upgrade policy of `record`,
or a plan still waiting on its first machine (ADR 0218), every machine
running the module differs, so `push <one>` sent the held build to all of
them (hq issue 259).

Each send now records which build of each module it carried
(node.sent_builds, migration 0061). The cascade, and the bus holder added
to a named push, skip a machine any of whose modules would move to a
build its policy records or an open plan has not sent it, and say which
module, which build, why, and that `push <node>` sends it. A machine
whose last send was not recorded is held until it is named. The named
machine itself, a whole-mesh push and `push --behind` are unchanged.
2026-10-05 22:22:03 +02:00
jochen 0ebd48a6a8 The private network writes nothing into the runtime's file (hq issue 190)
daemon.json and docker.service belong to the docker module, which holds node-container-runtime
and now states the registry itself through ${seat:mesh-artifact-store:reach} (hq ADR 0222). The
overlay stops generating registry-trust and registry-trust-reload. A generated resource is now
held to the collision check every module is, so a second writer cannot come back through
computed code; resolution never saw what a generator declares.
2026-10-05 22:19:43 +02:00
jochen 67e291c02a Tell a module where a mesh seat's holder is reached (hq ADR 0222)
The container runtime's module must state the mesh's registry to the runtime it owns, so the
controller can stop writing that into the runtime's file (hq issue 190). ${seat:<seat>:reach}
answers host:port without a binding: nothing required, granted or minted, and the address is
one the mesh already composes into every reference it built. Only mesh-artifact-store is
answered; another seat is refused by name. Unanswered in a file written into as JSON, the empty
member is dropped, so the runtime is never told to trust "".
2026-10-05 22:16:23 +02:00
mesh-admin 8a400d165e Merge pull request 'Delete node-dns-resolver; resolver config needs the uplink (hq ADR 0220)' (#61) from feat/resolver-config-needs-the-uplink into main 2026-10-05 20:11:18 +00:00
jochen 53d3cd7ce9 Delete node-dns-resolver and make resolver config need the uplink
Nothing has claimed node-dns-resolver since the mesh moved to one resolver
(hq ADR 0194); seeding never removes a row, so a migration deletes it.

resolv.conf stays the mesh's only while the network manager is told to keep
off it, which the node-uplink holder does (ADR 0117). A seat's Needs makes
that a dependency checked at assignment by the ADR 0207 mechanism (hq ADR
0220). The two-claimants test keeps its intent with a synthetic module now
that resolved-split-dns leaves the catalogue.
2026-10-05 21:57:04 +02:00
mesh-admin 6648e4a5c8 Merge pull request 'The controller writes no /etc/hosts (hq ADR 0199)' (#60) from fix/the-controller-writes-no-hosts-file into main 2026-10-05 19:23:41 +00:00
jochen 7fa2568ce7 The controller writes no /etc/hosts (hq ADR 0199)
/etc/hosts is the file of the node-hosts-file seat's holder; the controller writes into no file
another seat's holder owns, and asks that holder if it ever needs a line there. The private
network's module stops asking for the node-names fact; every machine already asks the mesh's
one resolver for these names, and the host gives the region back at the next push.
2026-10-05 21:23:25 +02:00
mesh-admin 4b382ceffd Merge pull request 'A mesh seat's holder elsewhere answers before this machine's own provider (hq issue 258)' (#59) from fix/a-mesh-seat-answers-before-the-machines-own into main 2026-10-05 19:14:24 +00:00
jochen ce86d09d22 A mesh seat's holder elsewhere answers before this machine's own provider (issue 258)
A mesh-wide provision a machine could answer itself was bound to the local provider, with the
seat's holder and any pin consulted only for a provider on another machine. With every machine
still running its own resolver, each bound its resolver configuration to itself while the mesh's
one resolver was held and pinned elsewhere.
2026-10-05 21:14:01 +02:00
jochen 853be00ebe The runtime's file is the runtime module's: the resolver test expects docker to write live-restore (issue 190, ADR 0196)
The catalogue moves daemon.json's live-restore and the reload from resolv-conf to the docker
module, so no module writes another software's configuration. The test composes docker beside
the resolver modules and refuses resolv-conf writing the runtime's file.
2026-10-05 21:14:01 +02:00
705 changed files with 207325 additions and 1056 deletions
+5 -3
View File
@@ -21,13 +21,15 @@ ARG GO_BASE=golang@sha256:8ac98ca534ac3f51e1f420a1dd2c15e74c75cfa0f23f3ad27eb5d7
FROM ${GO_BASE} AS build
WORKDIR /src
# Dependencies first, so a change to the source does not refetch them.
# **Nothing is fetched** (novox/hq to-be 45 Phase 1): every dependency is in vendor/, committed, so
# the image builds from this repository alone — the host's validator among them, whose module no
# public proxy is asked for. Dependencies first, so a change to the source does not re-copy them.
COPY go.mod go.sum ./
RUN go mod download
COPY vendor/ vendor/
COPY . .
ARG VERSION=development
RUN CGO_ENABLED=0 go build -trimpath \
RUN CGO_ENABLED=0 GOFLAGS=-mod=vendor GOPROXY=off go build -trimpath \
-ldflags "-s -w -X main.version=${VERSION}" \
-o /mesh-controller ./cmd/mesh-controller
+353
View File
@@ -0,0 +1,353 @@
package main
import (
"context"
"errors"
"fmt"
"os"
"sync"
"time"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/lease"
"github.com/novox/mesh-controller/internal/link"
)
// Acting under the lease (novox/hq to-be 45 §6, ADR 0227 rule 1).
//
// **Only the instance holding the lease acts**: sends a declaration, writes a plan, a condition or a
// call. Every one of those passes theLease.epoch, which answers the epoch the act carries or why it may
// not happen. Three ways a process stands to the lease:
//
// - **The serving controller** takes it before it does anything else — before it asserts the bus's
// objects, which are the controller's to write — waiting while another holds it, and renews it.
// A renewal refused or failed is the lease lost: the gate closes at once and the process exits, so
// its service manager restarts it as a candidate (serve, in push.go).
// - **A command run at a shell** — `push` in the installer, the lab, a person repairing a mesh whose
// controller is down (issue 201) — acts **under the holder's epoch** when a controller holds the
// lease: it is the same mesh's word, composed and sent under the store's hold of each machine like
// the serving controller's, and the epoch it carries is read at the moment it acts, so a handover
// between makes it stale and refused like any other. **When nobody holds the lease, the command
// takes it** for as long as it runs and gives it back; a controller starting meanwhile waits for
// it, as it would for another controller.
// - **A process with no bus** — a test, a command that only reads — acts with no epoch and is
// refused nothing: there is nothing to order against, and nothing it does reaches a machine.
//
// **Unleased, said and temporary.** A serving controller whose bus refuses it the lease's key — the bus's
// user list is older than this build and does not grant the bucket yet — and that sees no other holder
// serves without one, as every controller did before the lease: declarations carry no epoch, which no
// node-engine refuses. Said once, kept as a condition (S12), and tried again every renewal interval; the
// first push that sends the bus its new user list grants it, and the next try takes it. Refusing to act
// instead would be a controller that can never send the user list that lets it act.
// actor is this process's standing to the lease.
type actor struct {
mu sync.Mutex
// held is the lease this process holds: the serving controller's, or a command's own.
held *lease.Lease
// unleased is why a serving controller acts without the lease; empty while it holds it or is not
// serving.
unleased string
// serving is a serving controller, which never borrows another's epoch.
serving bool
// kv is the lease bucket, for a command to read the holder's epoch from.
kv jetstream.KeyValue
close func()
// noBus is a process with no bus configured.
noBus bool
// reset is when the lease bucket was found raised again from nothing and its revisions moved past
// the highest epoch issued, and what was said of it; zero when it was not (S12).
reset time.Time
resetSaid string
}
// theLease is this process's standing to the lease.
var theLease = &actor{}
// instance names this process among controller instances: its machine, its process and when it
// started. The lease's holder and every call this process keeps carry it.
var instance = func() string {
host, _ := os.Hostname()
return fmt.Sprintf("controller@%s pid %d since %s", host, os.Getpid(), time.Now().UTC().Format(time.RFC3339))
}()
// epoch is the gate: the epoch an act carries — zero for none — or why it may not happen.
func (a *actor) epoch(ctx context.Context) (uint64, error) {
a.mu.Lock()
held, serving, unleased, noBus := a.held, a.serving, a.unleased, a.noBus
a.mu.Unlock()
switch {
case held != nil:
return held.Epoch()
case serving && unleased != "":
return 0, nil
case serving:
return 0, lease.ErrNotHeld
case noBus:
return 0, nil
}
return a.forACommand(ctx)
}
// forACommand is a command's epoch: the holder's, or a lease of its own when nobody holds one.
func (a *actor) forACommand(ctx context.Context) (uint64, error) {
a.mu.Lock()
defer a.mu.Unlock()
if a.held != nil {
return a.held.Epoch()
}
if a.kv == nil {
address, err := broker.BusAddress()
if err != nil {
// No bus: this process reaches no machine, and has nothing to order against.
a.noBus = true
return 0, nil
}
js, err := broker.Dial(address)
if err != nil {
return 0, fmt.Errorf("the bus cannot be reached, so whether a controller holds the lease cannot be "+
"read and nothing is done: %w", err)
}
api, err := jetstream.New(js.Conn())
if err != nil {
js.Close()
return 0, err
}
reading, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
if err := broker.EnsureLeaseBucket(reading, api); err != nil {
js.Close()
return 0, err
}
kv, err := api.KeyValue(reading, broker.LeaseBucket)
if err != nil {
js.Close()
return 0, err
}
a.kv, a.close = kv, js.Close
if _, found, err := lease.Current(reading, kv); err == nil && !found {
// Nobody: this command takes it for as long as it runs.
l, err := lease.Open(reading, api, broker.LeaseBucket, lease.Options{Holder: holderOf(instance),
Say: func(format string, args ...any) { fmt.Printf(format+"\n", args...) }})
if err != nil {
return 0, err
}
epoch, err := l.TryTake(reading)
if err != nil {
return 0, fmt.Errorf("no controller holds the lease and this command could not take it: %w", err)
}
keeping, stop := context.WithCancel(context.Background())
go l.Keep(keeping)
a.held = l
closeBus := a.close
a.close = func() {
stop()
l.Release(context.Background())
closeBus()
}
return epoch, nil
}
}
reading, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
holder, found, err := lease.Current(reading, a.kv)
if err != nil {
return 0, fmt.Errorf("who holds the controller lease cannot be read, so nothing is done: %w", err)
}
if !found {
return 0, errors.New("the controller that held the lease while this command ran let go of it; nothing " +
"more is done under an epoch nobody holds — run the command again")
}
return holder.Epoch, nil
}
// release gives back what this process holds, at its end.
func (a *actor) release() {
a.mu.Lock()
closing := a.close
a.close = nil
a.mu.Unlock()
if closing != nil {
closing()
}
}
// holderOf is this process as the lease's holder.
func holderOf(instance string) lease.Holder {
host, _ := os.Hostname()
return lease.Holder{Instance: instance, Host: host, Build: version}
}
// serveUnderTheLease takes the lease for the serving controller, waiting while another holds it, and
// keeps it until ctx ends. Lost is closed when it is lost; the caller exits on it.
func (a *actor) serveUnderTheLease(ctx context.Context, inv *inventory.Inventory, address string) (lost <-chan struct{}, err error) {
a.mu.Lock()
a.serving = true
a.mu.Unlock()
js, err := broker.Dial(address)
if err != nil {
return nil, fmt.Errorf("the mesh is on the bus at %s and this control plane cannot reach it to take the "+
"lease: %w", broker.BareAddress(address), err)
}
api, err := jetstream.New(js.Conn())
if err != nil {
js.Close()
return nil, err
}
asserting, cancel := context.WithTimeout(ctx, 10*time.Second)
err = broker.EnsureLeaseBucket(asserting, api)
cancel()
if err != nil {
js.Close()
return nil, err
}
say := func(format string, args ...any) { fmt.Printf(format+"\n", args...) }
l, err := lease.Open(ctx, api, broker.LeaseBucket, lease.Options{Holder: holderOf(instance),
Floor: inv.HighestEpoch, Say: say, Moved: func(was, floor uint64) {
a.mu.Lock()
defer a.mu.Unlock()
a.reset = time.Now()
a.resetSaid = fmt.Sprintf("the lease bucket was at revision %d with epoch %d already issued: it was "+
"raised again from nothing (a bus whose data was replaced), and its revisions were moved past %d so "+
"no machine refuses the next epoch", was, floor, floor)
}})
if err != nil {
js.Close()
return nil, err
}
gone := make(chan struct{})
epoch, err := l.Take(ctx)
switch {
case ctx.Err() != nil:
js.Close()
return nil, ctx.Err()
case err != nil && !errors.Is(err, lease.ErrUnwritable):
// Whether another controller acts cannot be told: this one does not act, and exits to try again.
js.Close()
return nil, err
case err != nil:
// Nobody holds it and the bus will not let it be written: unleased, said, tried again (see above).
a.mu.Lock()
a.unleased = err.Error()
a.mu.Unlock()
say("this controller serves WITHOUT the lease: %v. Its declarations carry no epoch; it tries again "+
"every %s, and the first push that sends the bus its user list grants it", err, lease.RenewEvery)
go a.takeWhenGranted(ctx, l, inv, gone)
default:
a.took(ctx, l, inv, epoch, gone)
}
a.mu.Lock()
a.close = func() {
if held, err := l.Epoch(); err == nil {
ending, cancel := context.WithTimeout(context.Background(), 5*time.Second)
if err := inv.EndEpoch(ending, held, inventory.EpochReleased); err != nil {
say("how epoch %d ended could not be recorded: %v", held, err)
}
cancel()
}
l.Release(context.Background())
js.Close()
}
a.mu.Unlock()
return gone, nil
}
// took is the lease taken: recorded, earlier epochs nobody gave back ended as expired, kept.
func (a *actor) took(ctx context.Context, l *lease.Lease, inv *inventory.Inventory, epoch uint64, gone chan struct{}) {
a.mu.Lock()
a.held, a.unleased = l, ""
a.mu.Unlock()
h := holderOf(instance)
recording, cancel := context.WithTimeout(ctx, 10*time.Second)
expired, err := inv.TookEpoch(recording, inventory.Epoch{Epoch: epoch, Instance: h.Instance, Host: h.Host,
Build: h.Build, Taken: time.Now()})
cancel()
if err != nil {
fmt.Printf("epoch %d could not be recorded as taken, so a stale refusal from it will not name it: %v\n", epoch, err)
}
for _, e := range expired {
fmt.Printf("the controller of epoch %d (%s) stopped renewing the lease without giving it back: it is "+
"taken over at epoch %d\n", e.Epoch, e.Instance, epoch)
}
go l.Keep(ctx)
go func() {
<-l.Lost()
if ctx.Err() == nil {
why := l.LostWhy()
ending, cancel := context.WithTimeout(context.Background(), 5*time.Second)
_ = inv.EndEpoch(ending, epoch, inventory.EpochLost)
cancel()
fmt.Printf("the controller lease was lost (epoch %d): %v — this controller stops and exits, to "+
"be started again as a candidate\n", epoch, why)
}
close(gone)
}()
}
// takeWhenGranted tries the lease again every renewal interval while serving unleased, and stops this
// controller if another took it meanwhile: two serving at once is what the lease is for.
func (a *actor) takeWhenGranted(ctx context.Context, l *lease.Lease, inv *inventory.Inventory, gone chan struct{}) {
tick := time.NewTicker(lease.RenewEvery)
defer tick.Stop()
for {
select {
case <-ctx.Done():
return
case <-tick.C:
}
epoch, err := l.TryTake(ctx)
if errors.Is(err, lease.ErrTaken) {
fmt.Printf("another controller took the lease while this one served without it: %v — this one "+
"stops and exits\n", err)
close(gone)
return
}
if err != nil {
// Still not written, or not readable this time: unleased, said by S12, tried again.
a.mu.Lock()
a.unleased = err.Error()
a.mu.Unlock()
continue
}
a.took(ctx, l, inv, epoch, gone)
return
}
}
// standing is what `status` and the self-check say of this process and the lease.
type standing struct {
Epoch uint64
Held bool
Renewed time.Time
Unleased string
// Reset is when the lease bucket was found raised again from nothing, and ResetSaid what of it.
Reset time.Time
ResetSaid string
}
func (a *actor) standing() standing {
a.mu.Lock()
held, unleased, reset, resetSaid := a.held, a.unleased, a.reset, a.resetSaid
a.mu.Unlock()
st := standing{Unleased: unleased, Reset: reset, ResetSaid: resetSaid}
if held == nil {
return st
}
epoch, err := held.Epoch()
st.Epoch, st.Held, st.Renewed = epoch, err == nil, held.Renewed()
return st
}
// The gates, given to what acts: a declaration's send (link) and a plan's write (the inventory).
func init() {
link.ActingGate = func(ctx context.Context) error {
_, err := theLease.epoch(ctx)
if err != nil {
return fmt.Errorf("this controller may not send: %w", err)
}
return nil
}
}
+4
View File
@@ -224,6 +224,10 @@ func unassign(ctx context.Context, open *stores, node string, modules ...string)
for _, line := range unheldChange(shelf, node, assigned, left) {
answer += "\n " + line
}
// What it leaves behind that is irreplaceable is kept and retired, never removed (novox/hq ADR 0233).
for _, line := range keptOnUnassign(ctx, open.inventory, node, modules) {
answer += "\n " + line
}
return answer + blockedElsewhere(ctx, open, node), nil
}
+8
View File
@@ -111,6 +111,14 @@ func TestTheRegistryTrustAndEveryImageFollowThePortTheNodeGaveTheStore(t *testin
if _, err := assign(ctx, open, "laptop", "app"); err != nil {
t.Fatal(err)
}
// The runtime's trust is the runtime's module's to write (novox/hq ADR 0222): a stand-in for it
// asks where this machine reaches the store, as the docker module does.
register(t, open, catalogue.Manifest{Module: "runtime", Version: "1",
Resources: []map[string]any{{"id": "daemon", "type": "file", "path": "/etc/docker/daemon.json",
"into": "json", "content": `{"insecure-registries": ["${seat:mesh-artifact-store:reach}"]}` + "\n"}}})
if _, err := assign(ctx, open, "laptop", "runtime"); err != nil {
t.Fatal(err)
}
on := map[string]bool{"anchor": true, "laptop": true}
node, port, found, err := artifactStoreOnNetwork(ctx, open.inventory, on)
+1 -1
View File
@@ -85,7 +85,7 @@ func reportsReaching(t *testing.T, open *stores, reachable []link.Reach, held ..
if err != nil {
t.Fatal(err)
}
if err := open.inventory.RecordSent(ctx, record.ID, digestOf(body)); err != nil {
if err := open.inventory.RecordSent(ctx, record.ID, digestOf(body), nil); err != nil {
t.Fatal(err)
}
if _, err := (link.Enrolment{Inventory: open.inventory}).Heard(ctx, link.Report{
+281
View File
@@ -0,0 +1,281 @@
package main
import (
"errors"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/overlay"
)
// Between `assign` and `push` a module's own secrets are not made yet: the push makes them. D1 composed
// the machine's declaration without making anything, failed on the missing secret, and raised an urgent
// "nothing can be sent to <machine>" — seen live on 2026-10-06, a desktop notification for a machine
// the next push sent to without a word (novox/hq issue 275). D1 now composes as the push would, with a
// stand-in for what the push makes: pending, not broken, and said only past a bound, as a warning.
// aKeeper is a module with an own secret the mesh makes — a backup repository's password.
func aKeeper() catalogue.Manifest {
return catalogue.Manifest{Module: "keeper", Version: "1",
OwnSecrets: catalogue.OwnSecrets{"repository": {Path: "/var/lib/mesh/keeper/repository"}},
Resources: []map[string]any{
{"id": "state", "type": "directory", "path": "/var/lib/mesh/keeper", "mode": "0700"},
}}
}
// pushedBy records a send to a machine as a push does — composed on the send path, so its own secrets
// are made — without a bus to carry it.
func pushedBy(t *testing.T, open *stores, node string) string {
t.Helper()
ctx := t.Context()
plan, settings, err := planFor(ctx, open, node)
if err != nil {
t.Fatal(err)
}
declared, err := declarationFor(ctx, open, node, plan, settings)
if err != nil {
t.Fatal(err)
}
body, err := declared.Body()
if err != nil {
t.Fatal(err)
}
record, err := open.inventory.NodeByName(ctx, node)
if err != nil {
t.Fatal(err)
}
digest := digestOf(body)
if err := open.inventory.RecordSent(ctx, record.ID, digest, declared.Builds); err != nil {
t.Fatal(err)
}
return digest
}
// pushedAndApplied is pushedBy, and the machine reporting it applied that declaration.
func pushedAndApplied(t *testing.T, open *stores, node string) {
t.Helper()
digest := pushedBy(t, open, node)
record, err := open.inventory.NodeByName(t.Context(), node)
if err != nil {
t.Fatal(err)
}
if _, err := open.inventory.RecordDoing(t.Context(), record.ID, inventory.Doing{
Outcome: inventory.OutcomeApplied, Declared: digest}); err != nil {
t.Fatal(err)
}
}
// d1 runs D1 once.
func d1(t *testing.T, open *stores) []conditions.Observation {
t.Helper()
got, err := probeDeclarations(t.Context(), &doctor{open: open})
if err != nil {
t.Fatal(err)
}
return got
}
// **THE WINDOW, REPRODUCED**: assigned and not pushed, a module whose own secret the push makes is
// waiting, not uncomposable; past the bound it is a warning naming what the push makes; pushed, nothing.
func TestAModuleAssignedAndNotPushedIsAwaitingAPushNotUncomposable(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aKeeper())
pushedBy(t, open, "laptop") // the machine was pushed before; then the module is assigned
if _, err := assign(ctx, open, "laptop", "keeper"); err != nil {
t.Fatal(err)
}
// The read the rest of the mesh asks still refuses it, with the composer's typed error: nothing
// can be compared about a secret that does not exist (status), and nothing here string-matches.
plan, settings, err := planFor(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
gens, err := generators(ctx, open)
if err != nil {
t.Fatal(err)
}
_, err = declarationWith(ctx, open, "laptop", plan, settings, gens, Reading)
var notMade *catalogue.NotMadeError
if !errors.As(err, &notMade) || notMade.Module != "keeper" || notMade.Name != "repository" {
t.Fatalf("a read composition did not say which secret is not made, typed: %v", err)
}
// Foreseen, it composes, names what the push makes, and made nothing.
foreseen, err := declarationWith(ctx, open, "laptop", plan, settings, gens, Foreseeing)
if err != nil || len(foreseen.foreseen) != 1 || foreseen.foreseen[0] != "keeper/repository" {
t.Fatalf("foreseen: %v %v", foreseen.foreseen, err)
}
if _, held, err := open.inventory.ModuleSecretIfIssued(ctx, "laptop", "keeper", "repository"); err != nil || held {
t.Fatalf("asking ahead of the push made the secret (held %v, %v)", held, err)
}
// Within the bound: nothing at all — no urgent, no warning, nobody notified.
if got := d1(t, open); len(got) != 0 {
t.Fatalf("a machine waiting for a push raised %+v", got)
}
// Past the bound: a warning, not urgent, saying what the push will make.
before := awaitingPushBound
awaitingPushBound = -time.Minute
t.Cleanup(func() { awaitingPushBound = before })
got := d1(t, open)
if len(got) != 1 || got[0].Key() != "machine.laptop.awaiting-push" || got[0].Severity != conditions.Warning ||
!strings.Contains(got[0].Summary, "keeper/repository") || !strings.Contains(got[0].Summary, "push laptop") {
t.Fatalf("a machine left un-pushed past the bound: %+v", got)
}
// Pushed: the secret is made and D1 says nothing.
pushedBy(t, open, "laptop")
if got := d1(t, open); len(got) != 0 {
t.Fatalf("a pushed machine still raised %+v", got)
}
}
// **A REAL FAILURE IS STILL URGENT**: a secret the push would be refused on is not one it will make.
// A bus credential nobody issued (issue 203) fails the push, so D1 says it — urgent, in the push's words.
func TestASecretThePushCannotMakeIsStillUncomposable(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aTalker())
if _, err := assign(ctx, open, "laptop", "talker"); err != nil {
t.Fatal(err)
}
got := d1(t, open)
if len(got) != 1 || got[0].Key() != "machine.laptop.uncomposable" || got[0].Severity != conditions.Urgent ||
!strings.Contains(got[0].Summary, "module issue talker --node laptop") {
t.Fatalf("a push that will be refused was not said urgently: %+v", got)
}
// And a machine whose composition fails for anything else, with a secret waiting beside it, is
// uncomposable for that — the stand-in hides nothing.
register(t, open, aKeeper())
if _, err := assign(ctx, open, "anchor", "keeper"); err != nil {
t.Fatal(err)
}
one, two := rivals()
register(t, open, one)
register(t, open, two)
_, _ = assign(ctx, open, "anchor", "rival-one")
_, _ = assign(ctx, open, "anchor", "rival-two")
keys := map[string]conditions.Severity{}
for _, o := range d1(t, open) {
keys[o.Key()] = o.Severity
}
if keys["machine.anchor.uncomposable"] != conditions.Urgent {
t.Fatalf("a real failure beside a waiting secret: %v", keys)
}
}
// A given secret sealed to a key the machine no longer has is not the mesh's to make again: the push is
// refused on it, so D1 is too.
func TestAGivenSecretUnderAnOldKeyIsNotForeseen(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aKeeper())
if _, err := assign(ctx, open, "laptop", "keeper"); err != nil {
t.Fatal(err)
}
if err := open.inventory.AcceptSecretForModule(ctx, "laptop", "keeper", "repository", "given"); err != nil {
t.Fatal(err)
}
record, err := open.inventory.NodeByName(ctx, "laptop")
if err != nil {
t.Fatal(err)
}
if err := open.inventory.RecordSealingKey(ctx, record.ID, aPublicKey(t)); err != nil {
t.Fatal(err)
}
got := d1(t, open)
if len(got) != 1 || got[0].Key() != "machine.laptop.uncomposable" || !strings.Contains(got[0].Summary, "issue it again") {
t.Fatalf("a given secret under an old key: %+v", got)
}
}
// The bound is read from when the machine began waiting: the oldest assignment since its last send.
func TestAMachineAwaitsAPushSinceItsOldestAssignmentSinceTheLastSend(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aKeeper())
pushedBy(t, open, "laptop")
sent := time.Now()
since, err := open.inventory.AwaitingSince(ctx, "laptop")
if err != nil || since.After(sent) {
t.Fatalf("nothing assigned since the send: waiting since %v (%v), the send was before %v", since, err, sent)
}
if _, err := assign(ctx, open, "laptop", "keeper"); err != nil {
t.Fatal(err)
}
since, err = open.inventory.AwaitingSince(ctx, "laptop")
if err != nil || since.Before(sent.Add(-time.Second)) {
t.Fatalf("assigned after the send: waiting since %v (%v), not from the assignment", since, err)
}
}
// **D3 AND D13 WAIT FOR THE SEND**: a holder is expected to answer on a machine once the machine was sent
// it and had time to report — never between assign and push.
func TestAHolderIsExpectedOnlyOnceSentAndReported(t *testing.T) {
now := time.Now()
sent := now.Add(-time.Minute)
long := now.Add(-time.Hour)
cases := []struct {
name string
send lastSend
want bool
}{
{"never sent", lastSend{}, false},
{"sent without it", lastSend{sent: &long, current: true, carried: map[string]string{"other": "c"}}, false},
{"sent with it, not reported yet", lastSend{sent: &sent, carried: map[string]string{"keeper": "c"}}, false},
{"sent with it and reported", lastSend{sent: &sent, current: true, carried: map[string]string{"keeper": "c"}}, true},
{"sent with it long ago, never reported", lastSend{sent: &long, carried: map[string]string{"keeper": "c"}}, true},
{"sent before builds were kept", lastSend{sent: &long, current: true}, true},
}
for _, c := range cases {
if got := c.send.settled("keeper", now); got != c.want {
t.Errorf("%s: settled %v, want %v", c.name, got, c.want)
}
}
}
// And through the stores: assigned, not in the last send; pushed and reported, carried and settled.
func TestALastSendIsReadFromTheSendAndTheReport(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aKeeper())
pushedBy(t, open, "laptop")
if _, err := assign(ctx, open, "laptop", "keeper"); err != nil {
t.Fatal(err)
}
sends, err := readDeliveries(ctx, open.inventory)
if err != nil {
t.Fatal(err)
}
if sends["laptop"].settled("keeper", time.Now()) {
t.Fatal("a holder assigned and not pushed is expected to answer")
}
if !sends["laptop"].settled(overlay.Name, time.Now().Add(time.Hour)) {
t.Fatal("a module the machine was sent long ago is not expected to answer")
}
pushedBy(t, open, "laptop")
sends, err = readDeliveries(ctx, open.inventory)
if err != nil {
t.Fatal(err)
}
if sends["laptop"].settled("keeper", time.Now()) {
t.Fatal("pushed a moment ago and not reported, a holder is already expected")
}
if !sends["laptop"].settled("keeper", time.Now().Add(reportGrace+time.Minute)) {
t.Fatal("pushed past the grace, a holder is not expected")
}
if _, ok := sends["laptop"].carried["keeper"]; !ok {
t.Fatalf("the send's builds do not carry keeper: %v", sends["laptop"].carried)
}
pushedAndApplied(t, open, "laptop")
if sends, err = readDeliveries(ctx, open.inventory); err != nil || !sends["laptop"].settled("keeper", time.Now()) {
t.Fatalf("pushed and reported applied, a holder is not expected to answer (%v)", err)
}
}
+140
View File
@@ -0,0 +1,140 @@
package main
import (
"context"
"fmt"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
)
// A binding to data moves only by a person (novox/hq ADR 0232, issue 273).
//
// The resolver keeps each consumer of a provision that keeps its data at the provider it was last
// sent (catalogue/bound.go) and says what it would have moved. This is where that is said: on the
// push that composed it, and by the self-check's D12 every run, as an urgent condition naming the
// consumer, both providers and the pin that confirms the move.
// The condition kinds of D12.
const (
// kindBindingKept is a move the resolver refused: the consumer is still where its data is.
kindBindingKept = "binding-kept"
// kindBindingMoved is a consumer about to be sent another provider than the one on record with
// no pin naming it — what the resolver exists to make impossible, said if it ever is not.
kindBindingMoved = "binding-moved"
// kindBindingMoving is a move a pin asked for, not yet sent: a person's act, said so that the
// data is moved before the push that carries it.
kindBindingMoving = "binding-moving"
)
// probeBindingsID is the self-check's id for this probe, and the source of what it raises.
const probeBindingsID = "D12"
// keptObservation is the urgent condition for one refused move.
func keptObservation(k catalogue.KeptBinding) conditions.Observation {
return conditions.Observation{Scope: conditions.ScopeMachine, ID: bindingID(k.Machine, k.Consumer, k.Provision),
Token: kindBindingKept, Kind: kindBindingKept, Machine: k.Machine, Also: otherMachines(k.Machine, k.Bound.Node, k.Would.Node),
Severity: conditions.Urgent, Resolver: conditions.ResolverOperator,
Summary: fmt.Sprintf("on %s, the mesh %s", k.Machine, k.String())}
}
func bindingID(machine, consumer, provision string) string {
return machine + "." + consumer + "." + provision
}
func otherMachines(machine string, nodes ...string) []string {
var out []string
seen := map[string]bool{machine: true}
for _, n := range nodes {
if n != "" && !seen[n] {
seen[n] = true
out = append(out, n)
}
}
return out
}
// reportKept says every move a machine's resolution refused, on the push composing it, and raises its
// condition at once where this process keeps the conditions: a push is when a person is looking.
func reportKept(ctx context.Context, plan catalogue.Resolution) {
for _, k := range plan.Kept {
fmt.Printf("%s: the mesh %s\n", plan.Node, k)
if conditionsFrom != nil {
o := keptObservation(k)
o.Source = probeBindingsID
if _, err := conditionsFrom.Observe(ctx, o); err != nil {
fmt.Printf("%s: and the condition for it could not be raised: %v\n", plan.Node, err)
}
}
}
}
// probeBindings is D12: every consumer of a provision that keeps its data is bound where it was last
// sent, on every machine — the resolver kept it there (said, urgent, until a person pins), or a pin
// moves it (said, so the data goes first), and never anything else.
func probeBindings(ctx context.Context, d *doctor) ([]conditions.Observation, error) {
inv := d.open.inventory
nodes, err := inv.Nodes(ctx)
if err != nil {
return nil, err
}
var out []conditions.Observation
for _, n := range nodes {
plan, _, err := planFor(ctx, d.open, n.Name)
if err != nil {
if unresolvable(err) {
// D1 says it, with the binding that refused it when that is why.
continue
}
return nil, fmt.Errorf("%s cannot be worked out: %w", n.Name, err)
}
bound, err := inv.BindingsFor(ctx, n.Name)
if err != nil {
return nil, err
}
pins, err := inv.PinsFor(ctx, n.Name)
if err != nil {
return nil, err
}
out = append(out, bindingFindings(plan, bound, pins)...)
}
return out, nil
}
// bindingFindings is what one machine's resolution says against its record.
func bindingFindings(plan catalogue.Resolution, bound map[string]map[string]catalogue.Chosen,
pins map[string]catalogue.Chosen) []conditions.Observation {
var out []conditions.Observation
for _, k := range plan.Kept {
out = append(out, keptObservation(k))
}
said := map[string]bool{}
for _, need := range plan.Needs {
if !need.KeepsData || need.ByRecord {
continue
}
was, recorded := bound[need.For][need.Name]
now := catalogue.Chosen{Node: need.From, Module: need.Module}
if !recorded || was == now || (was.Module == "" && was.Node == now.Node) {
continue
}
id := bindingID(plan.Node, need.For, need.Name)
if said[id] {
continue
}
said[id] = true
o := conditions.Observation{Scope: conditions.ScopeMachine, ID: id, Machine: plan.Node,
Also: otherMachines(plan.Node, was.Node, now.Node), Resolver: conditions.ResolverOperator}
if pin, pinned := pins[need.Name]; pinned && pin.Node == now.Node && (pin.Module == "" || pin.Module == now.Module) {
o.Token, o.Kind, o.Severity = kindBindingMoving, kindBindingMoving, conditions.Warning
o.Summary = fmt.Sprintf("on %s, %s's %s moves from %s to %s at the next push, by the pin — its data "+
"is on %s: move it first", plan.Node, need.For, need.Name, was, now, was)
} else {
o.Token, o.Kind, o.Severity = kindBindingMoved, kindBindingMoved, conditions.Urgent
o.Summary = fmt.Sprintf("on %s, %s's %s would be sent %s, and it is bound to %s, where its data is, "+
"with no pin naming %s — a move nothing asked for", plan.Node, need.For, need.Name, now, was, now)
}
out = append(out, o)
}
return out
}
+183
View File
@@ -0,0 +1,183 @@
package main
import (
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
)
// novox/hq issue 273, ADR 0232: a consumer of a provision that keeps its data moves only by a pin.
func storeManifests() []catalogue.Manifest {
return []catalogue.Manifest{
{Module: "store", Version: "1",
Provides: []catalogue.Offer{{Name: "postgres-database", Scope: catalogue.ScopeMesh}},
Claims: []catalogue.Claim{{Name: "mesh-store", Scope: catalogue.ScopeMesh}},
Serves: map[string]map[string]any{"postgres-database": {"port": 5432}},
Grants: map[string]string{"postgres-database": "/var/lib/mesh/store/grants"}},
{Module: "resolver", Version: "1",
Provides: []catalogue.Offer{{Name: "wildcard-resolution", Scope: catalogue.ScopeMesh}},
Claims: []catalogue.Claim{{Name: "mesh-dns-resolver", Scope: catalogue.ScopeMesh}}},
{Module: "network", Version: "1", Requires: []string{"wildcard-resolution"}},
{Module: "board", Version: "1", Requires: []string{"postgres-database"}},
}
}
func need(t *testing.T, plan catalogue.Resolution, consumer, provision string) catalogue.Needed {
t.Helper()
for _, n := range plan.Needs {
if n.For == consumer && n.Name == provision {
return n
}
}
t.Fatalf("no %s for %s: %+v", provision, consumer, plan.Needs)
return catalogue.Needed{}
}
// The incident through the stores: the laptop runs its own store and a consumer of it, the anchor's
// store holds the mesh's seat. The consumer stays beside its data, the resolver follows its seat, the
// binding is recorded as sent, and a pin — only a pin — moves it, said before the push that carries it.
func TestTheIncidentAConsumerStaysBesideItsDataUntilAPersonPinsIt(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
if _, err := inv.SeedSeats(ctx, catalogue.DefaultSeats()); err != nil {
t.Fatal(err)
}
for _, m := range storeManifests() {
register(t, open, m)
}
assignAll := func(pairs ...[2]string) {
for _, a := range pairs {
if _, err := assign(ctx, open, a[0], a[1]); err != nil {
t.Fatalf("assign %s %s: %v", a[0], a[1], err)
}
}
}
// The anchor's store and resolver hold the mesh's seats, on record; the laptop runs its own of each.
assignAll([2]string{"anchor", "store"}, [2]string{"anchor", "resolver"})
for _, seat := range [][2]string{{"mesh-store", "store"}, {"mesh-dns-resolver", "resolver"}} {
if err := inv.HoldSeat(ctx, seat[0], catalogue.ScopeMesh, "anchor", seat[1]); err != nil {
t.Fatal(err)
}
}
assignAll([2]string{"laptop", "store"}, [2]string{"laptop", "resolver"}, [2]string{"laptop", "network"},
[2]string{"laptop", "board"})
plan, _, err := planFor(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
if n := need(t, plan, "board", "postgres-database"); n.From != "laptop" || n.Module != "store" || !n.KeepsData {
t.Fatalf("the consumer was bound to %s/%s (keeps data: %v); its data is beside it", n.From, n.Module, n.KeepsData)
}
if n := need(t, plan, "network", "wildcard-resolution"); n.From != "anchor" || n.KeepsData {
t.Fatalf("the resolver was bound to %s (keeps data: %v); its seat is held on anchor (issue 258)", n.From, n.KeepsData)
}
// Sent, and recorded: only the binding to data.
bindings := boundToData(plan, nil)
if len(bindings) != 1 || bindings[0].Provider != (catalogue.Chosen{Node: "laptop", Module: "store"}) {
t.Fatalf("recorded %+v", bindings)
}
if err := inv.RecordBindings(ctx, "laptop", bindings); err != nil {
t.Fatal(err)
}
if found, err := probeBindings(ctx, &doctor{open: open}); err != nil || len(found) != 0 {
t.Fatalf("a mesh bound where it was sent: %+v, %v", found, err)
}
// The laptop's store taken away: refused, not moved to the anchor's empty one.
if err := inv.Unassign(ctx, "laptop", "store"); err != nil {
t.Fatal(err)
}
if _, _, err := planFor(ctx, open, "laptop"); err == nil || !unresolvable(err) ||
!strings.Contains(err.Error(), "board on laptop is bound to laptop/store") ||
!strings.Contains(err.Error(), "pin laptop postgres-database anchor store") {
t.Fatalf("the consumer's store went and it was answered elsewhere: %v", err)
}
// A person pins the anchor's: it moves, said before it is sent, and the record keeps where it was.
if err := inv.PinProvision(ctx, "laptop", "postgres-database", "anchor", "store"); err != nil {
t.Fatal(err)
}
plan, _, err = planFor(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
if n := need(t, plan, "board", "postgres-database"); n.From != "anchor" {
t.Fatalf("pinned to the anchor and bound to %s", n.From)
}
found, err := probeBindings(ctx, &doctor{open: open})
if err != nil {
t.Fatal(err)
}
if len(found) != 1 || found[0].Kind != kindBindingMoving || found[0].Severity != conditions.Warning ||
!strings.Contains(found[0].Summary, "board's postgres-database moves from laptop/store to anchor/store") {
t.Fatalf("a pinned move is not said before it is sent: %+v", found)
}
if err := inv.RecordBindings(ctx, "laptop", boundToData(plan, nil)); err != nil {
t.Fatal(err)
}
all, err := inv.Bindings(ctx)
if err != nil || len(all) != 1 || all[0].MovedFrom != "laptop/store" {
t.Fatalf("%+v, %v", all, err)
}
if found, err := probeBindings(ctx, &doctor{open: open}); err != nil || len(found) != 0 {
t.Fatalf("a move sent is still said: %+v, %v", found, err)
}
}
// What one machine's resolution says against its record.
func TestBindingFindingsSayAKeptMoveAndAMoveNothingAskedFor(t *testing.T) {
home, anchor := catalogue.Chosen{Node: "home", Module: "store"}, catalogue.Chosen{Node: "anchor", Module: "store"}
plan := catalogue.Resolution{Node: "laptop",
Kept: []catalogue.KeptBinding{{Machine: "laptop", Consumer: "board", Provision: "postgres-database",
Bound: home, Would: anchor}},
Needs: []catalogue.Needed{
{Name: "postgres-database", For: "board", From: "home", Module: "store", KeepsData: true},
{Name: "postgres-database", For: "game", From: "anchor", Module: "store", KeepsData: true},
{Name: "wildcard-resolution", For: "network", From: "anchor", Module: "resolver"},
}}
bound := map[string]map[string]catalogue.Chosen{
"board": {"postgres-database": home},
"game": {"postgres-database": home},
"network": {"wildcard-resolution": {Node: "home", Module: "resolver"}},
}
found := bindingFindings(plan, bound, nil)
if len(found) != 2 {
t.Fatalf("found %+v", found)
}
kept, moved := found[0], found[1]
if kept.Kind != kindBindingKept || kept.Severity != conditions.Urgent || kept.Machine != "laptop" ||
!strings.Contains(kept.Summary, "would move board's postgres-database from home/store to anchor/store") ||
!strings.Contains(kept.Summary, "its data is on home/store") ||
!strings.Contains(kept.Summary, "`pin laptop postgres-database anchor store` to confirm a move (and move the data first)") {
t.Errorf("kept: %+v", kept)
}
if moved.Kind != kindBindingMoved || moved.Severity != conditions.Urgent ||
!strings.Contains(moved.Summary, "game's postgres-database would be sent anchor/store") {
t.Errorf("moved: %+v", moved)
}
if kept.Key() == moved.Key() {
t.Error("two consumers, one condition")
}
}
// A push says the move it refused, and raises its condition at once.
func TestAPushSaysAKeptMoveAndRaisesItsCondition(t *testing.T) {
k, _ := withConditionsInMemory(t)
reportKept(t.Context(), catalogue.Resolution{Node: "laptop", Kept: []catalogue.KeptBinding{{Machine: "laptop",
Consumer: "board", Provision: "postgres-database", Bound: catalogue.Chosen{Node: "home", Module: "store"},
Would: catalogue.Chosen{Node: "anchor", Module: "store"}}}})
open, err := k.Open(t.Context())
if err != nil {
t.Fatal(err)
}
if len(open) != 1 || open[0].Kind != kindBindingKept || open[0].Severity != conditions.Urgent ||
open[0].Source != probeBindingsID {
t.Fatalf("raised %+v", open)
}
}
+21
View File
@@ -6,6 +6,7 @@ import (
"errors"
"flag"
"fmt"
"github.com/novox/mesh-controller/internal/conditions"
"os"
"strings"
"time"
@@ -676,6 +677,24 @@ type answers struct {
// refused, until the switch — and while there is any, the mesh is not all well: the order the
// machines' modules are built in is the mesh's to keep, and this is where it says it is not kept.
unheld []catalogue.Unheld
// conditions is every open condition (novox/hq to-be 45 §2), urgent first and then oldest first:
// what leads status, and what its all-well sentence needs to be none of, silenced ones included.
// A provider failing a consumer is one of them (ADR 0224). conditionsUnread says why they could
// not be read when they could not — never read as none.
conditions []conditions.Condition
conditionsUnread string
// overflowing is every module whose identity overflows the bound of a provision it requires
// (novox/hq ADR 0225): its provider leaves it out of the grants and composes everything else, so
// this is the one place it is said across the mesh. Not well while there is any.
overflowing []catalogue.Overflow
// handActs is how many acts were done by hand in the last seven days (novox/hq to-be 45 §7), nil
// where the log is not on hand; handActsUnread why it could not be read when it could not.
handActs *int
handActsUnread string
// heals is what the healers did in the last seven days (novox/hq to-be 45 §7): acts, and conditions
// handed to the operator; healsUnread why it could not be read.
heals *healsCount
healsUnread string
}
// heldBy is every artifact this mesh has built, for a build that may need one as its base.
@@ -692,12 +711,14 @@ func heldBy(ctx context.Context) map[string]string {
if err != nil {
fmt.Fprintf(os.Stderr, "could not read what this mesh has built, so a module naming a "+
"base will be told that base is missing: %v\n", err)
// empty-on-error: said above; a build that names a base is refused by name for want of it
return nil
}
defer open.Close()
held, err := open.inventory.Held(ctx)
if err != nil {
fmt.Fprintf(os.Stderr, "could not read what this mesh has built: %v\n", err)
// empty-on-error: said above; a build that names a base is refused by name for want of it
return nil
}
address, err := whereABuilderReachesTheStore(ctx, open.inventory)
+162
View File
@@ -0,0 +1,162 @@
package main
import (
"context"
"errors"
"fmt"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
)
// The streams and durable consumers the mesh's own traffic needs, derived once (novox/hq to-be 45 §4,
// D6 and D7).
//
// **What the controller asserts at its start and what the self-check expects to find are one
// derivation**, run against the bus to make them and against a recorder to list them. Two lists would
// drift, and a self-check comparing the bus with a second opinion of what should be there would find
// the drift rather than the fault.
// assertBusObjects brings every stream and consumer into being on r, and answers the machines that
// can now hear a declaration.
func assertBusObjects(ctx context.Context, inv *inventory.Inventory, r broker.Raiser) ([]string, error) {
nodes, err := inv.Nodes(ctx)
if err != nil {
return nil, err
}
names := make([]string, 0, len(nodes))
for _, n := range nodes {
names = append(names, n.Name)
}
if err := broker.Raise(r, names); err != nil {
return nil, err
}
// The work queues of the mesh's own roles (novox/hq ADR 0121). The queue before the holder,
// deliberately: work queues until somebody arrives to do it, so assigning a build machine a week
// after something started asking for builds flushes the backlog instead of having lost it.
// With the seats' holders, so each role's work queue gets the consumer its holder takes
// work from. Passed as nil until the first live raise, which left the build machine bound to a
// consumer nothing had created (2026-09-28).
holders, err := seatHolders(ctx, inv)
if err != nil {
return nil, err
}
if err := broker.RaiseSeats(r, inventory.MeshSeats(), holders); err != nil {
return nil, err
}
// And how every module hears what it consumes. Derived from the same records the user list is
// composed from, so a module the mesh grants a consumer's subjects has that consumer waiting.
// Done on every raise, not only when a credential is issued: every module moved onto this bus
// by the rollout was issued on the old one, and came up with nothing to bind to (2026-09-28).
consumers, err := moduleConsumers(ctx, inv)
if err != nil {
return nil, err
}
// Every one tried, and every failure named: one module's consumer the bus refuses is no reason
// the modules after it in the list hear nothing (novox/hq issue 208, where this runs on each send).
var failed []error
for _, c := range consumers {
if err := r.EnsureConsumer(c.Consumer); err != nil {
failed = append(failed, fmt.Errorf("how %s on %s hears what it consumes: %w", c.Module, c.Node, err))
}
}
if len(failed) > 0 {
return nil, errors.Join(failed...)
}
return names, nil
}
// What raises the condition a send says when the objects it implies could not be asserted, and its kind.
const (
sourceBusObjects = "bus-objects"
kindBusObjectsUnasserted = "bus-objects-unasserted"
)
// assertOnSend asserts, on the bus a send is about to use, every object assertBusObjects derives —
// **whenever a declaration is sent, not only when the controller starts** (novox/hq issue 208).
//
// A module assigned after the controller started was sent its declaration and found no consumer to
// bind (`consumer not found`, messenger on 2026-10-06), and a seat holder assigned after it found no
// worker: the objects a declaration implies were asserted at start and nowhere else, so they existed
// only for what was assigned before the last restart. The same derivation, not a second list of what
// a send needs: what start asserts, the self-check expects and a send asserts are one answer. Every
// part is idempotent, so asserting the whole of it again is the no-op a restart already relies on.
//
// **A failure is said and raised, and the send goes on.** The objects are the mesh's, not the
// machines' being sent: holding every machine back for one consumer that none of them may use would
// turn one fault into all of them, and the declarations are not what is wrong. It is never silent —
// said in the send's own output and raised as a condition, which the next send that asserts them
// clears — and the start-time raise still refuses to serve without them.
func assertOnSend(ctx context.Context, inv *inventory.Inventory, r broker.Raiser, indent string) error {
_, err := assertBusObjects(ctx, inv, r)
var observed []conditions.Observation
if err != nil {
fmt.Printf("%sTHE BUS DOES NOT HOLD WHAT THIS SEND IMPLIES: %v\n", indent, err)
fmt.Printf("%s a module may find no consumer to bind, or a holder no worker; sent anyway, raised as "+
"condition %s, and asserted again by the next send\n", indent, unassertedObservation(err).Key())
observed = append(observed, unassertedObservation(err))
}
// Observed when it failed, cleared when it did not: a send that asserted everything is the
// observation that the bus holds what it should.
if kerr := withKeeper(ctx, func(k *conditions.Keeper) error {
return k.Reconcile(ctx, sourceBusObjects, observed)
}); kerr != nil {
fmt.Printf("%sand whether the bus holds what this send implies could not be kept as a condition: %v\n",
indent, kerr)
}
return err
}
// unassertedObservation is a send's failure to assert the bus's objects, as a condition.
func unassertedObservation(err error) conditions.Observation {
return conditions.Observation{Scope: conditions.ScopeBus, ID: "objects", Token: "unasserted",
Kind: kindBusObjectsUnasserted, Severity: conditions.Warning, Source: sourceBusObjects,
Summary: "the bus's streams and consumers could not be asserted when a declaration was sent: " +
"a module may find no consumer to bind, or a seat's holder no worker",
Said: err.Error()}
}
// moduleConsumers is every module's durable consumer, from the records the user list is composed from.
func moduleConsumers(ctx context.Context, inv *inventory.Inventory) ([]broker.ModuleConsumer, error) {
records, err := inv.BusRecords(ctx)
if err != nil {
return nil, err
}
users, err := broker.Users(records)
if err != nil {
return nil, err
}
return broker.ConsumersOf(users), nil
}
// moduleConsumerCount is how many modules hear what they consume, for the raise's one line.
func moduleConsumerCount(ctx context.Context, inv *inventory.Inventory) (int, error) {
consumers, err := moduleConsumers(ctx, inv)
return len(consumers), err
}
// expectedBusObjects is every stream and consumer assertBusObjects would make, made nowhere.
func expectedBusObjects(ctx context.Context, inv *inventory.Inventory) ([]broker.Stream, []broker.Consumer, error) {
var rec recordingRaiser
if _, err := assertBusObjects(ctx, inv, &rec); err != nil {
return nil, nil, fmt.Errorf("what the bus should hold cannot be worked out: %w", err)
}
return rec.streams, rec.consumers, nil
}
// recordingRaiser keeps what it was asked to assert and asserts nothing.
type recordingRaiser struct {
streams []broker.Stream
consumers []broker.Consumer
}
func (r *recordingRaiser) EnsureStream(s broker.Stream) error {
r.streams = append(r.streams, s)
return nil
}
func (r *recordingRaiser) EnsureConsumer(c broker.Consumer) error {
r.consumers = append(r.consumers, c)
return nil
}
+199
View File
@@ -0,0 +1,199 @@
package main
import (
"errors"
"os"
"strings"
"testing"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// novox/hq issue 208: the bus's objects a declaration implies are asserted whenever one is sent, not
// only when the controller starts.
// aCarriedConsumer is the runtime on laptop and, carried by it, a module that consumes an event: the
// shape of messenger on 2026-10-06, assigned after the controller started.
func aCarriedConsumer(t *testing.T, open *stores) broker.Consumer {
t.Helper()
register(t, open, catalogue.Manifest{Module: catalogue.RuntimeModule, Version: "1",
OwnSecrets: catalogue.OwnSecrets{"broker": {Path: "/var/lib/mesh/node-tools/broker"}}})
register(t, open, catalogue.Manifest{Module: "messenger", Version: "1",
Consumes: []string{"billing.order.placed"}})
for _, m := range []string{catalogue.RuntimeModule, "messenger"} {
if _, err := open.inventory.Assign(t.Context(), "laptop", m); err != nil {
t.Fatal(err)
}
}
consumers, err := moduleConsumers(t.Context(), open.inventory)
if err != nil {
t.Fatal(err)
}
for _, c := range consumers {
if c.Module == "messenger" && c.Node == "laptop" {
return c.Consumer
}
}
t.Fatalf("messenger on laptop is derived no consumer: %+v", consumers)
return broker.Consumer{}
}
// aLateHolder is a module holding the build agent's seat on anchor, assigned after the controller
// started, and the worker its seat's queue should have for it.
func aLateHolder(t *testing.T, open *stores) broker.Consumer {
t.Helper()
register(t, open, catalogue.Manifest{Module: "late-builder", Version: "1",
Claims: []catalogue.Claim{{Name: "node-build-agent", Scope: catalogue.ScopeNode}}})
if _, err := open.inventory.Assign(t.Context(), "anchor", "late-builder"); err != nil {
t.Fatal(err)
}
for _, s := range inventory.MeshSeats() {
if s.Name == "node-build-agent" {
c, needed := broker.HolderConsumerFor("anchor", "late-builder", s)
if !needed {
t.Fatal("the build agent's seat needs no worker")
}
return c
}
}
t.Fatal("the mesh declares no build agent's seat")
return broker.Consumer{}
}
// asserted says whether a recording holds a consumer by stream and name.
func asserted(rec *recordingRaiser, want broker.Consumer) bool {
for _, c := range rec.consumers {
if c.Stream == want.Stream && c.Name == want.Name {
return true
}
}
return false
}
// What a send asserts includes what was assigned after the start's assertion: a carried module's
// consumer and a late holder's worker. The derivation is the start's own, so this holds without a bus.
func TestASendAssertsWhatWasAssignedAfterStart(t *testing.T) {
open := aMesh(t)
var atStart recordingRaiser
if _, err := assertBusObjects(t.Context(), open.inventory, &atStart); err != nil {
t.Fatal(err)
}
consumer := aCarriedConsumer(t, open)
worker := aLateHolder(t, open)
if asserted(&atStart, consumer) || asserted(&atStart, worker) {
t.Fatal("the start asserted what was not yet assigned; the test proves nothing")
}
var onSend recordingRaiser
if err := assertOnSend(t.Context(), open.inventory, &onSend, ""); err != nil {
t.Fatal(err)
}
if !asserted(&onSend, consumer) {
t.Fatalf("messenger's consumer %s on %s is not asserted by the send: %+v", consumer.Name, consumer.Stream, onSend.consumers)
}
if !asserted(&onSend, worker) {
t.Fatalf("the late holder's worker %s on %s is not asserted by the send: %+v", worker.Name, worker.Stream, onSend.consumers)
}
}
// failingRaiser refuses one consumer by name and records everything it was asked.
type failingRaiser struct {
recordingRaiser
refuse string
}
func (f *failingRaiser) EnsureConsumer(c broker.Consumer) error {
f.consumers = append(f.consumers, c)
if c.Name == f.refuse {
return errors.New("nats: API error: code=503 description=insufficient resources")
}
return nil
}
// A send whose objects cannot be asserted says so in its output and raises a condition — and the next
// send that asserts them clears it. The consumers after the refused one are still asked for.
func TestASendThatCannotAssertTheBusSaysSoAndRaisesACondition(t *testing.T) {
open := aMesh(t)
consumer := aCarriedConsumer(t, open)
worker := aLateHolder(t, open)
failing := &failingRaiser{refuse: consumer.Name}
var sendErr error
out := stdoutOf(t, func() error {
sendErr = assertOnSend(t.Context(), open.inventory, failing, " ")
return nil
})
if sendErr == nil || !strings.Contains(sendErr.Error(), "how messenger on laptop hears what it consumes") {
t.Fatalf("the failure is not answered: %v", sendErr)
}
if !strings.Contains(out, "THE BUS DOES NOT HOLD WHAT THIS SEND IMPLIES") ||
!strings.Contains(out, "insufficient resources") || !strings.Contains(out, "bus.objects.unasserted") {
t.Fatalf("the failure is not said in the send's output:\n%s", out)
}
if !asserted(&failing.recordingRaiser, worker) {
t.Fatal("the seat's worker was not asked for")
}
c, open1, err := conditionsFrom.Get(t.Context(), "bus.objects.unasserted")
if err != nil || !open1 {
t.Fatalf("no condition raised: %v", err)
}
if c.Kind != kindBusObjectsUnasserted || c.Source != sourceBusObjects ||
len(c.Evidence) == 0 || !strings.Contains(c.Evidence[0].Said, "insufficient resources") {
t.Fatalf("the condition does not say what failed: %+v", c)
}
// The next send asserts them, and that observation clears it.
if err := assertOnSend(t.Context(), open.inventory, &recordingRaiser{}, ""); err != nil {
t.Fatal(err)
}
if _, stillOpen, err := conditionsFrom.Get(t.Context(), "bus.objects.unasserted"); err != nil || stillOpen {
t.Fatalf("a send that asserted everything left the condition open: %v", err)
}
}
// Against a real bus, through the send's own grant: a module assigned after start has its consumer
// once a declaration is sent, and a holder assigned after start its seat's worker.
func TestNatsAModuleAssignedAfterStartGetsItsConsumerAtItsFirstSend(t *testing.T) {
url := os.Getenv("MESH_TEST_NATS")
if url == "" {
t.Skip("MESH_TEST_NATS unset")
}
open := aMesh(t)
js, err := broker.Dial(url)
if err != nil {
t.Fatal(err)
}
t.Cleanup(js.Close)
for _, s := range []string{"CONTROL", "NODES", "ASSIGNMENTS", "EVENTS"} {
_ = js.Context().DeleteStream(s)
}
// The controller starting, before either was assigned.
if _, err := assertBusObjects(t.Context(), open.inventory, js); err != nil {
t.Fatal(err)
}
consumer := aCarriedConsumer(t, open)
worker := aLateHolder(t, open)
_ = js.Context().DeleteConsumer(worker.Stream, worker.Name)
for _, c := range []broker.Consumer{consumer, worker} {
if _, err := js.Context().ConsumerInfo(c.Stream, c.Name); !errors.Is(err, nats.ErrConsumerNotFound) {
t.Fatalf("%s on %s exists before any send; the test proves nothing: %v", c.Name, c.Stream, err)
}
}
server := link.ConnectNats(js, nil, nil)
if err := (overTheBus{open: open, server: server}).grant(t.Context(), nil); err != nil {
t.Fatal(err)
}
for _, c := range []broker.Consumer{consumer, worker} {
if _, err := js.Context().ConsumerInfo(c.Stream, c.Name); err != nil {
t.Fatalf("%s on %s is not on the bus after a send: %v", c.Name, c.Stream, err)
}
}
if _, raised, _ := conditionsFrom.Get(t.Context(), "bus.objects.unasserted"); raised {
t.Fatal("a send that asserted everything raised a condition")
}
}
+81
View File
@@ -0,0 +1,81 @@
package main
import (
"context"
"encoding/json"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/link"
)
// consoleWaits is how long the console waits for an answer (mesh-tools node-tools/internal/bus
// RequestTimeout) — the shortest wait of a caller the mesh ships.
const consoleWaits = 30 * time.Second
// **A call's one answer is never later than its caller or the bus allow** (novox/hq issue 265): a
// holder answers within AnswerWithin, which must be inside both the console's wait and the window the
// bus gives an answer. Before, the console waited 30s, the bus 60s, and a push ran as long as it ran.
func TestAVerbAnswersInsideEveryWaitOnIt(t *testing.T) {
if link.AnswerWithin >= consoleWaits/2 {
t.Errorf("a call answers within %s: not well inside the console's %s", link.AnswerWithin, consoleWaits)
}
if link.AnswerWithin >= broker.ResponseTTL {
t.Errorf("a call answers within %s, after the bus stops permitting an answer at %s", link.AnswerWithin, broker.ResponseTTL)
}
}
// A push — named or through command — answers before it runs: it sends the machine holding the bus
// first, and the broker reloading its user list forgets the answer it was about to permit.
func TestAPushAnswersBeforeItSends(t *testing.T) {
for _, c := range []struct {
verb string
args map[string]any
want bool
}{
{"push", map[string]any{"node": "anchor", "why": "w"}, true},
{"push", map[string]any{"why": "w"}, true},
{"command", map[string]any{"command": "push anchor --why w"}, true},
{"command", map[string]any{"command": "push --behind --why=w"}, true},
{"command", map[string]any{"command": "builds"}, false},
{"status", map[string]any{}, false},
{"assign", map[string]any{"node": "anchor", "module": "m"}, false},
} {
argv, err := argvFor(c.verb, c.args)
if err != nil {
t.Fatalf("%s %v: %v", c.verb, c.args, err)
}
if got := answersFirst(argv); got != c.want {
t.Errorf("%s %v answers first: %v, want %v", c.verb, c.args, got, c.want)
}
}
}
// `calls` is served, takes a call's id and nothing else, and says plainly when it holds no such call.
func TestCallsIsServedAndSaysWhatItKeeps(t *testing.T) {
handlers, behind, err := seatToolHandlers()
if err != nil || len(behind) != 0 {
t.Fatalf("%v %v", behind, err)
}
calls, ok := handlers["calls"]
if !ok {
t.Fatal("calls is not served")
}
if _, err := calls(context.Background(), json.RawMessage(`{"node":"anchor"}`)); err == nil ||
!strings.Contains(err.Error(), `"node"`) {
t.Errorf("calls took an argument it does not declare: %v", err)
}
if _, err := calls(context.Background(), json.RawMessage(`{"call":"call-0-0"}`)); err == nil ||
!strings.Contains(err.Error(), "not across a restart") {
t.Errorf("an unknown call was not said plainly: %v", err)
}
got, err := calls(context.Background(), json.RawMessage(`{}`))
if err != nil {
t.Fatal(err)
}
if _, listed := got.(map[string]any)["calls"]; !listed {
t.Errorf("calls answered %v", got)
}
}
+38 -1
View File
@@ -25,7 +25,17 @@ import (
// mesh seat is judged fully only at registration. A seat another module declares is unknown unless
// that module's manifest is passed too. Both are printed as a note, not as a problem — a check that
// refused what it could not see would teach people to ignore it.
//
// **And every identity against every bound it meets** (novox/hq ADR 0225, issue 263): each module's
// identity, on a machine whose name is `longestMachine` characters, against the bound of every
// provision it wants that a manifest given here offers. An overflow is refused in the pull request
// that introduces it — a new requirement, a lowered bound, a longer slug — instead of on the
// provider's machine when a real machine's name first meets the module's.
func moduleCheck(paths []string, out io.Writer) error {
return moduleCheckFor(paths, catalogue.DefaultLongestMachine, out)
}
func moduleCheckFor(paths []string, longestMachine int, out io.Writer) error {
if len(paths) == 0 {
return errors.New("module check <manifest.json>... — one file per module; pass every " +
"manifest of a repository together so the rules between them are checked too")
@@ -74,6 +84,24 @@ func moduleCheck(paths []string, out io.Writer) error {
}
failed += len(problems)
// Between the manifests too: an identity against the bounds of the provisions it wants, which
// only the provider's manifest states.
identities := catalogue.IdentityProblems(shelf, longestMachine)
sort.Strings(identities)
for _, p := range identities {
fmt.Fprintln(out, p)
}
failed += len(identities)
// And the data each module keeps (novox/hq ADR 0233): a provider that grants says what it keeps for
// its consumers, a directory a container writes is declared, and no backup line is written by hand.
data := catalogue.DataProblems(shelf)
sort.Strings(data)
for _, p := range data {
fmt.Fprintln(out, p)
}
failed += len(data)
var names []string
for name := range shelf {
names = append(names, name)
@@ -102,6 +130,14 @@ func moduleCheck(paths []string, out io.Writer) error {
if len(m.Reads) > 0 {
fmt.Fprintf(out, ", reads %s", strings.Join(m.Reads, ", "))
}
// The data it keeps, by class, so a reviewer sees what the mesh will protect and how.
if items := m.DataItems(); len(items) > 0 {
kept := make([]string, 0, len(items))
for _, it := range items {
kept = append(kept, it.ID+" ("+it.Class+")")
}
fmt.Fprintf(out, ", keeps %s", strings.Join(kept, ", "))
}
fmt.Fprintln(out)
}
if failed > 0 {
@@ -109,7 +145,8 @@ func moduleCheck(paths []string, out io.Writer) error {
}
fmt.Fprintf(out, "%d manifest(s) checked. Judged against the seats this binary carries; a claim on "+
"one of the mesh's own seats is judged fully at registration, and a seat declared by a "+
"module not given here reads as unknown\n", len(paths))
"module not given here reads as unknown. Identities judged on a %d-character machine name, "+
"against the bounds of the providers given here\n", len(paths), longestMachine)
return nil
}
+441
View File
@@ -0,0 +1,441 @@
package main
import (
"context"
"errors"
"flag"
"fmt"
"os"
"slices"
"strconv"
"strings"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
)
// What is wrong, kept until observation says it is not (novox/hq to-be 45 §2, ADR 0227).
//
// **`status` used to be the only place the mesh said it was wrong, and only to whoever asked.** Every
// core failure of research 031 was found by a person looking. A condition is the mesh saying it: raised
// by a watchdog when a signal is late (signals.go), by a probe when an invariant does not hold
// (doctor.go), or by an event a provider sends (standing.go); kept on the bus with since-when and
// evidence; said on the bus as it changes, for the operator's channel to carry; and cleared when an
// observation says it is resolved. Nobody resolves one by hand. A person who knows silences it, for a
// while, with a reason, and that is recorded as a hand act.
// conditionsFrom is the serving controller's keeper; nil in any other process, which opens its own.
var conditionsFrom *conditions.Keeper
// keeperOn is a keeper over the store on a connection, saying its transitions on that connection.
func keeperOn(ctx context.Context, conn *nats.Conn) (*conditions.Keeper, error) {
store, history, err := conditions.OnTheBus(ctx, conn)
if err != nil {
return nil, err
}
js, err := conn.JetStream()
if err != nil {
return nil, err
}
return conditions.NewKeeper(ctx, conditions.Options{Store: store, History: history,
Teller: link.OverNATS{Conn: conn, JS: js},
Say: func(format string, args ...any) { fmt.Fprintf(os.Stderr, format+"\n", args...) },
// What status leads with changed: composed again soon (a nudge outside the serving controller
// does nothing).
Changed: statusFrom.nudge,
// Written under the lease, carrying its epoch (novox/hq to-be 45 §6).
Epoch: func() (uint64, error) { return theLease.epoch(context.WithoutCancel(ctx)) }}), nil
}
// withKeeper runs f with the serving controller's keeper, or one of its own that says everything
// it was given before it returns.
func withKeeper(ctx context.Context, f func(*conditions.Keeper) error) error {
if conditionsFrom != nil {
return f(conditionsFrom)
}
return onTheBus(func(conn *nats.Conn) error {
k, err := keeperOn(ctx, conn)
if err != nil {
return err
}
defer func() {
flushing, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()
k.Close(flushing)
}()
return f(k)
})
}
// openConditions is every open condition, for `status` and `node show`: from the serving keeper, or
// read from the bus. Where there is no bus to read it from, it says so — never "none open".
func openConditions(ctx context.Context) ([]conditions.Condition, error) {
if conditionsFrom != nil {
return conditionsFrom.Open(ctx)
}
if _, err := broker.BusAddress(); err != nil {
return nil, fmt.Errorf("this process has no bus to read the conditions from: %w", err)
}
var out []conditions.Condition
err := onTheBus(func(conn *nats.Conn) error {
store, _, err := conditions.OnTheBus(ctx, conn)
if err != nil {
return err
}
reading, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
out, err = conditions.Read(reading, store)
return err
})
return out, err
}
// conditionsUsage is how the verb is typed.
const conditionsUsage = "conditions [--scope S] [--severity urgent|warning] [--machine M] [--json] | " +
"conditions show <key> | conditions silence <key> --for <duration> --why <text> | " +
"conditions history [--days N] [--key K] [--json]"
// conditionsCommand is `conditions`, `conditions show`, `conditions silence` and `conditions history`.
func conditionsCommand(ctx context.Context, args []string) error {
sub := "list"
if len(args) > 0 && !strings.HasPrefix(args[0], "-") {
sub, args = args[0], args[1:]
}
switch sub {
case "list":
return listConditions(ctx, args)
case "show":
return showCondition(ctx, args)
case "silence":
return silenceCondition(ctx, args)
case "history":
return conditionHistory(ctx, args)
}
return errors.New(conditionsUsage)
}
func listConditions(ctx context.Context, args []string) error {
set := flag.NewFlagSet("conditions", flag.ContinueOnError)
scope := set.String("scope", "", "only this scope: "+strings.Join(conditions.Scopes, ", "))
severity := set.String("severity", "", "only urgent, or only warning")
machine := set.String("machine", "", "only those about this machine")
asJSON := set.Bool("json", false, "as data")
if rest, err := parseAround(set, args); err != nil {
return err
} else if len(rest) > 0 {
return errors.New(conditionsUsage)
}
if *severity != "" && *severity != string(conditions.Urgent) && *severity != string(conditions.Warning) {
return fmt.Errorf("a severity is urgent or warning, not %q", *severity)
}
open, err := openConditions(ctx)
if err != nil {
return err
}
var out []conditions.Condition
for _, c := range open {
if (*scope == "" || c.Subject.Scope == *scope) && (*severity == "" || string(c.Severity) == *severity) &&
(*machine == "" || concerns(c, *machine)) {
out = append(out, c)
}
}
if *asJSON {
if out == nil {
out = []conditions.Condition{}
}
return printJSON(map[string]any{"conditions": out, "open": len(open),
"note": "urgent first, then oldest first; a condition clears when observation says so, never by hand"})
}
if len(out) == 0 {
if len(open) == 0 {
fmt.Println("no open conditions")
} else {
fmt.Printf("none of the %d open condition(s) is about that\n", len(open))
}
return nil
}
for _, line := range conditionLines(out, time.Now()) {
fmt.Println(line)
}
return nil
}
// concerns says whether a condition is about a machine: it names it, or its key does.
func concerns(c conditions.Condition, machine string) bool {
if c.Subject.Machine == machine || slices.Contains(c.Subject.Also, machine) {
return true
}
for _, part := range strings.Split(c.Key, ".") {
if part == machine {
return true
}
}
return false
}
// conditionLines is how a list of conditions reads: one line each, its silence under it.
func conditionLines(list []conditions.Condition, now time.Time) []string {
var out []string
for _, c := range list {
times := ""
if c.Count > 1 {
times = fmt.Sprintf(", raised %d times", c.Count)
}
out = append(out, fmt.Sprintf(" %-7s %s — %s (since %s%s)", strings.ToUpper(string(c.Severity)),
c.Key, c.Summary, c.Raised.Local().Format("2006-01-02 15:04"), times))
if c.SilencedAt(now) {
out = append(out, fmt.Sprintf(" silenced until %s by %s: %s",
c.Silenced.Until.Local().Format("2006-01-02 15:04"), c.Silenced.By, c.Silenced.Why))
}
}
return out
}
func showCondition(ctx context.Context, args []string) error {
set := flag.NewFlagSet("conditions show", flag.ContinueOnError)
asJSON := set.Bool("json", false, "as data")
rest, err := parseAround(set, args)
if err != nil {
return err
}
if len(rest) != 1 {
return errors.New("conditions show <key>")
}
key := rest[0]
var c conditions.Condition
var found bool
if conditionsFrom != nil {
c, found, err = conditionsFrom.Get(ctx, key)
} else {
err = onTheBus(func(conn *nats.Conn) error {
store, _, err := conditions.OnTheBus(ctx, conn)
if err != nil {
return err
}
c, found, err = conditions.ReadOne(ctx, store, key)
return err
})
}
if err != nil {
return err
}
if !found {
return fmt.Errorf("no condition %s is open — `conditions` lists those that are, and `conditions "+
"history --key %s` what became of it", key, key)
}
if *asJSON {
return printJSON(c)
}
now := time.Now()
fmt.Printf("%s %s\n %s\n\n", strings.ToUpper(string(c.Severity)), c.Key, c.Summary)
fmt.Printf(" kind %s\n about %s %s", c.Kind, c.Subject.Scope, c.Subject.ID)
if c.Subject.Machine != "" {
fmt.Printf(", on %s", c.Subject.Machine)
}
fmt.Printf("\n raised by %s\n since %s (%s ago), observed %d time(s), last %s ago\n",
c.Source, c.Raised.Local().Format("2006-01-02 15:04:05"), roughly(now.Sub(c.Raised)), c.Observations,
now.Sub(c.LastObserved).Round(time.Second))
if c.Count > 1 {
fmt.Printf(" raised %d times, each within ten minutes of clearing\n", c.Count)
}
fmt.Printf(" resolved by %s\n", resolverWords(c.Resolver))
if c.Silenced != nil {
fmt.Printf(" silenced until %s by %s: %s\n", c.Silenced.Until.Local().Format("2006-01-02 15:04"),
c.Silenced.By, c.Silenced.Why)
}
if len(c.Tried) > 0 {
fmt.Println("\n tried:")
for _, t := range c.Tried {
fmt.Printf(" %s %s — %s: %s\n", t.At.Local().Format("2006-01-02 15:04"), orHealer(t.By), t.What, t.Outcome)
}
}
fmt.Println("\n evidence, newest first:")
for _, e := range c.Evidence {
fmt.Printf(" %s %s\n", e.At.Local().Format("2006-01-02 15:04:05"), e.Said)
}
return nil
}
func resolverWords(r string) string {
switch r {
case conditions.ResolverSelf:
return "itself: it clears when observation says it is resolved"
case conditions.ResolverOperator:
return "the operator: nothing in the mesh will repair it"
case conditions.ResolverAgent:
return "an agent"
}
return r
}
// silenceCondition stops a condition's messages for a while (to-be 45 §2). A hand act: recorded with
// who and why before it is done, its cause the condition's kind unless one is given.
func silenceCondition(ctx context.Context, args []string) error {
set := flag.NewFlagSet("conditions silence", flag.ContinueOnError)
forFlag := set.String("for", "", "how long: 30m, 4h, 2d — at most 7d")
acts := addHandActFlags(set)
rest, err := parseAround(set, args)
if err != nil {
return err
}
if len(rest) != 1 {
return errors.New("conditions silence <key> --for <duration> --why <text>")
}
key := rest[0]
if err := acts.require("conditions silence"); err != nil {
return err
}
d, err := parseFor(*forFlag)
if err != nil {
return err
}
if d > conditions.MaxSilence {
return fmt.Errorf("a condition is silenced for at most %s at once; past it, say so again", conditions.MaxSilence)
}
return withKeeper(ctx, func(k *conditions.Keeper) error {
c, found, err := k.Get(ctx, key)
if err != nil {
return err
}
if !found {
return fmt.Errorf("no condition %s is open — `conditions` lists them. Nothing was silenced", key)
}
if strings.TrimSpace(*acts.cause) == "" {
*acts.cause = c.Kind
}
*acts.condition = key
acts.record(ctx, "conditions silence", []string{key, "--for", *forFlag})
held, err := k.Silence(ctx, key, d, link.Caller(), *acts.why)
if err != nil {
return err
}
fmt.Printf("%s is silenced until %s: no message is sent for it until then. It is still open, and "+
"`status` still says it; it clears when observation says it is resolved\n",
held.Key, held.Silenced.Until.Local().Format("2006-01-02 15:04"))
return nil
})
}
// parseFor reads a duration, days included.
func parseFor(s string) (time.Duration, error) {
s = strings.TrimSpace(s)
if s == "" {
return 0, errors.New("say for how long: --for 30m, 4h or 2d")
}
if days, ok := strings.CutSuffix(s, "d"); ok {
n, err := strconv.Atoi(days)
if err != nil || n <= 0 {
return 0, fmt.Errorf("%q is not a number of days", s)
}
return time.Duration(n) * 24 * time.Hour, nil
}
d, err := time.ParseDuration(s)
if err != nil || d <= 0 {
return 0, fmt.Errorf("%q is not a duration: 30m, 4h or 2d", s)
}
return d, nil
}
func conditionHistory(ctx context.Context, args []string) error {
set := flag.NewFlagSet("conditions history", flag.ContinueOnError)
days := set.Int("days", 7, "how many days back, at most 90")
key := set.String("key", "", "only this condition")
asJSON := set.Bool("json", false, "as data")
if rest, err := parseAround(set, args); err != nil {
return err
} else if len(rest) > 0 {
return errors.New("conditions history [--days N] [--key K] [--json]")
}
since := time.Now().Add(-time.Duration(*days) * 24 * time.Hour)
var events []conditions.Event
read := func(h conditions.History) error {
var err error
events, err = h.Since(ctx, since)
return err
}
var err error
if conditionsFrom != nil {
events, err = conditionsFrom.HistorySince(ctx, since)
} else {
err = onTheBus(func(conn *nats.Conn) error {
_, history, err := conditions.OnTheBus(ctx, conn)
if err != nil {
return err
}
return read(history)
})
}
if err != nil {
return err
}
var out []conditions.Event
for _, e := range events {
if *key == "" || e.Key == *key {
out = append(out, e)
}
}
if *asJSON {
if out == nil {
out = []conditions.Event{}
}
return printJSON(map[string]any{"history": out, "days": *days})
}
if len(out) == 0 {
fmt.Printf("nothing was raised, changed or cleared in the last %d day(s)\n", *days)
return nil
}
for _, e := range out {
line := fmt.Sprintf("%s %-13s %s", e.At.Local().Format("2006-01-02 15:04:05"), e.Change, e.Key)
switch e.Change {
case conditions.ChangeRaised, conditions.ChangeReopened:
line += " — " + e.Summary
case conditions.ChangeSeverity:
line += fmt.Sprintf(" — %s, was %s", e.Severity, e.Was)
case conditions.ChangeResolver:
line += fmt.Sprintf(" — %s, was %s", e.Resolver, e.Was)
default:
if e.Why != "" {
line += " — " + e.Why
}
}
fmt.Println(line)
}
return nil
}
// printConditions is the status section that leads it: every open condition, urgent first, oldest
// first, silenced ones with their expiry (to-be 45 §2). A store that could not be read is said, and
// is not "none open".
func printConditions(list []conditions.Condition, unread string, now time.Time) {
if unread != "" {
fmt.Printf("the open conditions could NOT be read, so whether anything is wrong is not known: %s\n\n", unread)
return
}
if len(list) == 0 {
return
}
urgent := 0
for _, c := range list {
if c.Severity == conditions.Urgent {
urgent++
}
}
fmt.Printf("%d open condition(s), %d urgent:\n\n", len(list), urgent)
for _, line := range conditionLines(list, now) {
fmt.Println(line)
}
fmt.Printf("\n `conditions show <key>` says more; each clears when observation says it is resolved, " +
"never by hand — `conditions silence <key> --for <d> --why <text>` stops its messages\n\n")
}
// orHealer is who tried, as an attempt names it.
func orHealer(by string) string {
if by == "" {
return "a healer"
}
return by
}
+107
View File
@@ -0,0 +1,107 @@
package main
import (
"errors"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/conditions"
)
// The verbs of the condition store (novox/hq to-be 45 §2) as the console reaches them, and status led
// by what is open.
func TestTheConditionsVerbComposesEachShape(t *testing.T) {
for _, c := range []struct {
args map[string]any
want string
}{
{map[string]any{}, "conditions --json"},
{map[string]any{"severity": "urgent", "machine": "ace"}, "conditions --json --severity urgent --machine ace"},
{map[string]any{"key": "machine.ace.silent"}, "conditions show machine.ace.silent --json"},
{map[string]any{"history": "true", "days": "3", "key": "machine.ace.silent"},
"conditions history --json --days 3 --key machine.ace.silent"},
{map[string]any{"silence": "machine.ace.silent", "for": "2h", "why": "on the train"},
"conditions silence machine.ace.silent --for 2h --why on the train"},
{map[string]any{"run": "true"}, "doctor run --json"},
{map[string]any{}, "doctor --json"},
} {
verb := "conditions"
if strings.HasPrefix(c.want, "doctor") {
verb = "doctor"
}
argv, err := argvFor(verb, c.args)
if err != nil || strings.Join(argv, " ") != c.want {
t.Errorf("%s %v composed %q (%v), want %q", verb, c.args, strings.Join(argv, " "), err, c.want)
}
}
for _, c := range []struct {
verb string
args map[string]any
}{
{"conditions", map[string]any{"silence": "machine.ace.silent", "why": "x"}}, // no for
{"doctor", map[string]any{"run": "true", "signals": "true"}}, // two at once
{"conditions", map[string]any{"silence": "machine.ace.silent", "for": "1h", "why": "x", "days": "3"}}, // passed over
} {
if argv, err := argvFor(c.verb, c.args); err == nil {
t.Errorf("%s %v composed %v", c.verb, c.args, argv)
}
}
if repairingCommand([]string{"conditions", "silence", "k"}) != "conditions silence" {
t.Error("a silence is not a hand act")
}
}
// **A silence through the verb is recorded and bounded**; the condition stays open.
func TestASilenceThroughTheVerbHoldsAndTheConditionStaysOpen(t *testing.T) {
k, _ := withConditionsInMemory(t)
ctx := t.Context()
if _, err := k.Observe(ctx, conditions.Observation{Scope: conditions.ScopeMachine, ID: "ace", Kind: "silent",
Severity: conditions.Warning, Summary: "ace is silent", Source: "S1"}); err != nil {
t.Fatal(err)
}
if err := conditionsCommand(ctx, []string{"silence", "machine.ace.silent", "--for", "2d"}); err == nil {
t.Fatal("silenced without saying why")
}
if err := conditionsCommand(ctx, []string{"silence", "machine.ace.silent", "--for", "8d", "--why", "x"}); err == nil {
t.Fatal("silenced for more than a week")
}
said := printed(t, func() error {
return conditionsCommand(ctx, []string{"silence", "machine.ace.silent", "--for", "2d", "--why", "on the train"})
})
if !strings.Contains(said, "is silenced until") {
t.Fatalf("%s", said)
}
c, found, _ := k.Get(ctx, "machine.ace.silent")
if !found || c.Silenced == nil || c.Silenced.Why != "on the train" {
t.Fatalf("%+v", c)
}
listed := printed(t, func() error { return conditionsCommand(ctx, nil) })
if !strings.Contains(listed, "machine.ace.silent") || !strings.Contains(listed, "silenced until") {
t.Fatalf("%s", listed)
}
}
// **Conditions that cannot be read are not none open**: status says so, and is not well.
func TestUnreadableConditionsAreNotAWellMesh(t *testing.T) {
open := aMesh(t)
_, store := withConditionsInMemory(t)
store.Fail = errors.New("the bus is away")
asked, err := theThreeQuestions(t.Context(), open)
if err != nil {
t.Fatal(err)
}
if asked.well() || asked.conditionsUnread == "" {
t.Fatalf("well with its conditions unread: %+v", asked.conditionsUnread)
}
said := printed(t, func() error { return printStatus(asked) })
if !strings.HasPrefix(said, "the open conditions could NOT be read") || strings.Contains(said, "no open conditions") {
t.Fatalf("%s", said)
}
store.Fail = nil
asked, _ = theThreeQuestions(t.Context(), open)
said = printed(t, func() error { return printStatus(asked) })
if asked.well() && !strings.Contains(said, "no open conditions;") {
t.Fatalf("the all-well sentence does not say no conditions are open:\n%s", said)
}
}
+926
View File
@@ -0,0 +1,926 @@
package main
import (
"context"
"encoding/json"
"errors"
"flag"
"fmt"
"log"
"sort"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A module declares the data it holds, and the mesh protects and watches it from that declaration
// (novox/hq ADR 0233).
//
// The self-check's D13 composes what every machine declares, asks each machine's backup holder what
// it measured of every item — size, newest write, newest good backup, the redundant storage it is on —
// and keeps both. From that, and from what every provider says it holds for its consumers, it raises,
// each URGENT for what is irreplaceable and a WARNING for what is valuable (the operator's ranking):
//
// - `data-shrank`: an item holds less than half of its largest size in seven days, and at least
// shrinkFloor less; `data-missing`: its path is gone;
// - `empty-replacement`: an item, or a consumer's data at a provider, is less than half the size of a
// copy of the same thing kept elsewhere — on 2026-10-05 five applications ran for twenty hours on
// empty databases while their real ones sat on another machine (issue 273);
// - `data-held-twice` (warning): a consumer has active data at two providers and their sizes cannot
// be compared;
// - `data-quiet`: an item said to be written all the time has not been, within its bound;
// - `backup-stale`: an item's newest good backup is older than its bound, or there is none;
// - `array-degraded`: the redundant storage an item is on is not healthy, or cannot be read;
// `protection-missing`: an item said to be protected by redundancy is on storage that is not;
// - `cleanup-waiting` (warning): an item retired more than thirty days, waiting for a person.
//
// And it retires: an irreplaceable or valuable item in a module's own directory that its machine no
// longer declares — its module unassigned — is kept, marked retired with when and why, and listed by
// `cleanup list` until `cleanup delete` removes it. The node-engine never deletes a directory with
// anything in it; this is the record of what it kept. An operator's path is never retired or deleted.
// The condition kinds of D13.
const (
kindDataShrank = "data-shrank"
kindEmptyReplacement = "empty-replacement"
kindDataHeldTwice = "data-held-twice"
kindDataQuiet = "data-quiet"
kindBackupStale = "backup-stale"
kindDataUnmeasured = "data-unmeasured"
// kindDataMissing is a watched item whose path is gone.
kindDataMissing = "data-missing"
// kindArrayDegraded is redundant storage watched data is on that is not healthy, or cannot be read.
kindArrayDegraded = "array-degraded"
// kindProtectionMissing is an item said to be protected by redundancy, on storage that is not.
kindProtectionMissing = "protection-missing"
)
// probeDataID is the self-check's id for this probe.
const probeDataID = "D13"
// The bounds the findings are read against.
var (
// shrinkWindow is how far back the largest size is looked for.
shrinkWindow = 7 * 24 * time.Hour
// shrinkFloor is the least loss that is worth saying: two empty databases differ by a few
// megabytes, and half of almost nothing is noise.
shrinkFloor int64 = 16 << 20
// dataAsk is how long one machine's holder, or one provider, is given to answer.
dataAsk = 8 * time.Second
)
// keyOfItem is one item's condition id: its machine, module and item.
func keyOfItem(machine, module, item string) string { return machine + "." + module + "." + item }
// holderAnswer is what a node-backup holder's `backed-up` says of one module (ADR 0233 adds Data).
type holderAnswer struct {
Module string `json:"module"`
Data []holderItem `json:"data"`
}
// holderItem is one item as the holder measured it.
type holderItem struct {
Item string `json:"item"`
Class string `json:"class"`
Path string `json:"path"`
SizeBytes *int64 `json:"size_bytes"`
LastWrite *time.Time `json:"last_write"`
MeasuredAt *time.Time `json:"measured_at"`
LastBackup *time.Time `json:"last_backup"`
Error string `json:"error,omitempty"`
// Precision is what the size is: exact, a dataset's, partial, or none (ADR 0233).
Precision string `json:"precision,omitempty"`
// Redundancy is the redundant storage the item is on, where the holder could tell (ADR 0233).
Redundancy *inventory.Redundancy `json:"redundancy,omitempty"`
}
// readHolder reads a holder's answer into measurements by module and item.
func readHolder(raw json.RawMessage) (map[string]map[string]inventory.Measurement, error) {
var modules []holderAnswer
if err := json.Unmarshal(raw, &modules); err != nil {
return nil, fmt.Errorf("its answer is not readable: %w", err)
}
out := map[string]map[string]inventory.Measurement{}
for _, m := range modules {
for _, it := range m.Data {
if out[m.Module] == nil {
out[m.Module] = map[string]inventory.Measurement{}
}
out[m.Module][it.Item] = inventory.Measurement{Path: it.Path, Size: it.SizeBytes, LastWrite: it.LastWrite,
MeasuredAt: it.MeasuredAt, LastBackup: it.LastBackup, Error: it.Error, Redundancy: it.Redundancy,
Precision: it.Precision}
}
}
return out, nil
}
// declaredOn is every data item a machine's composition declares, and the module there that holds
// node-backup to measure them — empty for none.
func declaredOn(plan catalogue.Resolution) ([]inventory.DeclaredData, string) {
var out []inventory.DeclaredData
held := ""
for _, m := range plan.Modules {
for _, c := range m.Claims {
if s, known := catalogue.SeatNamed(c.Name); known && s.Name == catalogue.BackupSeat {
held = m.Module
}
}
for _, it := range m.DataItems() {
out = append(out, inventory.DeclaredData{Module: m.Module, Item: it.ID, Class: it.Class,
Owned: it.OwnedByModule(), Protection: it.Protection()})
}
}
return out, held
}
// consumerCopy is one provider's account of one consumer: where, how big, and whether still active.
type consumerCopy struct {
Node, Module, Consumer string
Size *int64
Retired bool
// Class is how precious the consumer's data is: the stricter of what the provider keeps for its
// consumers and what the consumer says it keeps there (`kept-by`).
Class string
}
// probeData is D13.
func probeData(ctx context.Context, d *doctor) ([]conditions.Observation, error) {
if d.js == nil {
return nil, errors.New("no bus to ask the machines over")
}
open := d.open
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
return nil, err
}
nodes, err := open.inventory.Nodes(ctx)
if err != nil {
return nil, err
}
heard := heardMachines(d)
now := time.Now()
delivered, err := readDeliveries(ctx, open.inventory)
if err != nil {
return nil, err
}
type machine struct {
name string
declared []inventory.DeclaredData
held bool
measured map[string]map[string]inventory.Measurement
askErr error
}
var machines []*machine
for _, n := range nodes {
plan, _, err := planFor(ctx, open, n.Name)
if err != nil {
if ctx.Err() != nil {
return nil, ctx.Err()
}
// A machine that cannot be worked out declares nothing this run — which is not the same as
// declaring nothing: retiring its data on that would be acting on an unreadable result.
continue
}
declared, holder := declaredOn(plan)
// **A holder is asked only once its machine has been sent it and had time to report** (novox/hq
// issue 275): assigned and not pushed yet, it is not there to answer, and "did not say what it
// measured" about it was a warning for a push nobody had made yet.
held := holder != "" && delivered[n.Name].settled(holder, now)
machines = append(machines, &machine{name: n.Name, declared: declared, held: held})
}
// Every holder asked at once, as D8 asks every ban list.
var wg sync.WaitGroup
for _, m := range machines {
if !m.held || !heard[m.name] {
continue
}
wg.Add(1)
go func(m *machine) {
defer wg.Done()
asking, cancel := context.WithTimeout(ctx, dataAsk)
defer cancel()
raw, err := askSeatTool(asking, d.js.Conn(), catalogue.BackupSeat, "backed-up", m.name)
if err == nil {
m.measured, err = readHolder(raw)
}
m.askErr = err
}(m)
}
wg.Wait()
var out []conditions.Observation
for _, m := range machines {
if m.askErr != nil {
out = append(out, conditions.Observation{Scope: conditions.ScopeMachine, ID: m.name, Token: kindDataUnmeasured,
Kind: kindDataUnmeasured, Machine: m.name, Severity: conditions.Warning,
Summary: fmt.Sprintf("%s's backup holder did not say what it measured of the data declared there, so "+
"nothing about that data is known this run: %s", m.name, firstLine(m.askErr.Error())),
Said: firstLine(m.askErr.Error())})
}
why := fmt.Sprintf("no longer declared on %s: its module was unassigned there, or is no longer pulled in", m.name)
change, err := open.inventory.RecordData(ctx, m.name, m.declared, m.measured, why, now)
if err != nil {
return nil, fmt.Errorf("what %s holds could not be kept: %w", m.name, err)
}
for _, r := range change.Retired {
log.Printf("data: %s of %s on %s RETIRED, kept at %s: %s — `cleanup list` shows it, and only `cleanup "+
"delete` removes it (novox/hq ADR 0233)", r.Item, r.Module, r.Machine, orUnknownPath(r.Path), why)
}
for _, r := range change.Reenabled {
log.Printf("data: %s of %s on %s is declared again, no longer retired", r.Item, r.Module, r.Machine)
}
}
records, err := open.inventory.Data(ctx)
if err != nil {
return nil, err
}
peaks, err := open.inventory.DataPeaks(ctx, now.Add(-shrinkWindow))
if err != nil {
return nil, err
}
bindings, err := open.inventory.Bindings(ctx)
if err != nil {
return nil, err
}
upgraded, keptBy := keptByClasses(bindings, shelf)
copies, err := consumerCopies(ctx, d.js.Conn(), open.inventory, shelf, keptBy)
if err != nil {
return nil, err
}
out = append(out, dataFindings(records, peaks, shelf, copies, upgraded, now)...)
return out, nil
}
func orUnknownPath(p string) string {
if p == "" {
return "a path its backup holder never named"
}
return p
}
// consumerCopies asks every provider of a provision whose consumers' data is kept what it holds, at
// once. One that cannot answer is passed over: it says nothing about any copy, which is not a finding.
func consumerCopies(ctx context.Context, conn *nats.Conn, inv *inventory.Inventory,
shelf map[string]catalogue.Manifest, keptBy map[string]string) ([]consumerCopy, error) {
instances, err := providerInstances(ctx, inv)
if err != nil {
return nil, err
}
var asked []providerInstance
for _, p := range instances {
m := shelf[p.Module]
keeps := false
for provision := range m.Grants {
keeps = keeps || m.KeepsConsumerData(provision)
}
if keeps {
asked = append(asked, p)
}
}
states := make([]*link.RetirementState, len(asked))
var wg sync.WaitGroup
for i, p := range asked {
wg.Add(1)
go func(i int, p providerInstance) {
defer wg.Done()
asking, cancel := context.WithTimeout(ctx, dataAsk)
defer cancel()
if s, err := askRetirement(asking, conn, p); err == nil {
states[i] = &s
}
}(i, p)
}
wg.Wait()
var out []consumerCopy
for i, p := range asked {
s := states[i]
if s == nil {
continue
}
class := consumersClass(shelf[p.Module])
for _, c := range s.Held {
cp := consumerCopy{Node: p.Node, Module: p.Module, Consumer: c,
Class: catalogue.StricterClass(class, keptBy[p.Module+"/"+c])}
if size, ok := s.HeldSizes[c]; ok && size >= 0 {
size := size
cp.Size = &size
}
out = append(out, cp)
}
for _, r := range s.Retired {
if r.Kind != "" && r.Kind != "consumer" {
continue
}
cp := consumerCopy{Node: p.Node, Module: p.Module, Consumer: r.Consumer, Retired: true,
Class: catalogue.StricterClass(class, keptBy[p.Module+"/"+r.Consumer])}
if r.SizeBytes != nil && *r.SizeBytes >= 0 {
cp.Size = r.SizeBytes
}
out = append(out, cp)
}
}
return out, nil
}
// severityOf is how loud a finding about data of a class is: urgent for what is irreplaceable, a warning
// for anything else watched (the operator's ranking, ADR 0233).
func severityOf(class string) conditions.Severity {
if class == catalogue.ClassIrreplaceable {
return conditions.Urgent
}
return conditions.Warning
}
// consumersClass is the most precious class a provider keeps any of its consumers' data as.
func consumersClass(m catalogue.Manifest) string {
class := catalogue.ClassNone
for provision := range m.Grants {
if c, ok := m.ConsumerDataOf(provision); ok {
class = catalogue.StricterClass(class, c.Class)
} else if m.KeepsConsumerData(provision) {
class = catalogue.StricterClass(class, catalogue.ClassValuable)
}
}
return class
}
// keptByClasses is what consumers say of the data they keep with their providers (`kept-by`), read
// through where each is bound: by provider module and consumer identity, the class of that consumer's
// data there; and by provider item key (machine/module/item), the class the item holding it is held to.
func keptByClasses(bindings []inventory.Binding, shelf map[string]catalogue.Manifest) (map[string]string, map[string]string) {
upgraded, keptBy := map[string]string{}, map[string]string{}
for _, b := range bindings {
m, ok := shelf[b.Consumer]
if !ok {
continue
}
k, said := m.KeptByOf(b.Provision)
if !said {
continue
}
identity := catalogue.ConsumerIdentity(b.Machine, catalogue.IdentitySource(m.Slug, m.Module))
key := b.Provider.Module + "/" + identity
keptBy[key] = catalogue.StricterClass(keptBy[key], k.Class)
if pc, ok := shelf[b.Provider.Module].ConsumerDataOf(b.Provision); ok && pc.In != "" {
if _, own := shelf[b.Provider.Module].DataItem(pc.In); own {
item := b.Provider.Node + "/" + b.Provider.Module + "/" + pc.In
upgraded[item] = catalogue.StricterClass(upgraded[item], k.Class)
}
}
}
return upgraded, keptBy
}
// dataFindings is every condition the data on record raises now. A function of what is known, so the
// incident's shape is tested without a mesh. upgraded is the class an item is held to where a consumer
// of its module keeps data in it more precious than its own class says (`kept-by`), by its key.
func dataFindings(records []inventory.DataRecord, peaks map[string]int64, shelf map[string]catalogue.Manifest,
copies []consumerCopy, upgraded map[string]string, now time.Time) []conditions.Observation {
var out []conditions.Observation
byItem := map[string][]inventory.DataRecord{}
arrays := map[string][]inventory.DataRecord{}
type shrunk struct {
machine, dataset, class string
size, peak int64
items []string
}
shrunkDatasets := map[string]shrunk{}
for _, r := range records {
if r.DeletedAt != nil {
continue
}
class := catalogue.StricterClass(r.Class, upgraded[r.Key()])
r.Class = class
byItem[r.Module+"/"+r.Item] = append(byItem[r.Module+"/"+r.Item], r)
item, declared := shelf[r.Module].DataItem(r.Item)
id := keyOfItem(r.Machine, r.Module, r.Item)
if r.Retired() {
if now.Sub(*r.RetiredAt) > cleanupAfter {
out = append(out, conditions.Observation{Scope: conditions.ScopeMachine, ID: id, Token: "cleanup",
Kind: kindCleanupWaiting, Machine: r.Machine, Severity: conditions.Warning, Resolver: conditions.ResolverOperator,
Summary: fmt.Sprintf("%s of %s on %s (%s, %s) has been retired %d days — kept at %s since %s; `cleanup "+
"delete %s %s %s --why …` once a person has decided, or assign %s there again",
r.Item, r.Module, r.Machine, r.Class, sizeWords(r.Size), int(now.Sub(*r.RetiredAt).Hours()/24),
orUnknownPath(r.Path), r.RetiredWhy, r.Machine, r.Module, r.Item, r.Module)})
}
continue
}
if !catalogue.Watched(class) {
continue
}
severity := severityOf(class)
if r.Redundancy != nil {
where := r.Machine + "/" + r.Redundancy.Kind + ":" + r.Redundancy.Where
arrays[where] = append(arrays[where], r)
} else if declared && item.Redundancy != "" && r.MeasuredAt != nil && r.MeasureError == "" {
out = append(out, conditions.Observation{Scope: conditions.ScopeMachine, ID: id, Token: kindProtectionMissing,
Kind: kindProtectionMissing, Machine: r.Machine, Severity: severity, Resolver: conditions.ResolverOperator,
Summary: fmt.Sprintf("%s of %s on %s (%s) is said to be protected by the redundancy of the storage it is on, "+
"and %s is on nothing the backup holder can read as redundant: it has no protection the mesh can see",
r.Item, r.Module, r.Machine, class, orUnknownPath(r.Path))})
}
if r.MeasureError != "" && strings.Contains(r.MeasureError, "does not exist") {
out = append(out, conditions.Observation{Scope: conditions.ScopeMachine, ID: id, Token: kindDataMissing,
Kind: kindDataMissing, Machine: r.Machine, Severity: severity, Resolver: conditions.ResolverOperator,
Summary: fmt.Sprintf("%s of %s on %s (%s) is gone: %s does not exist any more", r.Item, r.Module, r.Machine,
class, orUnknownPath(r.Path))})
} else if peak, ok := peaks[r.Key()]; ok && r.Size != nil && inventory.Comparable(r.Precision) &&
*r.Size*2 < peak && peak-*r.Size >= shrinkFloor && inventory.Dataset(r.Precision) != "" {
// Several items on one dataset share its size: one condition for the dataset, as loud as the
// most precious item on it.
k := r.Machine + "/" + inventory.Dataset(r.Precision)
ds := shrunkDatasets[k]
ds.machine, ds.dataset, ds.size, ds.peak = r.Machine, inventory.Dataset(r.Precision), *r.Size, peak
ds.class = catalogue.StricterClass(ds.class, class)
ds.items = append(ds.items, r.Module+"/"+r.Item)
shrunkDatasets[k] = ds
} else if peak, ok := peaks[r.Key()]; ok && r.Size != nil && inventory.Comparable(r.Precision) &&
*r.Size*2 < peak && peak-*r.Size >= shrinkFloor {
out = append(out, conditions.Observation{Scope: conditions.ScopeMachine, ID: id, Token: kindDataShrank,
Kind: kindDataShrank, Machine: r.Machine, Severity: severity, Resolver: conditions.ResolverOperator,
Summary: fmt.Sprintf("%s of %s on %s (%s) shrank to %s from %s within %d days — more than half of what it "+
"held is gone. If that was meant, silence this with why; if not, `node-backup.restore` puts the last "+
"good copy beside it", r.Item, r.Module, r.Machine, class, sizeWords(r.Size), sizeWords(&peak),
int(shrinkWindow.Hours()/24))})
}
if !declared {
continue
}
if within := item.ActiveWithin(); within > 0 && r.LastWrite != nil && now.Sub(*r.LastWrite) > within {
out = append(out, conditions.Observation{Scope: conditions.ScopeMachine, ID: id, Token: kindDataQuiet,
Kind: kindDataQuiet, Machine: r.Machine, Severity: severity,
Summary: fmt.Sprintf("%s of %s on %s is written all the time, and has not been since %s (its bound is %s): "+
"whatever writes it has stopped", r.Item, r.Module, r.Machine, r.LastWrite.UTC().Format(time.RFC3339),
within)})
}
// A backup is required of what is irreplaceable and copied; of what is valuable it is the standard
// plan, said only where the machine was measured — where a holder is there to take it.
if item.BackedUp() && (class == catalogue.ClassIrreplaceable || r.MeasuredAt != nil) {
within := item.BackupWithin()
switch {
case r.LastBackup == nil && now.Sub(r.FirstSeen) > within:
out = append(out, conditions.Observation{Scope: conditions.ScopeMachine, ID: id, Token: kindBackupStale,
Kind: kindBackupStale, Machine: r.Machine, Severity: severity,
Summary: fmt.Sprintf("%s of %s on %s is %s and has no good backup on record, %s after it was first "+
"declared — is node-backup held there, and do its nights succeed? (`node-backup.backed-up`)",
r.Item, r.Module, r.Machine, class, now.Sub(r.FirstSeen).Round(time.Hour))})
case r.LastBackup != nil && now.Sub(*r.LastBackup) > within:
out = append(out, conditions.Observation{Scope: conditions.ScopeMachine, ID: id, Token: kindBackupStale,
Kind: kindBackupStale, Machine: r.Machine, Severity: severity,
Summary: fmt.Sprintf("%s of %s on %s is %s and its newest good backup is from %s, older than its bound "+
"of %s", r.Item, r.Module, r.Machine, class, r.LastBackup.UTC().Format(time.RFC3339), within)})
}
}
}
for _, k := range keysSorted(shrunkDatasets) {
ds := shrunkDatasets[k]
out = append(out, conditions.Observation{Scope: conditions.ScopeMachine,
ID: ds.machine + ".dataset." + strings.ReplaceAll(ds.dataset, "/", "-"), Token: kindDataShrank,
Kind: kindDataShrank, Machine: ds.machine, Severity: severityOf(ds.class), Resolver: conditions.ResolverOperator,
Summary: fmt.Sprintf("the dataset %s on %s shrank to %s from %s within %d days — more than half of what it held "+
"is gone; it holds %s", ds.dataset, ds.machine, sizeWords(&ds.size), sizeWords(&ds.peak),
int(shrinkWindow.Hours()/24), strings.Join(ds.items, ", "))})
}
// The redundant storage watched data is on: one condition per array, as loud as the most precious
// item on it — the array, not each item, is what degrades.
for _, where := range keysSorted(arrays) {
rs := arrays[where]
red := rs[0].Redundancy
if red.Healthy != nil && *red.Healthy {
continue
}
class, machine := catalogue.ClassValuable, rs[0].Machine
var names []string
for _, r := range rs {
class = catalogue.StricterClass(class, r.Class)
names = append(names, r.Module+"/"+r.Item)
}
state := "could not be read"
if red.Healthy != nil {
state = "is NOT healthy"
}
out = append(out, conditions.Observation{Scope: conditions.ScopeMachine,
ID: machine + ".array." + strings.NewReplacer("/", "-", ":", "-").Replace(red.Kind+"-"+red.Where),
Token: kindArrayDegraded, Kind: kindArrayDegraded, Machine: machine, Severity: severityOf(class),
Resolver: conditions.ResolverOperator,
Summary: fmt.Sprintf("the %s storage %s on %s %s: %s — and it is what protects %s", red.Kind, red.Where, machine,
state, firstLine(red.Said), strings.Join(names, ", "))})
}
// The same item on several machines: a copy that is in use and far smaller than one kept elsewhere is
// an empty replacement. Only against a retired copy — a module running on two machines on purpose
// keeps two different sets of data.
for _, key := range keysSorted(byItem) {
rs := byItem[key]
for _, a := range rs {
if a.Retired() || a.Size == nil || !catalogue.Watched(a.Class) || !inventory.Comparable(a.Precision) {
continue
}
for _, o := range rs {
if o.Machine == a.Machine || !o.Retired() || o.Size == nil || !inventory.Comparable(o.Precision) ||
!replacedByLess(*a.Size, *o.Size) {
continue
}
out = append(out, conditions.Observation{Scope: conditions.ScopeMachine,
ID: keyOfItem(a.Machine, a.Module, a.Item), Token: kindEmptyReplacement, Kind: kindEmptyReplacement,
Machine: a.Machine, Also: []string{o.Machine}, Severity: severityOf(a.Class), Resolver: conditions.ResolverOperator,
Summary: fmt.Sprintf("%s of %s on %s holds %s, and the copy %s kept on %s holds %s: %s is running on "+
"an empty replacement of its data. Move the data, or assign it back where its data is",
a.Item, a.Module, a.Machine, sizeWords(a.Size), o.Module, o.Machine, sizeWords(o.Size), a.Module)})
break
}
}
}
out = append(out, consumerFindings(copies)...)
return out
}
// replacedByLess is whether a copy in use is an empty replacement of a copy kept elsewhere: less than
// half of it, and at least shrinkFloor less.
func replacedByLess(inUse, kept int64) bool {
return inUse*2 < kept && kept-inUse >= shrinkFloor
}
// consumerFindings is the same question of consumers' data at providers: one consumer, the same
// provider module on two machines.
func consumerFindings(copies []consumerCopy) []conditions.Observation {
by := map[string][]consumerCopy{}
for _, c := range copies {
k := c.Module + "/" + c.Consumer
by[k] = append(by[k], c)
}
var out []conditions.Observation
for _, k := range keysSorted(by) {
cs := by[k]
if len(cs) < 2 {
continue
}
found := false
for _, a := range cs {
if a.Retired || a.Size == nil {
continue
}
for _, o := range cs {
if o.Node == a.Node || o.Size == nil || !replacedByLess(*a.Size, *o.Size) {
continue
}
state := "active"
if o.Retired {
state = "retired"
}
out = append(out, conditions.Observation{Scope: conditions.ScopeProvider,
ID: a.Module + "." + a.Node + "." + a.Consumer, Token: kindEmptyReplacement, Kind: kindEmptyReplacement,
Machine: a.Node, Also: []string{o.Node}, Severity: severityOf(catalogue.StricterClass(a.Class, o.Class)),
Resolver: conditions.ResolverOperator,
Summary: fmt.Sprintf("%s's data at %s on %s holds %s, and its %s copy at %s on %s holds %s: the "+
"consumer is using an empty replacement of its data (issue 273's shape). Pin it back to %s, or move "+
"the data first", a.Consumer, a.Module, a.Node, sizeWords(a.Size), state, o.Module, o.Node,
sizeWords(o.Size), o.Node)})
found = true
break
}
if found {
break
}
}
if found {
continue
}
var active []consumerCopy
for _, c := range cs {
if !c.Retired {
active = append(active, c)
}
}
if len(active) >= 2 {
var where []string
var also []string
for _, c := range active {
where = append(where, c.Node+" ("+sizeWords(c.Size)+")")
also = append(also, c.Node)
}
out = append(out, conditions.Observation{Scope: conditions.ScopeProvider,
ID: active[0].Module + "." + active[0].Consumer, Token: kindDataHeldTwice, Kind: kindDataHeldTwice,
Machine: active[0].Node, Also: also[1:], Severity: conditions.Warning, Resolver: conditions.ResolverOperator,
Summary: fmt.Sprintf("%s has active data at %s on %d machines — %s — and only one is the one it uses",
active[0].Consumer, active[0].Module, len(active), strings.Join(where, ", "))})
}
}
return out
}
func keysSorted[V any](m map[string]V) []string {
out := make([]string, 0, len(m))
for k := range m {
out = append(out, k)
}
sort.Strings(out)
return out
}
// ---- the `data` verb ---------------------------------------------------------------------------
const dataUsage = "data [--json] [--machine <name>] [--retired]"
// dataRow is one item as `data` lists it.
type dataRow struct {
Machine string `json:"machine"`
Module string `json:"module"`
Item string `json:"item"`
Class string `json:"class"`
Path string `json:"path,omitempty"`
Protection string `json:"protection,omitempty"`
// Array is the redundant storage it is on and its state, where its holder could tell.
Array string `json:"array,omitempty"`
Unmeasured string `json:"unmeasured,omitempty"`
// Precision says what the size is: exact, a dataset's whole size, partial, or none.
Precision string `json:"precision,omitempty"`
SizeBytes *int64 `json:"size-bytes,omitempty"`
LastWrite string `json:"last-write,omitempty"`
MeasuredAt string `json:"measured-at,omitempty"`
LastBackup string `json:"last-backup,omitempty"`
BackupDue string `json:"backup-within,omitempty"`
Retired string `json:"retired,omitempty"`
RetiredWhy string `json:"retired-why,omitempty"`
Deleted string `json:"deleted,omitempty"`
}
// dataCommand is `data`: every item every machine declares, or held retired, as the self-check last
// found it.
func dataCommand(ctx context.Context, args []string) error {
set := flag.NewFlagSet("data", flag.ContinueOnError)
asJSON := set.Bool("json", false, "as data")
only := set.String("machine", "", "one machine")
retiredOnly := set.Bool("retired", false, "only what is retired")
if rest, err := parseAround(set, args); err != nil {
return err
} else if len(rest) > 0 {
return errors.New(dataUsage)
}
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
records, err := open.inventory.Data(ctx)
if err != nil {
return err
}
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
return err
}
rows := dataRows(records, shelf, *only, *retiredOnly)
if *asJSON {
return printJSON(map[string]any{"data": rows})
}
if len(rows) == 0 {
fmt.Println("no data on record: the self-check (D13) records what each machine declares on its next run")
return nil
}
for _, r := range rows {
state := ""
switch {
case r.Deleted != "":
state = " DELETED " + r.Deleted
case r.Retired != "":
state = " RETIRED " + r.Retired + " — " + r.RetiredWhy
}
fmt.Printf("%s %s/%s %s %s %s protected by %s%s\n", r.Machine, r.Module, r.Item, r.Class,
sizeWords(r.SizeBytes), orUnknownPath(r.Path), orNothingWord(r.Protection), state)
if r.Array != "" {
fmt.Printf(" on %s\n", r.Array)
}
if r.Precision != "" && r.Precision != "exact" {
fmt.Printf(" size: %s\n", r.Precision)
}
if r.Unmeasured != "" {
fmt.Printf(" not measured: %s\n", r.Unmeasured)
}
if r.Class == catalogue.ClassCache {
continue
}
fmt.Printf(" last write %s, measured %s, last backup %s%s\n", orNever(r.LastWrite), orNever(r.MeasuredAt),
orNever(r.LastBackup), within(r.BackupDue))
}
return nil
}
func dataRows(records []inventory.DataRecord, shelf map[string]catalogue.Manifest, only string, retiredOnly bool) []dataRow {
rows := []dataRow{}
stamp := func(t *time.Time) string {
if t == nil {
return ""
}
return t.UTC().Format(time.RFC3339)
}
for _, r := range records {
if only != "" && r.Machine != only {
continue
}
if retiredOnly && !r.Retired() {
continue
}
row := dataRow{Machine: r.Machine, Module: r.Module, Item: r.Item, Class: r.Class, Path: r.Path,
Protection: r.Protection, Unmeasured: r.MeasureError, Precision: r.Precision, SizeBytes: r.Size, LastWrite: stamp(r.LastWrite), MeasuredAt: stamp(r.MeasuredAt),
LastBackup: stamp(r.LastBackup), Retired: stamp(r.RetiredAt), RetiredWhy: r.RetiredWhy,
Deleted: stamp(r.DeletedAt)}
if it, ok := shelf[r.Module].DataItem(r.Item); ok && it.BackedUp() {
row.BackupDue = it.BackupWithin().String()
}
if red := r.Redundancy; red != nil {
state := "state unread"
if red.Healthy != nil && *red.Healthy {
state = "healthy"
} else if red.Healthy != nil {
state = "NOT HEALTHY"
}
row.Array = red.Kind + " " + red.Where + ", " + state
}
rows = append(rows, row)
}
return rows
}
func orNothingWord(s string) string {
if s == "" || s == "none" {
return "nothing"
}
return s
}
func orNever(s string) string {
if s == "" {
return "never"
}
return s
}
func within(s string) string {
if s == "" {
return " (not backed up)"
}
return " (bound " + s + ")"
}
// ---- cleanup of retired own data ---------------------------------------------------------------
// The tools a node-backup holder serves to delete one retired item (novox/hq ADR 0233): the first
// takes a last restore point of it, tagged as retired, and only then removes it — in the background,
// because a large item outlasts any call — and the second says how that went. Module tools, not seat
// verbs: only the controller's `cleanup delete` calls them, as it calls a provider's provisioner_delete.
const (
ToolDeleteRetired = "backup_delete_retired"
ToolDeletedOutcome = "backup_deleted"
)
// deletionWait is how long `cleanup delete` follows a deletion before handing it back to the person.
var deletionWait = 8 * time.Minute
// deletionPoll is how often it asks.
var deletionPoll = 5 * time.Second
// deletion is a holder's account of one deletion.
type deletion struct {
Started bool `json:"started"`
Running bool `json:"running"`
Done bool `json:"done"`
OK bool `json:"ok"`
Snapshot string `json:"snapshot"`
Error string `json:"error"`
}
// retiredData is every retired item on record, as `cleanup list` shows them.
func retiredData(records []inventory.DataRecord, now time.Time) []retiredRow {
var out []retiredRow
for _, r := range records {
if !r.Retired() {
continue
}
out = append(out, retiredRow{Node: r.Machine, Module: r.Module, Consumer: r.Item, Kind: retiredDataKind,
RetiredAt: r.RetiredAt.UTC().Format(time.RFC3339), AgeDays: int(now.Sub(*r.RetiredAt).Hours() / 24),
SizeBytes: r.Size, Why: r.RetiredWhy, Path: r.Path, Class: r.Class})
}
return out
}
// retiredDataKind is what `cleanup list` calls a module's own retired data, beside a provider's consumer.
const retiredDataKind = "own-data"
// holderOn is the module holding node-backup on a machine.
func holderOn(ctx context.Context, inv *inventory.Inventory, machine string) (string, error) {
held, err := inv.Holdings(ctx)
if err != nil {
return "", err
}
for _, h := range held {
if s, known := catalogue.SeatNamed(h.Claim); known && s.Name == catalogue.BackupSeat && h.Node == machine {
return h.Module, nil
}
}
return "", fmt.Errorf("nothing holds %s on %s, and it is the backup holder that deletes retired data there "+
"(after a last restore point)", catalogue.BackupSeat, machine)
}
// deleteRetiredData has a machine's backup holder delete one retired item: never one declared now, and
// never one not retired. The holder takes a last restore point of it first, so the deletion can be
// undone until a person forgets that restore point; the record says deleted only once the holder says
// it is.
func deleteRetiredData(ctx context.Context, conn *nats.Conn, open *stores, r inventory.DataRecord, f handActFlags) error {
inv := open.inventory
if !r.Retired() {
return fmt.Errorf("%s of %s on %s is not retired — only retired data is deleted. Nothing was done",
r.Item, r.Module, r.Machine)
}
if r.Path == "" {
return fmt.Errorf("%s of %s on %s was never measured, so where it is was never said; nothing was deleted",
r.Item, r.Module, r.Machine)
}
if plan, _, err := planFor(ctx, open, r.Machine); err == nil {
for _, m := range plan.Modules {
if _, still := m.DataItem(r.Item); still && m.Module == r.Module {
return fmt.Errorf("%s runs on %s again and declares %s: it is not retired any more. Nothing was done",
r.Module, r.Machine, r.Item)
}
}
}
holder, err := holderOn(ctx, inv, r.Machine)
if err != nil {
return err
}
f.record(ctx, "cleanup delete", []string{r.Machine, r.Module, r.Item})
args := map[string]any{"module": r.Module, "item": r.Item, "path": r.Path, "confirm": r.Item,
"why": strings.TrimSpace(*f.why), "by": link.Caller(), "via": link.ViaController}
ask := func(tool string) (deletion, error) {
var d deletion
answer, err := link.AskModuleToolOn(ctx, conn, holder, tool, r.Machine, args, 25*time.Second)
if err != nil {
return d, err
}
if answer.Error != "" {
return d, fmt.Errorf("%s on %s refused: %s", holder, r.Machine, answer.Error)
}
return d, unmarshalAnswer(answer, &d)
}
d, err := ask(ToolDeleteRetired)
if err != nil {
return err
}
for waited := time.Duration(0); !d.Done && waited < deletionWait; waited += deletionPoll {
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(deletionPoll):
}
if d, err = ask(ToolDeletedOutcome); err != nil {
return err
}
}
switch {
case !d.Done:
fmt.Printf("%s on %s is still taking the last restore point of %s and deleting it; `cleanup list` keeps "+
"showing it until the holder says it is done — the same `cleanup delete` again reads how it went\n",
holder, r.Machine, r.Path)
return nil
case !d.OK:
return fmt.Errorf("%s on %s did NOT delete %s: %s", holder, r.Machine, r.Path, d.Error)
}
if err := inv.MarkDataDeleted(ctx, r.Machine, r.Module, r.Item, link.Caller(), strings.TrimSpace(*f.why), time.Now()); err != nil {
return fmt.Errorf("%s deleted %s on %s, and it could not be recorded: %w", holder, r.Path, r.Machine, err)
}
fmt.Printf("%s on %s deleted %s of %s (%s, %s); its last restore point is %s, kept until a person forgets it\n",
holder, r.Machine, r.Item, r.Module, r.Path, sizeWords(r.Size), orNever(d.Snapshot))
return nil
}
// keptOnUnassign says, for an unassignment, the irreplaceable and valuable data each module leaves in its
// own directories on the machine:
// kept, and retired at the self-check's next run.
func keptOnUnassign(ctx context.Context, inv *inventory.Inventory, machine string, modules []string) []string {
records, err := inv.Data(ctx)
if err != nil {
return []string{"what it leaves behind could not be read from the mesh's record: " + err.Error()}
}
var out []string
for _, r := range records {
if r.Machine != machine || r.DeletedAt != nil || !catalogue.Retires(r.Class) || !r.Owned {
continue
}
for _, m := range modules {
if r.Module == m {
out = append(out, fmt.Sprintf("%s's %s (%s, %s) stays where it is: it is %s, so it is retired, "+
"never removed — `cleanup list` shows it, `cleanup delete` alone removes it (novox/hq ADR 0233)",
r.Module, r.Item, orUnknownPath(r.Path), sizeWords(r.Size), r.Class))
}
}
}
return out
}
+464
View File
@@ -0,0 +1,464 @@
package main
import (
"context"
"encoding/json"
"os"
"strings"
"sync"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
)
func bytesOf(n int64) *int64 { return &n }
func when(t time.Time) *time.Time { return &t }
func shelfFor(t *testing.T, manifests ...string) map[string]catalogue.Manifest {
t.Helper()
out := map[string]catalogue.Manifest{}
for _, raw := range manifests {
m, err := catalogue.ParseManifest([]byte(raw))
if err != nil {
t.Fatal(err)
}
out[m.Module] = m
}
return out
}
const houseManifest = `{"module":"house","version":"1",
"data":{"own":[{"id":"config","path":"${dir:config}","class":"irreplaceable","active":"1d"}]},
"resources":[{"id":"config","type":"directory","mode":"0700"}]}`
func findingsByKind(obs []conditions.Observation) map[string]conditions.Observation {
out := map[string]conditions.Observation{}
for _, o := range obs {
out[o.Kind] = o
}
return out
}
// THE INCIDENT (issue 273), replayed against what D13 reads: five applications on the home server bound,
// by one changed rule, to the store on the control node, which made each an empty database — while
// their real databases, hundreds of megabytes each, sat on the home server's own store, by then retired
// because the mesh no longer asked for them there. Each is an empty replacement, naming both machines
// and the pin back: a warning for the store's consumers, whose data is valuable; urgent for one that says
// its data there is irreplaceable (`kept-by`).
func TestAnEmptyReplacementOfAConsumersDataIsSaid(t *testing.T) {
var copies []consumerCopy
for _, app := range []string{"mesh_home_board", "mesh_home_flows", "mesh_home_agents", "mesh_home_game", "mesh_home_cars"} {
copies = append(copies,
consumerCopy{Node: "home", Module: "postgres", Consumer: app, Size: bytesOf(400 << 20), Retired: true, Class: "valuable"},
consumerCopy{Node: "anchor", Module: "postgres", Consumer: app, Size: bytesOf(9 << 20), Class: "valuable"})
}
copies[1].Class = "irreplaceable" // the photo site's own database says so
got := dataFindings(nil, nil, nil, copies, nil, time.Now())
if len(got) != 5 {
t.Fatalf("%d findings for five empty replacements: %+v", len(got), got)
}
for i, o := range got {
want := conditions.Warning
if strings.Contains(o.ID, "mesh_home_board") {
want = conditions.Urgent
}
_ = i
if o.Kind != kindEmptyReplacement || o.Severity != want || o.Machine != "anchor" ||
len(o.Also) != 1 || o.Also[0] != "home" || !strings.Contains(o.Summary, "Pin it back to home") {
t.Errorf("%+v", o)
}
}
// While the old copy is still active (the first ten minutes), it is the same finding.
copies[0].Retired = false
copies[1].Class = "valuable"
if got := dataFindings(nil, nil, nil, copies[:2], nil, time.Now()); len(got) != 1 || got[0].Kind != kindEmptyReplacement {
t.Fatalf("with the old copy still active: %+v", got)
}
}
// A move a person made — the data moved first, then pinned — leaves a full copy at the new provider and
// a retired one at the old: nothing to say here; `cleanup` covers the old one.
func TestADeliberateMoveIsNoEmptyReplacement(t *testing.T) {
copies := []consumerCopy{
{Node: "home", Module: "postgres", Consumer: "mesh_home_board", Size: bytesOf(400 << 20), Retired: true},
{Node: "anchor", Module: "postgres", Consumer: "mesh_home_board", Size: bytesOf(402 << 20)},
}
if got := dataFindings(nil, nil, nil, copies, nil, time.Now()); len(got) != 0 {
t.Fatalf("a deliberate move raised %+v", got)
}
// Two small databases differing by less than the floor are not a finding either.
copies[0].Size, copies[1].Size = bytesOf(12<<20), bytesOf(8<<20)
if got := dataFindings(nil, nil, nil, copies, nil, time.Now()); len(got) != 0 {
t.Fatalf("noise between two empty databases raised %+v", got)
}
}
// Where a provider cannot say sizes, a consumer active at two providers is still said — as a warning,
// since which one is empty cannot be told.
func TestConsumerDataActiveTwiceWithoutSizesIsAWarning(t *testing.T) {
copies := []consumerCopy{
{Node: "home", Module: "minio", Consumer: "mesh_home_photos"},
{Node: "anchor", Module: "minio", Consumer: "mesh_home_photos"},
}
got := dataFindings(nil, nil, nil, copies, nil, time.Now())
if len(got) != 1 || got[0].Kind != kindDataHeldTwice || got[0].Severity != conditions.Warning {
t.Fatalf("%+v", got)
}
}
// The same incident for a module's own data: a module unassigned from one machine and assigned on
// another starts over in an empty directory while its full one is kept, retired, where it was.
func TestAnEmptyReplacementOfAModulesOwnDataIsUrgent(t *testing.T) {
now := time.Now()
retired := now.Add(-time.Hour)
records := []inventory.DataRecord{
{Machine: "home", Module: "house", Item: "config", Class: "irreplaceable", Path: "/var/lib/house/config",
Size: bytesOf(2 << 30), RetiredAt: &retired, FirstSeen: now.Add(-90 * 24 * time.Hour)},
{Machine: "anchor", Module: "house", Item: "config", Class: "irreplaceable", Path: "/var/lib/house/config",
Size: bytesOf(1 << 20), FirstSeen: now.Add(-time.Hour), LastWrite: when(now)},
}
got := findingsByKind(dataFindings(records, nil, shelfFor(t, houseManifest), nil, nil, now))
o, ok := got[kindEmptyReplacement]
if !ok || o.Severity != conditions.Urgent || o.Machine != "anchor" || o.Also[0] != "home" {
t.Fatalf("%+v", got)
}
// The same of a valuable item is a warning.
records[0].Class, records[1].Class = "valuable", "valuable"
if o := findingsByKind(dataFindings(records, nil, shelfFor(t, houseManifest), nil, nil, now))[kindEmptyReplacement]; o.Severity != conditions.Warning {
t.Fatalf("a valuable empty replacement: %+v", o)
}
records[0].Class, records[1].Class = "irreplaceable", "irreplaceable"
// Two machines running a module on purpose, both active, keep two sets of data: nothing to say.
records[0].RetiredAt = nil
if got := findingsByKind(dataFindings(records, nil, shelfFor(t, houseManifest), nil, nil, now)); got[kindEmptyReplacement].Kind != "" {
t.Fatalf("two active copies were read as a replacement: %+v", got)
}
}
// An irreplaceable item that lost more than half of its largest size in a week is urgent; a smaller loss,
// or a loss under the floor, is not a finding.
func TestAShrinkOfMoreThanHalfIsUrgent(t *testing.T) {
now := time.Now()
r := inventory.DataRecord{Machine: "home", Module: "house", Item: "config", Class: "irreplaceable",
Size: bytesOf(300 << 20), FirstSeen: now.Add(-30 * 24 * time.Hour), LastWrite: when(now), LastBackup: when(now)}
shelf := shelfFor(t, houseManifest)
o := findingsByKind(dataFindings([]inventory.DataRecord{r}, map[string]int64{r.Key(): 1 << 30}, shelf, nil, nil, now))[kindDataShrank]
if o.Severity != conditions.Urgent || !strings.Contains(o.Summary, "shrank") {
t.Fatalf("%+v", o)
}
for _, peak := range []int64{500 << 20, 20 << 20} {
if got := findingsByKind(dataFindings([]inventory.DataRecord{r}, map[string]int64{r.Key(): peak}, shelf, nil, nil, now)); got[kindDataShrank].Kind != "" {
t.Errorf("a peak of %d raised a shrink", peak)
}
}
small := r
small.Size = bytesOf(1 << 20)
if got := findingsByKind(dataFindings([]inventory.DataRecord{small}, map[string]int64{r.Key(): 10 << 20}, shelf, nil, nil, now)); got[kindDataShrank].Kind != "" {
t.Error("a loss under the floor raised a shrink")
}
}
// Data said to be written all the time and not written; data with no backup or an old one — urgent when
// irreplaceable, a warning when valuable; and a new item given its bound before it is said.
func TestQuietDataAndMissingBackupsAreSaidByClass(t *testing.T) {
now := time.Now()
shelf := shelfFor(t, houseManifest)
r := inventory.DataRecord{Machine: "home", Module: "house", Item: "config", Class: "irreplaceable",
Size: bytesOf(1 << 30), FirstSeen: now.Add(-10 * 24 * time.Hour), LastWrite: when(now.Add(-3 * 24 * time.Hour)),
LastBackup: when(now.Add(-72 * time.Hour))}
got := findingsByKind(dataFindings([]inventory.DataRecord{r}, nil, shelf, nil, nil, now))
if got[kindDataQuiet].Severity != conditions.Urgent || got[kindBackupStale].Severity != conditions.Urgent {
t.Fatalf("irreplaceable: %+v", got)
}
valuable := r
valuable.Class, valuable.MeasuredAt = "valuable", when(now) // measured: a holder is there to take its backup
if got := findingsByKind(dataFindings([]inventory.DataRecord{valuable}, nil, shelf, nil, nil, now)); got[kindDataQuiet].Severity != conditions.Warning ||
got[kindBackupStale].Severity != conditions.Warning {
t.Fatalf("valuable: %+v", got)
}
never := r
never.LastBackup = nil
if o := findingsByKind(dataFindings([]inventory.DataRecord{never}, nil, shelf, nil, nil, now))[kindBackupStale]; !strings.Contains(o.Summary, "no good backup") {
t.Fatalf("never backed up: %+v", o)
}
fresh := never
fresh.FirstSeen, fresh.LastWrite = now.Add(-time.Hour), when(now)
if got := dataFindings([]inventory.DataRecord{fresh}, nil, shelf, nil, nil, now); len(got) != 0 {
t.Fatalf("an item declared an hour ago, before its first night, raised %+v", got)
}
}
// An item retired more than thirty days waits for a person; less, it is only listed.
func TestRetiredDataWaitingThirtyDaysIsSaid(t *testing.T) {
now := time.Now()
old, recent := now.Add(-31*24*time.Hour), now.Add(-2*24*time.Hour)
records := []inventory.DataRecord{
{Machine: "home", Module: "house", Item: "config", Class: "irreplaceable", Size: bytesOf(1 << 30), RetiredAt: &old},
{Machine: "home", Module: "attic", Item: "boxes", Class: "irreplaceable", Size: bytesOf(1 << 30), RetiredAt: &recent},
}
got := dataFindings(records, nil, shelfFor(t, houseManifest), nil, nil, now)
if len(got) != 1 || got[0].Kind != kindCleanupWaiting || !strings.Contains(got[0].Summary, "cleanup delete home house config") {
t.Fatalf("%+v", got)
}
if rows := retiredData(records, now); len(rows) != 2 || rows[0].Kind != retiredDataKind {
t.Fatalf("cleanup list: %+v", rows)
}
}
// The holder's answer reads into measurements, by module and item.
func TestTheHoldersAnswerIsRead(t *testing.T) {
raw := []byte(`[{"module":"postgres","runs":1,"paths":["/var/lib/mesh-store/dumps"],"lastNight":null,"restorePoints":3,
"data":[{"item":"store","class":"irreplaceable","path":"/var/lib/mesh-store","covered_by":"/var/lib/mesh-store/dumps",
"size_bytes":1073741824,"last_write":"2026-10-06T10:00:00Z","measured_at":"2026-10-06T10:05:00Z","last_backup":"2026-10-06T03:10:00Z"}]}]`)
got, err := readHolder(raw)
if err != nil {
t.Fatal(err)
}
m := got["postgres"]["store"]
if m.Path != "/var/lib/mesh-store" || m.Size == nil || *m.Size != 1<<30 || m.LastBackup == nil || m.MeasuredAt == nil {
t.Fatalf("%+v", m)
}
// An older holder, which says no data, reads as nothing measured rather than a failure.
if got, err := readHolder([]byte(`[{"module":"postgres","runs":1,"paths":[]}]`)); err != nil || len(got) != 0 {
t.Fatalf("%v, %v", got, err)
}
}
// fakeHolder answers node-backup's `backed-up` on one machine over a real bus, with what it is told it
// measured.
type fakeHolder struct {
mu sync.Mutex
modules []map[string]any
}
func (f *fakeHolder) set(modules ...map[string]any) {
f.mu.Lock()
defer f.mu.Unlock()
f.modules = modules
}
func (f *fakeHolder) serve(t *testing.T, conn *nats.Conn, node string) {
t.Helper()
sub, err := conn.Subscribe(link.NodeSeatToolSubject(catalogue.BackupSeat, "backed-up", node), func(m *nats.Msg) {
f.mu.Lock()
defer f.mu.Unlock()
body, _ := json.Marshal(map[string]any{"result": f.modules, "error": "", "node": node})
_ = m.Respond(body)
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = sub.Unsubscribe() })
if err := conn.Flush(); err != nil {
t.Fatal(err)
}
}
func measuredHouse(path string, size int64, at time.Time) map[string]any {
return map[string]any{"module": "house", "runs": 0, "paths": []string{path}, "data": []map[string]any{{
"item": "config", "class": "irreplaceable", "path": path, "covered_by": path, "size_bytes": size,
"last_write": at, "measured_at": at, "last_backup": at}}}
}
// UNASSIGNING A MODULE WITH IRREPLACEABLE DATA KEEPS THE DATA, and assigning it elsewhere onto an empty
// directory is an empty replacement — through the real stores and a real bus. The unassignment says the
// data stays; the self-check's next run retires it (kept, listed by cleanup), and when the module comes
// up on another machine with an empty directory while the full one waits retired, that is urgent.
func TestNatsUnassigningIrreplaceableDataRetiresItAndAnEmptyReplacementIsUrgent(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
if _, err := inv.SeedSeats(ctx, catalogue.DefaultSeats()); err != nil {
t.Fatal(err)
}
register(t, open, catalogue.Manifest{Module: "keeper", Version: "1",
Claims: []catalogue.Claim{{Name: catalogue.BackupSeat, Scope: catalogue.ScopeNode, Serves: []string{"backed-up", "now", "restore"}}}})
house, err := catalogue.ParseManifest([]byte(houseManifest))
if err != nil {
t.Fatal(err)
}
register(t, open, house)
if _, err := assign(ctx, open, "laptop", "house"); err == nil {
t.Fatal("irreplaceable data was assigned to a machine with nothing to back it up")
}
for _, node := range []string{"laptop", "anchor"} {
if _, err := assign(ctx, open, node, "keeper"); err != nil {
t.Fatal(err)
}
}
if _, err := assign(ctx, open, "laptop", "house"); err != nil {
t.Fatal(err)
}
// Sent, and applied: a holder is asked only once it has been (novox/hq issue 275).
for _, node := range []string{"laptop", "anchor"} {
pushedAndApplied(t, open, node)
}
conn := onATestBus(t)
js, err := broker.Dial(os.Getenv("MESH_TEST_NATS"))
if err != nil {
t.Fatal(err)
}
t.Cleanup(js.Close)
laptop, anchor := &fakeHolder{}, &fakeHolder{}
laptop.serve(t, conn, "laptop")
anchor.serve(t, conn, "anchor")
now := time.Now()
heard := &watchdogs{last: &signalFacts{now: now, machines: []machineFacts{
{name: "laptop", lastHeard: now}, {name: "anchor", lastHeard: now}}}}
d := &doctor{open: open, js: js, watchdogs: heard}
var d13 probe
for _, p := range probeRegistry {
if p.ID == probeDataID {
d13 = p
}
}
run := func() []conditions.Observation {
t.Helper()
probing := context.WithValue(ctx, probeAsksKey{}, d13)
obs, err := probeData(probing, d)
if err != nil {
t.Fatal(err)
}
return obs
}
laptop.set(measuredHouse("/var/lib/house/config", 2<<30, now))
if obs := run(); len(obs) != 0 {
t.Fatalf("a measured, backed-up item raised %+v", obs)
}
r, err := inv.DataOf(ctx, "laptop", "house", "config")
if err != nil || r.Path != "/var/lib/house/config" || r.Size == nil || *r.Size != 2<<30 || r.LastBackup == nil {
t.Fatalf("what the holder measured was not kept: %+v, %v", r, err)
}
said, err := unassign(ctx, open, "laptop", "house")
if err != nil {
t.Fatal(err)
}
if !strings.Contains(said, "house's config (/var/lib/house/config, 2.0 GB) stays where it is") {
t.Fatalf("the unassignment does not say the data stays:\n%s", said)
}
laptop.set()
run()
r, err = inv.DataOf(ctx, "laptop", "house", "config")
if err != nil || !r.Retired() || r.Path != "/var/lib/house/config" {
t.Fatalf("unassigned, the irreplaceable item is not kept retired: %+v, %v", r, err)
}
records, _ := inv.Data(ctx)
if rows := retiredData(records, time.Now()); len(rows) != 1 || rows[0].Path != "/var/lib/house/config" {
t.Fatalf("cleanup list: %+v", rows)
}
// Assigned on the anchor, onto an empty directory.
if _, err := assign(ctx, open, "anchor", "house"); err != nil {
t.Fatal(err)
}
anchor.set(measuredHouse("/var/lib/house/config", 300<<10, time.Now()))
obs := findingsByKind(run())
o, ok := obs[kindEmptyReplacement]
if !ok || o.Severity != conditions.Urgent || o.Machine != "anchor" || o.Also[0] != "laptop" {
t.Fatalf("an empty replacement of a module's data was not urgent: %+v", obs)
}
}
// Data on redundant storage: the array it is on is watched, one condition per array as loud as the most
// precious item on it; an item said to be on redundancy and found on plain storage is said; an item
// whose path is gone is said.
func TestTheArrayUnderDataIsWatched(t *testing.T) {
now := time.Now()
media := `{"module":"media","version":"1","accesses":[{"id":"films","mode":"read"},{"id":"shows","mode":"read"}],
"data":{"own":[{"id":"films","path":"${access:films}","class":"irreplaceable","redundancy":"an array, no room to copy"},
{"id":"shows","path":"${access:shows}","class":"irreplaceable","redundancy":"an array, no room to copy"}]}}`
shelf := shelfFor(t, media)
sick := false
on := func(item string, healthy *bool) inventory.DataRecord {
return inventory.DataRecord{Machine: "home", Module: "media", Item: item, Class: "irreplaceable", Owned: false,
Path: "/tank/" + item, Size: bytesOf(40 << 40), FirstSeen: now.Add(-24 * time.Hour), MeasuredAt: when(now),
Redundancy: &inventory.Redundancy{Kind: "zfs", Where: "tank", Healthy: healthy, Said: "pool 'tank' is DEGRADED"}}
}
got := dataFindings([]inventory.DataRecord{on("films", &sick), on("shows", &sick)}, nil, shelf, nil, nil, now)
if len(got) != 1 || got[0].Kind != kindArrayDegraded || got[0].Severity != conditions.Urgent ||
!strings.Contains(got[0].Summary, "media/films, media/shows") {
t.Fatalf("%+v", got)
}
well := true
if got := dataFindings([]inventory.DataRecord{on("films", &well)}, nil, shelf, nil, nil, now); len(got) != 0 {
t.Fatalf("a healthy array raised %+v", got)
}
plain := on("films", nil)
plain.Redundancy = nil
if o := findingsByKind(dataFindings([]inventory.DataRecord{plain}, nil, shelf, nil, nil, now))[kindProtectionMissing]; o.Severity != conditions.Urgent {
t.Fatalf("redundancy said and not found: %+v", o)
}
gone := on("films", &well)
gone.MeasureError, gone.Size = "/tank/films does not exist", bytesOf(0)
if o := findingsByKind(dataFindings([]inventory.DataRecord{gone}, map[string]int64{gone.Key(): 40 << 40}, shelf, nil, nil, now))[kindDataMissing]; o.Severity != conditions.Urgent {
t.Fatalf("a vanished library: %+v", o)
}
}
// What a consumer keeps with its provider as irreplaceable holds the provider's item to that class:
// the photo site's objects make the object store's data an urgent matter.
func TestKeptByHoldsTheProvidersItemToTheConsumersClass(t *testing.T) {
objects := `{"module":"objects","version":"1","provides":[{"name":"s3-bucket","scope":"mesh"}],"grants":{"s3-bucket":"${dir:g}"},
"data":{"own":[{"id":"data","path":"${dir:data}","class":"valuable"}],"consumers":{"s3-bucket":{"class":"valuable","in":"data"}}},
"resources":[{"id":"g","type":"directory","mode":"0700"},{"id":"data","type":"directory","mode":"0700"}]}`
photos := `{"module":"photos","version":"1","requires":["s3-bucket"],"data":{"kept-by":{"s3-bucket":{"class":"irreplaceable"}}}}`
shelf := shelfFor(t, objects, photos)
bindings := []inventory.Binding{{Machine: "anchor", Consumer: "photos", Provision: "s3-bucket",
Provider: catalogue.Chosen{Node: "anchor", Module: "objects"}}}
upgraded, keptBy := keptByClasses(bindings, shelf)
if upgraded["anchor/objects/data"] != "irreplaceable" || keptBy["objects/mesh_anchor_photos"] != "irreplaceable" {
t.Fatalf("upgraded %v, kept by %v", upgraded, keptBy)
}
now := time.Now()
r := inventory.DataRecord{Machine: "anchor", Module: "objects", Item: "data", Class: "valuable", Owned: true,
Size: bytesOf(10 << 30), FirstSeen: now.Add(-10 * 24 * time.Hour), MeasuredAt: when(now), LastBackup: when(now.Add(-72 * time.Hour))}
if o := findingsByKind(dataFindings([]inventory.DataRecord{r}, nil, shelf, nil, upgraded, now))[kindBackupStale]; o.Severity != conditions.Urgent {
t.Fatalf("the photos' store without a backup: %+v", o)
}
if o := findingsByKind(dataFindings([]inventory.DataRecord{r}, nil, shelf, nil, nil, now))[kindBackupStale]; o.Severity != conditions.Warning {
t.Fatalf("a valuable store without a backup: %+v", o)
}
}
// Items measured from one dataset's counters share its size: a shrink of the dataset is one condition
// naming every item on it, not one per item; and a partial walk's lower bound is never compared.
func TestADatasetShrinksOnceAndAPartialSizeIsNeverCompared(t *testing.T) {
now := time.Now()
media := `{"module":"media","version":"1","accesses":[{"id":"films","mode":"read"},{"id":"shows","mode":"read"}],
"data":{"own":[{"id":"films","path":"${access:films}","class":"irreplaceable","redundancy":"an array","measure":"dataset"},
{"id":"shows","path":"${access:shows}","class":"irreplaceable","redundancy":"an array","measure":"dataset"}]}}`
shelf := shelfFor(t, media, houseManifest)
well := true
on := func(item string, size int64) inventory.DataRecord {
return inventory.DataRecord{Machine: "home", Module: "media", Item: item, Class: "irreplaceable",
Size: bytesOf(size), FirstSeen: now.Add(-24 * time.Hour), MeasuredAt: when(now),
Precision: "dataset tank/media: its whole size",
Redundancy: &inventory.Redundancy{Kind: "zfs", Where: "tank", Healthy: &well}}
}
films, shows := on("films", 30<<40), on("shows", 30<<40)
peaks := map[string]int64{films.Key(): 90 << 40, shows.Key(): 90 << 40}
got := dataFindings([]inventory.DataRecord{films, shows}, peaks, shelf, nil, nil, now)
if len(got) != 1 || got[0].Kind != kindDataShrank || got[0].Severity != conditions.Urgent ||
!strings.Contains(got[0].Summary, "media/films, media/shows") {
t.Fatalf("%+v", got)
}
partial := inventory.DataRecord{Machine: "home", Module: "house", Item: "config", Class: "irreplaceable",
Size: bytesOf(1 << 20), FirstSeen: now.Add(-24 * time.Hour), MeasuredAt: when(now), LastWrite: when(now),
LastBackup: when(now), Precision: "partial: measured partially"}
if got := dataFindings([]inventory.DataRecord{partial}, map[string]int64{partial.Key(): 1 << 30}, shelf, nil, nil, now); len(got) != 0 {
t.Fatalf("a partial size was compared: %+v", got)
}
}
+557
View File
@@ -0,0 +1,557 @@
package main
import (
"context"
"encoding/json"
"errors"
"flag"
"fmt"
"sort"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
)
// The self-check: `doctor` (novox/hq to-be 45 §4, ADR 0227 rule 6).
//
// **The design's invariants, run against the running mesh.** A probe is one live invariant of a
// design — every machine's declaration composes and validates, every resolver answers, every seat's
// holder answers, every stream and consumer is there as defined — with an id, a bound, and the
// condition it raises when the invariant does not hold. The serving controller runs the registry every
// five minutes, each probe given thirty seconds; a probe that errors or does not finish raises
// `probe-failed` for itself, because an unanswered probe is never a pass. Every run ends with a
// heartbeat on the bus (`doctor-heartbeat`, S10), which mesh-watcher listens for from a second
// machine: a controller that stops checking is itself said, through a channel that does not pass
// through it.
//
// `doctor` answers the last run's verdict at once; `doctor run` runs now; `doctor probes` lists the
// registry; `doctor signals` says, for every row of the signals table, the age of its newest signal.
// The self-check's clocks.
var (
// doctorEvery is how often the registry runs; doctorFirstAfter how long after the controller
// starts the first run waits, so what the controller hears at its start has arrived.
doctorEvery = 5 * time.Minute
doctorFirstAfter = time.Minute
// probeWithin is each probe's bound.
probeWithin = 30 * time.Second
)
// The verdicts a probe can have.
const (
verdictPass = "pass"
verdictFail = "fail"
verdictFailedToRun = "failed-to-run"
verdictDeferred = "deferred"
)
// probe is one live invariant.
type probe struct {
ID string
Asserts string
From string
// Kind is the condition kind raised when the invariant does not hold.
Kind string
// Raises are the other kinds its findings carry, each its own (a consumer missing, one far behind):
// what a healer may be registered against (healers.go).
Raises []string
Phase int
// Deferred says why it is not run yet; empty for one that is.
Deferred string
// Asks are the seat verbs it calls. A probe may call no other (askSeatTool refuses), and the
// controller's grant names every one (a test over this registry): a probe whose question the bus
// refuses checks nothing (D8, 2026-10-06).
Asks []broker.SeatVerb
run func(ctx context.Context, d *doctor) ([]conditions.Observation, error)
}
// probeRegistry is the registry, in to-be 45's order. **The registry is the design's live form**: a
// probe added to a design is a row added here.
var probeRegistry = []probe{
{ID: "D1", Asserts: "every machine's declaration composes, and passes the node-engine's validation",
From: "issues 236, 263, 275", Kind: "declaration-refused", Raises: []string{kindAwaitingPush}, Phase: 1,
run: probeDeclarations},
{ID: "D2", Asserts: "every holder of the mesh's resolver answers a machine name for IPv4, and NODATA for IPv6",
From: "issue 262", Kind: "resolver-wrong", Phase: 1, run: probeResolvers},
{ID: "D3", Asserts: "every seat on record that serves verbs has a live holder that answers, on every " +
"machine that is heard from", From: "issues 208, 218", Kind: "holder-silent", Phase: 1, run: probeHolders},
{ID: "D4", Asserts: "every kept archive is held by a manifest", From: "issue 253",
Kind: "archives-unheld", Phase: 1, run: probeArchives},
{ID: "D5", Asserts: "exactly one lease holder — the key names this controller at its epoch, and the record " +
"holds no other epoch open; no message from a stale epoch refused in the last interval",
From: "issue 204", Kind: "lease-split", Phase: 2, run: probeLease},
{ID: "D6", Asserts: "every durable consumer the mesh expects exists with its definition, and is near its " +
"stream's head", From: "issues 248, 266", Kind: "consumer-wrong", Raises: []string{"consumer-lost", kindConsumerBehind},
Phase: 1, run: probeConsumers},
{ID: "D7", Asserts: "every stream the controller defines exists with its definition, and its own buckets",
From: "issue 208", Kind: "stream-wrong", Phase: 1, run: probeStreams},
{ID: "D8", Asserts: "no address the mesh owns — a machine's private address or its endpoint — is in a ban list",
From: "issue 238", Kind: "own-address-banned", Phase: 1, run: probeBans,
Asks: []broker.SeatVerb{{Seat: "node-intrusion-prevention", Verb: "banned"}}},
{ID: "D9", Asserts: "status answers in full within ten seconds, from a summary composed lately",
From: "issue 265", Kind: "status-slow", Phase: 1, run: probeStatus},
{ID: "D10", Asserts: "every machine runs the node-engine and node tools builds the mesh holds, or is inside " +
"a plan's window", From: "the version split", Kind: "core-behind", Phase: 1, run: probeCoreBuilds},
{ID: "D11", Asserts: "no provider holds a consumer retired more than thirty days without a person deciding " +
"its cleanup", From: "ADR 0230", Kind: kindCleanupWaiting, Phase: 2, run: probeRetired},
{ID: probeBindingsID, Asserts: "every consumer of a provision that keeps its data is bound where it was last " +
"sent, or moves by a pin", From: "issue 273, ADR 0232", Kind: kindBindingMoved,
Raises: []string{kindBindingKept, kindBindingMoving}, Phase: 2, run: probeBindings},
{ID: probeDataID, Asserts: "every item of data a machine declares is measured, is there, holds what it held, is " +
"written where it should be, is backed up within its bound or sits on healthy redundant storage, and is no " +
"empty replacement of a copy kept elsewhere; what a machine no longer declares that is irreplaceable or " +
"valuable is retired, not forgotten", From: "issue 273, ADR 0233",
Kind: kindDataShrank, Raises: []string{kindEmptyReplacement, kindDataHeldTwice, kindDataQuiet, kindBackupStale,
kindDataUnmeasured, kindDataMissing, kindArrayDegraded, kindProtectionMissing, kindCleanupWaiting},
Phase: 2, run: probeData,
Asks: []broker.SeatVerb{{Seat: "node-backup", Verb: "backed-up"}}},
{ID: "DW", Asserts: "the watchdogs of the signals table ran within three of their intervals",
From: "ADR 0227 rule 6: the watchers are watched", Kind: "watchdogs-silent", Phase: 1, run: probeWatchdogs},
}
// probeVerdict is one probe's outcome in a run.
type probeVerdict struct {
ID string `json:"id"`
Verdict string `json:"verdict"`
Found []string `json:"found,omitempty"`
Error string `json:"error,omitempty"`
Took string `json:"took,omitempty"`
}
// doctorCounts are a run's verdicts, counted.
type doctorCounts struct {
Passed int `json:"passed"`
Failed int `json:"failed"`
FailedToRun int `json:"failed-to-run"`
Deferred int `json:"deferred"`
}
// doctorRun is one run of the registry, and the body of its heartbeat.
type doctorRun struct {
Run string `json:"run"`
At time.Time `json:"at"`
Started time.Time `json:"started"`
Took string `json:"took"`
IntervalSeconds int `json:"interval-seconds"`
Counts doctorCounts `json:"counts"`
Probes []probeVerdict `json:"probes"`
// Controller is the machine that ran it, and Why what started it: the schedule, or a person.
Controller string `json:"controller"`
Why string `json:"why"`
// Unsaid is how many condition transitions this controller could not say, since it started.
Unsaid int `json:"unsaid,omitempty"`
}
// doctor is the registry and what its probes need.
type doctor struct {
open *stores
js *broker.JetStream
keeper *conditions.Keeper
teller conditions.Teller
watchdogs *watchdogs
host string
running sync.Mutex
mu sync.Mutex
last *doctorRun
ended time.Time
}
// lastRunEnded is when the last run ended; zero before the first.
func (d *doctor) lastRunEnded() time.Time {
d.mu.Lock()
defer d.mu.Unlock()
return d.ended
}
// lastRun is the last run's verdict; nil before the first.
func (d *doctor) lastRun() *doctorRun {
d.mu.Lock()
defer d.mu.Unlock()
return d.last
}
// keep runs the registry on its schedule until ctx ends.
func (d *doctor) keep(ctx context.Context) {
select {
case <-ctx.Done():
return
case <-time.After(doctorFirstAfter):
}
tick := time.NewTicker(doctorEvery)
defer tick.Stop()
for {
// Only the controller acting checks the mesh on a schedule: one standing by would say a
// heartbeat for a self-check that is not the mesh's.
if d.watchdogs == nil || d.watchdogs.acting == nil || d.watchdogs.acting() {
d.runOnce(ctx, "the schedule")
}
select {
case <-ctx.Done():
return
case <-tick.C:
}
}
}
var doctorRuns struct {
sync.Mutex
n uint64
}
// runOnce runs every probe the registry runs, keeps what each found, and says the heartbeat. One run
// at a time: a person's `doctor run` during a scheduled one waits for it.
func (d *doctor) runOnce(ctx context.Context, why string) doctorRun {
d.running.Lock()
defer d.running.Unlock()
doctorRuns.Lock()
doctorRuns.n++
n := doctorRuns.n
doctorRuns.Unlock()
started := time.Now()
run := doctorRun{Run: fmt.Sprintf("doctor-%d-%d", started.Unix(), n), Started: started.UTC(),
IntervalSeconds: int(doctorEvery / time.Second), Controller: d.host, Why: why}
type result struct {
obs []conditions.Observation
err error
took time.Duration
}
results := make([]result, len(probeRegistry))
var wg sync.WaitGroup
for i, p := range probeRegistry {
if p.run == nil {
continue
}
wg.Add(1)
go func(i int, p probe) {
defer wg.Done()
probing, cancel := context.WithTimeout(context.WithValue(ctx, probeAsksKey{}, p), probeWithin)
defer cancel()
began := time.Now()
done := make(chan result, 1)
go func() {
defer func() {
if r := recover(); r != nil {
done <- result{err: fmt.Errorf("the probe panicked: %v", r)}
}
}()
obs, err := p.run(probing, d)
done <- result{obs: obs, err: err}
}()
select {
case r := <-done:
r.took = time.Since(began)
results[i] = r
case <-probing.Done():
results[i] = result{err: fmt.Errorf("it did not finish within %s", probeWithin), took: time.Since(began)}
}
}(i, p)
}
wg.Wait()
var blind []conditions.Observation
for i, p := range probeRegistry {
v := probeVerdict{ID: p.ID}
r := results[i]
switch {
case p.run == nil:
v.Verdict = verdictDeferred
run.Counts.Deferred++
case r.err != nil:
v.Verdict, v.Error = verdictFailedToRun, r.err.Error()
run.Counts.FailedToRun++
blind = append(blind, conditions.Observation{Scope: conditions.ScopeProbe, ID: p.ID, Kind: "probe-failed",
Token: "failed", Severity: conditions.Warning,
Summary: fmt.Sprintf("the probe %s (%s) could not run: what it checks is not known — never a pass", p.ID, p.Asserts),
Said: firstLine(r.err.Error())})
default:
if err := d.keeper.Reconcile(ctx, p.ID, kindedAs(r.obs, p.Kind)); err != nil {
v.Error = "what it found could not be kept: " + err.Error()
}
if len(r.obs) == 0 {
v.Verdict = verdictPass
run.Counts.Passed++
} else {
v.Verdict = verdictFail
run.Counts.Failed++
for _, o := range r.obs {
v.Found = append(v.Found, o.Summary)
}
}
}
if p.run != nil {
v.Took = r.took.Round(time.Millisecond).String()
}
run.Probes = append(run.Probes, v)
}
if err := d.keeper.Reconcile(ctx, sourceDoctor, blind); err != nil {
fmt.Printf("the self-check's own failures could not be kept: %v\n", err)
}
ended := time.Now()
run.At, run.Took, run.Unsaid = ended.UTC(), ended.Sub(started).Round(time.Millisecond).String(), d.keeper.Unsaid()
d.mu.Lock()
d.last, d.ended = &run, ended
d.mu.Unlock()
d.sayHeartbeat(ctx, run)
return run
}
// sourceDoctor is what raises a probe's own failure to run.
const sourceDoctor = "doctor"
// kindedAs gives each observation of a probe the probe's kind where it named none.
func kindedAs(obs []conditions.Observation, kind string) []conditions.Observation {
out := make([]conditions.Observation, 0, len(obs))
for _, o := range obs {
if o.Kind == "" {
o.Kind = kind
}
out = append(out, o)
}
return out
}
// sayHeartbeat publishes the run's heartbeat. Not said is said here, and S10 on the second machine
// says it outward: the watcher hears nothing.
func (d *doctor) sayHeartbeat(ctx context.Context, run doctorRun) {
if d.teller == nil {
return
}
body, err := json.Marshal(run)
if err != nil {
fmt.Printf("the self-check's heartbeat could not be written: %v\n", err)
return
}
saying, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
if err := d.teller.PublishSeatEvent(saying, conditions.Seat, conditions.HeartbeatEvent, body); err != nil {
fmt.Printf("the self-check's heartbeat (%s) could NOT be said, so the watcher on the second machine "+
"will say the self-check is silent: %v\n", run.Run, err)
}
}
// doctorFrom is the serving controller's self-check; nil in any other process.
var doctorFrom *doctor
// doctorCommand is `doctor`, `doctor run`, `doctor probes` and `doctor signals`.
func doctorCommand(ctx context.Context, args []string) error {
sub := ""
if len(args) > 0 && !strings.HasPrefix(args[0], "-") {
sub, args = args[0], args[1:]
}
set := flag.NewFlagSet("doctor", flag.ContinueOnError)
asJSON := set.Bool("json", false, "as data")
if rest, err := parseAround(set, args); err != nil {
return err
} else if len(rest) > 0 {
return errors.New("doctor [run|probes|signals] [--json]")
}
answer, err := doctorAnswer(ctx, sub)
if err != nil {
return err
}
if *asJSON {
return printJSON(answer)
}
fmt.Print(doctorText(answer))
return nil
}
// doctorAnswer is what the verb answers, as data.
func doctorAnswer(ctx context.Context, sub string) (any, error) {
switch sub {
case "":
if doctorFrom != nil {
if run := doctorFrom.lastRun(); run != nil {
return verdictAnswer(*run, time.Now()), nil
}
return nil, fmt.Errorf("the self-check has not finished its first run yet: it runs %s after the "+
"controller starts, then every %s — `doctor run` runs it now", doctorFirstAfter, doctorEvery)
}
run, err := lastHeartbeat(ctx)
if err != nil {
return nil, err
}
return verdictAnswer(run, time.Now()), nil
case "run":
d := doctorFrom
if d == nil {
local, closeIt, err := localDoctor(ctx)
if err != nil {
return nil, err
}
defer closeIt()
d = local
}
return verdictAnswer(d.runOnce(ctx, "asked by "+link.Caller()), time.Now()), nil
case "probes":
return probesAnswer(), nil
case "signals":
if doctorFrom == nil || doctorFrom.watchdogs == nil {
return nil, errors.New("the age of each signal is known to the serving controller alone, which " +
"hears them: ask it through the mesh-controller seat's doctor verb")
}
return signalsAnswer(doctorFrom.watchdogs.lastFacts(), doctorFrom.watchdogs.lastTick()), nil
}
return nil, fmt.Errorf("doctor answers the last run, or `run`, `probes` or `signals` — not %q", sub)
}
// verdictAnswer is a run as the verb answers it, with its age.
func verdictAnswer(run doctorRun, now time.Time) map[string]any {
return map[string]any{"run": run, "age": now.Sub(run.At).Round(time.Second).String(),
"note": "a probe that could not run is never a pass; each failure is an open condition until a run passes it"}
}
// probesAnswer is the registry.
func probesAnswer() map[string]any {
var out []map[string]any
for _, p := range probeRegistry {
row := map[string]any{"id": p.ID, "asserts": p.Asserts, "from": p.From, "kind": p.Kind, "phase": p.Phase}
if len(p.Raises) > 0 {
row["raises"] = p.Raises
}
if p.Deferred != "" {
row["deferred"] = p.Deferred
}
out = append(out, row)
}
return map[string]any{"probes": out, "every": doctorEvery.String(), "each within": probeWithin.String()}
}
// signalsAnswer is every row of the signals table with the age of its newest signal.
func signalsAnswer(f *signalFacts, ticked time.Time) map[string]any {
var rows []map[string]any
for _, r := range signalsTable {
row := map[string]any{"row": r.Row, "signal": r.Signal, "emitter": r.Emitter, "bound": r.Bound,
"kind": r.Kind, "severity": r.Severity, "phase": r.Phase}
switch {
case r.Deferred != "":
row["deferred"] = r.Deferred
case f == nil:
row["newest"] = "not yet looked at"
default:
if err := r.needs(f); err != nil {
row["blind"] = err.Error()
} else if newest := r.newest(f); newest.IsZero() {
row["newest"] = "none heard"
} else {
row["newest"] = newest.UTC().Format(time.RFC3339)
row["age"] = f.now.Sub(newest).Round(time.Second).String()
}
}
rows = append(rows, row)
}
out := map[string]any{"signals": rows, "every": watchEvery.String()}
if !ticked.IsZero() {
out["looked"] = ticked.UTC().Format(time.RFC3339)
}
return out
}
// doctorText is an answer as a person reads it.
func doctorText(answer any) string {
body, _ := json.Marshal(answer)
var b strings.Builder
var verdict struct {
Run doctorRun `json:"run"`
Age string `json:"age"`
}
if json.Unmarshal(body, &verdict) == nil && verdict.Run.Run != "" {
r := verdict.Run
fmt.Fprintf(&b, "%s, %s ago (took %s, %s): %d passed, %d failed, %d could not run, %d not built yet\n\n",
r.Run, verdict.Age, r.Took, r.Why, r.Counts.Passed, r.Counts.Failed, r.Counts.FailedToRun, r.Counts.Deferred)
for _, p := range r.Probes {
fmt.Fprintf(&b, " %-4s %-14s %s\n", p.ID, p.Verdict, p.Took)
for _, f := range p.Found {
fmt.Fprintf(&b, " %s\n", f)
}
if p.Error != "" {
fmt.Fprintf(&b, " %s\n", p.Error)
}
}
return b.String()
}
pretty, _ := json.MarshalIndent(answer, "", " ")
return string(pretty) + "\n"
}
// lastHeartbeat is the newest run's heartbeat, read from the events stream: what a process other than
// the serving controller answers `doctor` from.
func lastHeartbeat(ctx context.Context) (doctorRun, error) {
var run doctorRun
err := onTheBus(func(conn *nats.Conn) error {
js, err := conn.JetStream(nats.Context(ctx))
if err != nil {
return err
}
msg, err := js.GetLastMsg(broker.EventsStream, link.SeatEventSubject(conditions.Seat, conditions.HeartbeatEvent))
if errors.Is(err, nats.ErrMsgNotFound) {
return errors.New("the self-check has said no heartbeat on the bus in the last week: it is not running")
}
if err != nil {
return fmt.Errorf("the self-check's last heartbeat cannot be read: %w", err)
}
return json.Unmarshal(msg.Data, &run)
})
return run, err
}
// localDoctor is a self-check run by a process other than the serving controller: its own stores,
// its own connection, and its own keeper, closed after.
func localDoctor(ctx context.Context) (*doctor, func(), error) {
open, err := openStores(ctx)
if err != nil {
return nil, nil, err
}
js, err := dialTheBus()
if err != nil {
open.Close()
return nil, nil, err
}
k, err := keeperOn(ctx, js.Conn())
if err != nil {
js.Close()
open.Close()
return nil, nil, err
}
jsCtx := js.Context()
d := &doctor{open: open, js: js, keeper: k, teller: link.OverNATS{Conn: js.Conn(), JS: jsCtx},
host: controlHost(ctx, open.inventory)}
return d, func() {
flushing, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()
k.Close(flushing)
js.Close()
open.Close()
}, nil
}
// sortedFound is a probe's findings in a stated order, so two runs over one mesh say the same.
func sortedFound(obs []conditions.Observation) []conditions.Observation {
sort.Slice(obs, func(i, j int) bool { return obs[i].Key() < obs[j].Key() })
return obs
}
// probeAsksKey carries the running probe, so a seat verb it calls is checked against what it declares.
type probeAsksKey struct{}
// declaredBy says whether the probe running in ctx declared a seat verb; outside a probe, false.
func declaredBy(ctx context.Context, seat, verb string) (string, bool) {
p, ok := ctx.Value(probeAsksKey{}).(probe)
if !ok {
return "a caller outside the self-check", false
}
for _, v := range p.Asks {
if v.Seat == seat && v.Verb == verb {
return p.ID, true
}
}
return p.ID, false
}
+425
View File
@@ -0,0 +1,425 @@
package main
import (
"context"
"encoding/json"
"errors"
"net"
"os"
"slices"
"strings"
"sync/atomic"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"golang.org/x/net/dns/dnsmessage"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
)
// The self-check (novox/hq to-be 45 §4): a probe that fails raises its condition, one that cannot run
// raises probe-failed for itself and is never a pass, and every run ends with its heartbeat.
// withProbes runs the test with a registry of its own.
func withProbes(t *testing.T, probes ...probe) {
t.Helper()
before := probeRegistry
probeRegistry = probes
t.Cleanup(func() { probeRegistry = before })
}
func TestARunKeepsWhatEachProbeFoundAndSaysItsHeartbeat(t *testing.T) {
failing := conditions.Observation{Scope: conditions.ScopeMachine, ID: "anchor", Token: "refused",
Severity: conditions.Urgent, Summary: "anchor's node-engine would refuse its declaration"}
var broken atomic.Bool
broken.Store(true)
withProbes(t,
probe{ID: "P1", Asserts: "passes", Kind: "never", Phase: 1,
run: func(context.Context, *doctor) ([]conditions.Observation, error) { return nil, nil }},
probe{ID: "P2", Asserts: "finds a fault", Kind: "declaration-refused", Phase: 1,
run: func(context.Context, *doctor) ([]conditions.Observation, error) {
if broken.Load() {
return []conditions.Observation{failing}, nil
}
return nil, nil
}},
probe{ID: "P3", Asserts: "cannot run", Kind: "x", Phase: 1,
run: func(context.Context, *doctor) ([]conditions.Observation, error) {
if broken.Load() {
return nil, errors.New("the store is away")
}
return nil, nil
}},
probe{ID: "P4", Asserts: "hangs", Kind: "x", Phase: 1,
run: func(ctx context.Context, _ *doctor) ([]conditions.Observation, error) {
if broken.Load() {
<-ctx.Done()
time.Sleep(50 * time.Millisecond)
}
return nil, nil
}},
probe{ID: "P5", Asserts: "later", Kind: "x", Phase: 2, Deferred: "not yet"},
)
before := probeWithin
probeWithin = 200 * time.Millisecond
t.Cleanup(func() { probeWithin = before })
store := conditions.NewInMemory()
told := &conditions.Told{}
k := conditions.NewKeeper(t.Context(), conditions.Options{Store: store, History: store})
defer k.Close(context.Background())
d := &doctor{keeper: k, teller: told, host: "anchor"}
run := d.runOnce(t.Context(), "a test")
if run.Counts != (doctorCounts{Passed: 1, Failed: 1, FailedToRun: 2, Deferred: 1}) {
t.Fatalf("counted %+v", run.Counts)
}
open, err := k.Open(t.Context())
if err != nil {
t.Fatal(err)
}
var keys []string
for _, c := range open {
keys = append(keys, c.Key+"="+c.Kind)
}
for _, want := range []string{"machine.anchor.refused=declaration-refused", "probe.P3.failed=probe-failed",
"probe.P4.failed=probe-failed"} {
if !slices.Contains(keys, want) {
t.Errorf("%s is not open: %v", want, keys)
}
}
// The heartbeat, in the shape mesh-watcher reads (the contract with the operator's channel).
if len(told.Names) != 1 || told.Names[0] != conditions.HeartbeatEvent {
t.Fatalf("said %v", told.Names)
}
if d.lastRunEnded().IsZero() || d.lastRun().Run != run.Run {
t.Fatal("the run is not the last verdict")
}
body, _ := json.Marshal(run)
var shape map[string]any
_ = json.Unmarshal(body, &shape)
for _, field := range []string{"run", "at", "interval-seconds", "counts", "probes", "controller"} {
if _, ok := shape[field]; !ok {
t.Errorf("the heartbeat carries no %q: %s", field, body)
}
}
// Mended: the next run clears every one of them.
broken.Store(false)
d.runOnce(t.Context(), "a test")
if open, _ := k.Open(t.Context()); len(open) != 0 {
t.Fatalf("a passing run left open %+v", open)
}
}
// **The registry says what each probe asserts**, and a probe not built says why and when.
func TestTheRegistryIsTheDesignsLiveForm(t *testing.T) {
seen := map[string]bool{}
for _, p := range probeRegistry {
if seen[p.ID] {
t.Errorf("%s twice", p.ID)
}
seen[p.ID] = true
if p.Asserts == "" || p.From == "" || p.Kind == "" {
t.Errorf("%s does not say what it asserts, where from, or what it raises", p.ID)
}
if (p.run == nil) != (p.Deferred != "") || (p.Deferred != "" && p.Phase <= 1) {
t.Errorf("%s is run and deferred, or neither, or deferred out of Phase 1: %+v", p.ID, p)
}
}
for _, id := range []string{"D1", "D2", "D3", "D4", "D5", "D6", "D7", "D8", "D9", "D10"} {
if !seen[id] {
t.Errorf("to-be 45 §4 has %s and the registry does not", id)
}
}
}
// **The doctor and the watchdogs watch each other**: watchdogs that stopped are DW; a self-check that
// stopped is S10 (signals_test.go).
func TestWatchdogsThatStoppedAreSaid(t *testing.T) {
w := &watchdogs{started: time.Now().Add(-time.Hour)}
d := &doctor{watchdogs: w, host: "anchor"}
got, err := probeWatchdogs(t.Context(), d)
if err != nil || len(got) != 1 || got[0].Severity != conditions.Urgent {
t.Fatalf("%+v %v", got, err)
}
w.ticked = time.Now()
if got, _ := probeWatchdogs(t.Context(), d); len(got) != 0 {
t.Fatalf("%+v", got)
}
}
// **D1 composes every machine of a healthy mesh and the host's own validator takes each.**
func TestEveryMachineOfAHealthyMeshComposesAndValidates(t *testing.T) {
open := aMesh(t)
got, err := probeDeclarations(t.Context(), &doctor{open: open})
if err != nil {
t.Fatal(err)
}
if len(got) != 0 {
t.Fatalf("a healthy mesh failed D1: %+v", got)
}
}
// **D1 names a machine nothing can be sent to**, and the network that cannot be computed for it.
func TestAMachineWhoseDeclarationDoesNotComposeIsSaid(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
one, two := rivals()
register(t, open, one)
register(t, open, two)
for _, m := range []string{"rival-one", "rival-two"} {
if _, err := assign(ctx, open, "laptop", m); err != nil && m == "rival-one" {
t.Fatal(err)
}
}
got, err := probeDeclarations(ctx, &doctor{open: open})
if err != nil {
t.Fatal(err)
}
if len(got) != 1 || got[0].Key() != "machine.laptop.uncomposable" || !strings.Contains(got[0].Summary, "the-seat") {
t.Fatalf("%+v", got)
}
}
// **D2: a resolver answering NXDOMAIN for IPv6 is wrong** — musl takes it as no such name (issue 262).
func TestAResolverAnsweringNoSuchNameForIPv6IsWrong(t *testing.T) {
answerAs := func(rcode dnsmessage.RCode) string {
conn, err := net.ListenPacket("udp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = conn.Close() })
go func() {
buf := make([]byte, 1500)
for {
n, from, err := conn.ReadFrom(buf)
if err != nil {
return
}
var q dnsmessage.Message
if q.Unpack(buf[:n]) != nil {
continue
}
reply := dnsmessage.Message{Header: dnsmessage.Header{ID: q.ID, Response: true}, Questions: q.Questions}
if q.Questions[0].Type == dnsmessage.TypeA {
reply.Answers = []dnsmessage.Resource{{Header: dnsmessage.ResourceHeader{Name: q.Questions[0].Name,
Type: dnsmessage.TypeA, Class: dnsmessage.ClassINET}, Body: &dnsmessage.AResource{A: [4]byte{10, 77, 0, 1}}}}
} else {
reply.RCode = rcode
}
packed, _ := reply.Pack()
_, _ = conn.WriteTo(packed, from)
}
}()
_, port, _ := net.SplitHostPort(conn.LocalAddr().String())
return port
}
before := resolverPort
t.Cleanup(func() { resolverPort = before })
resolverPort = answerAs(dnsmessage.RCodeSuccess)
v4, rcode, err := askResolver(t.Context(), "127.0.0.1", "anchor.internal", dnsmessage.TypeA)
if err != nil || rcode != dnsmessage.RCodeSuccess || !slices.Equal(v4, []string{"10.77.0.1"}) {
t.Fatalf("%v %v %v", v4, rcode, err)
}
v6, rcode, err := askResolver(t.Context(), "127.0.0.1", "anchor.internal", dnsmessage.TypeAAAA)
if err != nil || rcode != dnsmessage.RCodeSuccess || len(v6) != 0 {
t.Fatalf("NODATA read as %v %v %v", v6, rcode, err)
}
resolverPort = answerAs(dnsmessage.RCodeNameError)
if _, rcode, _ := askResolver(t.Context(), "127.0.0.1", "anchor.internal", dnsmessage.TypeAAAA); rcode != dnsmessage.RCodeNameError {
t.Fatalf("NXDOMAIN read as %v", rcode)
}
}
// **D6, D7: what the controller defines is what it finds**, and a consumer deleted or a stream
// redefined is said — against a real bus, raised by the same derivation the controller starts with.
func TestNatsTheBusIsWhatTheControllerDefines(t *testing.T) {
url := os.Getenv("MESH_TEST_NATS")
if url == "" {
t.Skip("MESH_TEST_NATS unset")
}
open := aMesh(t)
js, err := broker.Dial(url)
if err != nil {
t.Fatal(err)
}
t.Cleanup(js.Close)
for _, s := range []string{"CONTROL", "NODES", "ASSIGNMENTS", "EVENTS"} {
_ = js.Context().DeleteStream(s)
}
if _, err := assertBusObjects(t.Context(), open.inventory, js); err != nil {
t.Fatal(err)
}
if err := js.EnsureControllerBuckets(); err != nil {
t.Fatal(err)
}
d := &doctor{open: open, js: js}
for _, p := range []func(context.Context, *doctor) ([]conditions.Observation, error){probeConsumers, probeStreams} {
got, err := p(t.Context(), d)
if err != nil || len(got) != 0 {
t.Fatalf("a bus just raised fails: %+v %v", got, err)
}
}
if err := js.Context().DeleteConsumer("NODES", "laptop"); err != nil {
t.Fatal(err)
}
info, err := js.Context().StreamInfo("EVENTS")
if err != nil {
t.Fatal(err)
}
cfg := info.Config
cfg.MaxMsgsPerSubject = 3
if _, err := js.Context().UpdateStream(&cfg); err != nil {
t.Fatal(err)
}
consumers, err := probeConsumers(t.Context(), d)
if err != nil || len(consumers) != 1 || consumers[0].Key() != "bus.NODES.laptop.missing" {
t.Fatalf("the deleted consumer: %+v %v", consumers, err)
}
streams, err := probeStreams(t.Context(), d)
if err != nil || len(streams) != 1 || !strings.Contains(streams[0].Summary, "per subject") {
t.Fatalf("the redefined stream: %+v %v", streams, err)
}
}
// **S9 hears the bus**: a consumer that gives up on a message, and one deleted, as the server says.
func TestNatsTheBusSaysAConsumerGaveUpAndOneWasDeleted(t *testing.T) {
url := os.Getenv("MESH_TEST_NATS")
if url == "" {
t.Skip("MESH_TEST_NATS unset")
}
conn, err := nats.Connect(url)
if err != nil {
t.Fatal(err)
}
defer conn.Close()
heard := make(chan *nats.Msg, 16)
for _, subject := range broker.BusAdvisories {
if _, err := conn.ChanSubscribe(subject, heard); err != nil {
t.Fatal(err)
}
}
api, _ := jetstream.New(conn)
_ = api.DeleteStream(t.Context(), "SEAT_ADVISED")
stream, err := api.CreateStream(t.Context(), jetstream.StreamConfig{Name: "SEAT_ADVISED", Subjects: []string{"advised.>"}})
if err != nil {
t.Fatal(err)
}
defer func() { _ = api.DeleteStream(context.Background(), "SEAT_ADVISED") }()
consumer, err := stream.CreateConsumer(t.Context(), jetstream.ConsumerConfig{Durable: "SEAT_ADVISED_worker",
AckPolicy: jetstream.AckExplicitPolicy, MaxDeliver: 1, AckWait: 100 * time.Millisecond})
if err != nil {
t.Fatal(err)
}
if _, err := api.Publish(t.Context(), "advised.x", []byte("x")); err != nil {
t.Fatal(err)
}
if _, err := consumer.Fetch(1, jetstream.FetchMaxWait(time.Second)); err != nil {
t.Fatal(err)
}
// A seat's worker, so the deletion is of a consumer the mesh names (link.MeshNamed).
// Not acknowledged: after its one delivery the consumer gives up on it — on the next fetch.
time.Sleep(300 * time.Millisecond)
_, _ = consumer.Fetch(1, jetstream.FetchMaxWait(300*time.Millisecond))
if err := stream.DeleteConsumer(t.Context(), "SEAT_ADVISED_worker"); err != nil {
t.Fatal(err)
}
kinds := map[string]string{}
deadline := time.After(5 * time.Second)
for len(kinds) < 2 {
select {
case m := <-heard:
if a, ok := link.ReadAdvisory(m.Subject, m.Data); ok {
kinds[a.Kind] = a.Said
}
case <-deadline:
t.Fatalf("the bus said only %v", kinds)
}
}
if !strings.Contains(kinds["max-deliveries"], "gave up") || !strings.Contains(kinds["consumer-lost"], "was deleted") {
t.Fatalf("%v", kinds)
}
}
// **D10 compares a node-engine with what it is delivered as, not with its commit** (2026-10-06: every
// machine read as behind right after a push sent it the current build — it says the digest-named
// directory it runs from, and the mesh holds a commit).
func TestANodeEngineIsJudgedByTheVersionItIsDeliveredAs(t *testing.T) {
m := catalogue.Manifest{Module: "mesh-host", Resources: []map[string]any{
{"id": "launcher", "type": "file", "path": "/usr/lib/nox-mesh-host/launch"},
{"id": "host", "type": "archive", "path": "/usr/lib/nox-mesh-host/versions/31045596c83a"},
{"id": "unfilled", "type": "archive", "path": "/usr/lib/x/versions/${version}"},
}}
delivered := deliveredVersions(m)
if !slices.Equal(delivered, []string{"31045596c83a"}) {
t.Fatalf("%v", delivered)
}
commit := "1545b00a9f0c"
for _, c := range []struct {
reported string
behind bool
}{
{"31045596c83a", false}, // the live case: current, and was called behind
{"0123456789ab", true}, // another delivery
{"1545b00a", false}, // placed by hand, stamped with the commit
{"", false}, // not said
} {
if got := engineBehind(c.reported, delivered, commit); got != c.behind {
t.Errorf("%q behind = %v, want %v", c.reported, got, c.behind)
}
}
if engineBehind("31045596c83a", nil, commit) {
t.Error("behind a mesh that holds no delivered build")
}
}
// **Every seat verb a probe calls is one it declares, and one the controller is granted** — derived
// from the registry, so a probe added with a question the bus would refuse fails here, not live.
func TestEverySeatVerbAProbeAsksIsGranted(t *testing.T) {
granted, err := broker.PermissionsFor(broker.Principal{Kind: broker.KindController, PasswordHash: "x"})
if err != nil {
t.Fatal(err)
}
asked := 0
for _, p := range probeRegistry {
for _, v := range p.Asks {
asked++
subject := link.NodeSeatToolSubject(v.Seat, v.Verb, "anchor")
if !slices.ContainsFunc(granted.Publish, func(pattern string) bool { return subjectMatches(pattern, subject) }) {
t.Errorf("%s asks %s.%s and the controller may not publish %s", p.ID, v.Seat, v.Verb, subject)
}
if !slices.Contains(broker.VerbsTheSelfCheckAsks, v) {
t.Errorf("%s asks %s.%s, which broker.VerbsTheSelfCheckAsks does not name", p.ID, v.Seat, v.Verb)
}
}
}
if asked == 0 {
t.Fatal("no probe asks a seat verb: D8 lost its declaration")
}
// And a probe asking what it did not declare is refused before anything is sent.
ctx := context.WithValue(t.Context(), probeAsksKey{}, probe{ID: "DX"})
if _, err := askSeatTool(ctx, nil, "node-intrusion-prevention", "banned", "anchor"); err == nil ||
!strings.Contains(err.Error(), "does not declare") {
t.Fatalf("an undeclared question was asked: %v", err)
}
}
// subjectMatches is the bus's matching of a permission pattern against a subject.
func subjectMatches(pattern, subject string) bool {
p, s := strings.Split(pattern, "."), strings.Split(subject, ".")
for i, tok := range p {
if tok == ">" {
return len(s) > i
}
if i >= len(s) || (tok != "*" && tok != s[i]) {
return false
}
}
return len(p) == len(s)
}
+170
View File
@@ -0,0 +1,170 @@
package main
import (
"context"
"encoding/json"
"flag"
"fmt"
"os"
"slices"
"sort"
"strings"
"time"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// What the core's bounds are set from (novox/hq to-be 45 Phase 0).
//
// **A bound is set from what was measured, not from what seemed reasonable.** Phase 1 puts a watchdog
// on each row of the signals table, and each has a bound: S1 three heartbeat intervals, S2 three times
// a machine's last apply, S3 a tier's build and apply time, S6 a build's timeout. Marked provisional
// in the design until a fortnight of these says what the mesh actually takes. Recorded by the serving
// controller as it hears each — a send's first report, a machine's next word, a plan leaving a tier, a
// build's outcome — and summarised here per machine, repository or module.
// recordBuildDuration measures one build from its ask to its outcome heard.
func recordBuildDuration(ctx context.Context, inv *inventory.Inventory, result link.BuildResult, asked time.Time) {
if asked.IsZero() || result.ID == "" {
return
}
subject := result.Module
if subject == "" {
subject = result.Repository
}
detail := "built"
if result.Failed != "" {
detail = "failed: " + firstLine(result.Failed)
}
if err := inv.RecordDuration(ctx, inventory.Duration{Kind: inventory.DurationBuild, Subject: subject,
Node: result.On, Ref: result.ID, Started: asked, Took: time.Since(asked), Detail: detail}); err != nil {
fmt.Fprintf(os.Stderr, "%s: how long it took could not be recorded: %v\n", result.ID, err)
}
}
// durationSummary is one subject's measurements of one kind.
type durationSummary struct {
Kind string `json:"kind"`
Subject string `json:"subject"`
Count int `json:"count"`
Median string `json:"median"`
P90 string `json:"p90"`
Max string `json:"max"`
// Bound is what to-be 45's rule would make of these, where the rule is a multiple of a measured
// time: three times the slowest apply (S2), three times the median word interval (S1).
Suggests string `json:"suggests,omitempty"`
}
func summarise(ds []inventory.Duration) []durationSummary {
type key struct{ kind, subject string }
by := map[key][]time.Duration{}
for _, d := range ds {
k := key{d.Kind, d.Subject}
by[k] = append(by[k], d.Took)
}
var out []durationSummary
for k, took := range by {
slices.Sort(took)
at := func(q float64) time.Duration { return took[int(q*float64(len(took)-1))] }
s := durationSummary{Kind: k.kind, Subject: k.subject, Count: len(took),
Median: round(at(0.5)), P90: round(at(0.9)), Max: round(took[len(took)-1])}
switch k.kind {
case inventory.DurationApply:
s.Suggests = "S2 bound max(2m, 3×last apply) ≈ " + round(max(2*time.Minute, 3*at(0.9))) + " at the p90"
case inventory.DurationHeartbeatGap:
s.Suggests = "S1 bound 3×interval ≈ " + round(3*at(0.5))
}
out = append(out, s)
}
sort.Slice(out, func(i, j int) bool {
ki, kj := slices.Index(inventory.DurationKinds, out[i].Kind), slices.Index(inventory.DurationKinds, out[j].Kind)
if ki != kj {
return ki < kj
}
return out[i].Subject < out[j].Subject
})
return out
}
func round(d time.Duration) string {
switch {
case d < time.Second:
return d.Round(time.Millisecond).String()
case d < time.Minute:
return d.Round(100 * time.Millisecond).String()
default:
return d.Round(time.Second).String()
}
}
// durationsCommand is `durations`: the summary per kind and subject, or every measurement as data.
func durationsCommand(ctx context.Context, args []string) error {
set := flag.NewFlagSet("durations", flag.ContinueOnError)
kind := set.String("kind", "", "one kind: "+strings.Join(inventory.DurationKinds, ", "))
days := set.Int("days", 14, "how many days back")
asJSON := set.Bool("json", false, "the summary as data")
all := set.Bool("all", false, "every measurement rather than the summary")
if _, err := parseAround(set, args); err != nil {
return err
}
if *kind != "" && !slices.Contains(inventory.DurationKinds, *kind) {
return fmt.Errorf("%q is not a kind of duration: %s", *kind, strings.Join(inventory.DurationKinds, ", "))
}
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
ds, err := open.inventory.Durations(ctx, *kind, time.Now().Add(-time.Duration(*days)*24*time.Hour))
if err != nil {
return err
}
if *all {
body, err := json.MarshalIndent(ds, "", " ")
if err != nil {
return err
}
fmt.Println(string(body))
return nil
}
summary := summarise(ds)
if *asJSON {
body, err := json.MarshalIndent(map[string]any{"days": *days, "durations": summary}, "", " ")
if err != nil {
return err
}
fmt.Println(string(body))
return nil
}
if len(summary) == 0 {
fmt.Printf("nothing measured in the last %d day(s): the serving controller records apply, heartbeat-gap, "+
"plan-tier and build durations as it hears them\n", *days)
return nil
}
fmt.Printf("durations over the last %d day(s) — what the core's bounds are set from (to-be 45 Phase 0)\n\n", *days)
fmt.Printf(" %-14s %-28s %6s %10s %10s %10s\n", "kind", "of", "count", "median", "p90", "max")
for _, s := range summary {
fmt.Printf(" %-14s %-28s %6d %10s %10s %10s\n", s.Kind, s.Subject, s.Count, s.Median, s.P90, s.Max)
if s.Suggests != "" {
fmt.Printf(" %-14s %-28s %s\n", "", "", s.Suggests)
}
}
return nil
}
// forgettingOldDurations removes what is older than a month, at start and daily after.
func forgettingOldDurations(ctx context.Context, inv *inventory.Inventory) {
for {
if n, err := inv.ForgetOldDurations(ctx); err != nil {
fmt.Fprintf(os.Stderr, "durations older than %s could not be removed: %v\n", inventory.DurationsKeptFor, err)
} else if n > 0 {
fmt.Printf("removed %d duration(s) older than %s\n", n, inventory.DurationsKeptFor)
}
select {
case <-ctx.Done():
return
case <-time.After(24 * time.Hour):
}
}
}
+115
View File
@@ -0,0 +1,115 @@
package main
import (
"context"
"encoding/json"
"errors"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/link"
)
// A declaration carries the lease's epoch (novox/hq to-be 45 §6) — to a machine whose node-engine said
// it reads one, and to no other: an older node-engine refuses a key it does not know, whole.
// bodiesDelivery records each send as the mesh does, and keeps the bodies.
type bodiesDelivery struct {
recordedDelivery
bodies map[string][]byte
}
func (b *bodiesDelivery) declare(ctx context.Context, s readyNode, body []byte) (string, error) {
b.bodies[s.node] = body
return b.recordedDelivery.declare(ctx, s, body)
}
func TestAMachineIsSentTheEpochOnlyOnceItSaysItReadsOne(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
epoch := uint64(57)
was := epochForActs
epochForActs = func(context.Context) (uint64, error) { return epoch, nil }
t.Cleanup(func() { epochForActs = was })
anchor, err := inv.NodeByName(ctx, "anchor")
if err != nil {
t.Fatal(err)
}
if err := inv.RecordReadsEpoch(ctx, anchor.ID, true); err != nil {
t.Fatal(err)
}
gens, err := generators(ctx, open)
if err != nil {
t.Fatal(err)
}
d := &bodiesDelivery{recordedDelivery: recordedDelivery{inv: inv}, bodies: map[string][]byte{}}
if _, err := sendRound(ctx, open, []string{"anchor", "laptop"}, composeForPush(open, gens), d, ""); err != nil {
t.Fatal(err)
}
carried := func(node string) (epoch float64, has bool) {
var envelope map[string]any
if err := json.Unmarshal(d.bodies[node], &envelope); err != nil {
t.Fatal(err)
}
epoch, has = envelope["epoch"].(float64)
return epoch, has
}
if e, has := carried("anchor"); !has || e != 57 {
t.Fatalf("the machine that reads an epoch was sent %v: %s", e, d.bodies["anchor"])
}
if _, has := carried("laptop"); has {
t.Fatalf("a machine that never said it reads an epoch was sent one: %s", d.bodies["laptop"])
}
// A new holder of the lease is not a change of the machine: neither reads as behind.
epoch = 58
would, err := wouldSend(ctx, open, mustNodes(t, open))
if err != nil {
t.Fatal(err)
}
for _, node := range []string{"anchor", "laptop"} {
sent, err := inv.Outstanding(ctx, node)
if err != nil {
t.Fatal(err)
}
if would[node] != sent {
t.Fatalf("%s reads as behind after the lease changed hands, with nothing else changed", node)
}
}
}
// A process that may not act composes nothing and sends nothing: its number is not taken.
func TestNothingIsComposedOrSentWithoutTheLease(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
was := epochForActs
epochForActs = func(context.Context) (uint64, error) { return 0, errors.New("this controller lost the lease") }
t.Cleanup(func() { epochForActs = was })
gens, err := generators(ctx, open)
if err != nil {
t.Fatal(err)
}
d := &bodiesDelivery{recordedDelivery: recordedDelivery{inv: open.inventory}, bodies: map[string][]byte{}}
refused, err := sendRound(ctx, open, []string{"anchor"}, composeForPush(open, gens), d, "")
if err != nil {
t.Fatal(err)
}
if len(d.bodies) != 0 || len(refused) != 1 || !strings.Contains(refused[0], "lost the lease") {
t.Fatalf("a controller without the lease composed %d and refused %v", len(d.bodies), refused)
}
anchor, _ := open.inventory.NodeByName(ctx, "anchor")
if seq, _ := open.inventory.Sequence(ctx, anchor.ID); seq != 0 {
t.Fatalf("a controller without the lease took sequence %d", seq)
}
// And at the send itself: the gate every declaration passes.
gate := link.ActingGate
link.ActingGate = func(context.Context) error { return errors.New("this controller lost the lease") }
t.Cleanup(func() { link.ActingGate = gate })
if err := link.Declare(ctx, nil, nil, "anchor", []byte(`{"declaration":1}`), 0); err == nil ||
!strings.Contains(err.Error(), "lost the lease") {
t.Fatalf("a declaration was let through the gate: %v", err)
}
}
+90
View File
@@ -0,0 +1,90 @@
package main
import (
"context"
"encoding/json"
"log"
"strings"
"sync"
"time"
"github.com/novox/mesh-controller/internal/link"
)
// A value given by hand lives only until the module's first good start (novox/hq ADR 0228).
//
// The serving controller hears every report; a clean one about the declaration a machine was last
// sent is the signal the store asks about (inventory.ReplaceGivenAfterStart). What it replaced is
// sent at once, said in the log and stated on the bus as the controller seat's `secret-replaced`,
// so a replacement is never silent. **Not in the hand-act log**: nobody acted by hand, and that log
// is read as the count of repairs a healer is wanted for.
// SecretReplaced is the controller seat's fact that a value given by hand was replaced
// (link.KeySecretReplaced). Never the value: neither the old one, which the mesh cannot read,
// nor the new one, sealed to the machine as it was made.
type SecretReplaced struct {
Node string `json:"node"`
Module string `json:"module"`
Name string `json:"name"`
Given time.Time `json:"given"`
// Sent are the machines sent so the module starts again on the new value; Unsent says why
// they could not be, in which case the next push carries it.
Sent []string `json:"sent"`
Unsent string `json:"unsent,omitempty"`
Why string `json:"why"`
}
// givenEvents is where the serving controller states it; nil in a command.
var givenEvents link.Bus
// replacing keeps one replacement per machine at a time: two reports arriving together find the
// same rows, and the store's claim makes one of them the replacer, but the sends need not race.
var replacing sync.Map
// startedWell says a report is a machine's clean account of a declaration: everything applied,
// nothing failed or refused. Whether it is the declaration last sent is the store's to answer.
func startedWell(report link.Report) bool {
return report.Declared != "" && report.Refused == "" && len(report.Failed) == 0 && report.Applied != nil
}
const givenWhy = "a value given by hand lives only until its module's first good start under the mesh (novox/hq ADR 0228)"
// replaceGiven replaces what the report makes due, sends the machines, and says so.
func replaceGiven(ctx context.Context, open *stores, report link.Report) {
if _, busy := replacing.LoadOrStore(report.Node, true); busy {
return
}
defer replacing.Delete(report.Node)
replaced, err := open.inventory.ReplaceGivenAfterStart(ctx, report.Node, report.Declared)
if err != nil {
log.Printf("a value given by hand on %s could not be replaced after its module started: %v", report.Node, err)
}
for _, r := range replaced {
said := SecretReplaced{Node: report.Node, Module: r.Module, Name: r.Name, Given: r.Given.UTC(),
Sent: r.Machines, Why: givenWhy}
log.Printf("replaced %q of %s on %s, given %s, with a value the mesh made: %s; sending %s",
r.Name, r.Module, report.Node, r.Given.UTC().Format(time.RFC3339), givenWhy, strings.Join(r.Machines, ", "))
if err := sendTo(ctx, open, r.Machines); err != nil {
said.Sent, said.Unsent = nil, err.Error()
log.Printf("the new %q of %s is sealed and not yet delivered to %s — the next push carries it: %v",
r.Name, r.Module, strings.Join(r.Machines, ", "), err)
}
stateReplaced(ctx, said)
}
}
func stateReplaced(ctx context.Context, said SecretReplaced) {
if givenEvents == nil {
return
}
body, err := json.Marshal(said)
if err != nil {
log.Printf("could not say that %s's %q was replaced: %v", said.Module, said.Name, err)
return
}
stating, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
if err := givenEvents.PublishSeatEvent(stating, link.MeshControllerSeat, link.KeySecretReplaced, body); err != nil {
log.Printf("could not say that %s's %q was replaced: %v", said.Module, said.Name, err)
}
}
+45
View File
@@ -0,0 +1,45 @@
package main
import (
"strings"
"testing"
"github.com/novox/mesh-controller/internal/link"
)
// A rotation asked through the seat carries why to the command, which records it in the hand-act
// log (novox/hq ADR 0228); why with a provision is refused as passed over, not dropped.
func TestARotationThroughTheSeatCarriesWhy(t *testing.T) {
argv, err := argvFor("rotate", map[string]any{"node": "anchor", "module": "letta",
"secret": "server-password", "why": "leaked into logs", "cause": "leaked"})
if err != nil || strings.Join(argv, " ") != "secret rotate anchor letta server-password --why leaked into logs --cause leaked" {
t.Fatalf("%v %v", argv, err)
}
argv, err = argvFor("rotate", map[string]any{"node": "anchor", "module": "letta", "secret": "server-password"})
if err != nil || strings.Join(argv, " ") != "secret rotate anchor letta server-password" {
t.Fatalf("without why: %v %v", argv, err)
}
if argv, err := argvFor("rotate", map[string]any{"provision": "postgres-database", "why": "leaked"}); err == nil {
t.Fatalf("why beside a provision was passed over: %v", argv)
}
}
// Only a clean account of a declaration is a good start; a refusal, a failure or a bare word that
// the machine is there is not (novox/hq ADR 0228).
func TestAGoodStartIsACleanAccountOfADeclaration(t *testing.T) {
for _, c := range []struct {
report link.Report
good bool
}{
{link.Report{Node: "anchor", Declared: "d", Applied: []string{"container:letta"}}, true},
{link.Report{Node: "anchor", Declared: "d", Applied: []string{}}, true},
{link.Report{Node: "anchor"}, false},
{link.Report{Node: "anchor", Applied: []string{"x"}}, false},
{link.Report{Node: "anchor", Declared: "d", Applied: []string{"x"}, Failed: map[string]string{"y": "no"}}, false},
{link.Report{Node: "anchor", Declared: "d", Refused: "older"}, false},
} {
if got := startedWell(c.report); got != c.good {
t.Errorf("%+v: a good start = %v, want %v", c.report, got, c.good)
}
}
}
@@ -0,0 +1,197 @@
package main
import (
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// novox/hq issue 274: a provider is granted exactly the consumers whose own resolution binds them to
// it — not every consumer a pair credential from it was ever made for.
// grantOf is the grant of one provision to one consuming module, and whether there is one at all.
func grantOf(grants []catalogue.Grant, provision, consumer, module string) (catalogue.Grant, bool) {
for _, g := range grants {
if g.Provision == provision && g.Consumer == consumer && (g.From == module || g.From == "") {
return g, true
}
}
return catalogue.Grant{}, false
}
// The morning after issue 273, through the stores: a consumer was bound to the store on another
// machine, a credential from there was made, and a person pinned it back to the store beside it. Both
// credentials are on record. The store it left is no longer granted it — so it retires it and keeps
// its data (ADR 0230) — and says so; the store it is bound to keeps its grant; and the credential from
// the store it left stays on record.
func TestAConsumerPinnedBackIsNoLongerGrantedByTheProviderItLeft(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
for _, m := range storeManifests() {
register(t, open, m)
}
if _, err := inv.SeedSeats(ctx, catalogue.DefaultSeats()); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "anchor", "store"); err != nil {
t.Fatal(err)
}
if err := inv.HoldSeat(ctx, "mesh-store", catalogue.ScopeMesh, "anchor", "store"); err != nil {
t.Fatal(err)
}
for _, a := range [][2]string{{"laptop", "store"}, {"laptop", "board"}} {
if _, err := assign(ctx, open, a[0], a[1]); err != nil {
t.Fatalf("assign %s %s: %v", a[0], a[1], err)
}
}
// Bound to the anchor's store, and sent so: the credential from the anchor is made and recorded.
if err := inv.PinProvision(ctx, "laptop", "postgres-database", "anchor", "store"); err != nil {
t.Fatal(err)
}
plan, _, err := planFor(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
if n := need(t, plan, "board", "postgres-database"); n.From != "anchor" {
t.Fatalf("pinned to the anchor and bound to %s", n.From)
}
if err := inv.RecordBindings(ctx, "laptop", boundToData(plan, nil)); err != nil {
t.Fatal(err)
}
grants, _, unbound, err := grantsFor(ctx, open, "anchor")
if err != nil {
t.Fatal(err)
}
if g, ok := grantOf(grants, "postgres-database", "laptop", "board"); !ok || g.From != "board" || len(unbound) != 0 {
t.Fatalf("a consumer bound to the anchor is not granted there: %+v, unbound %+v", grants, unbound)
}
// Pinned back beside its data, as the operator did.
if err := inv.PinProvision(ctx, "laptop", "postgres-database", "laptop", "store"); err != nil {
t.Fatal(err)
}
plan, _, err = planFor(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
if n := need(t, plan, "board", "postgres-database"); n.From != "laptop" {
t.Fatalf("pinned back to the laptop and bound to %s", n.From)
}
if err := inv.RecordBindings(ctx, "laptop", boundToData(plan, nil)); err != nil {
t.Fatal(err)
}
holders, err := inv.HoldersOf(ctx, "postgres-database", "laptop")
if err != nil {
t.Fatal(err)
}
if len(holders) != 2 {
t.Fatalf("today's state is two credentials on record, one from each store: %+v", holders)
}
// The store it left: no longer granted, and said.
grants, _, unbound, err = grantsFor(ctx, open, "anchor")
if err != nil {
t.Fatal(err)
}
if g, ok := grantOf(grants, "postgres-database", "laptop", "board"); ok && g.From != "" {
t.Fatalf("the anchor's store is still granted a consumer bound to the laptop's: %+v", g)
}
if len(unbound) != 1 || unbound[0].Module != "board" || unbound[0].Provider != "anchor" ||
strings.Join(unbound[0].BoundTo, ",") != "laptop" {
t.Fatalf("the consumer that moved is not the one said: %+v", unbound)
}
planned, settings, err := planFor(ctx, open, "anchor")
if err != nil {
t.Fatal(err)
}
declared, err := declarationFor(ctx, open, "anchor", planned, settings)
if err != nil {
t.Fatal(err)
}
if got := declared.Received["store"]["postgres-database"]; len(got) != 0 {
t.Fatalf("the anchor's store is still told about %+v", got)
}
said := printed(t, func() error { reportLeftOut("anchor", declared); return nil })
if !strings.Contains(said, "board on laptop is bound to laptop for postgres-database, not to anchor") ||
!strings.Contains(said, "cleanup delete") {
t.Fatalf("the push does not say whom the anchor no longer grants:\n%s", said)
}
// The store it is bound to: granted.
grants, _, unbound, err = grantsFor(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
if g, ok := grantOf(grants, "postgres-database", "laptop", "board"); !ok || g.From != "board" || len(unbound) != 0 {
t.Fatalf("the consumer is not granted by the store it is bound to: %+v, unbound %+v", grants, unbound)
}
// And the credential from the store it left is kept: the key to the login and data held there.
if holders, err = inv.HoldersOf(ctx, "postgres-database", "laptop"); err != nil || len(holders) != 2 {
t.Fatalf("a credential was forgotten while its provider still holds the login: %+v, %v", holders, err)
}
}
// A consumer whose resolution cannot be read is an error, never a consumer bound nowhere — withdrawing
// a grant on that reading would take its access away (issue 152).
func TestAConsumerWhoseResolutionCannotBeReadIsNotWithdrawn(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
for _, m := range storeManifests() {
register(t, open, m)
}
for _, a := range [][2]string{{"anchor", "store"}, {"laptop", "board"}} {
if _, err := assign(ctx, open, a[0], a[1]); err != nil {
t.Fatalf("assign %s %s: %v", a[0], a[1], err)
}
}
if _, _, err := planFor(ctx, open, "laptop"); err != nil {
t.Fatal(err)
}
// The laptop's key changes to one nothing can seal to: its resolution cannot be completed, and
// that is not its set failing to compose.
laptop, err := inv.NodeByName(ctx, "laptop")
if err != nil {
t.Fatal(err)
}
if err := inv.RecordSealingKey(ctx, laptop.ID, "not a key"); err != nil {
t.Fatal(err)
}
if _, _, err := planFor(ctx, open, "laptop"); err == nil || unresolvable(err) {
t.Fatalf("the seam this test relies on moved: %v", err)
}
grants, _, unbound, err := grantsFor(ctx, open, "anchor")
if err == nil {
t.Fatalf("an unreadable consumer was answered: grants %+v, unbound %+v", grants, unbound)
}
if !strings.Contains(err.Error(), "what laptop asked of postgres-database cannot be read") {
t.Fatalf("the error does not say whose resolution could not be read: %v", err)
}
}
// The rule itself, on a resolution: bound is the provider a need for exactly that credential is
// answered by, never one answered by a record, and a different local name is a different credential.
func TestBindsFromIsTheProviderOfThatCredential(t *testing.T) {
r := catalogue.Resolution{Needs: []catalogue.Needed{
{Name: "postgres-database", For: "board", From: "laptop"},
{Name: "postgres-database", For: "board", From: "laptop", Local: "reports"},
{Name: "postgres-database", For: "wiki", From: "anchor"},
{Name: "a-licence", For: "board", From: "the-licence", ByRecord: true},
}}
s := inventory.Secret{Name: "postgres-database", ConsumerModule: "board"}
if got := r.BindsFrom(s.Name, s.ConsumerModule, s.Local); strings.Join(got, ",") != "laptop" {
t.Fatalf("board's credential is bound to %v", got)
}
if got := r.BindsFrom("postgres-database", "board", "archive"); len(got) != 0 {
t.Fatalf("a local name nothing asks for is bound to %v", got)
}
if got := r.BindsFrom("a-licence", "board", ""); len(got) != 0 {
t.Fatalf("a need answered by a record is bound to %v", got)
}
}
+197
View File
@@ -0,0 +1,197 @@
package main
import (
"context"
"encoding/json"
"errors"
"flag"
"fmt"
"os"
"sort"
"strings"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/link"
)
// Acts done by hand, and why (novox/hq to-be 45 §7).
//
// **Which verbs ask why.** `plans close` and `plans stop`, `broker consumer-reset` and `hand-act
// record` refuse without it everywhere: nothing automated runs them, so a call without a reason is a
// person who has not given one. A named `push` asks for it through the mesh-controller seat, which is
// how a person or an agent acts by hand on the mesh; at a shell `--why` is recorded when given and not
// required, because the installer and the lab push by command line as a step of what they do, and a
// step of a procedure is not a repair. `conditions silence` joins them when the condition store does
// (Phase 1).
// handActFlags are the flags every repairing verb takes.
type handActFlags struct {
why, cause, condition *string
}
func addHandActFlags(set *flag.FlagSet) handActFlags {
return handActFlags{
why: set.String("why", "", "why this is done by hand — recorded in the hand-act log (novox/hq to-be 45 §7)"),
cause: set.String("cause", "", "the cause, in a word or a condition's kind; the verb's own name when not given"),
condition: set.String("condition", "", "the key of the condition this act addresses, if any"),
}
}
// given is whether a reason was given.
func (f handActFlags) given() bool { return strings.TrimSpace(*f.why) != "" }
// require refuses an act without a reason, before anything is done.
func (f handActFlags) require(verb string) error {
if f.given() {
return nil
}
return fmt.Errorf("%s is a repair done by hand, and says why: --why <text> (recorded in the hand-act "+
"log, novox/hq to-be 45 §7). Nothing was done", verb)
}
// handActConn is the serving controller's connection, for what it reads of the log itself; a
// command dials its own.
var handActConn *nats.Conn
// onTheBus runs f with a connection to the bus: the serving controller's, or one of its own.
func onTheBus(f func(*nats.Conn) error) error {
if handActConn != nil {
return f(handActConn)
}
address, err := broker.BusAddress()
if err != nil {
return err
}
js, err := broker.Dial(address)
if err != nil {
return fmt.Errorf("cannot reach the bus: %w", err)
}
defer js.Close()
return f(js.Conn())
}
// record writes the entry for an act about to be done. **Before the act, and never instead of it**:
// a log that cannot be written is said loudly, and the repair it was about still happens — a mesh
// whose bus is down is exactly the mesh somebody is repairing by hand.
func (f handActFlags) record(ctx context.Context, verb string, args []string) {
if !f.given() {
return
}
act := link.HandAct{Verb: verb, Args: args, Why: strings.TrimSpace(*f.why),
Cause: strings.TrimSpace(*f.cause), Condition: strings.TrimSpace(*f.condition)}
err := onTheBus(func(conn *nats.Conn) error {
written, err := link.RecordHandAct(ctx, conn, act)
act = written
return err
})
if err != nil {
fmt.Fprintf(os.Stderr, "this act by hand could NOT be recorded in the hand-act log, and is done anyway: %v\n", err)
return
}
fmt.Printf("recorded as %s in the hand-act log: %s, because %q (cause: %s)\n", act.ID, act.By, act.Why, act.Cause)
}
// handActCommand is `hand-act record` and `hand-acts`.
func handActCommand(ctx context.Context, args []string) error {
if len(args) > 0 && args[0] == "record" {
set := flag.NewFlagSet("hand-act record", flag.ContinueOnError)
f := addHandActFlags(set)
positionals, err := parseAround(set, args[1:])
if err != nil {
return err
}
what := strings.TrimSpace(strings.Join(positionals, " "))
if what == "" {
return errors.New("hand-act record <what was done> --why <text> [--cause <word>] [--condition <key>]")
}
if err := f.require("hand-act record"); err != nil {
return err
}
if strings.TrimSpace(*f.cause) == "" {
return errors.New("hand-act record says the cause too: --cause <word>, the word a second " +
"act for the same reason will use — it is how a repair done twice is found")
}
act := link.HandAct{Verb: "hand-act record", Args: []string{what}, Why: strings.TrimSpace(*f.why),
Cause: strings.TrimSpace(*f.cause), Condition: strings.TrimSpace(*f.condition)}
return onTheBus(func(conn *nats.Conn) error {
written, err := link.RecordHandAct(ctx, conn, act)
if err != nil {
return fmt.Errorf("the act could not be recorded: %w", err)
}
fmt.Printf("recorded as %s: %s did %q, because %q (cause: %s)\n", written.ID, written.By, what,
written.Why, written.Cause)
return nil
})
}
if len(args) > 0 && args[0] != "list" && !strings.HasPrefix(args[0], "-") {
return errors.New("hand-act record <what> --why <text> --cause <word> | hand-acts [--days N] [--json]")
}
if len(args) > 0 && args[0] == "list" {
args = args[1:]
}
set := flag.NewFlagSet("hand-acts", flag.ContinueOnError)
days := set.Int("days", 14, "how many days back")
asJSON := set.Bool("json", false, "as data")
if _, err := parseAround(set, args); err != nil {
return err
}
return onTheBus(func(conn *nats.Conn) error {
now := time.Now()
acts, err := link.HandActs(ctx, conn, now.Add(-time.Duration(*days)*24*time.Hour))
if err != nil {
return err
}
repeated := link.RepeatedCauses(acts, now)
if *asJSON {
body, err := json.MarshalIndent(map[string]any{"acts": acts, "repeated": repeated}, "", " ")
if err != nil {
return err
}
fmt.Println(string(body))
return nil
}
if len(acts) == 0 {
fmt.Printf("nothing was done by hand in the last %d day(s)\n", *days)
return nil
}
for i := len(acts) - 1; i >= 0; i-- {
a := acts[i]
fmt.Printf("%s %s %s %s\n by %s — %s (cause: %s", a.At.Local().Format("2006-01-02 15:04"), a.ID,
a.Verb, strings.Join(a.Args, " "), a.By, a.Why, a.Cause)
if a.Condition != "" {
fmt.Printf(", condition %s", a.Condition)
}
fmt.Println(")")
}
if len(repeated) > 0 {
causes := make([]string, 0, len(repeated))
for c, n := range repeated {
causes = append(causes, fmt.Sprintf("%s ×%d", c, n))
}
sort.Strings(causes)
fmt.Printf("\ndone by hand more than once in a fortnight — a healer is wanted (to-be 45 S15): %s\n",
strings.Join(causes, ", "))
}
return nil
})
}
// handActsThisWeek is how many acts were done by hand in the last seven days, for `status`; -1 when
// the log could not be read, which status says rather than reading as none.
func handActsThisWeek(ctx context.Context) (int, string) {
n := -1
err := onTheBus(func(conn *nats.Conn) error {
reading, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
acts, err := link.HandActs(reading, conn, time.Now().Add(-7*24*time.Hour))
n = len(acts)
return err
})
if err != nil {
return -1, err.Error()
}
return n, ""
}
+94
View File
@@ -0,0 +1,94 @@
package main
import (
"context"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
// **Every verb that repairs by hand takes a required why** (novox/hq to-be 45 §7): refused before
// anything is done, through the seat, through `command`, and at a shell where nothing automated runs
// the verb.
func TestARepairByHandWithoutAReasonIsRefused(t *testing.T) {
for _, c := range []struct {
verb string
args map[string]any
}{
{"push", map[string]any{"node": "anchor"}},
{"push", map[string]any{}},
{"plans", map[string]any{"close": "plan-1"}},
{"plans", map[string]any{"stop": "plan-1"}},
{"hand-act", map[string]any{"what": "restarted the proxy", "cause": "proxy-stuck"}},
{"command", map[string]any{"command": "push anchor"}},
{"command", map[string]any{"command": "plans close plan-1"}},
{"command", map[string]any{"command": "broker consumer-reset EVENTS controller"}},
{"command", map[string]any{"command": "hand-act record restarted --cause x"}},
} {
argv, err := argvFor(c.verb, c.args)
if c.verb == "plans" && err == nil {
// The seat composes the command line; the command refuses it, before opening anything.
err = plansCommand(context.Background(), argv[1:])
}
if err == nil || !strings.Contains(err.Error(), "why") {
t.Errorf("%s %v was not refused for want of why: %v %v", c.verb, c.args, argv, err)
}
}
for _, args := range [][]string{{"EVENTS", "controller"}} {
if err := consumerReset(context.Background(), args); err == nil || !strings.Contains(err.Error(), "--why") {
t.Errorf("consumer-reset without why: %v", err)
}
}
if err := handActCommand(context.Background(), []string{"record", "restarted the proxy", "--cause", "x"}); err == nil ||
!strings.Contains(err.Error(), "--why") {
t.Errorf("hand-act record without why: %v", err)
}
if err := handActCommand(context.Background(), []string{"record", "restarted the proxy", "--why", "it hung"}); err == nil ||
!strings.Contains(err.Error(), "--cause") {
t.Errorf("hand-act record without a cause: %v", err)
}
}
// With a reason, the seat passes it to the command, and a verb that only reads is not held to one.
func TestARepairByHandCarriesItsReason(t *testing.T) {
for _, c := range []struct {
verb string
args map[string]any
want string
}{
{"push", map[string]any{"node": "anchor", "why": "stuck", "cause": "sent-not-reported"},
"push anchor --wait 0 --why stuck --cause sent-not-reported"},
{"plans", map[string]any{"close": "plan-1", "why": "the report will not come"},
"plans close plan-1 --why the report will not come"},
{"plans", map[string]any{"retry": "plan-1"}, "plans retry plan-1"},
{"hand-act", map[string]any{"what": "restarted", "why": "hung", "cause": "proxy", "condition": "machine.a.silent"},
"hand-act record restarted --why hung --cause proxy --condition machine.a.silent"},
{"command", map[string]any{"command": "push anchor --why stuck"}, "push anchor --why stuck"},
{"command", map[string]any{"command": "plans plan-1"}, "plans plan-1"},
} {
argv, err := argvFor(c.verb, c.args)
if err != nil || strings.Join(argv, " ") != c.want {
t.Errorf("%s %v: %v %v, want %q", c.verb, c.args, argv, err, c.want)
}
}
}
// The summary of durations says, per kind and subject, what a bound would be set from.
func TestDurationsAreSummarisedPerSubject(t *testing.T) {
var ds []inventory.Duration
for i := 1; i <= 10; i++ {
ds = append(ds, inventory.Duration{Kind: inventory.DurationApply, Subject: "anchor",
Took: time.Duration(i) * time.Second})
}
ds = append(ds, inventory.Duration{Kind: inventory.DurationHeartbeatGap, Subject: "anchor", Took: time.Minute})
got := summarise(ds)
if len(got) != 2 || got[0].Kind != inventory.DurationApply || got[0].Count != 10 ||
got[0].Max != "10s" || got[0].Median != "5s" || got[0].P90 != "9s" {
t.Fatalf("%+v", got)
}
if !strings.Contains(got[1].Suggests, "3m0s") {
t.Fatalf("a minute between words suggests %q", got[1].Suggests)
}
}
+865
View File
@@ -0,0 +1,865 @@
package main
import (
"context"
"encoding/json"
"errors"
"flag"
"fmt"
"slices"
"sort"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// The healers (novox/hq to-be 45 §7, ADR 0227 rule 7, Phase 3).
//
// **A known failure heals itself, under a brake, and every repair is said.** Research 031 counted the
// repairs people made by hand in six days: a push to unstick a plan waiting on a report (four times),
// a controller restarted to make an object again (twice), a plan closed (twice), a consumer re-made from
// now (once). Each was the ordinary path, taken again by a person who noticed. A healer is that act,
// registered against the one condition kind it answers, so the mesh takes it itself:
//
// - **its repair is the ordinary path again** — the send a push makes, the plan's own close, the
// assertion every send makes, the consumer reset the verb makes — never a withdrawal, a deletion of
// data or a recreation of it;
// - **its budget** is how many acts it may take against one thing in a window, and **its settle** how
// long after an act it leaves the observation to say whether it worked;
// - **success is never the healer's to say.** The condition clears when the watchdog or the probe that
// raised it no longer sees it. A healer marking its own work done would be the second opinion of a
// fact that rule 1 forbids;
// - **a spent budget stops it**: the condition is the operator's, urgent, with every attempt in
// `tried`, and no healer touches it again until observation clears it;
// - **every act is said**: begun in the store before it is made (so a controller dying mid-act has
// still spent it), kept in the condition's `tried` as `healer Hn`, and said on the bus as the
// controller seat's `healer-acted`. A heal is not a hand act and is never in the hand-act log —
// which is how S15 can tell a repair the mesh made from one a person had to.
//
// **And the mesh-wide brake**: more than healBrakeLimit acts in an hour, all healers together, and every
// healer stops — an urgent condition says so — until an hour has passed with none. A healer looping is
// then at most a dozen acts, said, and never the incident itself.
//
// Only the controller holding the lease heals, under its epoch: a controller serving without the lease
// (S12) heals nothing, because a repair is the one act that can always wait for the lease.
// The mesh-wide brake (to-be 45 §7): how many acts all healers together may take in an hour.
const (
healBrakeLimit = 12
healBrakeWindow = time.Hour
)
// healEvery is how often the healers look at what is open.
var healEvery = 30 * time.Second
// What raises the healers' own conditions, and their kinds.
const (
sourceHealers = "healers"
kindHealersBraked = "healers-braked"
kindConsumerBehind = "consumer-behind"
kindHealersBlind = "probe-failed"
)
// healerRow is one row of the registry.
type healerRow struct {
ID string
// Kinds are the condition kinds it answers: from the signals table, the probe registry, or an event.
Kinds []string
// Condition, Repair, Then, From are the row in words, as to-be 45 §7 and `healers` say it.
Condition string
Repair string
Then string
From string
// Budget acts against one budget key within Window; Settle after an act before the next, or the
// escalation, so the observation has had its turn to say whether it worked.
Budget int
Window time.Duration
Settle time.Duration
// ActsIn is where the repair runs: the controller, or a module that repairs its own (H5).
ActsIn string
// Event is what each act is said as.
Event string
// applies says whether this condition is one the healer may act on, and what its budget is
// counted against; why when it is not. Reads, never acts. Nil where the repair is a module's.
applies func(ctx context.Context, h *healing, c conditions.Condition) (budget string, ok bool, why string, err error)
// repair is the act: what it did in words, what came of it, or an error when it could not be done.
repair func(ctx context.Context, h *healing, c conditions.Condition) (act, said string, err error)
}
// actsInController is a healer whose repair the controller makes.
const actsInController = "controller"
// healerRegistry is to-be 45 §7's table, compiled in. **The registry is the design's live form**: a
// test generated from it holds every row to a condition kind the mesh raises, a budget, a brake and its
// event (healers_test.go).
var healerRegistry = []healerRow{
{ID: "H1", Kinds: []string{"sent-not-reported"},
Condition: "a machine was sent a declaration and has not reported it (S2)",
Repair: "ask the machine's node-engine to say again what it last applied (the `report` verb); if what " +
"comes back is not the declaration it was sent, or nothing comes, send it its current declaration " +
"again — what a push of that one machine does, never moving a build a policy or a plan holds back",
Then: "resolver operator, urgent", From: "a push by hand to unstick a plan waiting on a report (230, 257, 264, 267)",
Budget: 2, Window: 6 * time.Hour, Settle: 3 * time.Minute, ActsIn: actsInController, Event: link.KeyHealerActed,
applies: appliesToAMachine, repair: repairReport},
{ID: "H2", Kinds: []string{"stalled"},
Condition: "a plan stalled on a wait that is superseded — a newer plan of its repository and branch " +
"exists — or already finished — every module of every tier built or failed, and every one that " +
"rolls out sent (S3)",
Repair: "close the plan with its note, as `plans close` does: superseded, naming the newer plan, or " +
"done; what it asked still builds and registers",
Then: "resolver operator, urgent", From: "a stuck plan closed by hand (214, 254)",
Budget: 1, Window: 24 * time.Hour, Settle: 2 * time.Minute, ActsIn: actsInController, Event: link.KeyHealerActed,
applies: appliesToAStalePlan, repair: repairPlan},
{ID: "H3", Kinds: []string{"holder-silent", "consumer-lost"},
Condition: "a seat's holder that does not answer (D3), or a durable consumer the mesh expects and the " +
"bus does not hold (D6, S9)",
Repair: "assert the bus's streams, consumers and seat workers again — the assertion every send makes " +
"(issue 208)",
Then: "resolver operator, urgent", From: "a controller restarted to make a missing object again (208, 248)",
Budget: 1, Window: time.Hour, Settle: 6 * time.Minute, ActsIn: actsInController, Event: link.KeyHealerActed,
applies: appliesToAnObject, repair: repairObjects},
{ID: "H4", Kinds: []string{kindConsumerBehind},
Condition: "a durable consumer far behind its stream's head (D6) that the stream table marks resettable",
Repair: "re-make the consumer to deliver from now — `broker consumer-reset` (issue 248); what it drops " +
"is caught up where the table says",
Then: "resolver operator, urgent", From: "a consumer re-made from now by hand (248)",
Budget: 1, Window: 24 * time.Hour, Settle: 6 * time.Minute, ActsIn: actsInController, Event: link.KeyHealerActed,
applies: appliesToAResettableConsumer, repair: repairConsumer},
{ID: "H5", Kinds: []string{kindProviderFailing},
Condition: "the identity provider's administrator refusing the mesh's secret (provider-failing, " +
"credentials-rejected)",
Repair: "the provider repairs it itself through the server's own bootstrap command, checks again and " +
"says what it did (ADR 0224 §5); the controller keeps its word as the condition",
Then: "announced failing by the provider, braked from ten minutes doubling to six hours (ADR 0224 §5)",
From: "the identity provider's admin reset through its bootstrap command (179)",
// Its budget and brake are the module's own: ten minutes, doubling, to six hours.
Budget: 1, Window: 10 * time.Minute, Settle: 10 * time.Minute,
ActsIn: "the provider module (keycloak), ADR 0224 §5", Event: "provisioner.failing, provisioner.recovered"},
}
// healerFor is the healer registered for a kind, and false when none acts in the controller.
func healerFor(kind string) (healerRow, bool) {
for _, r := range healerRegistry {
if r.repair != nil && slices.Contains(r.Kinds, kind) {
return r, true
}
}
return healerRow{}, false
}
// healerNamedFor is any healer registered for a kind, wherever it acts: what S15 says beside a cause.
func healerNamedFor(kind string) string {
for _, r := range healerRegistry {
if slices.Contains(r.Kinds, kind) {
return r.ID
}
}
return ""
}
// healing is the healers' runner and what their repairs reach.
type healing struct {
open *stores
keeper *conditions.Keeper
teller conditions.Teller
// js is the bus, for the repairs that assert or reset its objects; nil where there is none.
js *broker.JetStream
// epoch is the lease's gate; acting says this controller is the one acting, not one standing by.
epoch func(ctx context.Context) (uint64, error)
acting func() bool
now func() time.Time
say func(format string, args ...any)
// The acts, as seams a test replaces: asking a machine to report, sending it again, asserting the
// bus's objects, resetting a consumer.
askReport func(ctx context.Context, node string) error
sendAgain func(ctx context.Context, node string) error
assertObjects func(ctx context.Context) error
resetConsumer func(stream, name string) (string, error)
// reportWait is how long H1 waits for the machine's report after asking.
reportWait time.Duration
mu sync.Mutex
// declined is why each condition was last passed over, so it is said once and `healers` can show it.
declined map[string]string
paused string
}
// newHealing is the serving controller's runner, its acts the real ones.
func newHealing(open *stores, keeper *conditions.Keeper, teller conditions.Teller, js *broker.JetStream) *healing {
h := &healing{open: open, keeper: keeper, teller: teller, js: js, acting: link.Holding,
epoch: func(ctx context.Context) (uint64, error) { return theLease.epoch(ctx) },
now: time.Now, say: func(format string, args ...any) { fmt.Printf(format+"\n", args...) },
reportWait: 45 * time.Second, declined: map[string]string{}}
h.askReport = func(ctx context.Context, node string) error {
if h.js == nil {
return errors.New("this controller is not on the bus")
}
return askToReport(ctx, h.js.Conn(), node)
}
h.sendAgain = func(ctx context.Context, node string) error {
// **Never what a policy or a plan holds back** (ADR 0221): a send by the mesh itself is a push
// that did not name the machine — a person's word sends a held build, a healer's does not.
held, err := heldMachines(ctx, open, []string{node})
if err != nil {
return err
}
if why := held[node]; len(why) > 0 {
return fmt.Errorf("not sent again: it would move what a policy or a plan holds back (ADR 0221) — %s; "+
"`push %s` sends it, on a person's word", strings.Join(why, "; "), node)
}
return sendTo(ctx, open, []string{node})
}
h.assertObjects = func(ctx context.Context) error {
if h.js == nil {
return errors.New("this controller is not on the bus")
}
return assertOnSend(ctx, open.inventory, h.js, " ")
}
h.resetConsumer = func(stream, name string) (string, error) {
if h.js == nil {
return "", errors.New("this controller is not on the bus")
}
before, after, err := h.js.ResetConsumer(stream, name)
if err != nil {
return "", err
}
return fmt.Sprintf("it was %d behind with %d unacknowledged; it delivers from now, %d pending", before.Pending,
before.AckPending, after.Pending), nil
}
return h
}
// askToReport asks one machine's node-engine to say again what it last applied (to-be 45 §6). On core
// NATS, fired and flushed: the answer is the machine's ordinary report, read from the store.
func askToReport(ctx context.Context, conn *nats.Conn, node string) error {
body, err := json.Marshal(map[string]any{"asked": time.Now().UTC(), "by": "the controller's healer H1"})
if err != nil {
return err
}
if err := conn.Publish(broker.AskReportSubject(node), body); err != nil {
return err
}
flushing, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
return conn.FlushWithContext(flushing)
}
// keep runs the healers until ctx ends.
func (h *healing) keep(ctx context.Context) {
tick := time.NewTicker(healEvery)
defer tick.Stop()
for {
h.tick(ctx)
select {
case <-ctx.Done():
return
case <-tick.C:
}
}
}
// tick is one look at what is open, and every act it calls for.
func (h *healing) tick(ctx context.Context) {
if h.acting != nil && !h.acting() {
return
}
epoch, err := h.epoch(ctx)
switch {
case err != nil:
h.pause(fmt.Sprintf("this controller may not act: %v", err))
return
case epoch == 0:
h.pause("this controller serves without the lease (S12): a repair waits for it")
return
}
inv := h.open.inventory
now := h.now()
heals, err := inv.HealsSince(ctx, now.Add(-24*time.Hour))
if err != nil {
// Blind: nothing is done, and that is said — never read as "no heals, budgets whole".
h.reconcile(ctx, []conditions.Observation{{Scope: conditions.ScopeProbe, ID: "healers", Token: "failed",
Kind: kindHealersBlind, Severity: conditions.Warning,
Summary: "the healers cannot read what they did, so they count no budget and act on nothing",
Said: firstLine(err.Error())}})
return
}
open, err := h.keeper.Open(ctx)
if err != nil {
h.say("the healers cannot read the open conditions, and act on nothing: %v", err)
return
}
acts := actsWithin(heals, now.Add(-healBrakeWindow))
if braked, said := brakeHolds(acts, open, now); braked {
h.reconcile(ctx, []conditions.Observation{brakeObservation(acts, said)})
h.pause("the mesh-wide brake holds: " + said)
return
}
h.reconcile(ctx, nil)
h.resume()
for _, c := range open {
row, ok := healerFor(c.Kind)
if !ok || c.Escalated() {
continue
}
budget, applies, why, err := row.applies(ctx, h, c)
if err != nil {
h.decline(c.Key, row.ID, "could not tell whether it applies: "+err.Error())
continue
}
if !applies {
h.decline(c.Key, row.ID, why)
continue
}
h.forget(c.Key)
spent := spentOn(heals, row, budget, now)
if n := len(spent); n > 0 && now.Sub(spent[n-1].At) < row.Settle {
continue // its last act is still the observation's to judge
}
if len(spent) >= row.Budget {
h.escalate(ctx, row, c, budget, spent)
continue
}
if len(acts) >= healBrakeLimit {
return // the brake takes hold on the next look, said there
}
if h.act(ctx, row, c, budget, len(spent)) {
acts = append(acts, inventory.Heal{At: now})
}
}
}
// act is one healer's act on one condition: begun in the store, made, finished, kept in the
// condition's tried and said. False when it could not even be begun.
func (h *healing) act(ctx context.Context, row healerRow, c conditions.Condition, budget string, spent int) bool {
inv := h.open.inventory
begun, err := inv.BeginHeal(ctx, inventory.Heal{Healer: row.ID, ConditionKey: c.Key, Kind: c.Kind,
BudgetKey: budget, Act: row.Repair, At: h.now()})
if err != nil {
h.say("healer %s did not act on %s: %v", row.ID, c.Key, err)
return false
}
act, said, err := row.repair(ctx, h, c)
outcome := inventory.HealActed
if err != nil {
outcome, said = inventory.HealFailed, err.Error()
}
if act == "" {
act = row.Repair
}
finishing, cancel := context.WithTimeout(context.WithoutCancel(ctx), 10*time.Second)
defer cancel()
if ferr := inv.FinishHeal(finishing, begun.ID, outcome, act+" — "+said); ferr != nil {
h.say("healer %s acted on %s and could not record what came of it: %v", row.ID, c.Key, ferr)
}
budgetWords := fmt.Sprintf("act %d of %d within %s", spent+1, row.Budget, row.Window)
attempt := conditions.Attempt{What: act, Outcome: outcome + ": " + said + " (" + budgetWords + ")",
By: "healer " + row.ID}
if _, _, terr := h.keeper.Tried(finishing, c.Key, attempt, conditions.ResolverHealer(row.ID)); terr != nil {
h.say("healer %s acted on %s and could not keep it in the condition: %v", row.ID, c.Key, terr)
}
h.tell(finishing, healerActed{Healer: row.ID, Condition: c.Key, Kind: c.Kind, Act: act, Outcome: outcome,
Said: said, Budget: budgetWords, Epoch: begun.Epoch})
h.say("healer %s %s %s: %s — %s (%s)", row.ID, outcome, c.Key, act, said, budgetWords)
return true
}
// escalate is a spent budget: the condition is the operator's now, urgent, with what was tried. Said
// once: an escalated condition is passed over by every healer until observation clears it.
func (h *healing) escalate(ctx context.Context, row healerRow, c conditions.Condition, budget string, spent []inventory.Heal) {
var tried []string
for _, s := range spent {
tried = append(tried, fmt.Sprintf("%s at %s (%s)", s.Outcome, s.At.UTC().Format("15:04 MST"), firstLine(s.Said)))
}
said := fmt.Sprintf("its budget of %d act(s) within %s is spent and %s is still open: %s", row.Budget, row.Window,
c.Key, strings.Join(tried, "; "))
if _, err := h.open.inventory.BeginHeal(ctx, inventory.Heal{Healer: row.ID, ConditionKey: c.Key, Kind: c.Kind,
BudgetKey: budget, Act: "handed the condition to the operator", Outcome: inventory.HealEscalated, Said: said,
At: h.now()}); err != nil {
h.say("healer %s could not record handing %s to the operator, and does not: %v", row.ID, c.Key, err)
return
}
attempt := conditions.Attempt{What: "handed to the operator: nothing in the mesh will repair it now",
Outcome: inventory.HealEscalated + ": " + said, By: "healer " + row.ID}
if _, _, err := h.keeper.Escalate(ctx, c.Key, attempt); err != nil {
h.say("healer %s could not hand %s to the operator: %v", row.ID, c.Key, err)
}
h.tell(ctx, healerActed{Healer: row.ID, Condition: c.Key, Kind: c.Kind, Act: "handed to the operator",
Outcome: inventory.HealEscalated, Said: said, Budget: fmt.Sprintf("%d of %d within %s", len(spent), row.Budget, row.Window)})
h.say("healer %s handed %s to the operator: %s", row.ID, c.Key, said)
}
// healerActed is the body of `healer-acted` (to-be 45 §7): the healer, the condition, the act and its
// outcome, and where the budget stands. **A contract**, like a condition's events: the operator-channel's
// holder may say it.
type healerActed struct {
Event string `json:"event"`
At time.Time `json:"at"`
Healer string `json:"healer"`
By string `json:"by"`
Condition string `json:"condition"`
Kind string `json:"kind"`
Act string `json:"act"`
// Outcome is acted, failed or escalated. Acted says the act was made, not that it worked: the
// condition clearing says that.
Outcome string `json:"outcome"`
Said string `json:"said"`
Budget string `json:"budget"`
Epoch uint64 `json:"epoch,omitempty"`
Show string `json:"show"`
}
// tell says a heal on the bus. Not said is said in the log: the act and the condition's tried still
// hold it.
func (h *healing) tell(ctx context.Context, e healerActed) {
e.Event, e.At, e.By = link.KeyHealerActed, h.now().UTC(), "healer "+e.Healer
e.Show = "mesh-controller.conditions key=" + e.Condition
if h.teller == nil {
return
}
body, err := json.Marshal(e)
if err != nil {
return
}
saying, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
if err := h.teller.PublishSeatEvent(saying, conditions.Seat, link.KeyHealerActed, body); err != nil {
h.say("healer %s %s %s, and that could NOT be said on the bus: %v", e.Healer, e.Outcome, e.Condition, err)
}
}
// reconcile keeps the healers' own conditions: the brake, and their being blind.
func (h *healing) reconcile(ctx context.Context, observed []conditions.Observation) {
if err := h.keeper.Reconcile(ctx, sourceHealers, observed); err != nil {
h.say("the healers' own conditions could not be kept: %v", err)
}
}
func (h *healing) pause(why string) {
h.mu.Lock()
defer h.mu.Unlock()
if h.paused != why {
h.say("the healers act on nothing: %s", why)
}
h.paused = why
}
func (h *healing) resume() {
h.mu.Lock()
defer h.mu.Unlock()
if h.paused != "" {
h.say("the healers act again")
}
h.paused = ""
}
// decline remembers why a healer passed a condition over, said once.
func (h *healing) decline(key, healer, why string) {
h.mu.Lock()
defer h.mu.Unlock()
if h.declined[key] != why {
h.say("healer %s leaves %s alone: %s", healer, key, why)
}
h.declined[key] = why
}
func (h *healing) forget(key string) {
h.mu.Lock()
defer h.mu.Unlock()
delete(h.declined, key)
}
// actsWithin are the heals since a moment that were acts — begun, made or failed; an escalation is a
// saying, not an act, and the brake does not count it.
func actsWithin(heals []inventory.Heal, since time.Time) []inventory.Heal {
var out []inventory.Heal
for _, h := range heals {
if !h.At.Before(since) && h.Outcome != inventory.HealEscalated {
out = append(out, h)
}
}
return out
}
// spentOn is one healer's acts against one budget key within its window, oldest first.
func spentOn(heals []inventory.Heal, row healerRow, budget string, now time.Time) []inventory.Heal {
var out []inventory.Heal
for _, h := range actsWithin(heals, now.Add(-row.Window)) {
if h.Healer == row.ID && h.BudgetKey == budget {
out = append(out, h)
}
}
return out
}
// brakeHolds says whether the mesh-wide brake holds: the limit reached within the hour, or the brake
// already said and an act within the hour still — it lets go an hour after the last act, not the first.
func brakeHolds(acts []inventory.Heal, open []conditions.Condition, now time.Time) (bool, string) {
said := fmt.Sprintf("%d act(s) by the healers in the last %s, the limit %d", len(acts), healBrakeWindow, healBrakeLimit)
if len(acts) >= healBrakeLimit {
return true, said
}
held := slices.ContainsFunc(open, func(c conditions.Condition) bool { return c.Kind == kindHealersBraked })
if held && len(acts) > 0 {
last := acts[len(acts)-1].At
return true, fmt.Sprintf("%s; held until an hour after the last, at %s", said,
last.Add(healBrakeWindow).UTC().Format("15:04 MST"))
}
return false, said
}
// brakeObservation is the brake, as the urgent condition that says it.
func brakeObservation(acts []inventory.Heal, said string) conditions.Observation {
counts := map[string]int{}
for _, a := range acts {
if a.Healer != "" {
counts[a.Healer+" on "+a.ConditionKey]++
}
}
var most []string
for what, n := range counts {
most = append(most, fmt.Sprintf("%s ×%d", what, n))
}
sort.Strings(most)
return conditions.Observation{Scope: conditions.ScopeMesh, ID: "healers", Token: "braked", Kind: kindHealersBraked,
Severity: conditions.Urgent,
Summary: "every healer has stopped: the healers acted more often in an hour than the mesh allows, which is a " +
"healer looping, not repairing — " + said,
Said: strings.Join(most, ", ")}
}
// --- H1 ----------------------------------------------------------------------------------------------
// appliesToAMachine is H1's: a machine named, its budget counted against the condition.
func appliesToAMachine(_ context.Context, _ *healing, c conditions.Condition) (string, bool, string, error) {
if c.Subject.Machine == "" {
return "", false, "it names no machine", nil
}
return c.Key, true, "", nil
}
// repairReport is H1: ask the machine to report; if what comes back is not what it was sent, or nothing
// comes, send it again.
func repairReport(ctx context.Context, h *healing, c conditions.Condition) (string, string, error) {
node := c.Subject.Machine
inv := h.open.inventory
// The store's clock, as the report's time is: not the runner's, which a test moves by hand.
asked := time.Now()
askErr := h.askReport(ctx, node)
current, heard := false, false
if askErr == nil {
deadline := time.Now().Add(h.reportWait)
for {
r, found, err := lastReportOf(ctx, inv, node)
if err != nil {
return "", "", err
}
if found && r.At != nil && r.At.After(asked) {
heard, current = true, r.Current
if current {
break
}
}
if time.Now().After(deadline) {
break
}
select {
case <-ctx.Done():
return "", "", ctx.Err()
case <-time.After(min(2*time.Second, h.reportWait/4+time.Millisecond)):
}
}
}
if current {
return "asked " + node + "'s node-engine to say again what it last applied",
"it reported the declaration it was sent: its report had not reached the mesh", nil
}
why := "it said nothing within " + h.reportWait.String() + " — its node-engine may be older than the report verb"
switch {
case askErr != nil:
why = "it could not be asked: " + askErr.Error()
case heard:
why = "it reported a declaration other than the one it was sent"
}
if err := h.sendAgain(ctx, node); err != nil {
return "asked " + node + " to report, then sent it its current declaration again", why + "; the send failed",
err
}
return "asked " + node + " to report, then sent it its current declaration again", why + "; sent again", nil
}
// lastReportOf is one machine's last report beside its last send.
func lastReportOf(ctx context.Context, inv *inventory.Inventory, node string) (inventory.Reported, bool, error) {
reports, err := inv.LastReports(ctx)
if err != nil {
return inventory.Reported{}, false, err
}
for _, r := range reports {
if r.Node == node {
return r, true, nil
}
}
return inventory.Reported{}, false, nil
}
// --- H2 ----------------------------------------------------------------------------------------------
// planStale is why a plan's wait is superseded or finished, and the state closing it leaves it in; empty
// when it is neither, which is not H2's to repair.
func planStale(ctx context.Context, inv *inventory.Inventory, p inventory.Plan) (state, why string, err error) {
recent, err := inv.RecentPlans(ctx, 50)
if err != nil {
return "", "", err
}
for _, newer := range recent {
if newer.ID == p.ID || !newer.Created.After(p.Created) || !repositoryMatches(newer.Repository, p.Repository) ||
newer.Branch != p.Branch || newer.State == inventory.PlanSuperseded {
continue
}
return inventory.PlanSuperseded, fmt.Sprintf("superseded by %s (%s %s), a newer merge of the same repository "+
"and branch", newer.ID, newer.Repository, short(newer.Commit)), nil
}
for _, tier := range p.Tiers {
for _, m := range tier {
s := p.Modules[m]
if s == nil || (s.State != "built" && s.State != "failed") {
return "", "", nil
}
if s.State == "built" && s.SentAt == nil {
u, err := inv.UpgradeOf(ctx, m)
if err != nil {
return "", "", err
}
if u.RollOut {
return "", "", nil
}
}
}
}
return inventory.PlanDone, "finished: every module of every tier is built or failed, and every one that rolls " +
"out was sent — nothing is left to wait on", nil
}
// appliesToAStalePlan is H2's: the plan is open, and its wait is superseded or finished.
func appliesToAStalePlan(ctx context.Context, h *healing, c conditions.Condition) (string, bool, string, error) {
p, err := h.open.inventory.PlanByID(ctx, c.Subject.ID)
if err != nil {
return "", false, "", err
}
if !p.Open() {
return "", false, "the plan is " + p.State + " already: its condition clears on the next look", nil
}
state, _, err := planStale(ctx, h.open.inventory, p)
if err != nil {
return "", false, "", err
}
if state == "" {
return "", false, "its wait is neither superseded nor finished: it waits on something still to come, " +
"which its own signal says", nil
}
return c.Key, true, "", nil
}
// repairPlan is H2: the plan closed with its note, under the plans' hold, by compare-and-set.
func repairPlan(ctx context.Context, h *healing, c conditions.Condition) (string, string, error) {
inv := h.open.inventory
release, err := inv.HoldPlans(ctx, true)
if err != nil {
return "", "", err
}
defer release()
p, err := inv.PlanByID(ctx, c.Subject.ID)
if err != nil {
return "", "", err
}
if !p.Open() {
return "closed nothing", "the plan is " + p.State + " already", nil
}
state, why, err := planStale(ctx, inv, p)
if err != nil {
return "", "", err
}
if state == "" {
return "closed nothing", "its wait is no longer superseded or finished", nil
}
p.State = state
p.Note = fmt.Sprintf("closed by healer H2 at tier %d: %s", p.Tier, why)
if state != inventory.PlanDone {
sayUnsent(&p, func(m string) bool {
u, err := inv.UpgradeOf(ctx, m)
return err == nil && u.RollOut
})
}
if err := inv.SavePlan(ctx, &p); err != nil {
return "", "", err
}
return "closed the plan " + p.ID + " (" + state + ")", why, nil
}
// --- H3 ----------------------------------------------------------------------------------------------
// appliesToAnObject is H3's: every holder or consumer the probe or the bus names, its budget per object.
func appliesToAnObject(_ context.Context, h *healing, c conditions.Condition) (string, bool, string, error) {
if h.assertObjects == nil {
return "", false, "this controller is not on the bus", nil
}
return c.Key, true, "", nil
}
// repairObjects is H3: the send's own assertion of every stream, consumer and seat worker.
func repairObjects(ctx context.Context, h *healing, _ conditions.Condition) (string, string, error) {
if err := h.assertObjects(ctx); err != nil {
return "", "", err
}
return "asserted the bus's streams, consumers and seat workers again",
"every object the mesh defines is asserted; the probe that raised it says on its next run whether it is there", nil
}
// --- H4 ----------------------------------------------------------------------------------------------
// resettable is why a consumer may be reset by the mesh itself, from the stream table; empty when not.
func resettable(stream, name string) string {
for _, c := range broker.MeshConsumers() {
if c.Stream == stream && c.Name == name {
return c.Resettable
}
}
return ""
}
// consumerOf is the stream and consumer a bus condition names: `<stream>.<consumer>`.
func consumerOf(c conditions.Condition) (string, string, bool) {
stream, name, found := strings.Cut(c.Subject.ID, ".")
return stream, name, found && stream != "" && name != ""
}
// appliesToAResettableConsumer is H4's: only a consumer the stream table marks resettable.
func appliesToAResettableConsumer(_ context.Context, _ *healing, c conditions.Condition) (string, bool, string, error) {
stream, name, ok := consumerOf(c)
if !ok {
return "", false, "it names no consumer", nil
}
if resettable(stream, name) == "" {
return "", false, fmt.Sprintf("%s on %s is not marked resettable in the stream table: what a reset drops, "+
"nothing would catch up", name, stream), nil
}
return c.Key, true, "", nil
}
// repairConsumer is H4: `broker consumer-reset`, made by the mesh.
func repairConsumer(_ context.Context, h *healing, c conditions.Condition) (string, string, error) {
stream, name, _ := consumerOf(c)
said, err := h.resetConsumer(stream, name)
if err != nil {
return "", "", err
}
return fmt.Sprintf("re-made %s on %s to deliver from now", name, stream),
said + "; " + resettable(stream, name), nil
}
// --- the verb ----------------------------------------------------------------------------------------
// healersCommand is `healers`: the registry, what the healers did lately, and the brake.
func healersCommand(ctx context.Context, args []string) error {
set := flag.NewFlagSet("healers", flag.ContinueOnError)
daysFlag := set.Int("days", 7, "how many days of acts back")
jsonFlag := set.Bool("json", false, "as data")
if rest, err := parseAround(set, args); err != nil {
return err
} else if len(rest) > 0 {
return errors.New("healers [--days N] [--json]")
}
days, asJSON := *daysFlag, *jsonFlag
if days <= 0 {
return fmt.Errorf("healers --days takes a number of days, not %d", days)
}
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
now := time.Now()
heals, err := open.inventory.HealsSince(ctx, now.Add(-time.Duration(days)*24*time.Hour))
if err != nil {
return fmt.Errorf("what the healers did cannot be read: %w", err)
}
conds, condErr := openConditions(ctx)
braked, said := brakeHolds(actsWithin(heals, now.Add(-healBrakeWindow)), conds, now)
brake := map[string]any{"holds": braked, "said": said, "limit": healBrakeLimit, "window": healBrakeWindow.String()}
if condErr != nil {
brake["conditions"] = "the open conditions could not be read, so whether the brake was said is not known: " +
condErr.Error()
}
var rows []map[string]any
for _, r := range healerRegistry {
rows = append(rows, map[string]any{"id": r.ID, "kinds": r.Kinds, "condition": r.Condition, "repair": r.Repair,
"budget": fmt.Sprintf("%d act(s) within %s, %s apart at least", r.Budget, r.Window, r.Settle),
"then": r.Then, "acts-in": r.ActsIn, "event": r.Event, "from": r.From})
}
if heals == nil {
heals = []inventory.Heal{}
}
if asJSON {
return printJSON(map[string]any{"healers": rows, "heals": heals, "brake": brake, "days": days,
"note": "a heal is never a hand act; whether it repaired anything is the condition's clearing to say"})
}
for _, r := range healerRegistry {
fmt.Printf("%s %s\n → %s\n budget %d within %s; then %s (in %s)\n", r.ID, r.Condition, r.Repair, r.Budget,
r.Window, r.Then, r.ActsIn)
}
fmt.Printf("\nthe brake: %s — %s\n\n", map[bool]string{true: "HOLDS, every healer has stopped", false: "off"}[braked], said)
if len(heals) == 0 {
fmt.Printf("no healer acted in the last %d day(s)\n", days)
return nil
}
for i := len(heals) - 1; i >= 0; i-- {
x := heals[i]
fmt.Printf("%s %s %-9s %s\n %s\n", x.At.Local().Format("2006-01-02 15:04"), x.Healer, x.Outcome, x.ConditionKey,
firstLine(x.Said))
}
return nil
}
// forgettingOldHeals removes heals past their keeping once a day, by the controller acting only.
func forgettingOldHeals(ctx context.Context, inv *inventory.Inventory) {
for {
if link.Holding() {
if n, err := inv.ForgetOldHeals(ctx); err != nil {
fmt.Printf("heals older than %s could not be removed: %v\n", inventory.HealsKeptFor, err)
} else if n > 0 {
fmt.Printf("removed %d heal(s) older than %s\n", n, inventory.HealsKeptFor)
}
}
select {
case <-ctx.Done():
return
case <-time.After(24 * time.Hour):
}
}
}
// healsCount is what the healers did, counted for `status`.
type healsCount struct {
Acts int `json:"acts"`
Escalated int `json:"escalated"`
}
// countHeals counts acts and escalations.
func countHeals(heals []inventory.Heal) *healsCount {
out := &healsCount{}
for _, h := range heals {
if h.Outcome == inventory.HealEscalated {
out.Escalated++
} else {
out.Acts++
}
}
return out
}
+647
View File
@@ -0,0 +1,647 @@
package main
import (
"context"
"encoding/json"
"errors"
"os"
"slices"
"strconv"
"strings"
"sync"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// The test generated from the healer registry (novox/hq to-be 45 §7, ADR 0227 rule 7 "how it is
// checked"): **every row is walked.** Its kinds are ones the mesh raises — a row of the signals table, a
// probe of the self-check, or a provider's event — it has a budget, a window and a settle inside it,
// what happens when the budget is spent, and the event each act is said as; a healer the controller runs
// has its applies and its repair, and an induced failure below that sees it act, say so and brake. A
// row added without one fails, so the registry cannot grow a healer nobody has seen act.
// raisedKinds is every condition kind the mesh raises, and what raises it.
func raisedKinds() map[string]string {
out := map[string]string{kindProviderFailing: "the provisioner.failing event (ADR 0224)"}
for _, r := range signalsTable {
for _, k := range kindsOf(r) {
out[k] = r.Row
}
}
for _, p := range probeRegistry {
out[p.Kind] = p.ID
for _, k := range p.Raises {
out[k] = p.ID
}
}
return out
}
// inducedFailures are the healers seen acting in this file, by id: a row without one fails.
var inducedFailures = map[string]string{
"H1": "TestH1AsksAMachineToReportAndSendsItAgain",
"H2": "TestH2ClosesAPlanAnotherHasTakenOver",
"H3": "TestNatsH3AssertsAMissingConsumerAgainAndBrakesAfterItsBudget",
"H4": "TestNatsH4ResetsTheControllersEventsConsumerAndItStillDelivers",
}
func TestEveryHealerAnswersAKindTheMeshRaisesWithABudgetABrakeAndItsEvent(t *testing.T) {
kinds := raisedKinds()
source, err := os.ReadFile("healers_test.go")
if err != nil {
t.Fatal(err)
}
seen := map[string]bool{}
answered := map[string]string{}
for _, r := range healerRegistry {
t.Run(r.ID, func(t *testing.T) {
if seen[r.ID] {
t.Fatalf("%s is in the registry twice", r.ID)
}
seen[r.ID] = true
if len(r.Kinds) == 0 || r.Condition == "" || r.Repair == "" || r.Then == "" || r.From == "" || r.ActsIn == "" {
t.Fatalf("%s does not say what it answers, what it does, what then, and which hand act it replaces: %+v", r.ID, r)
}
for _, k := range r.Kinds {
if _, ok := kinds[k]; !ok {
t.Errorf("%s answers %q, which nothing in the mesh raises", r.ID, k)
}
if other, twice := answered[k]; twice {
t.Errorf("%q is answered by %s and %s: one healer per kind", k, other, r.ID)
}
answered[k] = r.ID
}
if r.Budget <= 0 || r.Window <= 0 || r.Settle <= 0 || r.Settle > r.Window {
t.Errorf("%s has no budget it can spend: %d within %s, settled after %s", r.ID, r.Budget, r.Window, r.Settle)
}
if r.Event == "" {
t.Errorf("%s says nothing when it acts", r.ID)
}
if r.ActsIn != actsInController {
if r.repair != nil || r.applies != nil {
t.Errorf("%s acts in %s and the controller would act for it too", r.ID, r.ActsIn)
}
return
}
if r.repair == nil || r.applies == nil {
t.Fatalf("%s acts in the controller and has no repair or no applies", r.ID)
}
if r.Event != link.KeyHealerActed || !slices.Contains(broker.ControllerStates, r.Event) {
t.Errorf("%s is said as %q, which the controller's grant does not permit", r.ID, r.Event)
}
if inducedFailures[r.ID] == "" {
t.Errorf("%s has no induced failure: a healer nobody has seen act", r.ID)
} else if !strings.Contains(string(source), "func "+inducedFailures[r.ID]+"(t *testing.T)") {
t.Errorf("%s's induced failure %s is not a test in this file", r.ID, inducedFailures[r.ID])
}
})
}
for id := range inducedFailures {
if !seen[id] {
t.Errorf("an induced failure for %s, which the registry does not have", id)
}
}
if healBrakeLimit <= 0 || healBrakeWindow <= 0 {
t.Error("the mesh-wide brake holds nothing")
}
}
// heard keeps the healer-acted events said.
type heard struct {
mu sync.Mutex
acts []healerActed
}
func (h *heard) PublishSeatEvent(_ context.Context, seat, event string, body []byte) error {
if seat != conditions.Seat || event != link.KeyHealerActed {
return errors.New("said under the wrong seat or name: " + seat + " " + event)
}
var e healerActed
if err := json.Unmarshal(body, &e); err != nil {
return err
}
h.mu.Lock()
defer h.mu.Unlock()
h.acts = append(h.acts, e)
return nil
}
func (h *heard) said() []healerActed {
h.mu.Lock()
defer h.mu.Unlock()
return append([]healerActed(nil), h.acts...)
}
// testClock is a moment a test moves by hand.
type testClock struct {
mu sync.Mutex
at time.Time
}
func (c *testClock) now() time.Time {
c.mu.Lock()
defer c.mu.Unlock()
return c.at
}
func (c *testClock) pass(d time.Duration) {
c.mu.Lock()
defer c.mu.Unlock()
c.at = c.at.Add(d)
}
// healingOn is a runner over a mesh's stores and its condition store, under epoch 57, every act a
// fake that fails the test unless the test gives it.
func healingOn(t *testing.T, open *stores) (*healing, *heard, *testClock) {
t.Helper()
if conditionsFrom == nil {
t.Fatal("the mesh has no condition store")
}
told, clock := &heard{}, &testClock{at: time.Now()}
// The store's gate, as the serving controller's is the lease's (stores.go).
open.inventory.ActsUnder(func(context.Context) (uint64, error) { return 57, nil })
h := &healing{open: open, keeper: conditionsFrom, teller: told,
epoch: func(context.Context) (uint64, error) { return 57, nil }, now: clock.now,
say: func(f string, a ...any) { t.Logf(f, a...) }, reportWait: 300 * time.Millisecond,
declined: map[string]string{}}
h.askReport = func(context.Context, string) error { t.Error("asked a machine to report"); return nil }
h.sendAgain = func(context.Context, string) error { t.Error("sent a machine again"); return nil }
h.assertObjects = func(context.Context) error { t.Error("asserted the bus's objects"); return nil }
h.resetConsumer = func(string, string) (string, error) { t.Error("reset a consumer"); return "", nil }
return h, told, clock
}
// sentNotReported raises S2 for a machine, as the watchdog does.
func sentNotReported(t *testing.T, node string) string {
t.Helper()
o := conditions.Observation{Scope: conditions.ScopeMachine, ID: node, Kind: "sent-not-reported", Machine: node,
Severity: conditions.Warning, Summary: node + " was sent a declaration and has not reported it", Source: "S2"}
if _, err := conditionsFrom.Observe(t.Context(), o); err != nil {
t.Fatal(err)
}
return o.Key()
}
// **H1, the commonest hand act** (031/01 §f: a push by hand to unstick a plan waiting on a report, four
// times): the machine is asked to report; a report that names what it was sent is all it takes, and one
// that does not — or none — is a send of its current declaration again. Each act kept in `tried` as
// `healer H1`, said as healer-acted, counted; twice, then the operator's, urgent; then nothing more.
func TestH1AsksAMachineToReportAndSendsItAgain(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
h, told, clock := healingOn(t, open)
record, err := open.inventory.NodeByName(ctx, "laptop")
if err != nil {
t.Fatal(err)
}
if err := open.inventory.RecordSent(ctx, record.ID, "d2", nil); err != nil {
t.Fatal(err)
}
key := sentNotReported(t, "laptop")
// First: the report had not reached the mesh, and asking for it brings it.
asked := 0
h.askReport = func(ctx context.Context, node string) error {
asked++
if node != "laptop" {
t.Errorf("asked %s", node)
}
_, err := open.inventory.RecordDoing(ctx, record.ID, inventory.Doing{Outcome: inventory.OutcomeApplied,
At: time.Now().Add(time.Second), Declared: "d2"})
return err
}
h.tick(ctx)
c, found, err := conditionsFrom.Get(ctx, key)
if err != nil || !found {
t.Fatalf("the condition is gone after the act — a healer cleared it, not an observation: %v", err)
}
if asked != 1 || len(c.Tried) != 1 || c.Tried[0].By != "healer H1" || !strings.Contains(c.Tried[0].Outcome, "acted") ||
c.Resolver != "healer:H1" {
t.Fatalf("after the first act: asked %d, %+v", asked, c)
}
if acts := told.said(); len(acts) != 1 || acts[0].Healer != "H1" || acts[0].Outcome != inventory.HealActed ||
acts[0].Condition != key || acts[0].By != "healer H1" || acts[0].Epoch != 57 {
t.Fatalf("the act was not said as healer-acted: %+v", acts)
}
// Within its settle, nothing more: the observation has its turn.
clock.pass(time.Minute)
h.tick(ctx)
if asked != 1 {
t.Fatalf("acted again inside the settle: asked %d", asked)
}
// Second: the machine says nothing, so it is sent again.
clock.pass(3 * time.Minute)
sent := 0
h.askReport = func(context.Context, string) error { asked++; return nil }
h.sendAgain = func(_ context.Context, node string) error { sent++; return nil }
h.tick(ctx)
if asked != 2 || sent != 1 {
t.Fatalf("the second act: asked %d, sent %d", asked, sent)
}
c, _, _ = conditionsFrom.Get(ctx, key)
if len(c.Tried) != 2 || !strings.Contains(c.Tried[1].Outcome, "sent again") || !strings.Contains(c.Tried[1].Outcome, "act 2 of 2") {
t.Fatalf("tried %+v", c.Tried)
}
// The budget is spent: the operator's, urgent, said; and no healer touches it again.
clock.pass(4 * time.Minute)
h.tick(ctx)
c, _, _ = conditionsFrom.Get(ctx, key)
if !c.Escalated() || c.Severity != conditions.Urgent || len(c.Tried) != 3 ||
!strings.Contains(c.Tried[2].Outcome, "budget of 2") {
t.Fatalf("not handed to the operator: %+v", c)
}
if acts := told.said(); len(acts) != 3 || acts[2].Outcome != inventory.HealEscalated {
t.Fatalf("the escalation was not said: %+v", acts)
}
clock.pass(time.Hour)
h.tick(ctx)
if asked != 2 || sent != 1 || len(told.said()) != 3 {
t.Fatalf("a healer acted on a condition the operator holds: asked %d sent %d said %d", asked, sent, len(told.said()))
}
heals, err := open.inventory.HealsSince(ctx, time.Now().Add(-time.Hour))
if err != nil || len(heals) != 3 || heals[0].Outcome != inventory.HealActed || heals[1].Outcome != inventory.HealActed ||
heals[2].Outcome != inventory.HealEscalated || heals[0].Epoch != 57 {
t.Fatalf("the heals kept: %+v %v", heals, err)
}
// And a heal is not a hand act: what S15 counts never sees it.
for _, x := range heals {
if x.Healer == "" || strings.HasPrefix(x.Act, "hand-act") {
t.Errorf("a heal reads as a hand act: %+v", x)
}
}
}
// **Only the controller holding the lease heals** (to-be 45 §6): unleased, or standing by, nothing.
func TestNoHealerActsWithoutTheLease(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
h, told, _ := healingOn(t, open)
sentNotReported(t, "laptop")
h.epoch = func(context.Context) (uint64, error) { return 0, nil }
h.tick(ctx)
h.epoch = func(context.Context) (uint64, error) { return 0, errors.New("the lease is not held") }
h.tick(ctx)
h.epoch = func(context.Context) (uint64, error) { return 57, nil }
h.acting = func() bool { return false }
h.tick(ctx)
if len(told.said()) != 0 {
t.Fatalf("a healer acted without the lease: %+v", told.said())
}
}
// **The mesh-wide brake**: a dozen acts in an hour and every healer stops, said urgently, until an hour
// after the last.
func TestTheBrakeStopsEveryHealerAndSaysSo(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
h, told, clock := healingOn(t, open)
for i := 0; i < healBrakeLimit; i++ {
if _, err := open.inventory.BeginHeal(ctx, inventory.Heal{Healer: "H3", ConditionKey: "seat.x.anchor.silent",
Kind: "holder-silent", Act: "asserted", Outcome: inventory.HealActed,
At: clock.now().Add(-time.Duration(healBrakeLimit-i) * time.Minute)}); err != nil {
t.Fatal(err)
}
}
sentNotReported(t, "laptop")
h.tick(ctx) // the fakes fail the test if anything acts
braked, found, err := conditionsFrom.Get(ctx, "mesh.healers.braked")
if err != nil || !found || braked.Severity != conditions.Urgent || !strings.Contains(braked.Evidence[0].Said, "H3 on seat.x.anchor.silent ×12") {
t.Fatalf("the brake was not said: %+v %v", braked, err)
}
if len(told.said()) != 0 {
t.Fatalf("a healer acted under the brake: %+v", told.said())
}
// Past the hour of the first act, still held: it lets go an hour after the last.
clock.pass(30 * time.Minute)
h.tick(ctx)
if _, held, _ := conditionsFrom.Get(ctx, "mesh.healers.braked"); !held {
t.Fatal("the brake let go before an hour had passed since the last act")
}
clock.pass(31 * time.Minute)
asked := 0
h.askReport = func(context.Context, string) error { asked++; return nil }
h.sendAgain = func(context.Context, string) error { return nil }
h.tick(ctx)
if _, held, _ := conditionsFrom.Get(ctx, "mesh.healers.braked"); held {
t.Fatal("the brake held an hour after the last act")
}
if asked != 1 {
t.Fatalf("the healers did not act again after the brake let go: asked %d", asked)
}
}
// **H2 closes a plan another has taken over**, with its note — and leaves a plan alone whose wait is
// still to come: that one is its own signal's, not a healer's.
func TestH2ClosesAPlanAnotherHasTakenOver(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
h, told, _ := healingOn(t, open)
created := time.Now().UTC().Add(-time.Hour)
older := inventory.Plan{ID: "plan-old", Repository: "novox/app", Branch: "main", Commit: "0ld0ld0", Created: created,
State: inventory.PlanBuilding, Tiers: [][]string{{"a"}}, Modules: map[string]*inventory.PlanModule{"a": {State: "asked"}}}
waiting := inventory.Plan{ID: "plan-other", Repository: "novox/other", Branch: "main", Commit: "07he707", Created: created,
State: inventory.PlanBuilding, Tiers: [][]string{{"b"}}, Modules: map[string]*inventory.PlanModule{"b": {State: "asked"}}}
newer := inventory.Plan{ID: "plan-new", Repository: "novox/app", Branch: "main", Commit: "new0new", Created: created.Add(time.Minute),
State: inventory.PlanDone, Tiers: [][]string{{"a"}}, Modules: map[string]*inventory.PlanModule{"a": {State: "built"}}}
for _, p := range []*inventory.Plan{&older, &waiting, &newer} {
if err := open.inventory.SavePlan(ctx, p); err != nil {
t.Fatal(err)
}
}
for _, id := range []string{"plan-old", "plan-other"} {
if _, err := conditionsFrom.Observe(ctx, conditions.Observation{Scope: conditions.ScopePlan, ID: id, Kind: "stalled",
Severity: conditions.Warning, Summary: id + " is stalled", Source: "S3"}); err != nil {
t.Fatal(err)
}
}
h.tick(ctx)
closed, err := open.inventory.PlanByID(ctx, "plan-old")
if err != nil || closed.State != inventory.PlanSuperseded || !strings.Contains(closed.Note, "closed by healer H2") ||
!strings.Contains(closed.Note, "plan-new") {
t.Fatalf("the superseded plan: %+v %v", closed, err)
}
if other, _ := open.inventory.PlanByID(ctx, "plan-other"); other.State != inventory.PlanBuilding {
t.Fatalf("a plan still waiting was closed: %+v", other)
}
if c, _, _ := conditionsFrom.Get(ctx, "plan.plan-other.stalled"); len(c.Tried) != 0 || c.Resolver != conditions.ResolverSelf {
t.Fatalf("a plan H2 does not repair was touched: %+v", c)
}
if acts := told.said(); len(acts) != 1 || acts[0].Healer != "H2" || acts[0].Condition != "plan.plan-old.stalled" {
t.Fatalf("said %+v", acts)
}
// Finished: every module built, none rolled out unsent — closed as done.
done := inventory.Plan{ID: "plan-done", Repository: "novox/third", Branch: "main", Commit: "d0ned0n", Created: created,
State: inventory.PlanRolling, Tier: 0, Tiers: [][]string{{"c"}}, Modules: map[string]*inventory.PlanModule{"c": {State: "built"}}}
if err := open.inventory.SavePlan(ctx, &done); err != nil {
t.Fatal(err)
}
if state, why, err := planStale(ctx, open.inventory, done); err != nil || state != inventory.PlanDone || !strings.Contains(why, "finished") {
t.Fatalf("a finished plan reads %q %q %v", state, why, err)
}
}
// **H4 only for a consumer the stream table marks resettable**: a module's consumer far behind is said
// and left — what a reset drops, nothing would catch up for it.
func TestH4ResetsOnlyWhatTheTableMarksResettable(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
h, _, _ := healingOn(t, open)
marked := 0
for _, c := range broker.MeshConsumers() {
if c.Resettable != "" {
marked++
if c.Stream != broker.EventsStream || c.Name != broker.ControllerName {
t.Errorf("%s on %s is marked resettable: only the controller's own events consumer is", c.Name, c.Stream)
}
}
}
if marked != 1 {
t.Fatalf("%d consumers are marked resettable, want the controller's events consumer alone", marked)
}
for _, who := range []string{"EVENTS.anchor_shop", "CONTROL.controller"} {
if _, err := conditionsFrom.Observe(ctx, conditions.Observation{Scope: conditions.ScopeBus, ID: who, Token: "behind",
Kind: kindConsumerBehind, Severity: conditions.Warning, Summary: who + " is far behind", Source: "D6"}); err != nil {
t.Fatal(err)
}
}
h.tick(ctx) // the fake reset fails the test if it is called
reset := ""
h.resetConsumer = func(stream, name string) (string, error) {
reset = stream + "." + name
return "it was 1500 behind", nil
}
if _, err := conditionsFrom.Observe(ctx, conditions.Observation{Scope: conditions.ScopeBus, ID: "EVENTS.controller",
Token: "behind", Kind: kindConsumerBehind, Severity: conditions.Warning, Summary: "far behind", Source: "D6"}); err != nil {
t.Fatal(err)
}
h.tick(ctx)
if reset != "EVENTS.controller" {
t.Fatalf("reset %q", reset)
}
}
// **H3 against a real bus** (issue 208's note): a consumer the mesh expects, deleted; D6 says so; H3
// asserts the bus's objects the way a send does, and D6's next run clears it — the healer never does.
// Deleted again within the hour, the budget is spent: the operator's, urgent, and H3 stops.
func TestNatsH3AssertsAMissingConsumerAgainAndBrakesAfterItsBudget(t *testing.T) {
url := os.Getenv("MESH_TEST_NATS")
if url == "" {
t.Skip("MESH_TEST_NATS unset")
}
open := aMesh(t)
ctx := t.Context()
js, err := broker.Dial(url)
if err != nil {
t.Fatal(err)
}
t.Cleanup(js.Close)
for _, s := range []string{"CONTROL", "NODES", "ASSIGNMENTS", "EVENTS"} {
_ = js.Context().DeleteStream(s)
}
if _, err := assertBusObjects(ctx, open.inventory, js); err != nil {
t.Fatal(err)
}
h, told, clock := healingOn(t, open)
h.js = js
h.assertObjects = func(ctx context.Context) error { return assertOnSend(ctx, open.inventory, js, " ") }
d := &doctor{open: open, js: js}
probe := func() {
t.Helper()
found, err := probeConsumers(ctx, d)
if err != nil {
t.Fatal(err)
}
if err := conditionsFrom.Reconcile(ctx, "D6", kindedAs(found, "consumer-wrong")); err != nil {
t.Fatal(err)
}
}
const key = "bus.NODES.laptop.missing"
if err := js.Context().DeleteConsumer("NODES", "laptop"); err != nil {
t.Fatal(err)
}
probe()
if c, raised, _ := conditionsFrom.Get(ctx, key); !raised || c.Kind != "consumer-lost" {
t.Fatalf("the deleted consumer was not raised: %+v", c)
}
h.tick(ctx)
if _, err := js.Context().ConsumerInfo("NODES", "laptop"); err != nil {
t.Fatalf("H3 did not make the consumer again: %v", err)
}
c, still, _ := conditionsFrom.Get(ctx, key)
if !still || len(c.Tried) != 1 || c.Tried[0].By != "healer H3" || c.Resolver != "healer:H3" {
t.Fatalf("the act is not in the condition, or the healer cleared it: %+v", c)
}
probe()
if _, still, _ := conditionsFrom.Get(ctx, key); still {
t.Fatal("the probe's next run did not clear what H3 repaired")
}
// Kept in the history as the keeper says it, a moment later.
var cleared *conditions.Event
for wait := time.Now().Add(5 * time.Second); cleared == nil && time.Now().Before(wait); time.Sleep(20 * time.Millisecond) {
history, err := conditionsFrom.HistorySince(ctx, time.Now().Add(-time.Minute))
if err != nil {
t.Fatal(err)
}
for i := range history {
if history[i].Key == key && history[i].Change == conditions.ChangeCleared {
cleared = &history[i]
}
}
}
if cleared == nil || !strings.Contains(cleared.Why, "D6 no longer observes it") || len(cleared.Tried) != 1 {
t.Fatalf("the clearing is not the probe's, or forgets what was tried: %+v", cleared)
}
// Again within the hour: the budget is one, so after its settle the operator is told, and H3 stops.
clock.pass(10 * time.Minute)
if err := js.Context().DeleteConsumer("NODES", "laptop"); err != nil {
t.Fatal(err)
}
probe()
h.assertObjects = func(context.Context) error { t.Error("H3 acted past its budget"); return nil }
h.tick(ctx)
c, _, _ = conditionsFrom.Get(ctx, key)
if !c.Escalated() || c.Severity != conditions.Urgent || c.Count != 2 {
t.Fatalf("the spent budget was not handed to the operator: %+v", c)
}
acts := told.said()
if len(acts) != 2 || acts[0].Outcome != inventory.HealActed || acts[1].Outcome != inventory.HealEscalated {
t.Fatalf("said %+v", acts)
}
clock.pass(2 * time.Hour)
h.tick(ctx)
if len(told.said()) != 2 {
t.Fatal("a healer acted on what the operator holds")
}
}
// **H4 against a real bus** (issue 248): the controller's events consumer a long way behind — the week it
// once replayed — is re-made from now by the mesh itself, and the controller's bound subscription still
// receives what comes next: a reset that left the controller deaf would be the incident.
func TestNatsH4ResetsTheControllersEventsConsumerAndItStillDelivers(t *testing.T) {
url := os.Getenv("MESH_TEST_NATS")
if url == "" {
t.Skip("MESH_TEST_NATS unset")
}
open := aMesh(t)
ctx := t.Context()
js, err := broker.Dial(url)
if err != nil {
t.Fatal(err)
}
t.Cleanup(js.Close)
for _, s := range []string{"CONTROL", "NODES", "ASSIGNMENTS", "EVENTS"} {
_ = js.Context().DeleteStream(s)
}
if _, err := assertBusObjects(ctx, open.inventory, js); err != nil {
t.Fatal(err)
}
// Bound as the controller binds it (link/receive_nats.go), taking one and acknowledging none.
events := make(chan *nats.Msg, 64)
sub, err := js.Context().ChanSubscribe("", events, nats.Bind(broker.EventsStream, broker.ControllerName))
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = sub.Unsubscribe() })
followed := broker.ControllerFollows[0]
for i := 0; i < consumerFarBehind+200; i++ {
if _, err := js.Context().Publish(followed, []byte(`{"n":`+strconv.Itoa(i)+`}`)); err != nil {
t.Fatal(err)
}
}
d := &doctor{open: open, js: js}
probe := func() []conditions.Observation {
t.Helper()
found, err := probeConsumers(ctx, d)
if err != nil {
t.Fatal(err)
}
if err := conditionsFrom.Reconcile(ctx, "D6", kindedAs(found, "consumer-wrong")); err != nil {
t.Fatal(err)
}
return found
}
probe()
const key = "bus.EVENTS.controller.behind"
if c, raised, _ := conditionsFrom.Get(ctx, key); !raised || c.Kind != kindConsumerBehind {
t.Fatalf("a consumer %d behind was not raised: %+v", consumerFarBehind+200, c)
}
h, told, _ := healingOn(t, open)
h.resetConsumer = func(stream, name string) (string, error) {
before, after, err := js.ResetConsumer(stream, name)
if err != nil {
return "", err
}
return "it was " + strconv.FormatUint(before.Pending, 10) + " behind; " + strconv.FormatUint(after.Pending, 10) +
" pending now", nil
}
h.tick(ctx)
if acts := told.said(); len(acts) != 1 || acts[0].Healer != "H4" || acts[0].Outcome != inventory.HealActed {
t.Fatalf("said %+v", acts)
}
if found := probe(); len(found) != 0 {
t.Fatalf("after the reset the probe still finds %+v", found)
}
if _, still, _ := conditionsFrom.Get(ctx, key); still {
t.Fatal("the probe's next run did not clear what H4 repaired")
}
// What comes next still reaches the controller.
for len(events) > 0 {
<-events
}
if _, err := js.Context().Publish(followed, []byte(`{"after":"the reset"}`)); err != nil {
t.Fatal(err)
}
deadline := time.After(10 * time.Second)
for {
select {
case m := <-events:
if strings.Contains(string(m.Data), "after") {
return
}
_ = m.Ack()
case <-deadline:
t.Fatal("the controller's subscription heard nothing after its consumer was reset: it would be deaf")
}
}
}
// **H1's question reaches the machine**, on the subject its grant lets it hear and nothing else.
func TestNatsAskToReportReachesTheMachine(t *testing.T) {
url := os.Getenv("MESH_TEST_NATS")
if url == "" {
t.Skip("MESH_TEST_NATS unset")
}
conn, err := nats.Connect(url)
if err != nil {
t.Fatal(err)
}
t.Cleanup(conn.Close)
asked, err := conn.SubscribeSync(broker.AskReportSubject("laptop"))
if err != nil {
t.Fatal(err)
}
if err := askToReport(t.Context(), conn, "laptop"); err != nil {
t.Fatal(err)
}
msg, err := asked.NextMsg(5 * time.Second)
if err != nil || !strings.Contains(string(msg.Data), "healer H1") {
t.Fatalf("the machine heard %v, %v", msg, err)
}
perms, err := broker.PermissionsFor(broker.Principal{Kind: broker.KindNode, Node: "laptop", PasswordHash: "x"})
if err != nil || !slices.Contains(perms.Subscribe, "mesh.node.laptop.ask.report") ||
slices.Contains(perms.Subscribe, "mesh.node.anchor.ask.report") {
t.Fatalf("a machine's grant for the question: %v %v", perms.Subscribe, err)
}
}
+242
View File
@@ -0,0 +1,242 @@
package main
import (
"context"
"fmt"
"io"
"sort"
"strings"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// A push sends no build a policy or a plan holds back, except to the machine it names (novox/hq
// issue 259, ADR 0221).
//
// A named push ends by sending every other machine whose declaration differs from what it was last
// sent (ADR 0083), so that a grant the push's work minted reaches the provider in the same act. A
// digest cannot say why a machine differs. A module whose upgrade policy records rather than rolls
// out makes every machine running it differ from the merge on, and so did a module whose plan was
// still waiting on its first machine (ADR 0218): `push <anchor>` sent all four machines the build
// that was meant to be walked through the mesh one machine at a time, and a fault in it was met
// everywhere at once.
//
// What tells the two apart is which build of each module the machine was last sent, kept with every
// send. A machine any of whose modules would move to a build its policy or an open plan holds back
// is not sent by a push that did not name it; the push says which, and why, and how to send it.
// heldBack is why a push that did not name a machine must not send it, empty when it may.
//
// `modules` is what the machine would be sent now; `sent` and `known` what it was last sent, as
// Inventory.SentBuilds answers. A module moves when the build it would carry is not the one the
// machine was last sent — including a module the machine was never sent at all. A move is held when
// the module's policy records rather than rolls out, or when an open plan has not yet sent this
// machine (planStillToSend). A machine whose last send was not recorded is held whole: what it carried
// is not known, so a held upgrade cannot be told from anything else.
func heldBack(node string, modules []string, sent map[string]string, known bool,
current map[string]inventory.CurrentBuild, plans []inventory.Plan) []string {
if !known {
return []string{"which builds it was last sent is not known — it was last sent before the " +
"mesh kept them, or sent a declaration by hand"}
}
var why []string
for _, m := range modules {
now := current[m]
was, carried := sent[m]
if carried && was == now.Commit {
continue
}
move := fmt.Sprintf("%s would move %sto %s", m, fromBuild(was, carried), buildName(now.Commit))
if !now.RollOut {
why = append(why, move+", which its upgrade policy records rather than rolls out")
continue
}
if id := planStillToSend(plans, m, node); id != "" {
why = append(why, move+", which "+id+" has not sent it yet (one machine first)")
}
}
sort.Strings(why)
return why
}
// planStillToSend is the open plan that has a module's new build still to send this machine, or empty:
// one holding the module that has neither finished sending it nor sent it here first, and has not
// failed it (novox/hq ADR 0218). The same reading rolledOutByAPlan makes for the whole module, made
// per machine.
func planStillToSend(plans []inventory.Plan, module, node string) string {
for _, p := range plans {
s, holds := p.Modules[module]
if !p.Open() || !holds {
continue
}
if s == nil {
return p.ID
}
if s.SentAt != nil || s.State == "failed" {
continue
}
first := false
for _, n := range s.First {
if n == node {
first = true
}
}
if !first {
return p.ID
}
}
return ""
}
func fromBuild(was string, carried bool) string {
if !carried {
return "(never sent it) "
}
return "from " + buildName(was) + " "
}
func buildName(commit string) string {
if commit == "" {
return "a build with no source"
}
return shortCommit(commit)
}
// heldMachines reads, for each machine named, why a push that did not name it must not send it
// (heldBack), and answers only the machines held. A machine whose set cannot be worked out is left
// to the send, which says why.
func heldMachines(ctx context.Context, open *stores, names []string) (map[string][]string, error) {
out := map[string][]string{}
if len(names) == 0 {
return out, nil
}
inv := open.inventory
current, err := inv.CurrentBuilds(ctx)
if err != nil {
return nil, err
}
plans, err := inv.OpenPlans(ctx)
if err != nil {
return nil, err
}
for _, node := range names {
plan, _, err := planFor(ctx, open, node)
if err != nil {
continue
}
modules := make([]string, 0, len(plan.Modules))
for _, m := range plan.Modules {
modules = append(modules, m.Module)
}
sent, known, err := inv.SentBuilds(ctx, node)
if err != nil {
return nil, err
}
if why := heldBack(node, modules, sent, known, current, plans); len(why) > 0 {
out[node] = why
}
}
return out, nil
}
// sayHeld is what a push says about a machine it left behind on purpose: that it is behind, why it
// was not sent, that whatever else it is owed waits with it, and the command that sends it.
func sayHeld(w io.Writer, node string, why []string) {
fmt.Fprintf(w, "\n%s is behind and was not sent: %s. A push sends no build a policy or a plan "+
"holds back to a machine it did not name (novox/hq ADR 0221), so anything else it is owed — a "+
"grant from this push among it — waits with it. `push %s` sends it\n",
node, strings.Join(why, "; "), node)
}
// flushBehind is the end of a named push: every other machine now behind is sent too, by name, over
// as many rounds as the sends take to settle (novox/hq issue 057, ADR 0083) — except a machine whose
// modules would move to a build a policy or a plan holds back, which is named and left (ADR 0221).
//
// `handled` is every machine already sent or already said; it is not considered again. Answers the
// machines that could not be composed, as refusals.
func flushBehind(ctx context.Context, open *stores, nodes []inventory.Node, handled map[string]bool,
compose func(held context.Context, node string) (sendable, error), d delivery, holder string,
w io.Writer) ([]string, error) {
inv := open.inventory
var refusals []string
// Bounded by the node count: a node is marked handled the round it is considered and is never
// considered twice, so the loop cannot run more than len(nodes) rounds. The bound is a guard
// against a logic error, not a real limit — if it were ever hit, that is a bug rather than a
// cascade legitimately still converging, so it is said rather than passed over in silence.
rounds := 0
for {
would, err := wouldSend(ctx, open, nodes)
if err != nil {
return refusals, err
}
behind, err := inv.Waiting(ctx, would)
if err != nil {
return refusals, err
}
var also []string
for _, m := range behind {
if !handled[m.Node] {
also = append(also, m.Node)
}
}
if len(also) == 0 {
return refusals, nil
}
if rounds++; rounds > len(nodes) {
fmt.Fprintf(w, "\nstopped cascading after %d rounds with %s still behind — this "+
"should not happen; run `push --behind` to finish\n",
rounds-1, strings.Join(also, ", "))
return refusals, nil
}
sort.Strings(also)
held, err := heldMachines(ctx, open, also)
if err != nil {
return refusals, err
}
var sending []string
for _, name := range also {
// Every candidate this round is marked handled — the sent ones so they are not
// re-listed, the held ones because they stay held, and the refused ones so a machine
// that cannot be composed does not make the loop spin on it for ever.
handled[name] = true
if why, isHeld := held[name]; isHeld {
sayHeld(w, name, why)
continue
}
sending = append(sending, name)
}
if len(sending) == 0 {
continue
}
fmt.Fprintf(w, "\nthis push left %s behind — a provision granted from there, or a "+
"declaration since changed; sending it too\n", strings.Join(sending, ", "))
// Tolerantly, exactly as the named send: a machine that cannot be composed is collected as
// a refusal and reported at the end, and the others are still sent (novox/hq ADR 0066).
// Held for this round only, and after the last round's were given back, so two pushes
// cascading into each other's machines never each wait on the other.
refused, err := sendRound(ctx, open, sending, compose, d, holder)
refusals = append(refusals, refused...)
if err != nil {
return refusals, err
}
}
}
// composeForPush is how a push composes one machine: its set resolved, what it cannot host and what
// is left out of it said, and its declaration allocated.
func composeForPush(open *stores, gens map[string]catalogue.Generator) func(held context.Context, node string) (sendable, error) {
return func(held context.Context, node string) (sendable, error) {
plan, settings, err := planFor(held, open, node)
if err != nil {
return sendable{}, err
}
reportUnhostable(node, plan)
reportKept(held, plan)
declared, err := declarationWith(held, open, node, plan, settings, gens, Allocating)
if err == nil {
reportLeftOut(node, declared)
}
return declared, err
}
}
+335
View File
@@ -0,0 +1,335 @@
package main
import (
"bytes"
"context"
"encoding/json"
"reflect"
"slices"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/overlay"
)
// novox/hq issue 259, ADR 0221: a push that did not name a machine does not send it a build its
// upgrade policy records rather than rolls out, nor one an open plan has not sent it yet. Anything
// else that moved is still a consequence the push sends (ADR 0083).
func TestAHeldBuildHoldsAMachineANamedPushDidNotName(t *testing.T) {
current := map[string]inventory.CurrentBuild{
"resolver": {Commit: "c2c2c2c2c2"},
"agent": {Commit: "a2", RollOut: true},
"network": {},
}
modules := []string{"network", "resolver", "agent"}
sent := map[string]string{"network": "", "resolver": "c1c1c1c1c1", "agent": "a2"}
// A module whose policy records moved: held, naming it, both builds and why.
why := heldBack("laptop", modules, sent, true, current, nil)
if len(why) != 1 || !strings.Contains(why[0], "resolver would move from c1c1c1c1 to c2c2c2c2") ||
!strings.Contains(why[0], "upgrade policy records") {
t.Fatalf("a recorded upgrade did not hold the machine: %v", why)
}
// Nothing moved — what differs is a grant, a peer, a setting: not held (issue 057).
sent["resolver"] = "c2c2c2c2c2"
if why := heldBack("laptop", modules, sent, true, current, nil); len(why) != 0 {
t.Fatalf("a machine whose builds are all current was held: %v", why)
}
// A module whose policy rolls out moved, and no plan holds it: sent, as before.
sent["agent"] = "a1"
if why := heldBack("laptop", modules, sent, true, current, nil); len(why) != 0 {
t.Fatalf("a rolled-out upgrade no plan holds was held: %v", why)
}
// The last send's builds are not known: held whole.
if why := heldBack("laptop", modules, nil, false, current, nil); len(why) != 1 ||
!strings.Contains(why[0], "not known") {
t.Fatalf("a machine whose last send was not recorded was not held: %v", why)
}
// A module the machine was never sent, under a recording policy: held, and said so.
delete(sent, "resolver")
sent["agent"] = "a2"
if why := heldBack("laptop", modules, sent, true, current, nil); len(why) != 1 ||
!strings.Contains(why[0], "resolver would move (never sent it) to c2c2c2c2") {
t.Fatalf("a module never sent under a recording policy: %v", why)
}
}
// ADR 0218 meets ADR 0083: a plan waiting on its first machine has not sent the rest, and a push
// naming some other machine must not send them for it.
func TestAPlanWaitingOnItsFirstMachineHoldsTheRest(t *testing.T) {
at := time.Now()
current := map[string]inventory.CurrentBuild{"agent": {Commit: "a2", RollOut: true}}
sent := map[string]string{"agent": "a1"}
waiting := []inventory.Plan{{ID: "plan-7", State: inventory.PlanRolling, Modules: map[string]*inventory.PlanModule{
"agent": {State: "built", First: []string{"ace"}, FirstAt: &at}}}}
why := heldBack("g14", []string{"agent"}, sent, true, current, waiting)
if len(why) != 1 || !strings.Contains(why[0], "plan-7 has not sent it yet") {
t.Fatalf("a machine the plan has not reached was not held: %v", why)
}
// The first machine itself was sent by the plan: not held by it.
if why := heldBack("ace", []string{"agent"}, sent, true, current, waiting); len(why) != 0 {
t.Fatalf("the plan's first machine was held: %v", why)
}
// Built but not yet sent anywhere, or not yet built: the plan has it still to send.
for what, s := range map[string]*inventory.PlanModule{"built, unsent": {State: "built"}, "unasked": nil} {
plans := []inventory.Plan{{ID: "plan-8", State: inventory.PlanBuilding,
Modules: map[string]*inventory.PlanModule{"agent": s}}}
if why := heldBack("ace", []string{"agent"}, sent, true, current, plans); len(why) != 1 {
t.Errorf("%s: not held: %v", what, why)
}
}
// Sent everywhere, failed, or a plan no longer open: the plan holds nothing back.
for what, plans := range map[string][]inventory.Plan{
"sent everywhere": {{ID: "p", State: inventory.PlanRolling, Modules: map[string]*inventory.PlanModule{
"agent": {State: "built", First: []string{"ace"}, FirstAt: &at, SentAt: &at}}}},
"failed": {{ID: "p", State: inventory.PlanRolling, Modules: map[string]*inventory.PlanModule{
"agent": {State: "failed"}}}},
"closed": {{ID: "p", State: inventory.PlanDone, Modules: map[string]*inventory.PlanModule{
"agent": {State: "built"}}}},
} {
if why := heldBack("g14", []string{"agent"}, sent, true, current, plans); len(why) != 0 {
t.Errorf("%s: held: %v", what, why)
}
}
}
// A send records the build of each module it carried; a module left out of it keeps the build it
// was last sent, since the machine keeps that one.
func TestASendCarriesTheCurrentBuildsAndALeftOutModuleKeepsItsOwn(t *testing.T) {
current := map[string]inventory.CurrentBuild{"a": {Commit: "a2"}, "b": {Commit: "b2"}, "c": {}}
got := carriedBuilds([]string{"a", "b", "c"}, map[string]string{"b": "a setting does not compose"},
current, map[string]string{"a": "a1", "b": "b1"})
if want := map[string]string{"a": "a2", "b": "b1", "c": ""}; !reflect.DeepEqual(got, want) {
t.Fatalf("carried %v, wanted %v", got, want)
}
// Not known before: the left-out module is not recorded at all, so it reads as never sent.
got = carriedBuilds([]string{"a", "b"}, map[string]string{"b": "x"}, current, nil)
if want := map[string]string{"a": "a2"}; !reflect.DeepEqual(got, want) {
t.Fatalf("carried %v, wanted %v", got, want)
}
}
// recordedDelivery sends nothing and records each send as the mesh does, so the next comparison
// reads the machine as current — and writes down which machines it declared.
type recordedDelivery struct {
inv *inventory.Inventory
declared []string
}
func (r *recordedDelivery) grant(context.Context, []readyNode) error { return nil }
func (r *recordedDelivery) declare(ctx context.Context, s readyNode, body []byte) (string, error) {
r.declared = append(r.declared, s.node)
return recordSent(ctx, r.inv, s.node, body, s.declared.Builds, s.declared.Epoch)
}
// aResolver is a module built from a repository, at a commit, with something on the machine that
// says which build it is.
func aResolver(t *testing.T, open *stores, commit string, asked time.Time) {
t.Helper()
m := catalogue.Manifest{Module: "resolver", Version: "1", Resources: []map[string]any{
{"id": "zones", "type": "file", "path": "/etc/resolver/zones", "content": "built from " + commit},
}}
if err := open.inventory.RegisterModule(t.Context(), m, inventory.Source{
Repository: "novox/mesh-catalog", Path: "modules/resolver", BuiltFrom: commit, Asked: asked}); err != nil {
t.Fatal(err)
}
}
// A third machine on the private network: once it is sent, every other machine's peers change with
// it, which is a consequence a push must still send — no build moved.
func aThirdMachine(t *testing.T, open *stores) {
t.Helper()
ctx := t.Context()
record, err := open.inventory.AddNode(ctx, "spare")
if err != nil {
t.Fatal(err)
}
if err := open.inventory.SetPlace(ctx, "spare", "spare.example:51820", "here", false, "10.77.0.3"); err != nil {
t.Fatal(err)
}
reported, err := json.Marshal(map[string]any{"capabilities": []map[string]any{
{"name": "container-runtime", "present": true}, {"name": "wireguard", "present": true},
{"name": "systemd", "present": true}}})
if err != nil {
t.Fatal(err)
}
var profile map[string]any
if err := json.Unmarshal(reported, &profile); err != nil {
t.Fatal(err)
}
if err := open.inventory.RecordProfile(ctx, record.ID, profile); err != nil {
t.Fatal(err)
}
if err := open.inventory.RecordSealingKey(ctx, record.ID, aPublicKey(t)); err != nil {
t.Fatal(err)
}
if err := open.inventory.RecordOverlayKey(ctx, record.ID, aPublicKey(t)); err != nil {
t.Fatal(err)
}
if _, err := open.inventory.Assign(ctx, "spare", overlay.Name); err != nil {
t.Fatal(err)
}
}
// The issue as it happened, against the real stores: a change merged with the policy `record`, a push
// naming the anchor, and the laptop — running the same module — left with what it had, by name.
func TestANamedPushLeavesAMachineAPolicyHoldsBack(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
asked := time.Now().Add(-time.Hour)
aResolver(t, open, "c1c1c1c1c1", asked)
for _, node := range []string{"anchor", "laptop"} {
if _, err := inv.Assign(ctx, node, "resolver"); err != nil {
t.Fatal(err)
}
}
gens, err := generators(ctx, open)
if err != nil {
t.Fatal(err)
}
compose := composeForPush(open, gens)
d := &recordedDelivery{inv: inv}
if _, err := sendRound(ctx, open, []string{"anchor", "laptop"}, compose, d, ""); err != nil {
t.Fatal(err)
}
if builds, known, err := inv.SentBuilds(ctx, "laptop"); err != nil || !known || builds["resolver"] != "c1c1c1c1c1" {
t.Fatalf("the send did not record the build it carried: %v %v %v", builds, known, err)
}
digestOfLaptop := func() string {
sent, err := inv.Outstanding(ctx, "laptop")
if err != nil {
t.Fatal(err)
}
return sent
}
before := digestOfLaptop()
// The change merges; the policy is the default, record. `push anchor` sends the anchor...
aResolver(t, open, "c2c2c2c2c2", asked.Add(time.Minute))
d.declared = nil
if _, err := sendRound(ctx, open, []string{"anchor"}, compose, d, ""); err != nil {
t.Fatal(err)
}
// ...and its cascade leaves the laptop, saying so.
var said bytes.Buffer
d.declared = nil
refused, err := flushBehind(ctx, open, mustNodes(t, open), map[string]bool{"anchor": true}, compose, d, "", &said)
if err != nil || len(refused) != 0 {
t.Fatalf("the cascade failed: %v %v", refused, err)
}
if len(d.declared) != 0 {
t.Fatalf("the cascade sent %v a build its policy records", d.declared)
}
if digestOfLaptop() != before {
t.Fatal("the laptop's last send moved: it was sent the held build")
}
for _, want := range []string{"laptop is behind and was not sent", "resolver would move from c1c1c1c1 to c2c2c2c2",
"upgrade policy records", "`push laptop` sends it"} {
if !strings.Contains(said.String(), want) {
t.Errorf("the push did not say %q:\n%s", want, said.String())
}
}
// Held and owed something else at once — a peer joined: still not sent, and both said: why it
// is held, and that what else it is owed waits with it.
aThirdMachine(t, open)
d.declared = nil
if _, err := sendRound(ctx, open, []string{"spare"}, compose, d, ""); err != nil {
t.Fatal(err)
}
d.declared = nil
said.Reset()
if _, err := flushBehind(ctx, open, mustNodes(t, open), map[string]bool{"anchor": true, "spare": true},
compose, d, "", &said); err != nil {
t.Fatal(err)
}
if len(d.declared) != 0 || digestOfLaptop() != before {
t.Fatalf("a held machine owed a consequence was sent: %v", d.declared)
}
if !strings.Contains(said.String(), "resolver would move") || !strings.Contains(said.String(), "anything else it is owed") {
t.Fatalf("the push did not say both:\n%s", said.String())
}
// A policy that rolls out: the laptop is a consequence like any other, and sent.
if err := inv.SetUpgradeOf(ctx, "resolver", inventory.Upgrade{RollOut: true}); err != nil {
t.Fatal(err)
}
said.Reset()
if _, err := flushBehind(ctx, open, mustNodes(t, open), map[string]bool{"anchor": true, "spare": true},
compose, d, "", &said); err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(d.declared, []string{"laptop"}) || digestOfLaptop() == before {
t.Fatalf("a rolled-out upgrade's machine was not sent: %v\n%s", d.declared, said.String())
}
if builds, _, _ := inv.SentBuilds(ctx, "laptop"); builds["resolver"] != "c2c2c2c2c2" {
t.Fatalf("the new send did not record the new build: %v", builds)
}
}
// Issue 057's case is unchanged: a machine whose builds are all current and whose declaration moved
// for another reason is sent by a push that names someone else.
func TestANamedPushStillSendsAConsequenceNothingHolds(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
aResolver(t, open, "c1c1c1c1c1", time.Now().Add(-time.Hour))
if _, err := inv.Assign(ctx, "laptop", "resolver"); err != nil {
t.Fatal(err)
}
gens, err := generators(ctx, open)
if err != nil {
t.Fatal(err)
}
compose := composeForPush(open, gens)
d := &recordedDelivery{inv: inv}
if _, err := sendRound(ctx, open, []string{"anchor", "laptop"}, compose, d, ""); err != nil {
t.Fatal(err)
}
// `push spare`, the machine just placed: the others' peers change with it.
aThirdMachine(t, open)
if _, err := sendRound(ctx, open, []string{"spare"}, compose, d, ""); err != nil {
t.Fatal(err)
}
d.declared = nil
var said bytes.Buffer
if _, err := flushBehind(ctx, open, mustNodes(t, open), map[string]bool{"spare": true}, compose, d, "", &said); err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(d.declared, []string{"anchor", "laptop"}) {
t.Fatalf("a consequence nothing holds was not sent: %v\n%s", d.declared, said.String())
}
if strings.Contains(said.String(), "was not sent") {
t.Fatalf("a machine nothing holds was said to be held:\n%s", said.String())
}
// A machine whose last send was not recorded — a declaration sent by hand — is held until named.
record, err := inv.NodeByName(ctx, "laptop")
if err != nil {
t.Fatal(err)
}
if err := inv.RecordSent(ctx, record.ID, "sent-by-hand", nil); err != nil {
t.Fatal(err)
}
d.declared = nil
said.Reset()
if _, err := flushBehind(ctx, open, mustNodes(t, open), map[string]bool{"spare": true}, compose, d, "", &said); err != nil {
t.Fatal(err)
}
// The anchor, the hub, may still be settling from the machine placed above; the laptop is the
// question.
if slices.Contains(d.declared, "laptop") || !strings.Contains(said.String(), "laptop is behind and was not sent") {
t.Fatalf("a machine whose last send is not known was sent: %v\n%s", d.declared, said.String())
}
}
+71
View File
@@ -6,6 +6,8 @@ import (
"sort"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/overlay"
)
// recordDerivedHolders writes down who holds each mesh-scoped seat that nobody was ever recorded
@@ -90,3 +92,72 @@ func recordDerivedHolders(ctx context.Context, open *stores) ([]string, error) {
}
return said, nil
}
// replicatedHolders is, for each replicated mesh seat, every machine on the private network holding
// it — by internal name, at its private address (novox/hq ADR 0223). What a machine's resolver file
// lists for `mesh-dns-resolver`; the rendering puts the machine itself first when it is one.
//
// **The holders on record, and only the sole claimant when there are none** — the same answer the
// resolver gives about who holds (ADR 0131, issue 170). An assignment standing beside the holders,
// eligible and silent, is not listed: it becomes a holder by `seat <name> --add`, an act, never by
// being assigned. Two claimants with nothing on record are refused at resolution, so neither is
// listed here. A holder off the private network is left out: a resolver named at an address nothing
// answers is a lookup that waits out its timeout on every name.
func replicatedHolders(ctx context.Context, inv *inventory.Inventory,
shelf map[string]catalogue.Manifest) (map[string]map[string]string, error) {
var replicated []catalogue.Seat
for _, s := range catalogue.Seats() {
if s.Replicated && s.Scope == catalogue.ScopeMesh {
replicated = append(replicated, s)
}
}
if len(replicated) == 0 {
return nil, nil
}
recorded, err := inv.Holdings(ctx)
if err != nil {
return nil, err
}
places, err := onTheNetwork(ctx, inv, shelf)
if err != nil {
return nil, err
}
address := map[string]string{}
for _, p := range places {
address[p.Name] = p.Address
}
entries, err := inv.Catalogued(ctx)
if err != nil {
return nil, err
}
out := map[string]map[string]string{}
for _, seat := range replicated {
var nodes []string
for _, h := range recorded {
if hs, ok := catalogue.SeatNamed(h.Claim); ok && hs.Name == seat.Name && h.Scope == seat.Scope {
nodes = append(nodes, h.Node)
}
}
if len(nodes) == 0 {
var derived []string
for _, e := range entries {
for _, c := range e.Manifest.Claims {
if cs, ok := catalogue.SeatNamed(c.Name); ok && cs.Name == seat.Name && c.At() == seat.Scope {
derived = append(derived, e.On...)
}
}
}
if len(derived) == 1 {
nodes = derived
}
}
at := map[string]string{}
for _, n := range nodes {
if address[n] != "" {
at[overlay.InternalName(n)] = address[n]
}
}
out[seat.Name] = at
}
return out, nil
}
+150
View File
@@ -0,0 +1,150 @@
package main
import (
"bytes"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// novox/hq ADR 0225, issue 263: a consumer's identity is bounded by the provision it requires, an
// overflow is refused before merge by `module check`, and a provider's machine is never refused for
// one consumer's identity.
// `module check` refuses the pull request that introduces an overflow, naming the module.
func TestModuleCheckRefusesAnIdentityThatOverflowsWhatItRequires(t *testing.T) {
dir := t.TempDir()
write := func(name, body string) string {
p := filepath.Join(dir, name+".json")
if err := os.WriteFile(p, []byte(body), 0o600); err != nil {
t.Fatal(err)
}
return p
}
objects := write("objects", `{"module":"objects","version":"1",
"provides":[{"name":"s3-bucket","scope":"mesh","identity":{"max":20,"in":"an S3 access key"}}],
"receives":{"s3-bucket":"/var/lib/mesh/objects/mesh.json"}}`)
resolver := write("resolver", `{"module":"resolver","version":"1",
"provides":[{"name":"wildcard-resolution","scope":"mesh","identity":false}]}`)
album := write("photoalbum", `{"module":"photoalbum","version":"1","requires":["s3-bucket"]}`)
nm := write("networkmanager", `{"module":"networkmanager","version":"1","requires":["wildcard-resolution"]}`)
var out bytes.Buffer
if err := moduleCheckFor([]string{resolver, nm}, 6, &out); err != nil {
t.Fatalf("a long name requiring a keyless provision was refused (issue 263): %v\n%s", err, out.String())
}
out.Reset()
err := moduleCheckFor([]string{objects, album, resolver, nm}, 6, &out)
if err == nil {
t.Fatalf("an identity overflowing an S3 access key passed:\n%s", out.String())
}
if !strings.Contains(out.String(), "photoalbum wants s3-bucket") ||
!strings.Contains(out.String(), "`slug` of at most 8 characters") ||
strings.Contains(out.String(), "networkmanager wants") {
t.Fatalf("the refusal does not name the one overflowing module and its remedy:\n%s", out.String())
}
}
// Tonight's case, through the commands: networkmanager on a six-character machine requires the
// resolver provision, and a second consumer there overflows an object store's access key. The
// provider's machine still composes; the overflowing consumer is left out of its grants and named,
// by push and by `status`, and the keyless consumer is granted with its long name.
func TestAnOverflowingConsumerNeverRefusesItsProvidersMachine(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, catalogue.Manifest{Module: "objects", Version: "1",
Provides: []catalogue.Offer{{Name: "s3-bucket", Scope: catalogue.ScopeMesh,
Identity: &catalogue.OfferIdentity{Max: 20, In: "an S3 access key"}}},
Receives: map[string]string{"s3-bucket": "/var/lib/mesh/objects/mesh.json"}})
register(t, open, catalogue.Manifest{Module: "resolver", Version: "1",
Provides: []catalogue.Offer{{Name: "wildcard-resolution", Scope: catalogue.ScopeMesh}}})
register(t, open, catalogue.Manifest{Module: "networkmanager", Version: "1",
Requires: []string{"wildcard-resolution"}})
register(t, open, catalogue.Manifest{Module: "photoalbum", Version: "1", Requires: []string{"s3-bucket"}})
register(t, open, catalogue.Manifest{Module: "files", Version: "1", Requires: []string{"s3-bucket"}})
for _, a := range [][2]string{{"anchor", "objects"}, {"anchor", "resolver"},
{"laptop", "networkmanager"}, {"laptop", "photoalbum"}, {"laptop", "files"}} {
if _, err := assign(ctx, open, a[0], a[1]); err != nil {
t.Fatalf("assign %s %s: %v", a[0], a[1], err)
}
}
// The consumer's machine resolves, and says which of its modules no provider will grant.
consumer, _, err := planFor(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
over := consumer.Overflowing()
if len(over) != 1 || over[0].Module != "photoalbum" || over[0].Provision != "s3-bucket" {
t.Fatalf("the consumer's side does not name exactly photoalbum: %+v", over)
}
// The provider's machine composes. Under ADR 0049's one bound this was a refusal naming
// networkmanager, and no push to the provider could go through.
plan, settings, err := planFor(ctx, open, "anchor")
if err != nil {
t.Fatal(err)
}
declared, err := declarationFor(ctx, open, "anchor", plan, settings)
if err != nil {
t.Fatalf("one consumer's identity refused its provider's whole machine: %v", err)
}
if len(declared.withheld) != 1 || declared.withheld[0].Identity != "mesh_laptop_photoalbum" {
t.Fatalf("the overflowing consumer is not the one withheld: %+v", declared.withheld)
}
grants, _, _, err := grantsFor(ctx, open, "anchor")
if err != nil {
t.Fatal(err)
}
var keyless bool
for _, g := range grants {
keyless = keyless || g.Provision == "wildcard-resolution" && g.From == "networkmanager"
}
if !keyless {
t.Fatalf("networkmanager, 26 characters, is not granted the keyless resolver provision: %+v", grants)
}
var granted []string
for _, c := range declared.Received["objects"]["s3-bucket"] {
granted = append(granted, c.From)
}
if strings.Join(granted, ",") != "files" {
t.Fatalf("the object store grants %v; files and only files fit", granted)
}
said := printed(t, func() error { reportLeftOut("anchor", declared); return nil })
if !strings.Contains(said, `photoalbum on laptop requires s3-bucket from anchor`) ||
!strings.Contains(said, "left out of anchor's grants") {
t.Fatalf("the push does not say whom it leaves out:\n%s", said)
}
// And `status` names it, and does not call the mesh well while it stands.
asked, err := theThreeQuestions(ctx, open)
if err != nil {
t.Fatal(err)
}
if len(asked.overflowing) != 1 || asked.overflowing[0].Module != "photoalbum" {
t.Fatalf("status does not carry the overflow: %+v", asked.overflowing)
}
if asked.well() {
t.Fatal("a mesh with a consumer left out of its grants reads as well")
}
shown := printed(t, func() error { return printStatus(asked) })
if !strings.Contains(shown, "identified too long for a provision they require") ||
!strings.Contains(shown, "mesh_laptop_photoalbum") {
t.Fatalf("status does not say it:\n%s", shown)
}
body, err := statusAsJSON(asked)
if err != nil {
t.Fatal(err)
}
var doc struct {
Overflowing []catalogue.Overflow `json:"overflowing"`
}
if err := json.Unmarshal(body, &doc); err != nil || len(doc.Overflowing) != 1 ||
doc.Overflowing[0].Bound.Max != 20 {
t.Fatalf("the document does not carry it: %v\n%s", err, body)
}
}
+4 -4
View File
@@ -45,11 +45,11 @@ func TestADeclarationComposedEarlierIsNumberedLowerWhateverOrderItIsSent(t *test
// fails leaves nothing composed for that machine, and the others are still composed.
func TestTheNumberIsTakenBeforeComposingAndItsFailureIsARefusal(t *testing.T) {
calls := 0
allot := func(node string) (int64, error) {
allot := func(node string) (order, error) {
if node == "anchor" {
return 0, context.DeadlineExceeded
return order{}, context.DeadlineExceeded
}
return 7, nil
return order{sequence: 7}, nil
}
sending, refusals := composeEach([]string{"anchor", "laptop"}, allot, func(node string) (sendable, error) {
calls++
@@ -77,7 +77,7 @@ func TestASendIsRecordedEvenWhenTheSenderIsBeingCancelled(t *testing.T) {
}
cancel() // the sender is going away: its context is cancelled between the send and the record
body := []byte(`{"declaration":1,"resources":[]}`)
digest, err := recordSent(ctx, inv, "anchor", body)
digest, err := recordSent(ctx, inv, "anchor", body, nil, 0)
if err != nil {
// NodeByName on the cancelled context may itself refuse; the record must still be possible
// through the detached context, so look the node up again on a live one.
+68
View File
@@ -0,0 +1,68 @@
package main
import (
"testing"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/lease"
)
// A command run at a shell (novox/hq to-be 45 §6): under the holder's epoch while a controller holds the
// lease, read at the moment it acts; under a lease of its own while none does, given back as it ends.
func TestACommandActsUnderTheHoldersEpochOrItsOwn(t *testing.T) {
url, kv := aBusForTheLease(t)
t.Setenv(broker.NATSVar, url)
ctx := t.Context()
// Nobody holds it: the command takes it, and gives it back.
cmd := &actor{}
own, err := cmd.epoch(ctx)
if err != nil || own == 0 {
t.Fatalf("a command with nobody holding the lease acts as %d (%v)", own, err)
}
if h, found, _ := lease.Current(ctx, kv); !found || h.Epoch != own {
t.Fatalf("the command's lease is not on the bus: %+v", h)
}
cmd.release()
if _, found, _ := lease.Current(ctx, kv); found {
t.Fatal("the command did not give its lease back as it ended")
}
// A controller holds it: a command acts under that epoch.
l, err := lease.Open(ctx, mustJetStream(t, url), broker.LeaseBucket, lease.Options{Holder: lease.Holder{Instance: "serving"}})
if err != nil {
t.Fatal(err)
}
held, err := l.TryTake(ctx)
if err != nil {
t.Fatal(err)
}
borrower := &actor{}
defer borrower.release()
if got, err := borrower.epoch(ctx); err != nil || got != held {
t.Fatalf("a command acts as %d (%v), want the holder's %d", got, err, held)
}
// The holder lets go: the command does not go on under an epoch nobody holds.
l.Release(ctx)
if _, err := borrower.epoch(ctx); err == nil {
t.Fatal("a command acted under an epoch nobody holds any more")
}
}
// mustJetStream is a connection of its own to the test bus.
func mustJetStream(t *testing.T, url string) jetstream.JetStream {
t.Helper()
conn, err := nats.Connect(url)
if err != nil {
t.Fatal(err)
}
t.Cleanup(conn.Close)
js, err := jetstream.New(conn)
if err != nil {
t.Fatal(err)
}
return js
}
+175
View File
@@ -0,0 +1,175 @@
package main
import (
"context"
"errors"
"os"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/lease"
)
// Two controllers at once (novox/hq to-be 45 §6, issue 204; the half of replay R1 that lives here): two
// serving controllers over one store and one bus. The second waits while the first holds the lease; on
// a handover it takes it at a higher epoch, the record says which held what and how each ended; and the
// one that lost it acts no more — no declaration composed, no plan and no condition written — the
// moment it lost it.
//
// MESH_TEST_POSTGRES=… MESH_TEST_NATS=nats://127.0.0.1:14222 go test ./cmd/mesh-controller/ -run Controllers
// aBusForTheLease is the test bus with the controller's lease bucket new.
func aBusForTheLease(t *testing.T) (string, jetstream.KeyValue) {
t.Helper()
url := os.Getenv("MESH_TEST_NATS")
if url == "" {
t.Skip("MESH_TEST_NATS unset")
}
conn, err := nats.Connect(url)
if err != nil {
t.Fatal(err)
}
t.Cleanup(conn.Close)
js, err := jetstream.New(conn)
if err != nil {
t.Fatal(err)
}
_ = js.DeleteKeyValue(t.Context(), broker.LeaseBucket)
if err := broker.EnsureLeaseBucket(t.Context(), js); err != nil {
t.Fatal(err)
}
kv, err := js.KeyValue(t.Context(), broker.LeaseBucket)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = js.DeleteKeyValue(context.Background(), broker.LeaseBucket) })
return url, kv
}
func TestTwoControllersOneActs(t *testing.T) {
url, kv := aBusForTheLease(t)
inv := inventory.ForTest(t)
ctx := t.Context()
// Controller A takes the lease.
a := &actor{}
aCtx, stopA := context.WithCancel(ctx)
defer stopA()
lostA, err := a.serveUnderTheLease(aCtx, inv, url)
if err != nil {
t.Fatal(err)
}
epochA, err := a.epoch(ctx)
if err != nil || epochA == 0 {
t.Fatalf("A holds no epoch: %d, %v", epochA, err)
}
// Controller B starts while A holds it, and waits — acting on nothing meanwhile.
b := &actor{}
bCtx, stopB := context.WithCancel(ctx)
defer stopB()
tookB := make(chan (<-chan struct{}), 1)
go func() {
lost, err := b.serveUnderTheLease(bCtx, inv, url)
if err != nil {
t.Errorf("B: %v", err)
close(tookB)
return
}
tookB <- lost
}()
select {
case <-tookB:
t.Fatal("B took the lease while A held it")
case <-time.After(3 * time.Second):
}
if _, err := b.epoch(ctx); !errors.Is(err, lease.ErrNotHeld) {
t.Fatalf("B, waiting, may act: %v", err)
}
// A hands over, as a controller being replaced does: B takes the lease at once, at a higher epoch.
stopA()
a.release()
var lostB <-chan struct{}
select {
case lostB = <-tookB:
case <-time.After(10 * time.Second):
t.Fatal("B did not take the lease A gave back")
}
select {
case <-lostA:
default:
t.Fatal("A, having given the lease back, is not told it no longer holds it")
}
epochB, err := b.epoch(ctx)
if err != nil || epochB <= epochA {
t.Fatalf("B acts as epoch %d after A's %d (%v): an epoch only grows", epochB, epochA, err)
}
if _, err := a.epoch(ctx); err == nil {
t.Fatal("A acts after giving the lease back")
}
ea, _, _ := inv.EpochOf(ctx, epochA)
eb, _, _ := inv.EpochOf(ctx, epochB)
if ea.How != inventory.EpochReleased || eb.Ended != nil || eb.Instance != instance {
t.Fatalf("the record of the handover reads %+v then %+v", ea, eb)
}
// Something else writes the lease's key — a third controller on a clock that read it as expired:
// B's next renewal is refused, and B stops acting at once.
if _, err := kv.Put(ctx, lease.Key, []byte(`{"instance":"a third controller","epoch":1}`)); err != nil {
t.Fatal(err)
}
select {
case <-lostB:
case <-time.After(2 * lease.RenewEvery):
t.Fatal("B was not told it lost the lease")
}
if _, err := b.epoch(ctx); !errors.Is(err, lease.ErrNotHeld) {
t.Fatalf("B acts after losing the lease: %v", err)
}
// Nothing B does is written: a declaration's number is not taken, a plan is not saved, a condition
// is not raised.
inv.ActsUnder(b.epoch)
if _, err := inv.AddNode(ctx, "anchor"); err != nil {
t.Fatal(err)
}
saved := inventory.Plan{ID: "plan-after-loss", Repository: "novox/app", Commit: "c0ffee00", Created: time.Now(),
State: inventory.PlanBuilding}
if err := inv.SavePlan(ctx, &saved); err == nil {
t.Fatal("B wrote a plan after losing the lease")
}
store := conditions.NewInMemory()
keeper := conditions.NewKeeper(ctx, conditions.Options{Store: store, History: store,
Epoch: func() (uint64, error) { return b.epoch(ctx) }})
defer keeper.Close(context.Background())
if _, err := keeper.Observe(ctx, conditions.Observation{Scope: conditions.ScopeCore, ID: "x", Kind: "x",
Severity: conditions.Warning, Summary: "x", Source: "test"}); err == nil {
t.Fatal("B raised a condition after losing the lease")
}
if ended, _, _ := inv.EpochOf(ctx, epochB); ended.How != inventory.EpochLost {
t.Fatalf("B's epoch does not say it was lost: %+v", ended)
}
}
// A controller whose bus refuses it the lease's key, with nobody holding it, serves without the lease:
// it acts with no epoch — refused by no node-engine — says so, and takes the lease once it can.
func TestAControllerTheBusRefusesTheLeaseServesUnleasedAndSaysSo(t *testing.T) {
a := &actor{serving: true, unleased: "the bus refused the lease's key"}
if epoch, err := a.epoch(context.Background()); err != nil || epoch != 0 {
t.Fatalf("an unleased controller answers %d, %v: it acts, claiming no epoch", epoch, err)
}
if st := a.standing(); st.Unleased == "" || st.Held {
t.Fatalf("its standing says %+v", st)
}
// One that neither holds nor is unleased — still waiting — acts on nothing.
waiting := &actor{serving: true}
if _, err := waiting.epoch(context.Background()); !errors.Is(err, lease.ErrNotHeld) {
t.Fatalf("a controller waiting for the lease may act: %v", err)
}
}
+57 -2
View File
@@ -55,6 +55,9 @@ func run() error {
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
// Whatever this process holds of the controller's lease is given back as it ends (novox/hq to-be
// 45 §6), so the next controller takes it at once rather than after its age.
defer theLease.release()
switch args[0] {
case "build":
@@ -145,6 +148,31 @@ func run() error {
return seatCommand(ctx, args[1:])
case "status":
return statusCommand(ctx, args[1:])
// Acts done by hand, and why (novox/hq to-be 45 §7).
case "hand-act":
return handActCommand(ctx, args[1:])
case "hand-acts":
return handActCommand(ctx, append([]string{"list"}, args[1:]...))
// What the mesh's bounds will be set from (novox/hq to-be 45 Phase 0).
case "durations":
return durationsCommand(ctx, args[1:])
// What is wrong, and the self-check (novox/hq to-be 45 §2, §4).
case "conditions":
return conditionsCommand(ctx, args[1:])
case "doctor":
return doctorCommand(ctx, args[1:])
// What the healers did, and their brake (novox/hq to-be 45 §7).
case "healers":
return healersCommand(ctx, args[1:])
// A consumer the mesh stopped asking for: retired, waiting for a person, deleted only by one
// (novox/hq ADR 0230).
case "retire":
return retireCommand(ctx, args[1:])
case "cleanup":
return cleanupCommand(ctx, args[1:])
// The data every machine declares, as the self-check last found it (novox/hq ADR 0233).
case "data":
return dataCommand(ctx, args[1:])
case "version":
fmt.Println(version)
return nil
@@ -191,9 +219,16 @@ func usage() {
upgrade <name> roll-out [--together] ...send it to the machines running it
upgrade <name> record ...record that they are behind, and send nothing
status [--json] what is wrong, what is quiet, and what is out of date
retire [--json] every provider waiting for a person to approve a retirement (ADR 0230)
retire approve <node> <module> --why <text> retire what it waits with: access off, data kept
retire reject <node> <module> --why <text> keep them active; a warning stays open
cleanup [list] [--json] every retired consumer per provider: age, size, why
cleanup delete <node> <module> <consumer> --why <text> the provider deletes that one retired consumer
cleanup delete --older-than <days> --why <text> [--confirm] list those older; delete only with --confirm
seats [--json] every seat this mesh defines, what it delivers, and who holds it
seat rename <from> <to> rename a seat; its former name still resolves (ADR 0122)
seat <name> --to <node>/<module> hand a seat to that assignment as one act; never empty in between (ADR 0131)
seat <name> --add <node>/<module> add a holder beside the others, for a replicated seat (ADR 0223)
board [--listen ADDR] the same three questions, as a page that holds nothing
api --issuer URL [--listen A] assign and unassign over http, for a surface that is not here
assign <node> <module>... put modules on a node, judged together (ADR 0207)
@@ -225,19 +260,34 @@ func usage() {
kill <id> end a build where it runs; recorded failed, killed by hand
pause [<node>] / resume [<node>] the build seat's holder there, or every holder, takes nothing new / again
plans retry <id> ask a failed plan's failed builds again, and carry the plan on
plans stop|close <id> --why <text> end a plan by hand; recorded in the hand-act log
hand-act record <what> --why <text> --cause <word> [--condition <key>]
record an act done by hand outside the mesh (to-be 45 §7)
hand-acts [--days N] [--json] what was done by hand lately, why, and which causes repeat
conditions [--scope S] [--severity S] [--machine M] [--json]
what is wrong now: every open condition, urgent first (to-be 45 §2)
conditions show <key> one condition whole, with its evidence
conditions silence <key> --for <d> --why <text> send no message for it a while; a hand act
conditions history [--days N] [--key K] every raising, change and clearing lately
doctor [run|probes|signals] [--json]
the self-check: the last verdict, a run now, the probes, the signals' ages
healers [--days N] [--json] the healers, what they did lately, and their brake (to-be 45 §7)
durations [--kind K] [--days N] [--json]
apply, heartbeat, plan-tier and build durations, per machine or module
collection [--json] kept archives held/unheld by a manifest, and what the sweep may let go
builder issue <name> a broker account for a build machine, scoped to build work,
delivered as the builder module's broker secret (module add it first)
licence add|list|use|key model access, under the name a person calls it
licence manager <name> <node> the node that holds a refreshable licence's refresh token
licence refresh <name> mint a new access token and seal it to every holder
rotate <provision> [--consumer <n>] a new credential for every holder, both ends at once
rotate <provision> [--consumer <n>] [--module <m>] a new credential for every holder, both ends at once
ask <module> <tool> [json] call one of a module's tools over the broker, and print its answer
pin <node> <provision> <from-node> <module>
which provider this one gets a provision from: the module, and its node
unpin <node> <provision> put that question back
plan <node> [--files|--json] what that node would run, and why
push [<node>] [--behind] send a node everything it should be, or only those that need it
push [<node>] [--behind] [--why <text>] send a node everything it should be, or only those
that need it; --why records it in the hand-act log
version what this binary is
Each context reaches its own store through its own credential (novox/hq ADR 0008), named
@@ -290,6 +340,9 @@ func (b builds) Built(ctx context.Context, result link.BuildResult) error {
// When it was asked, so a plan takes as its outcome only a build asked for it or after it
// (novox/hq 04-ISSUES/219). Zero when the id does not say.
asked, _ := link.BuildAskedAt(result.ID)
// And how long it took, asked to heard, which a build's bound will be set from (novox/hq to-be 45
// Phase 0). Said if lost; never a reason not to take the build in.
recordBuildDuration(ctx, b.inv, result, asked)
switch {
case err != nil && result.Failed != "":
fmt.Printf("%s: %v\n", result.ID, err)
@@ -316,5 +369,7 @@ func (b builds) Built(ctx context.Context, result link.BuildResult) error {
result.ID, manifest.Module, manifest.Version, result.On, short(result.Commit))
saysWhenThePolicyActs(ctx, b.inv, manifest.Module)
planBuilt(ctx, b.open, manifest.Module, result.Commit, "", asked, result.ID)
// A module registered may be one a machine is now behind: `status` is composed again.
statusFrom.nudge()
return nil
}
+20
View File
@@ -1,11 +1,14 @@
package main
import (
"context"
"crypto/ecdh"
"crypto/rand"
"encoding/base64"
"encoding/json"
"fmt"
"github.com/novox/mesh-controller/internal/conditions"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
@@ -37,6 +40,9 @@ func aMesh(t *testing.T) *stores {
t.Fatal(err)
}
t.Cleanup(open.Close)
// A condition store of its own, held in memory (novox/hq to-be 45 §2): status leads with what is
// open, and a mesh with no bus would otherwise read as one whose conditions cannot be read.
withConditionsInMemory(t)
for _, m := range provided {
if err := open.inventory.Provide(t.Context(), m); err != nil {
t.Fatal(err)
@@ -106,3 +112,17 @@ func rivals() (catalogue.Manifest, catalogue.Manifest) {
return catalogue.Manifest{Module: "rival-one", Version: "1", Claims: claim},
catalogue.Manifest{Module: "rival-two", Version: "1", Claims: claim}
}
// withConditionsInMemory gives the test a condition store in memory, as the serving controller's.
func withConditionsInMemory(t *testing.T) (*conditions.Keeper, *conditions.InMemory) {
t.Helper()
store := conditions.NewInMemory()
k := conditions.NewKeeper(t.Context(), conditions.Options{Store: store, History: store})
before := conditionsFrom
conditionsFrom = k
t.Cleanup(func() {
conditionsFrom = before
k.Close(context.Background())
})
return k, store
}
+189
View File
@@ -0,0 +1,189 @@
package main
import (
"context"
"fmt"
"sync"
"time"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// **A merge the bus announced and the controller never acted on is caught up** (novox/hq issue 266).
//
// The forge's poll announces every merge on the events stream, and the controller acts on what its
// consumer there hands it. On 2026-10-06 one merge was on the stream and never handed over: the bus
// server moved the consumer past it — a fault of consumers with several filters in the server the
// mesh ran — and the controller, which only acts on what it is handed, said nothing. The modules
// built from that repository stayed behind and were built by hand.
//
// So the stream is read back on a timer, on a consumer of its own filtered on merges alone, and every
// announcement older than mergeGrace is judged as SourceMoved would judge it. **No record of what
// was handled is kept, because none is needed**: acting on a merge marks every module it moved as
// looked at since, so an announcement already acted on reads as history and moves nothing. One that
// would still move something was never acted on — it is said, and acted on now.
const (
// mergeGrace is how long an announcement is left to the controller's own consumer before it is
// judged missed. That consumer hands over one event at a time, and a merge waits behind a build
// outcome that is being acted on; acting on a merge itself asks builds and does not wait for them.
mergeGrace = 10 * time.Minute
// mergeLookBack is how far back a pass reads. A merge missed longer ago than this was missed by a
// controller that was not running this, and is the operator's to look at, not a surprise rebuild.
mergeLookBack = 24 * time.Hour
// mergeCatchUpEvery is how often the stream is read back.
mergeCatchUpEvery = 5 * time.Minute
)
// merges is what reads back the forge's announcements; the link server, or a test's list.
type merges interface {
AnnouncedMerges(ctx context.Context, since time.Time) ([]link.AnnouncedMerge, error)
}
// catchingUpOnMerges reads back the forge's announcements on a timer, until the context ends.
func catchingUpOnMerges(ctx context.Context, open *stores, announced merges) {
f := following{open}
catalogued := func(ctx context.Context) ([]inventory.Entry, map[string][]inventory.ReadRepository, error) {
entries, err := open.inventory.Catalogued(ctx)
if err != nil {
return nil, nil, err
}
read, err := open.inventory.ReadRepositories(ctx)
return entries, read, err
}
failing := ""
tick := time.NewTicker(mergeCatchUpEvery)
defer tick.Stop()
for {
select {
case <-ctx.Done():
return
case <-tick.C:
}
watchedMerges.begin()
err := catchUpOnMerges(ctx, time.Now(), announced, catalogued, f.SourceMoved, func(format string, args ...any) {
fmt.Printf(format+"\n", args...)
})
// What the pass found is what S5 says (novox/hq to-be 45 §3); a pass that could not read
// says that instead, and leaves what the last one found standing.
watchedMerges.end(time.Now(), err)
// A pass that cannot read says so once, not every five minutes, and says when it reads again.
why := ""
if err != nil {
why = err.Error()
}
if why != failing {
if why != "" {
fmt.Printf("merges the bus may not have handed over cannot be looked for: %s\n", why)
} else {
fmt.Println("merges the bus may not have handed over are looked for again")
}
failing = why
}
}
}
// catchUpOnMerges is one pass: every announcement older than mergeGrace that acting on would still
// move something is said and acted on, oldest first.
//
// Judged twice: once against the catalogue as the pass found it, and again just before acting,
// because acting on an earlier missed merge of the same repository may have moved what a later one
// would have.
func catchUpOnMerges(ctx context.Context, now time.Time, announced merges,
catalogued func(context.Context) ([]inventory.Entry, map[string][]inventory.ReadRepository, error),
act func(context.Context, link.SourceMoved) error, say func(string, ...any)) error {
all, err := announced.AnnouncedMerges(ctx, now.Add(-mergeLookBack))
if err != nil {
return err
}
entries, read, err := catalogued(ctx)
if err != nil {
return err
}
for _, a := range all {
if now.Sub(a.At) < mergeGrace {
continue
}
if len(wouldMove(a.SourceMoved, entries, read)) == 0 {
continue
}
if entries, read, err = catalogued(ctx); err != nil {
return err
}
moves := wouldMove(a.SourceMoved, entries, read)
if len(moves) == 0 {
continue
}
var names []string
for _, e := range moves {
names = append(names, e.Manifest.Module)
}
watchedMerges.found(missedMerge{Owner: a.Owner, Repo: a.Repo, Base: a.Base, Commit: a.Commit,
At: a.At, Modules: names})
say("%s/%s merged into %s (%.8s), announced %s ago, and the controller never acted on it: the bus "+
"did not hand the announcement over (novox/hq issue 266). %s %s behind it; acting on it now",
a.Owner, a.Repo, a.Base, a.Commit, now.Sub(a.At).Round(time.Minute), readableList(names),
isAre(len(names)))
if err := act(ctx, a.SourceMoved); err != nil {
say("%s/%s moved to %.8s and the mesh could not act on it: %v; the next pass tries again",
a.Owner, a.Repo, a.Commit, err)
continue
}
if entries, read, err = catalogued(ctx); err != nil {
return err
}
}
return nil
}
// missedMerge is one merge the bus announced and never handed over, as a pass found it.
type missedMerge struct {
Owner, Repo, Base, Commit string
// At is when the bus took the announcement.
At time.Time
Modules []string
}
// mergeWatch is what the passes found, for S5 (novox/hq to-be 45 §3): **a merge nothing read is
// said**, urgent, even though the pass acts on it at once — the bus skipping a message is a fault of
// the transport the mesh's every change rides on, and acting late is the repair, not the absence of
// the fault. The next pass, finding it acted on, clears it.
type mergeWatch struct {
mu sync.Mutex
passed time.Time
err error
finding []missedMerge
missed []missedMerge
}
var watchedMerges = &mergeWatch{}
func (w *mergeWatch) begin() {
w.mu.Lock()
defer w.mu.Unlock()
w.finding = nil
}
func (w *mergeWatch) found(m missedMerge) {
w.mu.Lock()
defer w.mu.Unlock()
w.finding = append(w.finding, m)
}
func (w *mergeWatch) end(at time.Time, err error) {
w.mu.Lock()
defer w.mu.Unlock()
w.passed, w.err = at, err
if err == nil {
w.missed = w.finding
}
}
// last is when the last pass ended, what the last pass that read found, and what the last pass
// could not read.
func (w *mergeWatch) last() (time.Time, []missedMerge, error) {
w.mu.Lock()
defer w.mu.Unlock()
return w.passed, append([]missedMerge(nil), w.missed...), w.err
}
+180
View File
@@ -0,0 +1,180 @@
package main
import (
"context"
"errors"
"fmt"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// announcedList is the events stream's merges as a test gives them.
type announcedList []link.AnnouncedMerge
func (a announcedList) AnnouncedMerges(_ context.Context, since time.Time) ([]link.AnnouncedMerge, error) {
var out []link.AnnouncedMerge
for _, m := range a {
if !m.At.Before(since) {
out = append(out, m)
}
}
return out, nil
}
// aCatalogue is what the inventory holds, changed the way acting on a merge changes it: every module
// the merge moved is marked as looked at (inventory.SourceMoved writes source_seen = now()).
type aCatalogue struct {
entries []inventory.Entry
acted []string
fail error
}
func (c *aCatalogue) read(context.Context) ([]inventory.Entry, map[string][]inventory.ReadRepository, error) {
return append([]inventory.Entry(nil), c.entries...), nil, nil
}
func (c *aCatalogue) act(now func() time.Time) func(context.Context, link.SourceMoved) error {
return func(_ context.Context, m link.SourceMoved) error {
if c.fail != nil {
return c.fail
}
c.acted = append(c.acted, m.Repo+"@"+m.Commit[:8])
for _, moved := range wouldMove(m, c.entries, nil) {
for i := range c.entries {
if c.entries[i].Manifest.Module == moved.Manifest.Module {
c.entries[i].Source.Head = m.Commit
c.entries[i].Source.Seen = now()
}
}
}
return nil
}
}
func at(s string) time.Time {
t, err := time.Parse(time.RFC3339, s)
if err != nil {
panic(err)
}
return t
}
func announced(repo, commit, mergedAt, onTheBus string, paths ...string) link.AnnouncedMerge {
return link.AnnouncedMerge{
SourceMoved: link.SourceMoved{Owner: "novox", Repo: repo, Base: "main", Commit: commit,
MergedAt: mergedAt, Paths: paths,
CloneURL: "http://forge.internal:20000/novox/" + repo + ".git"},
At: at(onTheBus),
}
}
func built(module, repo, path, commit, seen string) inventory.Entry {
e := fromRepo(module, "http://forge.internal:20000/novox/"+repo+".git", path)
e.Source.BuiltFrom, e.Source.Head, e.Source.Seen = commit, commit, at(seen)
return e
}
// **novox/hq issue 266, as it happened.** The forge announced a merge of the tools repository on the
// events stream; the bus never handed it to the controller, which acted on the merges around it and
// not on this one, and said nothing. Read back from the stream, it is the one merge that would still
// move something — so it is said and acted on, once, and only after the controller's own consumer
// has had its time with it.
func TestAMergeTheBusNeverHandedOverIsActedOnLate(t *testing.T) {
cat := &aCatalogue{entries: []inventory.Entry{
built("mesh-tools", "mesh-tools", "", "8b789578aaaaaaaa", "2026-10-04T15:24:32Z"),
built("node-tools", "mesh-tools", "node-tools", "8b789578aaaaaaaa", "2026-10-04T15:24:32Z"),
// Acted on when it was announced: looked at after it was merged.
built("gitea", "mesh-catalog", "modules/gitea", "5c2157b8bbbbbbbb", "2026-10-05T22:39:21Z"),
}}
stream := announcedList{
// Nothing the mesh holds is built from the records repository.
announced("hq", "88f7f79fcccccccc", "2026-10-05T22:43:00Z", "2026-10-05T22:43:04Z", "04-ISSUES/x.md"),
// Acted on: its module was looked at since.
announced("mesh-catalog", "78328d4adddddddd", "2026-10-05T22:39:00Z", "2026-10-05T22:39:21Z", "modules/gitea/x.ts"),
// Never handed over.
announced("mesh-tools", "9730bd89c3e48d0e", "2026-10-05T22:46:47Z", "2026-10-05T22:47:06Z",
"node-tools/internal/console/console.go"),
}
var said []string
say := func(format string, args ...any) { said = append(said, fmt.Sprintf(format, args...)) }
clock := at("2026-10-05T22:50:00Z")
now := func() time.Time { return clock }
pass := func() {
t.Helper()
if err := catchUpOnMerges(context.Background(), clock, stream, cat.read, cat.act(now), say); err != nil {
t.Fatal(err)
}
}
pass()
if len(cat.acted) != 0 {
t.Fatalf("a merge three minutes old was taken from the controller's own consumer: %v", cat.acted)
}
clock = at("2026-10-05T22:58:00Z")
pass()
if strings.Join(cat.acted, ",") != "mesh-tools@9730bd89" {
t.Fatalf("acted on %v, wanted the one merge never handed over", cat.acted)
}
if len(said) != 1 || !strings.Contains(said[0], "novox/mesh-tools merged into main (9730bd89)") ||
!strings.Contains(said[0], "mesh-tools and node-tools are behind it") {
t.Fatalf("the missed merge was not said as one: %q", said)
}
clock = at("2026-10-05T23:03:00Z")
pass()
if len(cat.acted) != 1 || len(said) != 1 {
t.Fatalf("a merge acted on was acted on again: %v %q", cat.acted, said)
}
}
// A merge that changed none of the held modules' files moves nothing, so it is never "missed"; one
// that could not be acted on is said and tried again on the next pass.
func TestAMissedMergeThatCouldNotBeActedOnIsTriedAgain(t *testing.T) {
cat := &aCatalogue{
entries: []inventory.Entry{built("gitea", "mesh-catalog", "modules/gitea", "5c2157b8bbbbbbbb", "2026-10-05T20:00:00Z")},
fail: errors.New("the store is restarting"),
}
stream := announcedList{
announced("mesh-catalog", "aaaaaaaa11111111", "2026-10-05T21:00:00Z", "2026-10-05T21:00:10Z",
"modules/plex/module.json", "modules/plex/x.ts"),
announced("mesh-catalog", "bbbbbbbb22222222", "2026-10-05T21:10:00Z", "2026-10-05T21:10:10Z", "modules/gitea/x.ts"),
}
var said []string
say := func(format string, args ...any) { said = append(said, fmt.Sprintf(format, args...)) }
clock := at("2026-10-05T22:00:00Z")
now := func() time.Time { return clock }
if err := catchUpOnMerges(context.Background(), clock, stream, cat.read, cat.act(now), say); err != nil {
t.Fatal(err)
}
if len(said) != 2 || !strings.Contains(said[1], "could not act on it") {
t.Fatalf("a failed catch-up was not said: %q", said)
}
cat.fail = nil
clock = at("2026-10-05T22:05:00Z")
if err := catchUpOnMerges(context.Background(), clock, stream, cat.read, cat.act(now), say); err != nil {
t.Fatal(err)
}
if strings.Join(cat.acted, ",") != "mesh-catalog@bbbbbbbb" {
t.Fatalf("acted on %v, wanted only the merge that changed a held module", cat.acted)
}
}
// A merge older than the look-back is left to the operator: a controller that did not run this
// missed it, and acting on it days later would be a surprise rebuild.
func TestAMergeOlderThanTheLookBackIsLeftAlone(t *testing.T) {
cat := &aCatalogue{entries: []inventory.Entry{built("gitea", "mesh-catalog", "modules/gitea", "5c2157b8bbbbbbbb", "2026-10-01T00:00:00Z")}}
stream := announcedList{announced("mesh-catalog", "cccccccc33333333", "2026-10-03T00:00:00Z", "2026-10-03T00:00:05Z", "modules/gitea/x.ts")}
clock := at("2026-10-05T22:00:00Z")
if err := catchUpOnMerges(context.Background(), clock, stream, cat.read, cat.act(func() time.Time { return clock }),
func(string, ...any) {}); err != nil {
t.Fatal(err)
}
if len(cat.acted) != 0 {
t.Fatalf("a merge of three days ago was acted on: %v", cat.acted)
}
}
+22 -6
View File
@@ -40,7 +40,12 @@ func providedModules() []catalogue.Manifest {
// and neither could be swapped for anything, which is the test of whether a thing is a
// module at all (novox/hq ADR 0040). They existed because computed output needed somewhere
// to live, and now a module says where it wants it — `facts` in its own manifest.
overlay.Manifest(), overlay.DomainManifest(),
//
// **And the bundle that required it is gone** (novox/hq ADR 0226): `networking` named this
// module's requirement and nothing else, so every machine carried two modules for one
// network. A machine is assigned the private network itself; what a release stops shipping
// is retired at the next start (stores.go, Inventory.RetireUnshipped).
overlay.Manifest(),
} {
var m catalogue.Manifest
b, _ := json.Marshal(raw)
@@ -59,8 +64,19 @@ func moduleCommand(ctx context.Context, args []string) error {
// `check` needs no mesh, and must not: it is what somebody runs in their own repository before
// there is a mesh in reach (novox/hq issue 148). A directory expands to every manifest under it.
if args[0] == "check" {
set := flag.NewFlagSet("module check", flag.ContinueOnError)
// The longest machine name an identity must fit on (novox/hq ADR 0225): a mesh passes its own.
longest := set.Int("longest-machine-name", catalogue.DefaultLongestMachine,
"judge each module's identity on a machine name this many characters long")
given, err := parseAround(set, args[1:])
if err != nil {
return err
}
if *longest < 1 {
return errors.New("--longest-machine-name is a length, at least 1")
}
var paths []string
for _, a := range args[1:] {
for _, a := range given {
if info, err := os.Stat(a); err == nil && info.IsDir() {
under, err := manifestsUnder(a)
if err != nil {
@@ -71,7 +87,7 @@ func moduleCommand(ctx context.Context, args []string) error {
}
paths = append(paths, a)
}
return moduleCheck(paths, os.Stdout)
return moduleCheckFor(paths, *longest, os.Stdout)
}
open, err := openStores(ctx)
if err != nil {
@@ -494,8 +510,8 @@ const theBrokerSeat = "mesh-broker"
// says. Only when nothing holds the seat yet (genesis raised the broker as plumbing and no module
// has adopted it) does the hub stand in, which is where the foundation is by convention.
//
// "On the overlay" is what `whereEveryoneIs` answers — a machine that RESOLVED the networking
// module — not "has an address", which is true of every placed machine and says nothing about
// "On the overlay" is what `whereEveryoneIs` answers — a machine that RESOLVED the private network
// — not "has an address", which is true of every placed machine and says nothing about
// whether anything can reach it (novox/hq issue 059). A node not on the overlay — at genesis,
// before any `overlay place`, which is when the builder's account is issued — keeps the genesis
// address, so nothing about bring-up changes. This is issue 055, corrected by 059.
@@ -661,7 +677,7 @@ func issueWith(ctx context.Context, inv *inventory.Inventory, m catalogue.Manife
// durable subscription nobody reads.
if consumer, needed := broker.ConsumerFor(broker.Principal{
Kind: broker.KindModule, Node: node, Module: m.Module,
Emits: m.Emits, Consumes: m.Consumes, Serves: m.Tools,
Emits: m.EmitsAll(), Consumes: m.Consumes, Serves: m.Tools,
}); needed {
if busAddress == "" {
fmt.Printf(" %s consumes; its consumer is created when the bus is reachable (`push`, then "+
+2 -18
View File
@@ -275,25 +275,9 @@ func network(ctx context.Context, inv *inventory.Inventory, on map[string]bool,
if err != nil {
return nil, err
}
// No registry trust is composed here any more: the container runtime's module states it, told
// where the store is reached by ${seat:mesh-artifact-store:reach} (novox/hq ADR 0222, issue 190).
g, err := overlay.From(nodes, cidr, "")
if g != nil {
// The artifact store, as this network reaches it. Found rather than configured: the
// provider is whichever module offers it, on whichever machine holds that module — and if
// nothing does yet (genesis raises the registry before the catalogue knows it), there is
// no trust to write and nothing is written (novox/hq ADR 0082).
//
// Refused rather than composed without it when the question could not be answered: a
// declaration missing the trust because a lookup failed is a machine that cannot pull,
// delivered by a push that reported success — and nothing recomposes it until the next
// push (the shape of novox/hq issues 042/048, reappearing as a race).
at, port, found, storeErr := artifactStoreOnNetwork(ctx, inv, on)
if storeErr != nil {
return nil, fmt.Errorf("finding the artifact store this network reaches: %w", storeErr)
}
if found {
g.TrustRegistry(overlay.InternalName(at) + ":" + port)
}
}
if err != nil && len(refused) > 0 {
// The network is missing something, and some machines could not be resolved at all. Those
// are almost always the same fact: a node that does not resolve contributes nothing, so
+36 -4
View File
@@ -6,6 +6,7 @@ import (
"errors"
"flag"
"fmt"
"github.com/novox/mesh-controller/internal/conditions"
"strings"
"time"
@@ -387,7 +388,7 @@ func brokerCommand(ctx context.Context, args []string) error {
return busAccounts(ctx, args[1:])
}
if len(args) > 0 && args[0] == "consumer-reset" {
return consumerReset(args[1:])
return consumerReset(ctx, args[1:])
}
if len(args) == 0 || args[0] != "show" {
return errors.New("broker show | broker certificate [--check] --into <directory> | broker accounts --into <file> | " +
@@ -414,10 +415,21 @@ func brokerCommand(ctx context.Context, args []string) error {
// consumerReset re-makes one consumer on a stream that keeps history to start from now (novox/hq issue
// 248): the way out of a consumer replaying a week of announcements, said rather than done by hand. A
// person's act — what was pending is dropped — so it is a command, and nothing calls it on its own.
func consumerReset(args []string) error {
if len(args) != 2 {
return errors.New("broker consumer-reset <stream> <consumer>, e.g. broker consumer-reset EVENTS controller")
func consumerReset(ctx context.Context, args []string) error {
set := flag.NewFlagSet("broker consumer-reset", flag.ContinueOnError)
// A repair by hand, which says why (novox/hq to-be 45 §7).
why := addHandActFlags(set)
args, err := parseAround(set, args)
if err != nil {
return err
}
if len(args) != 2 {
return errors.New("broker consumer-reset <stream> <consumer> --why <text>, e.g. broker consumer-reset EVENTS controller --why ...")
}
if err := why.require("broker consumer-reset"); err != nil {
return err
}
why.record(ctx, "broker consumer-reset", args)
address, err := broker.BusAddress()
if err != nil {
return err
@@ -505,6 +517,26 @@ func showNode(ctx context.Context, inv *inventory.Inventory, name string) error
fmt.Printf(" public domain %s\n", domain)
}
// Every open condition about this machine (novox/hq to-be 45 §2) — a provider here failing a
// consumer, or a consumer here failed, among them (ADR 0224). Before the capabilities, because it
// is something not working now and they are a description. Unreadable is said, not passed over.
open, err := openConditions(ctx)
if err != nil {
fmt.Printf("\n the open conditions could NOT be read, so whether anything here is wrong is not known: %v\n", err)
}
var here []conditions.Condition
for _, c := range open {
if concerns(c, name) {
here = append(here, c)
}
}
if len(here) > 0 {
fmt.Printf("\n %d open condition(s) about this machine:\n", len(here))
for _, line := range conditionLines(here, time.Now()) {
fmt.Printf(" %s\n", line)
}
}
held, err := inv.Profile(ctx, name)
if err != nil {
return err
+218 -41
View File
@@ -90,6 +90,12 @@ func planFor(ctx context.Context, open *stores, nodeName string) (catalogue.Reso
if err != nil {
return catalogue.Resolution{}, nil, err
}
// Where each of its consumers of a provision that keeps data was last sent (novox/hq ADR 0232):
// a resolution that would answer one from anywhere else keeps it there, and says so.
world.Bound, err = inv.BindingsFor(ctx, nodeName)
if err != nil {
return catalogue.Resolution{}, nil, err
}
onNetwork, err := whereEveryoneIs(ctx, inv, shelf)
if err != nil {
@@ -372,13 +378,19 @@ func declarationFor(ctx context.Context, open *stores, node string,
// So the mesh chooses a port when it commits to sending one, and every other caller reads what
// was chosen. A module with nothing assigned yet has never been sent, which is exactly what a
// machine "waiting" means — the read needs no number to be right about that.
type Choosing bool
type Choosing int
const (
// Allocating is the send path: what is not assigned yet is assigned now and kept.
Allocating Choosing = true
// Reading is every question: what is assigned is used, and nothing is created.
Reading Choosing = false
Reading Choosing = iota
// Allocating is the send path: what is not assigned yet is assigned now and kept.
Allocating
// Foreseeing is Reading, asked ahead of a send (the self-check's D1, novox/hq issue 275): nothing
// is created, and an own secret the next send WOULD make is composed with a stand-in and named
// (sendable.foreseen) rather than failing the composition — while one the send would be refused
// (a bus credential nobody issued, a given secret under an old key) is refused here as it would
// be there. Never sent: the stand-in is not a sealed value.
Foreseeing
)
func declarationWith(ctx context.Context, open *stores, node string,
@@ -397,9 +409,75 @@ func declarationWith(ctx context.Context, open *stores, node string,
if err != nil {
return sendable{}, err
}
return sendable{Resources: composed.Resources, Adoption: adoption,
out := sendable{Resources: composed.Resources, Adoption: adoption,
Received: composed.Received, Mesh: with.Mesh, BusUsers: with.BusUsers,
LeftOut: sortedKeysOf(composed.LeftOut), leftOutWhy: composed.LeftOut}, nil
LeftOut: sortedKeysOf(composed.LeftOut), leftOutWhy: composed.LeftOut, withheld: with.Withheld,
unbound: with.Unbound, foreseen: composed.Foreseen}
// And which build of each module it carries, for the send to record (novox/hq issue 259, ADR
// 0221). Read only on the send path: a question about what would be sent records nothing.
if choosing == Allocating {
current, err := open.inventory.CurrentBuilds(ctx)
if err != nil {
return sendable{}, err
}
before, known, err := open.inventory.SentBuilds(ctx, node)
if err != nil {
return sendable{}, err
}
if !known {
before = nil
}
names := make([]string, 0, len(plan.Modules))
for _, m := range plan.Modules {
names = append(names, m.Module)
}
out.Builds = carriedBuilds(names, composed.LeftOut, current, before)
out.Bindings = boundToData(plan, composed.LeftOut)
}
return out, nil
}
// boundToData is every binding of this machine's consumers to a provision that keeps their data
// (novox/hq ADR 0232), as a send records it. Not a consumer left out of the declaration: the machine
// is not told anything new about it, so nothing about where it is bound has been sent.
func boundToData(plan catalogue.Resolution, leftOut map[string]string) []inventory.Binding {
var out []inventory.Binding
seen := map[[2]string]bool{}
for _, n := range plan.Needs {
if !n.KeepsData || n.ByRecord || n.Module == "" {
continue
}
if _, left := leftOut[n.For]; left {
continue
}
key := [2]string{n.For, n.Name}
if seen[key] {
continue
}
seen[key] = true
out = append(out, inventory.Binding{Machine: plan.Node, Consumer: n.For, Provision: n.Name,
Provider: catalogue.Chosen{Node: n.From, Module: n.Module}})
}
return out
}
// carriedBuilds is the build of each module a declaration carries, as a send records it (novox/hq
// issue 259): the module's current build for each module in it, and for a module left out of it
// (ADR 0163, rule 6) the build it was last sent, since the machine keeps that one — or nothing, when
// that is not known. Never nil, so a send through here always records what it knows.
func carriedBuilds(modules []string, leftOut map[string]string, current map[string]inventory.CurrentBuild,
before map[string]string) map[string]string {
out := map[string]string{}
for _, m := range modules {
if _, left := leftOut[m]; left {
if was, kept := before[m]; kept {
out[m] = was
}
continue
}
out[m] = current[m].Commit
}
return out
}
// sortedKeysOf is a map's keys, sorted — so what a declaration says it left out does not move
@@ -424,6 +502,39 @@ func reportLeftOut(node string, declared sendable) {
"what the machine holds for it is kept and its containers are untouched. %s\n",
node, m, declared.leftOutWhy[m])
}
// And whom it serves nothing, because their identity overflows what the provision keeps (ADR
// 0225): the machine is sent everything else, and the consumer is named.
for _, o := range declared.withheld {
fmt.Printf("%s: %s\n", node, o)
}
// And whom it no longer serves because they are bound elsewhere (novox/hq issue 274).
for _, u := range declared.unbound {
fmt.Printf("%s: %s\n", node, u)
}
}
// busCredentialIssued refuses an own secret called `broker` whose bus account nobody issued.
//
// **The broker credential is never invented here** (novox/hq issue 203). Every other own secret is
// the mesh's to make — a password nobody else knows — but this one is an account on the bus, minted
// by `module issue` and sealed by it; a push that made a random one would deliver a file the process
// cannot read and report the machine applied. Refused by name, with the verb — on the send, and on
// a question asked ahead of it (Foreseeing), so that one is never told the push will make it.
func busCredentialIssued(ctx context.Context, inv *inventory.Inventory, node, module, name string) error {
if name != "broker" {
return nil
}
user := broker.Principal{Kind: broker.KindModule, Node: node, Module: module}.Username()
if _, minted, err := inv.BusUserHash(ctx, user); err != nil {
return err
} else if !minted {
return fmt.Errorf(
"%s on %s has no bus credential: nothing was issued for %s, and a push "+
"would seal a placeholder its process cannot read (novox/hq issue 203). "+
"`module issue %s --node %s`, then push again",
module, node, user, module, node)
}
return nil
}
// renderingFor is everything a node's declaration is composed with, and the node's record.
@@ -431,7 +542,7 @@ func renderingFor(ctx context.Context, open *stores, node string,
plan catalogue.Resolution, settings catalogue.SettingsBy,
gens map[string]catalogue.Generator, choosing Choosing) (catalogue.Rendering, inventory.Node, error) {
inv := open.inventory
grants, err := grantsFor(ctx, open, node)
grants, withheld, unbound, err := grantsFor(ctx, open, node)
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
@@ -443,7 +554,7 @@ func renderingFor(ctx context.Context, open *stores, node string,
// consumer is told are all derived from it.
// What this machine was already given, for a composition that may not allocate.
already := map[string]map[int]int{}
if choosing == Reading {
if choosing != Allocating {
held, err := inv.PortsFor(ctx, node)
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
@@ -529,6 +640,7 @@ func renderingFor(ctx context.Context, open *stores, node string,
// And each module's own secrets — a superuser password, an administrator, an account. Made
// per node, so a module running on three machines has three.
needed := map[string]map[string]string{}
foreseen := map[string]map[string]bool{}
for _, m := range plan.Modules {
for name := range m.OwnSecrets {
// Minted on the send path and only read on every other. Making one is an insert, and
@@ -536,27 +648,29 @@ func renderingFor(ctx context.Context, open *stores, node string,
var sealed string
var err error
if choosing == Allocating {
// **The broker credential is never invented here** (novox/hq issue 203). Every other
// own secret is the mesh's to make — a password nobody else knows — but this one
// is an account on the bus, minted by `module issue` and sealed by it; a push that
// made a random one would deliver a file the process cannot read and report the
// machine applied. Refused by name, with the verb.
if name == "broker" {
user := broker.Principal{Kind: broker.KindModule, Node: node, Module: m.Module}.Username()
if _, minted, err := inv.BusUserHash(ctx, user); err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
} else if !minted {
return catalogue.Rendering{}, inventory.Node{}, fmt.Errorf(
"%s on %s has no bus credential: nothing was issued for %s, and a push "+
"would seal a placeholder its process cannot read (novox/hq issue 203). "+
"`module issue %s --node %s`, then push again",
m.Module, node, user, m.Module, node)
}
if err := busCredentialIssued(ctx, inv, node, m.Module, name); err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
sealed, err = inv.SecretForModule(ctx, node, m.Module, name)
} else {
var held bool
sealed, held, err = inv.ModuleSecretIfIssued(ctx, node, m.Module, name)
if err == nil && !held && choosing == Foreseeing {
// Asked ahead of the send: what the send would do about it, by the send's own
// rules. Made by it — a stand-in, named; refused by it — refused here, the same
// words (novox/hq issue 275).
if err := busCredentialIssued(ctx, inv, node, m.Module, name); err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
if err := inv.WouldMakeSecretForModule(ctx, node, m.Module, name); err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
if foreseen[m.Module] == nil {
foreseen[m.Module] = map[string]bool{}
}
foreseen[m.Module][name] = true
continue
}
if err == nil && !held {
// Never issued, so this machine cannot be running it. Left out rather than
// invented: an empty string here would compose a declaration that differs
@@ -667,6 +781,13 @@ func renderingFor(ctx context.Context, open *stores, node string,
return catalogue.Rendering{}, inventory.Node{}, err
}
// And who holds each replicated seat, where (novox/hq ADR 0223): every machine's resolver file
// lists every holder of the mesh's resolver.
replicas, err := replicatedHolders(ctx, inv, shelf)
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
// **The bus is never public** (novox/hq ADR 0169). It was a foundation port — widened from the
// broker's own `from: mesh` to from-anywhere on the broker's host, so a machine could enrol
// before it had an address on the private network. A machine joins through the tunnel now, and
@@ -732,15 +853,22 @@ func renderingFor(ctx context.Context, open *stores, node string,
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
// Where this machine reaches each mesh seat's holder, for ${seat:<seat>:reach} (novox/hq ADR
// 0222): the artifact store as this network reaches it, which the container runtime is told to
// trust (ADR 0082). The same address composed into every reference the mesh built.
var reach map[string]string
if artifactStore != "" {
reach = map[string]string{"mesh-artifact-store": artifactStore}
}
return catalogue.Rendering{
BusMembership: memberships[node],
Settings: settings, Generators: gens, Grants: grants, Needed: needed, Ports: ports,
Settings: settings, Generators: gens, Grants: grants, Needed: needed, Foreseen: foreseen, Ports: ports,
Certificate: certificate, Authority: authority, Mesh: private, Names: names,
Machines: machines, Zones: zones,
Machines: machines, Zones: zones, Holders: replicas,
Suffix: overlay.Suffix(), MeshRange: meshRange, TunnelInterface: overlay.Interface, Accounts: accounts, Foundation: foundation,
Kept: kept, Adopted: record.Adopted, OutwardLinks: outwardLinks,
Given: given, Taken: taken, Seats: seats, ArtifactStore: artifactStore, Built: built,
BusUsers: busUsers,
Given: given, Taken: taken, Seats: seats, ArtifactStore: artifactStore, SeatReach: reach, Built: built,
BusUsers: busUsers, Withheld: withheld, Unbound: unbound,
}, record, nil
}
@@ -876,28 +1004,45 @@ func certificateFor(ctx context.Context, open *stores, node string) (string, str
// The mirror of what a consumer is given, and the half that makes the credential real: a password
// nothing was told to create is a password that authenticates nowhere. Sealed to this node, so
// the mesh hands over something it cannot itself use.
func grantsFor(ctx context.Context, open *stores, node string) ([]catalogue.Grant, error) {
//
// **One consumer's identity never refuses the provider's machine** (novox/hq ADR 0225, issue 263).
// A consumer whose identity overflows the provision's bound is left out of the grants and returned
// beside them, for push, plan and `status` to say; every other consumer is granted and the provider's
// declaration composes. Refusing here once made a whole machine unpushable for one module elsewhere.
//
// **A provider is granted exactly the consumers whose own resolution binds them to it** (novox/hq
// issue 274). The pair credentials on record say only whom this node was ever asked by: after a
// consumer of a provision that keeps its data was moved and pinned back (issue 273, ADR 0232), the
// credential from the provider it left was still on record, so that provider went on being asked for
// five databases nobody used and never retired them. A credential whose consumer is bound elsewhere
// is withdrawn here like one nobody asks for — the provider retires it and keeps its data (ADR 0230)
// — and returned beside the grants, for plan and push to say. It stays on record: it is the key to
// the login the provider keeps until `cleanup delete`, and to that data should a person pin it back.
func grantsFor(ctx context.Context, open *stores, node string) (
[]catalogue.Grant, []catalogue.Overflow, []catalogue.Unbound, error) {
inv := open.inventory
issued, err := inv.SecretsFrom(ctx, node)
if err != nil {
return nil, err
return nil, nil, nil, err
}
// Where each consumer is, so a provider that must reach back to one does not have to know how
// the mesh names machines.
shelf, err := inv.Catalogue(ctx)
if err != nil {
return nil, err
return nil, nil, nil, err
}
onNetwork, err := whereEveryoneIs(ctx, inv, shelf)
if err != nil {
return nil, err
return nil, nil, nil, err
}
// What each consumer actually asked for, taken from that machine's own resolution rather than
// from a record beside it. A provider told to create a password and not what to create it for
// can do nothing with it, and the name a consumer wants is the consumer's to say.
out := make([]catalogue.Grant, 0, len(issued))
var withheld []catalogue.Overflow
var unbound []catalogue.Unbound
for _, s := range issued {
plan, settings, err := planFor(ctx, open, s.Consumer)
switch {
@@ -910,11 +1055,11 @@ func grantsFor(ctx context.Context, open *stores, node string) ([]catalogue.Gran
// The mesh could not be asked what they wanted, which is not the same as their wanting
// nothing — and withholding a grant on that reading takes a consumer's access away
// (novox/hq 04-ISSUES/152).
return nil, fmt.Errorf("what %s asked of %s cannot be read: %w", s.Consumer, s.Name, err)
return nil, nil, nil, fmt.Errorf("what %s asked of %s cannot be read: %w", s.Consumer, s.Name, err)
}
values, asks, err := plan.ContributionsFrom(s.Name, s.ConsumerModule, settings)
if err != nil {
return nil, err
return nil, nil, nil, err
}
// A port in there is the consumer's software port until this. The consumer is on another
// machine, so the assignment that moved it is that machine's — fetched here rather than
@@ -922,7 +1067,7 @@ func grantsFor(ctx context.Context, open *stores, node string) ([]catalogue.Gran
// this case (novox/hq 04-ISSUES/038, the cross-node half).
published, err := portsOn(ctx, inv, s.Consumer, s.ConsumerModule)
if err != nil {
return nil, err
return nil, nil, nil, err
}
values = catalogue.AtPublishedPort(values, s.ConsumerModule, published)
from := s.ConsumerModule
@@ -932,10 +1077,20 @@ func grantsFor(ctx context.Context, open *stores, node string) ([]catalogue.Gran
// working for ever after its consumer went away.
from = ""
}
if bound := plan.BindsFrom(s.Name, s.ConsumerModule, s.Local); from != "" && !slices.Contains(bound, node) {
// It still asks, and not of this node: its resolution — the same one read above, so an
// unreadable one is the error above and never an empty answer here (issue 152) — binds
// this credential to another provider, or under this local name to none. Withdrawn
// exactly as a credential nobody asks for, and said.
from = ""
unbound = append(unbound, catalogue.Unbound{Provision: s.Name, Provider: node,
Consumer: s.Consumer, Module: s.ConsumerModule, Local: s.Local, BoundTo: bound})
}
// The consumer's identity slug, from its own manifest, carried on the grant so the provider
// derives the same login the consumer does (novox/hq ADR 0049). Refused here if it still would
// not fit the tightest backend — the mesh chose the name, so the mesh refuses it, with the
// remedy a short slug rather than a login a provider silently shortened.
// derives the same login the consumer does (novox/hq ADR 0049). Judged against the bound of
// this provision, as the consumer's resolution states it from the provider's offer (ADR
// 0225): a consumer it would not fit is left out of the grants and said, rather than a login a
// provider silently shortened — and rather than this whole machine refused for it.
slug := ""
for _, mm := range plan.Modules {
if mm.Module == s.ConsumerModule {
@@ -944,15 +1099,32 @@ func grantsFor(ctx context.Context, open *stores, node string) ([]catalogue.Gran
}
}
if from != "" {
if err := catalogue.CheckIdentity(s.Consumer, catalogue.IdentitySource(slug, s.ConsumerModule)); err != nil {
return nil, err
bound := boundOfGrant(plan, s, node)
source := catalogue.IdentitySource(slug, s.ConsumerModule)
if catalogue.CheckIdentityWithin(s.Consumer, source, bound) != nil {
withheld = append(withheld, catalogue.Overflow{Provision: s.Name, Provider: node,
Consumer: s.Consumer, Module: s.ConsumerModule,
Identity: catalogue.ConsumerIdentity(s.Consumer, source), Bound: bound})
continue
}
}
out = append(out, catalogue.Grant{
Provision: s.Name, Consumer: s.Consumer, At: onNetwork[s.Consumer],
From: from, Values: values, Slug: slug, Sealed: s.ForProvider, Local: s.Local})
}
return out, nil
return out, withheld, unbound, nil
}
// boundOfGrant is the identity bound the consumer's own resolution states for the requirement this
// grant answers. A requirement not found there is held to the tightest bound the mesh knows rather
// than to none: what the provider keeps of it is not known here.
func boundOfGrant(consumer catalogue.Resolution, s inventory.Secret, provider string) catalogue.IdentityBound {
for _, n := range consumer.Needs {
if n.Name == s.Name && n.For == s.ConsumerModule && n.From == provider {
return n.Identity
}
}
return catalogue.DefaultIdentityBound
}
// listensLines is what a person is told about what this module would open, and why — the same
@@ -1028,6 +1200,11 @@ func planCommand(ctx context.Context, args []string) error {
left := plan.LeftOut(settings, record.Adopted)
reportLeftOut(args[0], sendable{LeftOut: sortedKeysOf(left), leftOutWhy: left})
}
// And which of its modules no provider will grant, because the identity overflows the bound of
// what it requires (novox/hq ADR 0225) — said on the machine the remedy is for.
for _, o := range plan.Overflowing() {
fmt.Printf("%s: %s\n", args[0], o)
}
// And a setting that reaches nothing — refused where it is stored, and said here for one
// stored before its definition moved from under it.
for _, m := range plan.Modules {
+3 -3
View File
@@ -280,7 +280,7 @@ func retryPlan(ctx context.Context, open *stores, id string) (string, error) {
}
}
resumed(&p, fmt.Sprintf("tier %d retried by hand: %s asked again", p.Tier, strings.Join(failed, ", ")))
if err := inv.SavePlan(ctx, p); err != nil {
if err := inv.SavePlan(ctx, &p); err != nil {
return "", err
}
if p.State != inventory.PlanBuilding {
@@ -323,7 +323,7 @@ func joinAPlan(ctx context.Context, open *stores, module string) (bool, string,
askModule(ctx, &p, module, byName)
s := p.Modules[module]
resumed(&p, fmt.Sprintf("tier %d: %s rebuilt by hand", p.Tier, module))
if err := inv.SavePlan(ctx, p); err != nil {
if err := inv.SavePlan(ctx, &p); err != nil {
return false, "", err
}
if s.State != "asked" {
@@ -393,7 +393,7 @@ func retryRollouts(ctx context.Context, open *stores, p *inventory.Plan) (string
}
p.State = inventory.PlanRolling
p.Note = fmt.Sprintf("tier %d retried by hand; sent %s first again", p.Tier, strings.Join(said, "; "))
if err := open.inventory.SavePlan(ctx, *p); err != nil {
if err := open.inventory.SavePlan(ctx, p); err != nil {
return "", err
}
return fmt.Sprintf("%s retried at tier %d of %d: sent %s first again; the rest follow once it reports it "+
@@ -18,7 +18,7 @@ func TestAControllerLeavesThePlansToTheOneHoldingThem(t *testing.T) {
plan := inventory.Plan{ID: "plan-213", Repository: "r", Commit: "abc", Created: now, Updated: now,
State: inventory.PlanRolling, Tier: 1, Tiers: [][]string{{"app"}},
Modules: map[string]*inventory.PlanModule{"app": {State: "built"}}}
if err := open.inventory.SavePlan(ctx, plan); err != nil {
if err := open.inventory.SavePlan(ctx, &plan); err != nil {
t.Fatal(err)
}
File diff suppressed because it is too large Load Diff
+250 -128
View File
@@ -8,6 +8,7 @@ import (
"errors"
"flag"
"fmt"
"github.com/novox/mesh-controller/internal/conditions"
"io"
"log"
"os"
@@ -62,7 +63,7 @@ func connectLink(ctx context.Context, inv *inventory.Inventory, enroller link.En
return link.ConnectNats(js, enroller, listener), nil
}
func serve(ctx context.Context) error {
func serve(ctx context.Context) (err error) {
// The one process whose log is read over time, so the one that says each change to a node's
// unmet seat dependencies once (novox/hq ADR 0207).
logUnheldChanges = true
@@ -103,9 +104,47 @@ func serve(ctx context.Context) error {
// being live is refused, because a mesh half on each is one where a declaration goes out on one
// and the report comes back on the other, and every component logs success while it happens.
// **The lease, before anything that acts** (novox/hq to-be 45 §6): asserting the bus's objects is the
// controller's to do, and so is everything after. A controller starting while another holds it waits
// here, said; one that loses it stops: every act's gate closes at once, and ctx ends so the process
// exits and is started again as a candidate.
busAddress, err := broker.BusAddress()
if err != nil {
return err
}
lost, err := theLease.serveUnderTheLease(ctx, inv, busAddress)
if err != nil {
return err
}
// Given back before the store closes, so the epoch is recorded as given back rather than found
// expired by the next holder.
defer theLease.release()
ctx, stopActing := context.WithCancel(ctx)
defer stopActing()
go func() {
select {
case <-lost:
stopActing()
case <-ctx.Done():
}
}()
defer func() {
select {
case <-lost:
if err == nil {
err = errors.New("the controller lease was lost; this controller stopped acting and exits, to " +
"be started again as a candidate")
}
default:
}
}()
work := link.Enrolment{Inventory: inv, Identity: ident, Broker: known,
OnNATS: true}
server, err := connectLink(ctx, inv, work, work)
// `status` from a summary kept current here (novox/hq to-be 45 Phase 0): a machine saying
// something new is one thing that moves it, so the listener nudges it.
statusFrom = newStatusSummary(composeStatus(open))
server, err := connectLink(ctx, inv, work, nudgingListener{Enrolment: work, summary: statusFrom, open: open})
if err != nil {
return err
}
@@ -121,17 +160,33 @@ func serve(ctx context.Context) error {
// Open plans move on a timer as well as on outcomes (novox/hq ADR 0162): a tier waiting for
// machines to report moves when they have, and a plan left by a replaced controller resumes.
go planTicker(ctx, open)
// The durations the core's bounds are set from are kept a month (novox/hq to-be 45 Phase 0).
go forgettingOldDurations(ctx, inv)
// And what the catalogue decided a build meant. The builder's own result is already handled
// above; this is the other half — the control plane is the only one of the three that knows
// which machines run the thing, so it is the one that acts (novox/hq ADR 0072).
if err := server.Follows(following{open}); err != nil {
return err
}
// And the merges the bus announced and never handed over, read back on a timer and acted on late
// rather than never (novox/hq issue 266).
go catchingUpOnMerges(ctx, open, server)
// And a catalogue that has just started, asking for what it missed. The same type answers
// both: what a build meant and what the builds were are two questions about one record.
if err := server.Answers(following{open}); err != nil {
return err
}
// And what providers say about consumers they keep failing, kept for `status` (novox/hq ADR
// 0224): a provider's journal must not be the only place that says so.
if err := server.Watches(standings{keeper: func() *conditions.Keeper { return conditionsFrom }}); err != nil {
return err
}
// And what they say about consumers the mesh stopped asking for: one waiting for a person is an
// urgent condition, and an act asked of a provider some other way is recorded by hand (ADR 0230).
if err := server.KeepsRetirements(retirements{keeper: func() *conditions.Keeper { return conditionsFrom },
record: recordHandActOnTheBus}); err != nil {
return err
}
// And the mesh's own verbs, as the seat this control plane holds (novox/hq ADR 0154). Served
// from the store's row, so what the seat declares is what is answered.
@@ -150,6 +205,33 @@ func serve(ctx context.Context) error {
if !isNATS {
return errors.New("the mesh's verbs are served over the bus, and this control plane is not on it")
}
// The hand-act log is counted for `status` on this connection rather than a new one a minute.
handActConn = bus.Conn
// And says when it replaced a value given by hand (novox/hq ADR 0228).
givenEvents = bus
// Composed now and kept current, before the verb that answers from it is served.
go statusFrom.keep(ctx)
// Every call carries the lease's epoch, and its record is written only under the lease (novox/hq
// to-be 45 §6).
link.Calls.UnderLease(func() (uint64, error) { return theLease.epoch(ctx) })
// A call that outlasts its caller's patience is followed by `calls` (novox/hq issue 265).
link.Calls.Follow = catalogue.ControllerSeatName + ".calls"
// And every call is kept on the bus, so a restart of this process keeps what came of each
// (novox/hq to-be 45 §6). A bus without the bucket is said and served from memory, as before:
// answering no calls at all would be worse than answering them without the record.
said := log.New(os.Stdout, "", log.LstdFlags)
if keeper, err := link.CallsOnTheBus(ctx, bus.Conn); err != nil {
fmt.Printf("calls are kept in memory only, and lost when this controller stops: %v\n", err)
} else if err := link.Calls.Durably(ctx, keeper, instance, said); err != nil {
fmt.Printf("calls are kept on the bus from now on; the ones kept before could not be read: %v\n", err)
}
// What is wrong, kept and said (novox/hq to-be 45 §2): the condition store, the watchdogs of the
// signals table, what the bus says about itself, and the self-check. A store that cannot be opened
// is said and the controller serves on: status then says the conditions cannot be read, and is
// not well — louder than not serving at all, and the push that repairs the bus still runs.
if stopWatching := watchTheMesh(ctx, open, server, bus); stopWatching != nil {
defer stopWatching()
}
stopServing, err := bus.ServeSeatTools(catalogue.ControllerSeatName, handlers, log.New(os.Stdout, "", log.LstdFlags))
if err != nil {
return err
@@ -217,8 +299,14 @@ func declare(ctx context.Context, args []string) error {
}
// Written down like every other send (novox/hq issue 204): a declaration a person sent by hand
// is still what the machine was last told, and status must not read it as current for the one
// the mesh would compose.
if _, err := recordSent(ctx, inv, node, raw); err != nil {
// the mesh would compose. Which builds it carried is recorded as not known (novox/hq issue 259):
// the mesh did not compose it, so a push that does not name this machine treats it as held.
// The epoch it carried, if a person wrote one in, is what the machine heard.
var carried struct {
Epoch uint64 `json:"epoch"`
}
_ = json.Unmarshal(raw, &carried)
if _, err := recordSent(ctx, inv, node, raw, nil, carried.Epoch); err != nil {
return err
}
fmt.Printf("sent %s a signed declaration (%d bytes)\n", node, len(raw))
@@ -249,6 +337,9 @@ func pushCommand(ctx context.Context, args []string) error {
// (novox/hq ADR 0010). 0 waits for nothing, which is the old fire-and-forget.
wait := set.Duration("wait", 0,
"for a named node, how long to wait for it to report applying what it was sent (0: do not wait)")
// A push by hand is a repair, and says why (novox/hq to-be 45 §7): required through the seat,
// recorded when given at a shell — see handacts.go for why a shell is not refused.
why := addHandActFlags(set)
positionals, err := parseAround(set, args)
if err != nil {
return err
@@ -263,6 +354,11 @@ func pushCommand(ctx context.Context, args []string) error {
return errors.New("push <node> or push --behind, not both: one names a machine and the " +
"other asks which machines need one")
}
recorded := append([]string(nil), args...)
if *behind {
recorded = append(recorded, "--behind")
}
why.record(ctx, "push", recorded)
open, err := openStores(ctx)
if err != nil {
return err
@@ -316,7 +412,7 @@ func pushCommand(ctx context.Context, args []string) error {
if len(needsOne) == 0 {
// Said rather than doing nothing quietly. "Nothing needed one" and "this did not run"
// must never look the same.
fmt.Println("every machine is doing what it was told")
fmt.Println("no machine named: a push of the whole mesh, and every machine is doing what it was told — nothing sent")
return nil
}
}
@@ -357,6 +453,18 @@ func pushCommand(ctx context.Context, args []string) error {
}
asked = append(asked, n.Name)
}
// **A push that named no machine says so, first.** Through the console a machine the caller
// meant to name could be lost on the way (novox/hq issue 244): the call arrived empty, ran as
// `push --behind`, and every machine behind was pushed by someone who thought they had pushed one.
// Its whole-mesh reach is the first line of the answer, with the machines it is about to send.
if len(args) == 0 {
which := "every machine"
if *behind {
which = "every machine that is behind"
}
fmt.Printf("no machine named: this is a push of the WHOLE mesh — %s (%d): %s\n",
which, len(asked), strings.Join(asked, ", "))
}
// **The machine holding the bus first** (novox/hq issue 249): its declaration carries the bus's
// user list, and a module's new grants are refused by the bus until that list says them. Among
@@ -367,6 +475,23 @@ func pushCommand(ctx context.Context, args []string) error {
if err != nil {
return err
}
// **Not when its own modules are held back** (novox/hq issue 259, ADR 0221): added rather than
// named, it is sent its whole declaration, and a build its policy records or a plan has not sent
// it yet would go with the user list. Named and left, with what that costs.
saidHeld := map[string]bool{}
if holderBehind && len(args) == 1 {
held, err := heldMachines(ctx, open, []string{holder})
if err != nil {
return err
}
if why, isHeld := held[holder]; isHeld {
sayHeld(os.Stdout, holder, why)
fmt.Printf("%s holds the bus, and the user list it would carry has changed: until it is "+
"sent, the bus may refuse what this push's machines were newly granted\n", holder)
holderBehind = false
saidHeld[holder] = true
}
}
asked = brokerFirst(asked, holder, holderBehind)
// Held from composing to sending, so a converge on one of them cannot send between the two
@@ -387,6 +512,7 @@ func pushCommand(ctx context.Context, args []string) error {
// healthy modules beside it are still resolved and sent. Reported so it is not silently
// dropped — the remedy is to move it, and until then the rest of the node converges.
reportUnhostable(node, plan)
reportKept(held, plan)
unheld[node] = plan.Unheld
// The private network is in here with everything else. It used to be composed separately
// and prepended, which meant every machine with an address was on it and no machine could
@@ -407,7 +533,11 @@ func pushCommand(ctx context.Context, args []string) error {
return err
}
release()
fmt.Printf("\n%d node(s) told\n", len(sending))
told := make([]string, 0, len(sending))
for _, r := range sending {
told = append(told, r.node)
}
fmt.Printf("\n%d node(s) told: %s\n", len(sending), strings.Join(told, ", "))
reportUnheldPushed(os.Stdout, len(args) == 1, asked, unheld)
// **A named push leaves the mesh consistent, not just the machine it named** (novox/hq
@@ -418,76 +548,21 @@ func pushCommand(ctx context.Context, args []string) error {
//
// Compared against what each machine was last SENT, not against a before/after of this push:
// the mint usually happened at `assign` or `module issue`, before this command ran, so the
// only durable signal is "what it should be" versus "what it last received". A machine behind
// for an unrelated reason is caught here too, which is not a cost — a named push that knew a
// machine was behind and left it so would be the very silence this removes. Bounded: a
// only durable signal is "what it should be" versus "what it last received". Bounded: a
// flushed send may itself mint, so this converges over a few rounds.
//
// **Except a machine a policy or a plan holds back** (novox/hq issue 259, ADR 0221): one whose
// modules would move to a build their upgrade policy records rather than rolls out, or that an
// open plan has not sent it yet. It is named, with why, and left for a push that names it.
if len(args) == 1 {
flushed := map[string]bool{args[0]: true}
// Bounded by the node count: a node is marked flushed the round it is handled and is
// never handled twice, so the loop cannot run more than len(nodes) rounds. The bound is
// a guard against a logic error, not a real limit — if it were ever hit, that is a bug
// rather than a cascade legitimately still converging, so it is said rather than passed
// over in silence, unlike the earlier fixed cap that could stop a real cascade short.
rounds := 0
for {
would, err := wouldSend(ctx, open, nodes)
if err != nil {
return err
}
behind, err := inv.Waiting(ctx, would)
if err != nil {
return err
}
var also []string
for _, m := range behind {
if !flushed[m.Node] {
also = append(also, m.Node)
}
}
if len(also) == 0 {
break
}
if rounds++; rounds > len(nodes) {
fmt.Printf("\nstopped cascading after %d rounds with %s still behind — this "+
"should not happen; run `push --behind` to finish\n",
rounds-1, strings.Join(also, ", "))
break
}
sort.Strings(also)
fmt.Printf("\nthis push left %s behind — a provision granted from there, or a "+
"declaration since changed; sending it too\n", strings.Join(also, ", "))
// Tolerantly, exactly as the named send above: a machine that cannot be composed is
// collected as a refusal and reported at the end, and the others are still sent
// (novox/hq ADR 0066). The earlier cut routed these through sendTo, which is
// all-or-nothing — so one swept machine's compose error failed the operator's named
// push and skipped its --wait, the very intolerance the main path exists to avoid.
// Held for this round only, and after the last round's were given back, so two pushes
// cascading into each other's machines never each wait on the other.
refused, err := sendRound(ctx, open, also,
func(held context.Context, node string) (sendable, error) {
plan, settings, err := planFor(held, open, node)
if err != nil {
return sendable{}, err
}
reportUnhostable(node, plan)
declared, err := declarationWith(held, open, node, plan, settings, gens, Allocating)
if err == nil {
reportLeftOut(node, declared)
}
return declared, err
},
bus, holder)
refusals = append(refusals, refused...)
if err != nil {
return err
}
// Every candidate this round is marked handled — the sent ones so they are not
// re-listed, and the refused ones so a machine that cannot be composed does not make
// the loop spin on it for ever. Its refusal is already in the report.
for _, name := range also {
flushed[name] = true
}
handled := map[string]bool{args[0]: true}
for n := range saidHeld {
handled[n] = true
}
refused, err := flushBehind(ctx, open, nodes, handled, composeForPush(open, gens), bus, holder, os.Stdout)
refusals = append(refusals, refused...)
if err != nil {
return err
}
}
@@ -561,7 +636,7 @@ type readyNode struct {
//
// The all-or-nothing rule is kept where it means something — sendTo, which rotates a credential
// across two machines that must agree — and dropped here, where it never did.
func composeEach(names []string, allot func(node string) (int64, error),
func composeEach(names []string, allot func(node string) (order, error),
compose func(node string) (sendable, error)) ([]readyNode, []string) {
var sending []readyNode
@@ -575,7 +650,7 @@ func composeEach(names []string, allot func(node string) (int64, error),
// took the older content as the newer word: on 2026-10-02 a runtime assigned and applied on
// two machines was undone two seconds later by exactly that. Taken here, before the first
// read, what was composed earlier is numbered lower whatever order the sends happen in.
seq, err := allot(name)
numbered, err := allot(name)
if err != nil {
refusals = append(refusals, fmt.Sprintf("%s:\n%v", name, err))
continue
@@ -585,7 +660,7 @@ func composeEach(names []string, allot func(node string) (int64, error),
refusals = append(refusals, fmt.Sprintf("%s:\n%v", name, err))
continue
}
declared.Sequence = seq
declared.Sequence, declared.Epoch = numbered.sequence, numbered.epoch
if len(declared.Resources) == 0 {
// Sent, not skipped (novox/hq issue 127). A node whose declaration composes to
// nothing may have HELD something before — the broker opening a placement gave it,
@@ -753,6 +828,13 @@ type overTheBus struct {
}
func (b overTheBus) grant(ctx context.Context, sending []readyNode) error {
// The bus's objects first, which every declaration implies (novox/hq issue 208): said and raised
// when they cannot be, and never what holds the send back (assertOnSend).
if bus, ok := b.server.Bus().(link.OverNATS); ok {
js := broker.OnConn(bus.Conn)
js.Note = func(format string, args ...any) { fmt.Printf(b.indent+" "+format+"\n", args...) }
_ = assertOnSend(ctx, b.open.inventory, js, b.indent)
}
return issueMemberships(ctx, b.open, b.server, sending)
}
@@ -762,10 +844,21 @@ func (b overTheBus) declare(ctx context.Context, s readyNode, body []byte) (stri
}
// After it is away, not before. A digest recorded for something that failed to send would make
// the machine look current for a declaration it never received.
digest, err := recordSent(ctx, b.open.inventory, s.node, body)
digest, err := recordSent(ctx, b.open.inventory, s.node, body, s.declared.Builds, s.declared.Epoch)
if err != nil {
return "", err
}
if len(s.declared.Bindings) > 0 {
// And where it bound each consumer of a provision that keeps its data (novox/hq ADR 0232):
// what the next resolution keeps it at. On the same outliving context as the send's record.
kept, cancel := context.WithTimeout(context.WithoutCancel(ctx), 10*time.Second)
err := b.open.inventory.RecordBindings(kept, s.node, s.declared.Bindings)
cancel()
if err != nil {
return "", fmt.Errorf("%s was sent its declaration, and where its consumers are bound to their "+
"data could not be recorded: %w", s.node, err)
}
}
if s.declared.BusUsers != "" {
// And the user list it carried, so the next send reads whether it must go first from the
// list alone (novox/hq issue 249). On the same outliving context as the send's record.
@@ -915,7 +1008,7 @@ func sendToEach(ctx context.Context, open *stores, names []string) ([]string, er
var refusals []string
for _, name := range names {
// Numbered before composing, for the reason composeEach gives (novox/hq issue 204).
seq, err := allot(ctx, inv, name)
numbered, err := allot(ctx, inv, name)
if err != nil {
refusals = append(refusals, fmt.Sprintf("%s:\n%v", name, err))
continue
@@ -926,12 +1019,13 @@ func sendToEach(ctx context.Context, open *stores, names []string) ([]string, er
continue
}
reportUnhostable(name, plan)
reportKept(ctx, plan)
declared, err := declarationWith(ctx, open, name, plan, settings, gens, Allocating)
if err != nil {
refusals = append(refusals, fmt.Sprintf("%s:\n%v", name, err))
continue
}
declared.Sequence = seq
declared.Sequence, declared.Epoch = numbered.sequence, numbered.epoch
reportLeftOut(name, declared)
sending = append(sending, readyNode{name, declared})
}
@@ -1050,6 +1144,20 @@ func digestOf(body []byte) string {
// be worked out" is a different problem with a different remedy, and `plan` is where it is said.
func wouldSend(ctx context.Context, open *stores,
nodes []inventory.Node) (map[string]string, error) {
return wouldSendFrom(ctx, open, nodes, nil)
}
// planned is one machine's plan as planFor answered it, for a caller that already asked.
type planned struct {
plan catalogue.Resolution
settings catalogue.SettingsBy
}
// wouldSendFrom is wouldSend reusing the plans a caller worked out a moment before: resolving a
// machine is most of what `status` costs, and it used to resolve every machine twice (novox/hq
// to-be 45 Phase 0). A machine absent from plans is worked out here.
func wouldSendFrom(ctx context.Context, open *stores,
nodes []inventory.Node, plans map[string]planned) (map[string]string, error) {
gens, err := generators(ctx, open)
if err != nil {
@@ -1057,9 +1165,12 @@ func wouldSend(ctx context.Context, open *stores,
}
out := map[string]string{}
for _, n := range nodes {
plan, settings, err := planFor(ctx, open, n.Name)
if err != nil {
continue
known, have := plans[n.Name]
plan, settings := known.plan, known.settings
if !have {
if plan, settings, err = planFor(ctx, open, n.Name); err != nil {
continue
}
}
declared, err := declarationWith(ctx, open, n.Name, plan, settings, gens, Reading)
if err != nil {
@@ -1071,6 +1182,11 @@ func wouldSend(ctx context.Context, open *stores,
if declared.Sequence, err = open.inventory.Sequence(ctx, n.ID); err != nil {
return nil, err
}
// And the epoch it was last sent under, for the same reason: a new holder of the lease is not a
// change of the machine (novox/hq to-be 45 §6).
if declared.Epoch, err = open.inventory.SentEpoch(ctx, n.ID); err != nil {
return nil, err
}
body, err := declared.Body()
if err != nil {
return nil, err
@@ -1118,35 +1234,22 @@ func raiseTheBus(ctx context.Context, inv *inventory.Inventory, address string)
}
}
nodes, err := inv.Nodes(ctx)
// The mesh's own streams and consumers, the seats' work queues and their workers, and how every
// machine hears its declaration — one derivation, which the self-check reads as well (D6, D7).
names, err := assertBusObjects(ctx, inv, js)
if err != nil {
return err
}
names := make([]string, 0, len(nodes))
for _, n := range nodes {
names = append(names, n.Name)
}
if err := broker.Raise(js, names); err != nil {
return err
}
// The work queues of the mesh's own roles (novox/hq ADR 0121). The queue before the holder,
// deliberately: work queues until somebody arrives to do it, so assigning a build machine a week
// after something started asking for builds flushes the backlog instead of having lost it.
// With the seats' holders, so each role's work queue gets the consumer its holder takes
// work from. Passed as nil until the first live raise, which left the build machine bound to a
// consumer nothing had created (2026-09-28).
holders, err := seatHolders(ctx, inv)
if err != nil {
return err
}
if err := broker.RaiseSeats(js, inventory.MeshSeats(), holders); err != nil {
return err
}
// And each work queue's cancelled set (novox/hq ADR 0219), so a holder taking an ask can ask
// whether it was cancelled the moment it took it.
if err := broker.RaiseCancelledSets(js, inventory.MeshSeats()); err != nil {
return err
}
// And the controller's own buckets (novox/hq to-be 45 §1): the calls it serves and the acts done
// by hand, kept where a restart of this process does not take them.
if err := js.EnsureControllerBuckets(); err != nil {
return err
}
// Every module's state (novox/hq ADR 0201), from the catalogue: a bucket exists from
// registration, so a module reading one may watch it before its owner runs anywhere. One that
// nothing declares any more is said and kept — what it holds is data.
@@ -1162,25 +1265,11 @@ func raiseTheBus(ctx context.Context, inv *inventory.Inventory, address string)
fmt.Printf("the bus holds state nothing declares any more, kept because it is data: %s — "+
"removing it is a person's act\n", strings.Join(undeclared, ", "))
}
// And how every module hears what it consumes. Derived from the same records the user list is
// composed from, so a module the mesh grants a consumer's subjects has that consumer waiting.
// Done on every raise, not only when a credential is issued: every module moved onto this bus
// by the rollout was issued on the old one, and came up with nothing to bind to (2026-09-28).
records, err := inv.BusRecords(ctx)
// And how every module hears what it consumes: asserted with the rest above, counted here.
hearing, err := moduleConsumerCount(ctx, inv)
if err != nil {
return err
}
users, err := broker.Users(records)
if err != nil {
return err
}
hearing := 0
for _, c := range broker.ConsumersOf(users) {
if err := js.EnsureConsumer(c.Consumer); err != nil {
return fmt.Errorf("how %s on %s hears what it consumes: %w", c.Module, c.Node, err)
}
hearing++
}
fmt.Printf("the bus at %s has its streams, %d machine(s) can hear a declaration, %d module(s) "+
"can hear what they consume, and %d bucket(s) of state\n", broker.BareAddress(address), len(names), hearing, len(buckets))
return nil
@@ -1223,17 +1312,46 @@ func seatHolders(ctx context.Context, inv *inventory.Inventory) (map[string]brok
// number gives one send the next sequence for its node (novox/hq 04-ISSUES/107).
// allotting is allot over one inventory, in the shape composeEach takes.
func allotting(ctx context.Context, inv *inventory.Inventory) func(node string) (int64, error) {
return func(node string) (int64, error) { return allot(ctx, inv, node) }
func allotting(ctx context.Context, inv *inventory.Inventory) func(node string) (order, error) {
return func(node string) (order, error) { return allot(ctx, inv, node) }
}
// allot takes the next sequence for a machine — the number its next declaration carries.
func allot(ctx context.Context, inv *inventory.Inventory, node string) (int64, error) {
// epochForActs is the lease's gate as a composition asks it; a variable so a test can act under an epoch
// without a bus.
var epochForActs = func(ctx context.Context) (uint64, error) { return theLease.epoch(ctx) }
// order is what a declaration carries of its writer's order (link/order.go): its sequence, and the
// epoch of the lease it is composed under — zero for a machine that has not said it reads one.
type order struct {
sequence int64
epoch uint64
}
// allot takes the next sequence for a machine — the number its next declaration carries — under the
// lease: a process that may not act takes none, and composes nothing (novox/hq to-be 45 §6).
func allot(ctx context.Context, inv *inventory.Inventory, node string) (order, error) {
epoch, err := epochForActs(ctx)
if err != nil {
return order{}, fmt.Errorf("nothing was composed for %s: %w", node, err)
}
record, err := inv.NodeByName(ctx, node)
if err != nil {
return 0, err
return order{}, err
}
return inv.NextSequence(ctx, record.ID)
if epoch > 0 {
reads, err := inv.ReadsEpoch(ctx, record.ID)
if err != nil {
return order{}, err
}
if !reads {
epoch = 0
}
}
seq, err := inv.NextSequence(ctx, record.ID)
if err != nil {
return order{}, err
}
return order{sequence: seq, epoch: epoch}, nil
}
// recordSent writes down what a machine was just sent, and returns the digest.
@@ -1244,7 +1362,11 @@ func allot(ctx context.Context, inv *inventory.Inventory, node string) (int64, e
// never wrote it down: status read "applied, current" over a machine that had just been sent
// something else. What was sent was sent; the record of it must not depend on the sender living
// another second. Bounded, so a store that is away does not hold a dying process open for ever.
func recordSent(ctx context.Context, inv *inventory.Inventory, node string, body []byte) (string, error) {
//
// And the build of each module it carried (novox/hq issue 259, ADR 0221), nil when that is not known:
// what tells a machine held back by a policy or a plan from one a push left behind.
func recordSent(ctx context.Context, inv *inventory.Inventory, node string, body []byte,
builds map[string]string, epoch uint64) (string, error) {
kept, cancel := context.WithTimeout(context.WithoutCancel(ctx), 10*time.Second)
defer cancel()
record, err := inv.NodeByName(kept, node)
@@ -1252,7 +1374,7 @@ func recordSent(ctx context.Context, inv *inventory.Inventory, node string, body
return "", err
}
digest := digestOf(body)
if err := inv.RecordSent(kept, record.ID, digest); err != nil {
if err := inv.RecordSentUnder(kept, record.ID, digest, builds, epoch); err != nil {
return "", err
}
return digest, nil
+2 -2
View File
@@ -76,7 +76,7 @@ func TestASkippedMachineIsStillAnError(t *testing.T) {
}
// numbered is an allotter for tests: one higher per call, as the inventory's is per machine.
func numbered() func(string) (int64, error) {
func numbered() func(string) (order, error) {
var n int64
return func(string) (int64, error) { n++; return n, nil }
return func(string) (order, error) { n++; return order{sequence: n}, nil }
}
+7 -7
View File
@@ -64,7 +64,7 @@ func TestAFailedPlanIsRetriedAndGoesOnThroughItsLaterTiers(t *testing.T) {
Note: "a failed to build in tier 0",
Modules: map[string]*inventory.PlanModule{"a": {State: "failed", AskedAt: &before, Build: "build-1",
Why: link.KilledByHand}}}
if err := open.inventory.SavePlan(ctx, failed); err != nil {
if err := open.inventory.SavePlan(ctx, &failed); err != nil {
t.Fatal(err)
}
@@ -156,7 +156,7 @@ func TestARebuildJoinsThePlanHoldingTheModule(t *testing.T) {
Created: before, State: inventory.PlanFailed, Tiers: [][]string{{"a"}, {"b"}},
Note: "a failed to build in tier 0",
Modules: map[string]*inventory.PlanModule{"a": {State: "failed", AskedAt: &before, Build: "build-1", Why: link.CancelledByHand}}}
if err := open.inventory.SavePlan(ctx, failed); err != nil {
if err := open.inventory.SavePlan(ctx, &failed); err != nil {
t.Fatal(err)
}
if err := rebuildCommand(ctx, []string{"a"}); err != nil {
@@ -433,7 +433,7 @@ func TestCancelDeletesTheAskAndFailsThePlanThatAskedIt(t *testing.T) {
plan := inventory.Plan{ID: "plan-cancel", Repository: "novox/a", Commit: "c0ffee", Created: asked,
State: inventory.PlanBuilding, Tiers: [][]string{{"a"}, {"b"}},
Modules: map[string]*inventory.PlanModule{"a": {State: "asked", AskedAt: &asked, Build: id}}}
if err := open.inventory.SavePlan(ctx, plan); err != nil {
if err := open.inventory.SavePlan(ctx, &plan); err != nil {
t.Fatal(err)
}
@@ -631,21 +631,21 @@ func TestAPlanStoppedAtItsFirstMachineIsRetried(t *testing.T) {
Note: "a stopped at its first machine in tier 0: laptop refused what it was sent",
Modules: map[string]*inventory.PlanModule{"a": {State: "built", BuiltAt: &long, Commit: "c0ffee",
First: []string{"laptop"}, FirstAt: &long, Why: "laptop refused what it was sent"}}}
if err := open.inventory.SavePlan(ctx, stopped); err != nil {
if err := open.inventory.SavePlan(ctx, &stopped); err != nil {
t.Fatal(err)
}
// A newer plan holding a refuses it: sending the older build would put it back.
newer := inventory.Plan{ID: "plan-newer", Repository: "novox/other", Commit: "d00d", Created: long.Add(time.Hour),
State: inventory.PlanDone, Tiers: [][]string{{"a"}}, Modules: map[string]*inventory.PlanModule{}}
if err := open.inventory.SavePlan(ctx, newer); err != nil {
if err := open.inventory.SavePlan(ctx, &newer); err != nil {
t.Fatal(err)
}
if _, err := retryPlan(ctx, open, stopped.ID); err == nil || !strings.Contains(err.Error(), "plan-newer") {
t.Fatalf("retried under a newer plan: %v", err)
}
newer.State = inventory.PlanSuperseded
if err := open.inventory.SavePlan(ctx, newer); err != nil {
if err := open.inventory.SavePlan(ctx, &newer); err != nil {
t.Fatal(err)
}
@@ -683,7 +683,7 @@ func TestAPlanIsAnsweredOnlyByTheBuildItAskedFor(t *testing.T) {
plan := inventory.Plan{ID: "plan-own", Repository: "novox/a", Commit: "c0ffee", Created: asked,
State: inventory.PlanBuilding, Tiers: [][]string{{"a"}},
Modules: map[string]*inventory.PlanModule{"a": {State: "asked", AskedAt: &asked, Build: "build-own"}}}
if err := open.inventory.SavePlan(ctx, plan); err != nil {
if err := open.inventory.SavePlan(ctx, &plan); err != nil {
t.Fatal(err)
}
planBuilt(ctx, open, "a", "0ldc0mm1t", "", time.Now().UTC(), "build-replay")
+31
View File
@@ -4,6 +4,7 @@ import (
"encoding/json"
"fmt"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"sort"
"time"
@@ -78,6 +79,28 @@ type meshStatus struct {
// that machine holds, with the modules that could hold it (novox/hq ADR 0207). Absent when every
// dependency is met. Reported, not refused, until the switch.
Unheld []catalogue.Unheld `json:"unheld,omitempty"`
// HandActsThisWeek is how many acts were done by hand in the last seven days (novox/hq to-be 45
// §7): every one is a repair a healer could have made. Absent where the log is not on hand;
// HandActsUnread says why when it could not be read, rather than reading as none.
HandActsThisWeek *int `json:"handActsThisWeek,omitempty"`
HandActsUnread string `json:"handActsUnread,omitempty"`
// HealsThisWeek is what the healers did in the last seven days (novox/hq to-be 45 §7); HealsUnread
// why it could not be read.
HealsThisWeek *healsCount `json:"healsThisWeek,omitempty"`
HealsUnread string `json:"healsUnread,omitempty"`
// Conditions is every open condition, urgent first and then oldest first (novox/hq to-be 45 §2):
// what is wrong, as the watchdogs, the self-check and the providers say it. Always present — an
// empty list is "none open" — unless they could not be read, which ConditionsUnread says.
Conditions []conditions.Condition `json:"conditions"`
ConditionsUnread string `json:"conditionsUnread,omitempty"`
// Failing is every consumer a provider says it keeps failing (novox/hq ADR 0224): the open
// conditions of that kind, carried here as well because ADR 0224 names this field. Absent when no
// provider says so. A document without it called the mesh well while the identity provider
// refused every consumer for a day (04-ISSUES/179).
Failing []conditions.Condition `json:"failing,omitempty"`
// Overflowing is every module whose identity overflows the bound of a provision it requires, and
// so is left out of its provider's grants (novox/hq ADR 0225). Absent when every identity fits.
Overflowing []catalogue.Overflow `json:"overflowing,omitempty"`
}
// machineFiltered is one rule set on a converged machine that the mesh did not write and that
@@ -210,6 +233,14 @@ func statusAsJSON(asked answers) ([]byte, error) {
}
}
out.Unheld = asked.unheld
out.HandActsThisWeek, out.HandActsUnread = asked.handActs, asked.handActsUnread
out.HealsThisWeek, out.HealsUnread = asked.heals, asked.healsUnread
out.Conditions, out.ConditionsUnread = asked.conditions, asked.conditionsUnread
if out.Conditions == nil {
out.Conditions = []conditions.Condition{}
}
out.Failing = providerStandings(asked.conditions)
out.Overflowing = asked.overflowing
for name := range asked.refused {
out.Unresolved = append(out.Unresolved, machineUnresolved{
Node: name, Problem: asked.refused[name]})
+14 -5
View File
@@ -441,7 +441,7 @@ func planBuilt(ctx context.Context, open *stores, module, commit, failed string,
state.BuiltAt = &now
state.Commit = commit
}
if err := inv.SavePlan(ctx, *p); err != nil {
if err := inv.SavePlan(ctx, p); err != nil {
fmt.Printf("%s: cannot keep the plan: %v\n", p.ID, err)
continue
}
@@ -500,7 +500,7 @@ func advanceHeld(ctx context.Context, open *stores) {
// Kept in the plan, so `plans` says why it has not moved rather than the log alone;
// the state is left as it was and the step is tried again on the next tick.
p.Note = "tier " + fmt.Sprint(p.Tier) + ": " + err.Error() + " — tried again"
if err := inv.SavePlan(ctx, *p); err != nil {
if err := inv.SavePlan(ctx, p); err != nil {
fmt.Printf("%s: cannot keep the plan: %v\n", p.ID, err)
}
break
@@ -508,7 +508,7 @@ func advanceHeld(ctx context.Context, open *stores) {
if p.State == inventory.PlanFailed {
sayUnsent(p, rollsOut)
}
if err := inv.SavePlan(ctx, *p); err != nil {
if err := inv.SavePlan(ctx, p); err != nil {
fmt.Printf("%s: cannot keep the plan: %v\n", p.ID, err)
break
}
@@ -986,10 +986,18 @@ func plansCommand(ctx context.Context, args []string) error {
whatIf := set.String("what-if", "", "owner/repository: the plan a merge there would produce, saving nothing — with --paths or --modules")
paths := set.String("paths", "", "the files the merge would change, comma-separated, from the repository's root")
modules := set.String("modules", "", "or the modules it would change, comma-separated")
// Ending a plan by hand is a repair, and says why (novox/hq to-be 45 §7).
why := addHandActFlags(set)
positionals, err := parseAround(set, args)
if err != nil {
return err
}
if len(positionals) == 2 && (positionals[0] == "stop" || positionals[0] == "close") {
// Refused before anything is opened: a repair by hand says why.
if err := why.require("plans " + positionals[0]); err != nil {
return err
}
}
open, err := openStores(ctx)
if err != nil {
return err
@@ -1061,13 +1069,14 @@ func plansCommand(ctx context.Context, args []string) error {
if !p.Open() {
return fmt.Errorf("%s is already %s", p.ID, p.State)
}
why.record(ctx, "plans "+positionals[0], positionals[1:])
p.State = inventory.PlanFailed
p.Note = how + " by hand at tier " + fmt.Sprint(p.Tier)
p.Note = how + " by hand at tier " + fmt.Sprint(p.Tier) + ": " + strings.TrimSpace(*why.why)
sayUnsent(&p, func(m string) bool {
u, err := inv.UpgradeOf(ctx, m)
return err == nil && u.RollOut
})
if err := inv.SavePlan(ctx, p); err != nil {
if err := inv.SavePlan(ctx, &p); err != nil {
return err
}
fmt.Printf("%s %s at tier %d of %d; what was asked still builds and registers, nothing further is asked\n",
+479
View File
@@ -0,0 +1,479 @@
package main
import (
"context"
"encoding/json"
"errors"
"flag"
"fmt"
"sort"
"strconv"
"strings"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// The verbs a person answers a provider's retirement with, and cleans up with (novox/hq ADR 0230).
//
// **The controller asks; the provider acts.** Approving, rejecting and deleting are each a question to
// the provider on the machine it runs on — its `provisioner_*` tools — because the provider owns its
// backend and the controller touches none. Every one says why, is written in the hand-act log before
// it is asked, and the provider says what it did as its `provisioner.retirement` event.
const retireUsage = "retire [--json] | retire approve <node> <module> --why <text> | retire reject <node> <module> --why <text>"
const cleanupUsage = "cleanup [list] [--json] | cleanup delete <node> <module> <consumer> --why <text> | " +
"cleanup delete --older-than <days> --why <text> [--confirm]"
func isNothingServes(err error) bool { return errors.Is(err, link.ErrNothingServes) }
func unmarshalAnswer(a link.Answer, v any) error {
if len(a.Result) == 0 {
return errors.New("an empty answer")
}
return json.Unmarshal(a.Result, v)
}
// retireCommand is `retire`, `retire approve` and `retire reject`.
func retireCommand(ctx context.Context, args []string) error {
sub := "list"
if len(args) > 0 && !strings.HasPrefix(args[0], "-") {
sub, args = args[0], args[1:]
}
switch sub {
case "list":
set := flag.NewFlagSet("retire", flag.ContinueOnError)
asJSON := set.Bool("json", false, "as data")
if rest, err := parseAround(set, args); err != nil {
return err
} else if len(rest) > 0 {
return errors.New(retireUsage)
}
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
return onTheBus(func(conn *nats.Conn) error { return listRetiring(ctx, open.inventory, conn, *asJSON) })
case "approve", "reject":
set := flag.NewFlagSet("retire "+sub, flag.ContinueOnError)
f := addHandActFlags(set)
rest, err := parseAround(set, args)
if err != nil {
return err
}
if len(rest) != 2 {
return errors.New(retireUsage)
}
if err := f.require("retire " + sub); err != nil {
return err
}
return onTheBus(func(conn *nats.Conn) error {
return answerRetirement(ctx, conn, providerInstance{Node: rest[0], Module: rest[1]}, sub == "approve", f)
})
}
return errors.New(retireUsage)
}
// retiringRow is one provider's waiting or rejected set, as `retire` lists it.
type retiringRow struct {
Node string `json:"node"`
Module string `json:"module"`
Waiting []link.RetiredConsumer `json:"waiting,omitempty"`
Since string `json:"since,omitempty"`
Held int `json:"held,omitempty"`
Bound string `json:"bound,omitempty"`
Rejected []link.RetiredConsumer `json:"rejected,omitempty"`
RejectWhy string `json:"rejected-why,omitempty"`
Unasked string `json:"unasked,omitempty"`
}
func listRetiring(ctx context.Context, inv *inventory.Inventory, conn *nats.Conn, asJSON bool) error {
instances, err := providerInstances(ctx, inv)
if err != nil {
return err
}
var rows []retiringRow
for _, p := range instances {
row := retiringRow{Node: p.Node, Module: p.Module}
state, err := askRetirement(ctx, conn, p)
switch {
case isNothingServes(err):
row.Unasked = "answers no retirement question: it predates ADR 0230"
case err != nil:
row.Unasked = err.Error()
default:
if state.Waiting != nil {
row.Waiting, row.Since, row.Held = state.Waiting.Consumers, state.Waiting.Since, state.Waiting.Held
}
if state.Rejected != nil {
row.Rejected, row.RejectWhy = state.Rejected.Consumers, orWhy(state.Rejected.By, state.Rejected.Why)
}
row.Bound = state.Bound
if row.Waiting == nil && row.Rejected == nil {
continue
}
}
rows = append(rows, row)
}
if asJSON {
if rows == nil {
rows = []retiringRow{}
}
return printJSON(map[string]any{"providers": rows})
}
said := false
for _, r := range rows {
switch {
case r.Unasked != "":
fmt.Printf("%s on %s: not asked — %s\n", r.Module, r.Node, r.Unasked)
default:
if r.Waiting != nil {
said = true
fmt.Printf("%s on %s WAITS since %s to retire %d of the %d it holds (%s): %s\n"+
" retire approve %s %s --why … | retire reject %s %s --why …\n",
r.Module, r.Node, r.Since, len(r.Waiting), r.Held, orBound(r.Bound), consumerList(r.Waiting),
r.Node, r.Module, r.Node, r.Module)
}
if r.Rejected != nil {
said = true
fmt.Printf("%s on %s keeps active, by a rejection (%s): %s\n", r.Module, r.Node, r.RejectWhy,
consumerList(r.Rejected))
}
}
}
if !said {
fmt.Println("no provider waits for a person to approve a retirement")
}
return nil
}
// answerRetirement approves or rejects what one provider waits with. **The set sent is the set the
// provider says it waits with, read now**, and the provider refuses any other: a person approves what
// they were shown, never a set that moved since.
func answerRetirement(ctx context.Context, conn *nats.Conn, p providerInstance, approve bool, f handActFlags) error {
state, err := askRetirement(ctx, conn, p)
if err != nil {
return err
}
verb, tool := "retire reject", link.ToolRetireReject
var set []link.RetiredConsumer
if state.Waiting != nil {
set = state.Waiting.Consumers
}
if approve {
verb, tool = "retire approve", link.ToolRetireApprove
if set == nil && state.Rejected != nil {
// A rejection can be taken back: the consumers it kept are retired after all.
set = state.Rejected.Consumers
}
}
if len(set) == 0 {
return fmt.Errorf("%s on %s waits for nobody to approve or reject a retirement. Nothing was done", p.Module, p.Node)
}
names := make([]string, 0, len(set))
for _, c := range set {
names = append(names, c.Consumer)
}
if strings.TrimSpace(*f.cause) == "" {
*f.cause = kindRetireWaiting
}
if strings.TrimSpace(*f.condition) == "" {
*f.condition = retireWaitingKey(p.Module, p.Node)
}
f.record(ctx, verb, append([]string{p.Node, p.Module}, names...))
answer, err := link.AskModuleToolOn(ctx, conn, p.Module, tool, p.Node, map[string]any{
"consumers": names, "why": strings.TrimSpace(*f.why), "by": link.Caller(), "via": link.ViaController,
}, retirementAsk)
if err != nil {
return err
}
if answer.Error != "" {
return fmt.Errorf("%s on %s refused: %s", p.Module, p.Node, answer.Error)
}
if approve && state.RetiresBy == "mark-only" {
fmt.Printf("%s on %s marked %s retired, MARK ONLY: this provider cannot disable a consumer, so they keep "+
"their access until `cleanup delete`; `cleanup list` shows them\n", p.Module, p.Node, strings.Join(names, ", "))
} else if approve {
fmt.Printf("%s on %s retired %s: access disabled, data kept; `cleanup list` shows them\n", p.Module, p.Node,
strings.Join(names, ", "))
} else {
fmt.Printf("%s on %s keeps %s active; the warning stays open until the mesh asks for them again or the "+
"retirement is approved\n", p.Module, p.Node, strings.Join(names, ", "))
}
return nil
}
// cleanupCommand is `cleanup list` and `cleanup delete`.
func cleanupCommand(ctx context.Context, args []string) error {
sub := "list"
if len(args) > 0 && !strings.HasPrefix(args[0], "-") {
sub, args = args[0], args[1:]
}
switch sub {
case "list":
set := flag.NewFlagSet("cleanup", flag.ContinueOnError)
asJSON := set.Bool("json", false, "as data")
if rest, err := parseAround(set, args); err != nil {
return err
} else if len(rest) > 0 {
return errors.New(cleanupUsage)
}
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
return onTheBus(func(conn *nats.Conn) error {
listing, err := gatherRetired(ctx, open.inventory, conn, time.Now())
if err != nil {
return err
}
return printRetired(listing, *asJSON)
})
case "delete":
set := flag.NewFlagSet("cleanup delete", flag.ContinueOnError)
f := addHandActFlags(set)
olderThan := set.Int("older-than", 0, "every retired consumer older than this many days")
confirm := set.Bool("confirm", false, "with --older-than: delete what is listed, rather than only list it")
rest, err := parseAround(set, args)
if err != nil {
return err
}
if err := f.require("cleanup delete"); err != nil {
return err
}
if strings.TrimSpace(*f.cause) == "" {
*f.cause = kindCleanupWaiting
}
switch {
case *olderThan > 0 && len(rest) == 0:
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
return onTheBus(func(conn *nats.Conn) error {
return deleteOlderThan(ctx, open, conn, *olderThan, *confirm, f, time.Now())
})
case *olderThan == 0 && len(rest) == 3 && !*confirm:
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
return onTheBus(func(conn *nats.Conn) error {
// A module's own retired data, when the mesh holds such an item (novox/hq ADR 0233); a
// provider's retired consumer otherwise.
if r, err := open.inventory.DataOf(ctx, rest[0], rest[1], rest[2]); err == nil && r.DeletedAt == nil &&
r.RetiredAt != nil {
return deleteRetiredData(ctx, conn, open, r, f)
}
return deleteRetired(ctx, conn, providerInstance{Node: rest[0], Module: rest[1]}, rest[2], f)
})
}
return errors.New(cleanupUsage)
}
return errors.New(cleanupUsage)
}
// retiredRow is one retired consumer, as `cleanup list` shows it.
type retiredRow struct {
Node string `json:"node"`
Module string `json:"module"`
Consumer string `json:"consumer"`
// ConsumerNode is where the consumer was, when the provider knows.
ConsumerNode string `json:"consumer-node,omitempty"`
Kind string `json:"kind,omitempty"`
RetiredAt string `json:"retired-at,omitempty"`
// AgeDays is whole days since it was retired; -1 when the provider could not say when.
AgeDays int `json:"age-days"`
SizeBytes *int64 `json:"size-bytes,omitempty"`
Why string `json:"why,omitempty"`
// Access is "kept" for a consumer of a mark-only provider: retired on record, still reachable.
Access string `json:"access,omitempty"`
// Path and Class are a module's own retired data's (Kind own-data, novox/hq ADR 0233): where it is
// kept on its machine, and its class. Consumer is then the item.
Path string `json:"path,omitempty"`
Class string `json:"class,omitempty"`
}
// retiredListing is every provider's retired consumers, and the providers that could not say.
type retiredListing struct {
Retired []retiredRow `json:"retired"`
// Unasked names each provider not asked, and why: one older than the question, or one that failed.
Unasked []string `json:"unasked,omitempty"`
}
func gatherRetired(ctx context.Context, inv *inventory.Inventory, conn *nats.Conn, now time.Time) (retiredListing, error) {
instances, err := providerInstances(ctx, inv)
if err != nil {
return retiredListing{}, err
}
listing := retiredOf(ctx, conn, instances, now)
// And every module's own data retired on its machine (novox/hq ADR 0233).
records, err := inv.Data(ctx)
if err != nil {
return retiredListing{}, err
}
listing.Retired = append(listing.Retired, retiredData(records, now)...)
sort.SliceStable(listing.Retired, func(i, j int) bool { return listing.Retired[i].AgeDays > listing.Retired[j].AgeDays })
return listing, nil
}
func retiredOf(ctx context.Context, conn *nats.Conn, instances []providerInstance, now time.Time) retiredListing {
out := retiredListing{Retired: []retiredRow{}}
for _, p := range instances {
state, err := askRetirement(ctx, conn, p)
if err != nil {
if isNothingServes(err) {
out.Unasked = append(out.Unasked, fmt.Sprintf("%s on %s answers no retirement question: it predates ADR 0230", p.Module, p.Node))
} else {
out.Unasked = append(out.Unasked, err.Error())
}
continue
}
for _, c := range state.Retired {
row := retiredRow{Node: p.Node, Module: p.Module, Consumer: c.Consumer, ConsumerNode: c.Node, Kind: c.Kind,
RetiredAt: c.RetiredAt, AgeDays: -1, SizeBytes: c.SizeBytes, Why: c.Why, Access: c.Access}
if at := retiredAt(c); !at.IsZero() {
row.AgeDays = int(now.Sub(at).Hours() / 24)
}
out.Retired = append(out.Retired, row)
}
}
sort.SliceStable(out.Retired, func(i, j int) bool { return out.Retired[i].AgeDays > out.Retired[j].AgeDays })
return out
}
func printRetired(l retiredListing, asJSON bool) error {
if asJSON {
return printJSON(l)
}
if len(l.Retired) == 0 {
fmt.Println("no provider holds a retired consumer, and no machine holds retired data")
}
for _, r := range l.Retired {
age := "age unknown"
if r.AgeDays >= 0 {
age = strconv.Itoa(r.AgeDays) + " day(s)"
}
kind := ""
if r.Kind != "" && r.Kind != "consumer" {
kind = " [" + r.Kind + "]"
}
if r.Access == "kept" {
kind += " [MARK ONLY: access kept until deleted]"
}
if r.Kind == retiredDataKind {
fmt.Printf("%s on %s: its own %s, %s, at %s — retired %s, %s, %s\n %s\n", r.Module, r.Node, r.Consumer,
r.Class, r.Path, age, sizeWords(r.SizeBytes), r.RetiredAt, orWhy("", r.Why))
continue
}
fmt.Printf("%s on %s: %s%s — retired %s, %s, %s\n %s\n", r.Module, r.Node, r.Consumer, kind, age,
sizeWords(r.SizeBytes), r.RetiredAt, orWhy("", r.Why))
}
for _, u := range l.Unasked {
fmt.Printf("not asked: %s\n", u)
}
return nil
}
// deleteRetired asks one provider to delete one consumer it holds retired — never an active one: the
// provider refuses that, and this refuses it first, from what the provider says it holds.
func deleteRetired(ctx context.Context, conn *nats.Conn, p providerInstance, consumer string, f handActFlags) error {
state, err := askRetirement(ctx, conn, p)
if err != nil {
return err
}
found := false
for _, c := range state.Retired {
found = found || c.Consumer == consumer
}
if !found {
return fmt.Errorf("%s on %s holds no retired consumer %s — only a retired consumer is deleted. Nothing was done",
p.Module, p.Node, consumer)
}
f.record(ctx, "cleanup delete", []string{p.Node, p.Module, consumer})
answer, err := link.AskModuleToolOn(ctx, conn, p.Module, link.ToolRetiredDelete, p.Node, map[string]any{
"consumer": consumer, "confirm": consumer, "why": strings.TrimSpace(*f.why), "by": link.Caller(),
"via": link.ViaController,
}, retirementAsk)
if err != nil {
return err
}
if answer.Error != "" {
return fmt.Errorf("%s on %s refused to delete %s: %s", p.Module, p.Node, consumer, answer.Error)
}
var done struct {
FreedBytes *int64 `json:"freed_bytes"`
}
_ = unmarshalAnswer(answer, &done)
fmt.Printf("%s on %s deleted %s (%s freed)\n", p.Module, p.Node, consumer, sizeWords(done.FreedBytes))
return nil
}
// deleteOlderThan lists every consumer retired more than days ago, and deletes them only with confirm.
// One whose age the provider cannot say is never in it.
func deleteOlderThan(ctx context.Context, open *stores, conn *nats.Conn, days int, confirm bool,
f handActFlags, now time.Time) error {
listing, err := gatherRetired(ctx, open.inventory, conn, now)
if err != nil {
return err
}
return deleteFrom(ctx, conn, open, listing, days, confirm, f)
}
func deleteFrom(ctx context.Context, conn *nats.Conn, open *stores, listing retiredListing, days int, confirm bool, f handActFlags) error {
var due []retiredRow
unknown := 0
for _, r := range listing.Retired {
switch {
case r.AgeDays < 0:
unknown++
case r.AgeDays > days:
due = append(due, r)
}
}
for _, u := range listing.Unasked {
fmt.Printf("not asked: %s\n", u)
}
if unknown > 0 {
fmt.Printf("%d retired consumer(s) whose age their provider cannot say are left out\n", unknown)
}
if len(due) == 0 {
fmt.Printf("nothing has been retired more than %d day(s)\n", days)
return nil
}
fmt.Printf("retired more than %d day(s):\n", days)
for _, r := range due {
fmt.Printf(" %s on %s: %s — %d day(s), %s\n", r.Module, r.Node, r.Consumer, r.AgeDays, sizeWords(r.SizeBytes))
}
if !confirm {
fmt.Printf("nothing was deleted: add --confirm to delete these %d\n", len(due))
return nil
}
var failed []string
for _, r := range due {
var err error
if r.Kind == retiredDataKind {
var rec inventory.DataRecord
if rec, err = open.inventory.DataOf(ctx, r.Node, r.Module, r.Consumer); err == nil {
err = deleteRetiredData(ctx, conn, open, rec, f)
}
} else {
err = deleteRetired(ctx, conn, providerInstance{Node: r.Node, Module: r.Module}, r.Consumer, f)
}
if err != nil {
failed = append(failed, err.Error())
}
}
if len(failed) > 0 {
return fmt.Errorf("%d of %d not deleted: %s", len(failed), len(due), strings.Join(failed, "; "))
}
return nil
}
+343
View File
@@ -0,0 +1,343 @@
package main
import (
"context"
"fmt"
"sort"
"strings"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A consumer the mesh stops asking for is retired, not withdrawn, and deleted only by a person
// (novox/hq ADR 0230, the operator's decision of 2026-10-06; it replaces ADR 0229's withdrawal brake).
//
// A provider retires a consumer — disables its access, keeps its data, marks it with when and why —
// once it has seen the same consumers go unasked for in five passes. More than three at once, or more
// than half of those it holds where it holds more than one, is a person actively working on the mesh,
// so the provider retires nothing and waits: the controller keeps that as an urgent condition naming
// what would go and the two verbs that answer it. A retired consumer is deleted only by `cleanup
// delete`, which the provider executes on its own backend — the controller never touches one — and
// one retired more than thirty days is a warning that cleanup is waiting (D11).
// The kinds a provider's retirement word raises.
const (
kindRetireWaiting = "retire-waiting"
kindRetireRejected = "retire-rejected"
kindCleanupWaiting = "cleanup-waiting"
// sourceRetirement is what raised a retirement condition: the provider's own event.
sourceRetirement = "provisioner.retirement"
)
// cleanupAfter is how long a consumer may stay retired before cleanup is said to be waiting (D11).
const cleanupAfter = 30 * 24 * time.Hour
// retirementAsk is how long one provider is given to answer one question about its retired consumers.
var retirementAsk = 20 * time.Second
func retireWaitingKey(module, node string) string {
return conditions.Key(conditions.ScopeProvider, module+"."+node, "retire")
}
func retireRejectedKey(module, node string) string {
return conditions.Key(conditions.ScopeProvider, module+"."+node, "retire-rejected")
}
// retirements keeps what providers say about consumers they retire, as conditions.
type retirements struct {
keeper func() *conditions.Keeper
// record writes an act done by hand that reached a provider some other way than this controller's
// verbs; nil records nothing (a test).
record func(ctx context.Context, act link.HandAct) error
}
func consumerList(cs []link.RetiredConsumer) string {
parts := make([]string, 0, len(cs))
for _, c := range cs {
p := c.Consumer
if c.Node != "" {
p += " (" + c.Node + ")"
}
parts = append(parts, p)
}
return strings.Join(parts, ", ")
}
// waitingObservation is a provider waiting for a person to approve or reject a retirement.
func waitingObservation(r link.Retirement) conditions.Observation {
since := r.Since
if since.IsZero() {
since = r.At
}
summary := fmt.Sprintf("%s on %s would retire %d consumer(s) the mesh no longer asks for — %s — more than its "+
"bound (%s), so it retires nothing until a person answers: `retire approve %s %s --why …` or `retire reject "+
"%s %s --why …`", r.Module, r.ProviderNode, len(r.Consumers), consumerList(r.Consumers), orBound(r.Bound),
r.ProviderNode, r.Module, r.ProviderNode, r.Module)
said := fmt.Sprintf("waiting since %s, %d held: %s", since.UTC().Format("2006-01-02 15:04 MST"), r.Held,
consumerList(r.Consumers))
return conditions.Observation{Scope: conditions.ScopeProvider, ID: r.Module + "." + r.ProviderNode,
Token: "retire", Kind: kindRetireWaiting, Machine: r.ProviderNode, Also: consumerNodes(r),
Severity: conditions.Urgent, Summary: summary, Said: said, Source: sourceRetirement,
Resolver: conditions.ResolverOperator}
}
// rejectedObservation is consumers kept active by a person's rejection though the mesh asks for them no more.
func rejectedObservation(r link.Retirement) conditions.Observation {
summary := fmt.Sprintf("%s on %s keeps %s active although the mesh no longer asks for them: a person rejected "+
"their retirement (%s). Assign them again, or `retire approve %s %s --why …`", r.Module, r.ProviderNode,
consumerList(r.Consumers), orWhy(r.By, r.Why), r.ProviderNode, r.Module)
return conditions.Observation{Scope: conditions.ScopeProvider, ID: r.Module + "." + r.ProviderNode,
Token: "retire-rejected", Kind: kindRetireRejected, Machine: r.ProviderNode, Also: consumerNodes(r),
Severity: conditions.Warning, Summary: summary, Said: "rejected: " + orWhy(r.By, r.Why),
Source: sourceRetirement, Resolver: conditions.ResolverOperator}
}
func orBound(b string) string {
if b == "" {
return "more than 3, or more than half of those held"
}
return b
}
func orWhy(by, why string) string {
switch {
case by != "" && why != "":
return by + ": " + why
case why != "":
return why
case by != "":
return "by " + by
}
return "no reason given"
}
// consumerNodes are the other machines a retirement concerns: where its consumers are.
func consumerNodes(r link.Retirement) []string {
seen := map[string]bool{r.ProviderNode: true}
var out []string
for _, c := range r.Consumers {
if c.Node != "" && !seen[c.Node] {
seen[c.Node] = true
out = append(out, c.Node)
}
}
sort.Strings(out)
return out
}
// Retired keeps one word. Waiting raises the urgent condition; a rejection turns it into a warning; a
// retirement, an approval or the set settling clears both. An approval, a rejection or a deletion that
// did not come through this controller's verbs is recorded in the hand-act log here, so every one is.
func (s retirements) Retired(ctx context.Context, r link.Retirement) error {
k := s.keeper()
if k == nil {
return fmt.Errorf("the condition store is not open in this controller: %w", link.ErrTryAgain)
}
waiting, rejected := retireWaitingKey(r.Module, r.ProviderNode), retireRejectedKey(r.Module, r.ProviderNode)
clear := func(keys ...string) error {
for _, key := range keys {
if _, err := k.Clear(ctx, key, fmt.Sprintf("%s on %s says %s", r.Module, r.ProviderNode, r.Change)); err != nil {
return storeAway(err)
}
}
return nil
}
var err error
switch r.Change {
case link.RetireWaiting:
if _, err = k.Observe(ctx, waitingObservation(r)); err != nil {
return storeAway(err)
}
case link.RetireRejected:
if err = clear(waiting); err != nil {
return err
}
if _, err = k.Observe(ctx, rejectedObservation(r)); err != nil {
return storeAway(err)
}
case link.RetireApproved, link.RetireRetired, link.RetireSettled:
if err = clear(waiting, rejected); err != nil {
return err
}
}
switch r.Change {
case link.RetireApproved, link.RetireRejected, link.RetireDeleted:
if r.Via != link.ViaController && s.record != nil {
verb := map[string]string{link.RetireApproved: "retire approve", link.RetireRejected: "retire reject",
link.RetireDeleted: "cleanup delete"}[r.Change]
cause := kindRetireWaiting
if r.Change == link.RetireDeleted {
cause = kindCleanupWaiting
}
why := r.Why
if strings.TrimSpace(why) == "" {
why = "no reason given to the provider"
}
act := link.HandAct{Verb: verb, Args: append([]string{r.ProviderNode, r.Module}, r.Names()...),
Why: why + " (asked of the provider directly, not through the controller)", Cause: cause,
By: r.By, At: r.At}
if act.By == "" {
act.By = "unknown, asked of " + r.Module + " on " + r.ProviderNode + " directly"
}
if err := s.record(ctx, act); err != nil {
return fmt.Errorf("%s on %s %s by hand, and it could not be recorded: %v: %w", r.Module,
r.ProviderNode, r.Change, err, link.ErrTryAgain)
}
}
}
return nil
}
// recordHandActOnTheBus writes an act through the serving controller's connection.
func recordHandActOnTheBus(ctx context.Context, act link.HandAct) error {
return onTheBus(func(conn *nats.Conn) error {
_, err := link.RecordHandAct(ctx, conn, act)
return err
})
}
// providerInstance is one provider module on one machine.
type providerInstance struct {
Node, Module string
}
// providerInstances are every module assigned on every machine whose manifest receives contributions:
// every provider, which serves the retirement tools (ADR 0230) — or is older than them.
func providerInstances(ctx context.Context, inv *inventory.Inventory) ([]providerInstance, error) {
shelf, err := inv.Catalogue(ctx)
if err != nil {
return nil, err
}
nodes, err := inv.Nodes(ctx)
if err != nil {
return nil, err
}
var out []providerInstance
for _, n := range nodes {
modules, err := inv.Assigned(ctx, n.Name)
if err != nil {
return nil, fmt.Errorf("what %s is assigned cannot be read: %w", n.Name, err)
}
for _, m := range modules {
if man, ok := shelf[m]; ok && len(man.Receives) > 0 {
out = append(out, providerInstance{Node: n.Name, Module: m})
}
}
}
sort.Slice(out, func(i, j int) bool {
if out[i].Node != out[j].Node {
return out[i].Node < out[j].Node
}
return out[i].Module < out[j].Module
})
return out, nil
}
// askRetirement asks one provider for what it holds retired and what waits.
func askRetirement(ctx context.Context, conn *nats.Conn, p providerInstance) (link.RetirementState, error) {
var state link.RetirementState
answer, err := link.AskModuleToolOn(ctx, conn, p.Module, link.ToolRetirement, p.Node, map[string]any{}, retirementAsk)
if err != nil {
return state, err
}
if answer.Error != "" {
return state, fmt.Errorf("%s on %s answered %s with an error: %s", p.Module, p.Node, link.ToolRetirement, answer.Error)
}
if err := unmarshalAnswer(answer, &state); err != nil {
return state, fmt.Errorf("%s on %s answered %s with something unreadable: %w", p.Module, p.Node, link.ToolRetirement, err)
}
return state, nil
}
// retiredAt reads a retired consumer's moment; zero when the provider could not say.
func retiredAt(c link.RetiredConsumer) time.Time {
t, err := time.Parse(time.RFC3339, c.RetiredAt)
if err != nil {
return time.Time{}
}
return t
}
// cleanupObservation is a provider holding consumers retired longer than cleanupAfter.
func cleanupObservation(p providerInstance, old []link.RetiredConsumer, now time.Time) conditions.Observation {
parts := make([]string, 0, len(old))
for _, c := range old {
parts = append(parts, fmt.Sprintf("%s (%d days, %s)", c.Consumer, int(now.Sub(retiredAt(c)).Hours()/24),
sizeWords(c.SizeBytes)))
}
return conditions.Observation{Scope: conditions.ScopeProvider, ID: p.Module + "." + p.Node, Token: "cleanup",
Kind: kindCleanupWaiting, Machine: p.Node, Severity: conditions.Warning,
Summary: fmt.Sprintf("%s on %s holds %d consumer(s) retired more than %d days, waiting for a person to "+
"delete or bring them back: %s — `cleanup list`, then `cleanup delete %s %s <consumer> --why …`",
p.Module, p.Node, len(old), int(cleanupAfter.Hours()/24), strings.Join(parts, ", "), p.Node, p.Module),
Resolver: conditions.ResolverOperator}
}
func sizeWords(size *int64) string {
if size == nil || *size < 0 {
return "size unknown"
}
b := float64(*size)
for _, unit := range []string{"B", "KB", "MB", "GB"} {
if b < 1024 || unit == "GB" {
if unit == "B" {
return fmt.Sprintf("%d B", *size)
}
return fmt.Sprintf("%.1f %s", b, unit)
}
b /= 1024
}
return ""
}
// probeRetired is D11: no provider holds a consumer retired more than thirty days. Every provider
// assigned is asked what it holds retired; one that does not serve the question — an older build, a
// TypeScript provider not yet on the SDK that retires — has nothing it can say and is passed over,
// which is not a failure of the probe. A provider that cannot be asked otherwise is.
func probeRetired(ctx context.Context, d *doctor) ([]conditions.Observation, error) {
if d.js == nil {
return nil, fmt.Errorf("no bus to ask the providers over")
}
instances, err := providerInstances(ctx, d.open.inventory)
if err != nil {
return nil, err
}
return retiredTooLong(ctx, d.js.Conn(), instances, time.Now())
}
// retiredTooLong asks each provider and answers one observation per provider holding anything retired
// longer than cleanupAfter.
func retiredTooLong(ctx context.Context, conn *nats.Conn, instances []providerInstance,
now time.Time) ([]conditions.Observation, error) {
var out []conditions.Observation
var problems []string
for _, p := range instances {
state, err := askRetirement(ctx, conn, p)
if err != nil {
if isNothingServes(err) {
continue
}
problems = append(problems, err.Error())
continue
}
var old []link.RetiredConsumer
for _, c := range state.Retired {
if at := retiredAt(c); !at.IsZero() && now.Sub(at) > cleanupAfter {
old = append(old, c)
}
}
if len(old) > 0 {
out = append(out, cleanupObservation(p, old, now))
}
}
if len(problems) > 0 {
// Not "nothing retired": a provider that could not be asked may hold the oldest of all.
return nil, fmt.Errorf("%s", strings.Join(problems, "; "))
}
return out, nil
}
+483
View File
@@ -0,0 +1,483 @@
package main
import (
"context"
"encoding/json"
"flag"
"os"
"slices"
"sort"
"strings"
"sync"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
)
// A consumer the mesh stops asking for is retired, not withdrawn, and deleted only by a person
// (novox/hq ADR 0230): what a provider says becomes a condition a person answers, and the verbs that
// answer it ask the provider — never anything else — for exactly what it said.
func waitingWord(module, node string, names ...string) link.Retirement {
r := link.Retirement{Module: module, Provider: "postgres-database", ProviderNode: node, Change: link.RetireWaiting,
Held: 7, Bound: "more than 3, or more than half of the 7 held", At: time.Now(), Since: time.Now()}
for _, n := range names {
r.Consumers = append(r.Consumers, link.RetiredConsumer{Consumer: n, Node: "laptop"})
}
return r
}
func openKeys(t *testing.T, k *conditions.Keeper) map[string]conditions.Condition {
t.Helper()
all, err := k.Open(t.Context())
if err != nil {
t.Fatal(err)
}
out := map[string]conditions.Condition{}
for _, c := range all {
out[c.Key] = c
}
return out
}
func TestARetirementWaitingIsUrgentARejectionAWarningAndARetirementClears(t *testing.T) {
k, _ := withConditionsInMemory(t)
var recorded []link.HandAct
r := retirements{keeper: func() *conditions.Keeper { return k },
record: func(_ context.Context, a link.HandAct) error { recorded = append(recorded, a); return nil }}
ctx := t.Context()
waiting := waitingWord("postgres", "anchor", "a", "b", "c", "d")
if err := r.Retired(ctx, waiting); err != nil {
t.Fatal(err)
}
open := openKeys(t, k)
c, ok := open["provider.postgres.anchor.retire"]
if !ok || c.Kind != kindRetireWaiting || c.Severity != conditions.Urgent {
t.Fatalf("waiting is not an urgent retire-waiting condition: %+v", open)
}
for _, want := range []string{"a (laptop), b (laptop), c (laptop), d (laptop)", "retire approve anchor postgres",
"retire reject anchor postgres", "more than 3"} {
if !strings.Contains(c.Summary, want) {
t.Errorf("the condition does not say %q: %s", want, c.Summary)
}
}
// Rejected, through the controller: the urgent one becomes a warning, and nothing is recorded here —
// the verb recorded it before it asked.
rejected := waiting
rejected.Change, rejected.By, rejected.Why, rejected.Via = link.RetireRejected, "operator", "moving them", link.ViaController
if err := r.Retired(ctx, rejected); err != nil {
t.Fatal(err)
}
open = openKeys(t, k)
if _, still := open["provider.postgres.anchor.retire"]; still {
t.Fatal("a rejection left the provider waiting")
}
if c, ok := open["provider.postgres.anchor.retire-rejected"]; !ok || c.Severity != conditions.Warning ||
c.Kind != kindRetireRejected || !strings.Contains(c.Summary, "moving them") {
t.Fatalf("a rejection is not a warning saying why: %+v", open)
}
if len(recorded) != 0 {
t.Fatalf("an act through the controller was recorded twice: %+v", recorded)
}
// Approved afterwards, asked of the provider directly: both cleared, and recorded by hand here.
approved := waiting
approved.Change, approved.By, approved.Why, approved.Via = link.RetireApproved, "someone", "done moving", ""
if err := r.Retired(ctx, approved); err != nil {
t.Fatal(err)
}
if open := openKeys(t, k); len(open) != 0 {
t.Fatalf("an approval left conditions open: %+v", open)
}
if len(recorded) != 1 || recorded[0].Verb != "retire approve" || recorded[0].By != "someone" ||
!strings.Contains(recorded[0].Why, "directly") || !slices.Contains(recorded[0].Args, "d") {
t.Fatalf("an approval outside the controller was not recorded: %+v", recorded)
}
// Waiting again, then the set settles (asked for again): cleared. And retired clears too.
for _, change := range []string{link.RetireSettled, link.RetireRetired} {
if err := r.Retired(ctx, waiting); err != nil {
t.Fatal(err)
}
done := waiting
done.Change = change
if err := r.Retired(ctx, done); err != nil {
t.Fatal(err)
}
if open := openKeys(t, k); len(open) != 0 {
t.Fatalf("%s left conditions open: %+v", change, open)
}
}
// A deletion asked of the provider directly is recorded too.
deleted := link.Retirement{Module: "postgres", ProviderNode: "anchor", Change: link.RetireDeleted, By: "x",
Why: "gone for good", At: time.Now(), Consumers: []link.RetiredConsumer{{Consumer: "a"}}}
if err := r.Retired(ctx, deleted); err != nil {
t.Fatal(err)
}
if len(recorded) != 2 || recorded[1].Verb != "cleanup delete" || recorded[1].Cause != kindCleanupWaiting {
t.Fatalf("a deletion outside the controller was not recorded: %+v", recorded)
}
}
func TestARetirementWordIsKeptAsItsConditionNamingTheEmitterFromTheSubject(t *testing.T) {
body, _ := json.Marshal(map[string]any{"provider": "oidc-client", "provider-node": "anchor", "change": "waiting",
"held": 2, "consumers": []map[string]any{{"consumer": "x"}, {"consumer": "y"}}, "module": "liar"})
r, err := link.ReadRetirement("mesh.mod.idp.event.provisioner.retirement", body)
if err != nil || r.Module != "idp" || len(r.Consumers) != 2 {
t.Fatalf("%+v %v", r, err)
}
if _, err := link.ReadRetirement("mesh.mod.idp.event.provisioner.retirement",
[]byte(`{"provider-node":"anchor","change":"vanished"}`)); err == nil {
t.Fatal("a change the mesh has no name for was read")
}
if _, err := link.ReadRetirement("mesh.mod.idp.event.provisioner.failing", body); err == nil {
t.Fatal("a failing word was read as a retirement")
}
k, _ := withConditionsInMemory(t)
if err := (retirements{keeper: func() *conditions.Keeper { return k }}).Retired(t.Context(), r); err != nil {
t.Fatal(err)
}
if _, ok := openKeys(t, k)["provider.idp.anchor.retire"]; !ok {
t.Fatal("not kept under the emitter the subject names")
}
}
func TestARetirementConditionOfAnUnassignedProviderClears(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, catalogue.Manifest{Module: "pg", Version: "1",
Receives: map[string]string{"postgres-database": "/var/lib/mesh/pg/mesh.json"}})
if _, err := assign(ctx, open, "anchor", "pg"); err != nil {
t.Fatal(err)
}
r := retirements{keeper: func() *conditions.Keeper { return conditionsFrom }}
for _, w := range []link.Retirement{waitingWord("pg", "anchor", "a", "b", "c", "d"), waitingWord("gone", "anchor", "a", "b", "c", "d")} {
if err := r.Retired(ctx, w); err != nil {
t.Fatal(err)
}
}
if err := conditionsFrom.Reconcile(ctx, "D11", []conditions.Observation{
cleanupObservation(providerInstance{Node: "anchor", Module: "gone"},
[]link.RetiredConsumer{{Consumer: "a", RetiredAt: time.Now().Add(-40 * 24 * time.Hour).Format(time.RFC3339)}}, time.Now()),
}); err != nil {
t.Fatal(err)
}
all, _ := conditionsFrom.Open(ctx)
if len(all) != 3 {
t.Fatalf("%+v", all)
}
if err := unassignedProviders(ctx, open.inventory, conditionsFrom, providerConditions(all)); err != nil {
t.Fatal(err)
}
left := openKeys(t, conditionsFrom)
if len(left) != 1 || left["provider.pg.anchor.retire"].Key == "" {
t.Fatalf("only the assigned provider's waiting should stay: %+v", left)
}
}
// fakeProvider answers the retirement tools on one machine, over a real bus, keeping what it was asked.
type fakeProvider struct {
mu sync.Mutex
state link.RetirementState
asked []map[string]any
deleted []string
approved []string
rejected []string
}
func (f *fakeProvider) serve(t *testing.T, conn *nats.Conn, module, node string) {
t.Helper()
answer := func(m *nats.Msg, result any, refusal string) {
body, _ := json.Marshal(map[string]any{"result": result, "error": refusal, "node": node})
_ = m.Respond(body)
}
names := func(args map[string]any) []string {
var out []string
for _, v := range args["consumers"].([]any) {
out = append(out, v.(string))
}
sort.Strings(out)
return out
}
set := func(cs []link.RetiredConsumer) []string {
var out []string
for _, c := range cs {
out = append(out, c.Consumer)
}
sort.Strings(out)
return out
}
for _, tool := range []string{link.ToolRetirement, link.ToolRetireApprove, link.ToolRetireReject, link.ToolRetiredDelete} {
tool := tool
sub, err := conn.Subscribe(link.ModuleToolOn(module, tool, node), func(m *nats.Msg) {
f.mu.Lock()
defer f.mu.Unlock()
var args map[string]any
_ = json.Unmarshal(m.Data, &args)
f.asked = append(f.asked, map[string]any{"tool": tool, "args": args})
switch tool {
case link.ToolRetirement:
answer(m, f.state, "")
case link.ToolRetireApprove:
if f.state.Waiting == nil || !slices.Equal(names(args), set(f.state.Waiting.Consumers)) {
answer(m, nil, "not the set I wait with")
return
}
f.approved = names(args)
answer(m, map[string]any{"retired": f.approved}, "")
case link.ToolRetireReject:
if f.state.Waiting == nil || !slices.Equal(names(args), set(f.state.Waiting.Consumers)) {
answer(m, nil, "not the set I wait with")
return
}
f.rejected = names(args)
answer(m, map[string]any{"kept": f.rejected}, "")
case link.ToolRetiredDelete:
if args["confirm"] != args["consumer"] || args["why"] == "" || args["via"] != link.ViaController {
answer(m, nil, "confirm, why and via")
return
}
f.deleted = append(f.deleted, args["consumer"].(string))
answer(m, map[string]any{"deleted": args["consumer"], "freed_bytes": 1024}, "")
}
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = sub.Unsubscribe() })
}
if err := conn.Flush(); err != nil {
t.Fatal(err)
}
}
func onATestBus(t *testing.T) *nats.Conn {
t.Helper()
url := os.Getenv("MESH_TEST_NATS")
if url == "" {
t.Skip("MESH_TEST_NATS unset")
}
js, err := broker.Dial(url)
if err != nil {
t.Fatal(err)
}
t.Cleanup(js.Close)
if err := js.EnsureControllerBuckets(); err != nil {
t.Fatal(err)
}
before := handActConn
handActConn = js.Conn()
t.Cleanup(func() { handActConn = before })
return js.Conn()
}
func whyFlags(t *testing.T, why string) handActFlags {
t.Helper()
set := flag.NewFlagSet("t", flag.ContinueOnError)
f := addHandActFlags(set)
if err := set.Parse([]string{"--why", why}); err != nil {
t.Fatal(err)
}
return f
}
func handActsBy(t *testing.T, conn *nats.Conn, verb string) []link.HandAct {
t.Helper()
acts, err := link.HandActs(t.Context(), conn, time.Now().Add(-time.Minute))
if err != nil {
t.Fatal(err)
}
var out []link.HandAct
for _, a := range acts {
if a.Verb == verb {
out = append(out, a)
}
}
return out
}
func retiredDaysAgo(name string, days int) link.RetiredConsumer {
size := int64(4096)
return link.RetiredConsumer{Consumer: name, Kind: "consumer", Why: "the mesh stopped asking for it",
RetiredAt: time.Now().Add(-time.Duration(days) * 24 * time.Hour).UTC().Format(time.RFC3339), SizeBytes: &size}
}
func TestNatsApproveSendsTheExactSetTheProviderWaitsWith(t *testing.T) {
conn := onATestBus(t)
fake := &fakeProvider{}
fake.state.Waiting = &link.RetirementWaiting{Consumers: []link.RetiredConsumer{{Consumer: "d"}, {Consumer: "b"}, {Consumer: "a"}, {Consumer: "c"}}, Held: 6}
fake.serve(t, conn, "pg-approve", "anchor")
p := providerInstance{Node: "anchor", Module: "pg-approve"}
said := printed(t, func() error { return answerRetirement(t.Context(), conn, p, true, whyFlags(t, "moved them")) })
if !slices.Equal(fake.approved, []string{"a", "b", "c", "d"}) || !strings.Contains(said, "retired d, b, a, c") {
t.Fatalf("approved %v; said %s", fake.approved, said)
}
acts := handActsBy(t, conn, "retire approve")
if len(acts) == 0 || acts[len(acts)-1].Cause != kindRetireWaiting ||
acts[len(acts)-1].Condition != "provider.pg-approve.anchor.retire" {
t.Fatalf("the approval is not in the hand-act log: %+v", acts)
}
// Reject, on another provider: the same set, and nothing approved.
other := &fakeProvider{state: fake.state}
other.serve(t, conn, "pg-reject", "anchor")
printed(t, func() error {
return answerRetirement(t.Context(), conn, providerInstance{Node: "anchor", Module: "pg-reject"}, false,
whyFlags(t, "still moving"))
})
if !slices.Equal(other.rejected, []string{"a", "b", "c", "d"}) || other.approved != nil {
t.Fatalf("rejected %v approved %v", other.rejected, other.approved)
}
// Nothing waiting: refused, nothing asked but the question.
idle := &fakeProvider{}
idle.serve(t, conn, "pg-idle", "anchor")
if err := answerRetirement(t.Context(), conn, providerInstance{Node: "anchor", Module: "pg-idle"}, true,
whyFlags(t, "x")); err == nil || !strings.Contains(err.Error(), "waits for nobody") {
t.Fatalf("%v", err)
}
if len(idle.asked) != 1 {
t.Fatalf("an idle provider was asked more than its state: %+v", idle.asked)
}
}
func TestNatsCleanupDeletesOnlyTheNamedRetiredConsumer(t *testing.T) {
conn := onATestBus(t)
fake := &fakeProvider{}
fake.state.Held = []string{"active"}
fake.state.Retired = []link.RetiredConsumer{retiredDaysAgo("old", 40), retiredDaysAgo("young", 2)}
fake.serve(t, conn, "pg-clean", "anchor")
p := providerInstance{Node: "anchor", Module: "pg-clean"}
said := printed(t, func() error { return deleteRetired(t.Context(), conn, p, "old", whyFlags(t, "not needed")) })
if !slices.Equal(fake.deleted, []string{"old"}) || !strings.Contains(said, "deleted old (1.0 KB freed)") {
t.Fatalf("deleted %v; said %s", fake.deleted, said)
}
// An active consumer, or one it does not hold, is refused before the provider is asked to delete.
for _, name := range []string{"active", "nobody"} {
if err := deleteRetired(t.Context(), conn, p, name, whyFlags(t, "x")); err == nil ||
!strings.Contains(err.Error(), "only a retired consumer is deleted") {
t.Fatalf("%s: %v", name, err)
}
}
if !slices.Equal(fake.deleted, []string{"old"}) {
t.Fatalf("more was deleted: %v", fake.deleted)
}
if acts := handActsBy(t, conn, "cleanup delete"); len(acts) == 0 || acts[len(acts)-1].Args[2] != "old" {
t.Fatalf("the deletion is not in the hand-act log: %+v", acts)
}
// Older than: listed, and nothing deleted without confirm; with it, only the old one.
listing := retiredOf(t.Context(), conn, []providerInstance{p}, time.Now())
if len(listing.Retired) != 2 || listing.Retired[0].Consumer != "old" || listing.Retired[0].AgeDays != 40 {
t.Fatalf("%+v", listing)
}
fake.deleted = nil
said = printed(t, func() error { return deleteFrom(t.Context(), conn, nil, listing, 30, false, whyFlags(t, "tidy")) })
if fake.deleted != nil || !strings.Contains(said, "nothing was deleted: add --confirm") || !strings.Contains(said, "old") ||
strings.Contains(said, "young") {
t.Fatalf("deleted %v; said %s", fake.deleted, said)
}
printed(t, func() error { return deleteFrom(t.Context(), conn, nil, listing, 30, true, whyFlags(t, "tidy")) })
if !slices.Equal(fake.deleted, []string{"old"}) {
t.Fatalf("confirmed, deleted %v", fake.deleted)
}
}
func TestNatsD11SaysCleanupWaitsAfterThirtyDays(t *testing.T) {
conn := onATestBus(t)
old := &fakeProvider{}
old.state.Retired = []link.RetiredConsumer{retiredDaysAgo("mesh_a_letta", 31), retiredDaysAgo("fresh", 1)}
old.serve(t, conn, "pg-d11-old", "anchor")
young := &fakeProvider{}
young.state.Retired = []link.RetiredConsumer{retiredDaysAgo("x", 29)}
young.serve(t, conn, "pg-d11-young", "anchor")
instances := []providerInstance{{Node: "anchor", Module: "pg-d11-old"}, {Node: "anchor", Module: "pg-d11-young"},
// Nothing serves this one: a provider older than the question is passed over, not a failure.
{Node: "anchor", Module: "pg-d11-predates"}}
found, err := retiredTooLong(t.Context(), conn, instances, time.Now())
if err != nil {
t.Fatal(err)
}
if len(found) != 1 || found[0].Key() != "provider.pg-d11-old.anchor.cleanup" || found[0].Kind != kindCleanupWaiting ||
found[0].Severity != conditions.Warning || !strings.Contains(found[0].Summary, "mesh_a_letta (31 days") ||
strings.Contains(found[0].Summary, "fresh") {
t.Fatalf("%+v", found)
}
}
func TestTheRetireAndCleanupVerbsComposeTheirCommandLines(t *testing.T) {
for _, c := range []struct {
verb string
args map[string]any
want string
}{
{"retire", map[string]any{}, "retire --json"},
{"retire", map[string]any{"answer": "approve", "node": "anchor", "module": "postgres", "why": "moved"},
"retire approve anchor postgres --why moved"},
{"retire", map[string]any{"answer": "reject", "node": "anchor", "module": "postgres", "why": "no"},
"retire reject anchor postgres --why no"},
{"cleanup", map[string]any{}, "cleanup list --json"},
{"cleanup", map[string]any{"node": "anchor", "module": "postgres", "consumer": "x", "why": "gone"},
"cleanup delete anchor postgres x --why gone"},
{"cleanup", map[string]any{"older-than": "30", "why": "tidy"}, "cleanup delete --older-than 30 --why tidy"},
{"cleanup", map[string]any{"older-than": "30", "why": "tidy", "confirm": "true"},
"cleanup delete --older-than 30 --why tidy --confirm"},
} {
argv, err := argvFor(c.verb, c.args)
if err != nil || strings.Join(argv, " ") != c.want {
t.Errorf("%s %v: %q %v, want %q", c.verb, c.args, argv, err, c.want)
}
}
for _, args := range []map[string]any{
{"answer": "approve", "node": "anchor", "module": "postgres"}, // no why
{"answer": "maybe", "node": "anchor", "module": "postgres", "why": "x"},
} {
if _, err := argvFor("retire", args); err == nil {
t.Errorf("retire %v was composed", args)
}
}
for _, args := range []map[string]any{
{"node": "anchor", "module": "postgres", "consumer": "x"}, // no why
{"older-than": "30"},
{"consumer": "x", "older-than": "30", "node": "a", "module": "b", "why": "y"},
} {
if _, err := argvFor("cleanup", args); err == nil {
t.Errorf("cleanup %v was composed", args)
}
}
if repairingCommand([]string{"cleanup", "delete", "a", "b", "c"}) == "" ||
repairingCommand([]string{"retire", "approve", "a", "b"}) == "" || repairingCommand([]string{"retire"}) != "" {
t.Error("the generic verb would let a retirement or a deletion through without a why")
}
}
// A mark-only provider's retired consumer keeps its access; the listing and the approval say so
// rather than claiming it was disabled (ADR 0230).
func TestNatsAMarkOnlyRetirementIsSaidAsSuch(t *testing.T) {
conn := onATestBus(t)
fake := &fakeProvider{}
kept := retiredDaysAgo("ledger", 3)
kept.Access = "kept"
fake.state.Retired = []link.RetiredConsumer{kept}
fake.state.RetiresBy = "mark-only"
fake.state.Waiting = &link.RetirementWaiting{Consumers: []link.RetiredConsumer{{Consumer: "a"}, {Consumer: "b"}}, Held: 2}
fake.serve(t, conn, "vault-mark", "anchor")
p := providerInstance{Node: "anchor", Module: "vault-mark"}
listing := retiredOf(t.Context(), conn, []providerInstance{p}, time.Now())
said := printed(t, func() error { return printRetired(listing, false) })
if !strings.Contains(said, "MARK ONLY") || listing.Retired[0].Access != "kept" {
t.Fatalf("%s %+v", said, listing)
}
said = printed(t, func() error { return answerRetirement(t.Context(), conn, p, true, whyFlags(t, "gone")) })
if !strings.Contains(said, "MARK ONLY") || strings.Contains(said, "access disabled") {
t.Fatal(said)
}
}
+1 -1
View File
@@ -189,7 +189,7 @@ func readinessOf(ctx context.Context, inv *inventory.Inventory) (broker.Readines
// A third of the catalogue never does (novox/hq ADR 0120), and counting those as missing a credential
// would bury the ones that matter under a list nobody can act on.
func speaksOnTheBus(m catalogue.Manifest) bool {
return len(m.Emits) > 0 || len(m.Consumes) > 0 || len(m.Tools) > 0 ||
return len(m.EmitsAll()) > 0 || len(m.Consumes) > 0 || len(m.Tools) > 0 ||
len(m.DefinesSeats) > 0 || len(m.Uses) > 0 || len(m.Claims) > 0
}
+34 -1
View File
@@ -6,6 +6,8 @@ import (
"flag"
"fmt"
"sort"
"github.com/novox/mesh-controller/internal/inventory"
)
// rotateCommand replaces a credential and moves both ends together.
@@ -33,12 +35,16 @@ func rotateCommand(ctx context.Context, args []string) error {
// One consumer rather than all of them. Ordinary: a credential is suspected on one machine,
// and rotating the other nine would be a great deal of disruption for one suspicion.
only := set.String("consumer", "", "only this machine's credential, rather than every holder's")
// One consuming module rather than every module on the machine. A machine runs many consumers
// of one provision, each with its own credential; one module that leaked its credential (novox/hq
// issue 268) is no reason to restart every other one on the machine.
module := set.String("module", "", "only this consuming module's credential")
positionals, err := parseAround(set, args)
if err != nil {
return err
}
if len(positionals) != 1 {
return errors.New("rotate <provision> [--consumer <machine>]")
return errors.New("rotate <provision> [--consumer <machine>] [--module <module>]")
}
provision := positionals[0]
@@ -53,6 +59,12 @@ func rotateCommand(ctx context.Context, args []string) error {
if err != nil {
return err
}
holders = ofModule(holders, *module)
if len(holders) == 0 && *module != "" {
return fmt.Errorf(
"no module %s%s holds a credential for %q, so there is nothing to rotate. `plan <machine>` "+
"says what a machine holds", *module, onMachine(*only), provision)
}
if len(holders) == 0 {
// Said, not silent. "Nobody holds this" and "this did not run" must never look the same —
// and a rotation somebody believes happened is worse than one they know did not.
@@ -116,6 +128,27 @@ func rotateCommand(ctx context.Context, args []string) error {
return nil
}
// ofModule is the holders whose consuming module is this one; all of them when none is named.
func ofModule(holders []inventory.Holder, module string) []inventory.Holder {
if module == "" {
return holders
}
var out []inventory.Holder
for _, h := range holders {
if h.ConsumerModule == module {
out = append(out, h)
}
}
return out
}
func onMachine(machine string) string {
if machine == "" {
return ""
}
return " on " + machine
}
// asLocal names the credential inside the consumer where it holds several (ADR 0094).
func asLocal(local string) string {
if local == "" {
+27
View File
@@ -0,0 +1,27 @@
package main
import (
"testing"
"github.com/novox/mesh-controller/internal/inventory"
)
// One consuming module's credential, and not its neighbours' on the same machine (novox/hq issue
// 268): a module that leaked its database password is no reason to restart every other consumer.
func TestARotationNarrowedToAModuleTouchesOnlyThatModulesCredential(t *testing.T) {
holders := []inventory.Holder{
{Provision: "postgres-database", Consumer: "ace", ConsumerModule: "letta", Provider: "ace"},
{Provision: "postgres-database", Consumer: "ace", ConsumerModule: "n8n", Provider: "ace"},
{Provision: "postgres-database", Consumer: "ace", ConsumerModule: "letta", Local: "reader", Provider: "ace"},
}
got := ofModule(holders, "letta")
if len(got) != 2 || got[0].ConsumerModule != "letta" || got[1].Local != "reader" {
t.Fatalf("narrowed to letta: %+v", got)
}
if len(ofModule(holders, "")) != 3 {
t.Fatal("no module named narrowed anyway")
}
if len(ofModule(holders, "absent")) != 0 {
t.Fatal("a module holding nothing matched")
}
}
+35 -9
View File
@@ -29,11 +29,13 @@ type seatHolder struct {
// seatRow is one seat and who holds it. Unheld is an answer — "this mesh has no X" — not a fault.
type seatRow struct {
Seat string `json:"seat"`
Scope string `json:"scope"`
Delivers string `json:"delivers,omitempty"`
Decision string `json:"decision"`
Holders []seatHolder `json:"holders"`
Seat string `json:"seat"`
Scope string `json:"scope"`
Delivers string `json:"delivers,omitempty"`
Decision string `json:"decision"`
// Replicated says the seat may be held on several machines at once (novox/hq ADR 0223).
Replicated bool `json:"replicated,omitempty"`
Holders []seatHolder `json:"holders"`
}
// seatsHeld is every seat the mesh defines with its holders, and every claim held that names no
@@ -47,7 +49,7 @@ func seatsHeld(seats []catalogue.Seat, held []catalogue.Held) ([]seatRow, []cata
rows := make([]seatRow, 0, len(seats))
for _, s := range seats {
row := seatRow{Seat: s.Name, Scope: s.Scope, Delivers: s.Delivers, Decision: s.Decision,
Holders: []seatHolder{}}
Replicated: s.Replicated, Holders: []seatHolder{}}
seen := map[seatHolder]bool{}
for _, h := range held {
// Resolve the held claim to a seat rather than comparing names, so a record naming a
@@ -104,9 +106,13 @@ func seatCommand(ctx context.Context, args []string) error {
return nil
}
if len(args) == 3 && args[1] == "--to" {
return handOver(ctx, args[0], args[2])
return handOver(ctx, args[0], args[2], false)
}
return fmt.Errorf("seat rename <from> <to> | seat <name> --to <node>/<module>")
if len(args) == 3 && args[1] == "--add" {
return handOver(ctx, args[0], args[2], true)
}
return fmt.Errorf("seat rename <from> <to> | seat <name> --to <node>/<module> | " +
"seat <name> --add <node>/<module>")
}
// handOver makes one assignment the holder of a seat, as one act, so the seat is never without a
@@ -119,7 +125,12 @@ func seatCommand(ctx context.Context, args []string) error {
// **not** checked is whether the module is running yet: that is what `push` confirms afterwards,
// and refusing to record a handover to a module the node has not started would make the handover
// impossible to do before the switch instead of as the switch.
func handOver(ctx context.Context, seatName, to string) error {
//
// **Or adds one holder beside the others, for a replicated seat** (novox/hq ADR 0223): `--add`
// records the named assignment as a further holder and leaves every holder on record as it is. A
// seat held once refuses it, naming `--to`; `--to` on a replicated seat replaces every holder with
// the one named, as it always did.
func handOver(ctx context.Context, seatName, to string, adding bool) error {
nodeName, module, ok := strings.Cut(to, "/")
if !ok || nodeName == "" || module == "" {
return fmt.Errorf("the new holder is named <node>/<module>, not %q", to)
@@ -135,6 +146,10 @@ func handOver(ctx context.Context, seatName, to string) error {
if !known {
return fmt.Errorf("%q is not a seat this mesh defines — `seats` lists them", seatName)
}
if adding && !seat.Replicated {
return fmt.Errorf("%s is held once per %s, so a second holder cannot be added beside the first — "+
"`seat %s --to %s` hands it over", seat.Name, seat.Scope, seat.Name, to)
}
assigned, err := inv.Assigned(ctx, nodeName)
if err != nil {
return err
@@ -157,6 +172,7 @@ func handOver(ctx context.Context, seatName, to string) error {
return fmt.Errorf("%s is assigned but not in the catalogue, which should not happen", module)
}
var was string
var held []string
holdings, err := inv.Holdings(ctx)
if err != nil {
return err
@@ -164,6 +180,7 @@ func handOver(ctx context.Context, seatName, to string) error {
for _, h := range holdings {
if hs, ok := catalogue.SeatNamed(h.Claim); ok && hs.Name == seat.Name {
was = h.Node
held = append(held, h.Node)
}
}
@@ -187,6 +204,15 @@ func handOver(ctx context.Context, seatName, to string) error {
} else if err := catalogue.CanHold(*m, seat); err != nil {
return fmt.Errorf("%s cannot hold %s: %w", module, seat.Name, err)
}
if adding {
if err := inv.AddSeatHolder(ctx, seat.Name, seat.Scope, nodeName, module); err != nil {
return err
}
fmt.Printf("%s is held by %s on %s, beside what was on record: %s\n", seat.Name, module, nodeName,
strings.Join(held, ", "))
fmt.Printf(" `push --behind` re-declares every machine that reads the seat's holders\n")
return nil
}
if err := inv.HoldSeat(ctx, seat.Name, seat.Scope, nodeName, module); err != nil {
return err
}
+544 -43
View File
@@ -9,9 +9,11 @@ import (
"github.com/nats-io/nats.go/micro"
"os"
"os/exec"
"slices"
"sort"
"strings"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/link"
)
@@ -36,19 +38,194 @@ type verbAnswer struct {
// argvFor is the command line a verb and its arguments become. Only the verbs the seat declares, and
// only the arguments each declares: a caller cannot reach a flag the schema did not name.
//
// **Nothing a caller sends is passed over** (novox/hq issue 244). An argument the verb does not
// declare is refused, naming it; a switch that is not "true" or "false" is refused; and an argument
// the verb declares but did not use for the command line it composed — given beside another that
// wins, or half of a shape — is refused too. On 2026-10-05 a push naming one machine reached the
// verb without the machine and ran as a push of every machine behind; a verb that answers "I did
// not take that" would have stopped it before anything was sent.
func argvFor(verb string, args map[string]any) ([]string, error) {
str := func(key string) string {
v, _ := args[key].(string)
return strings.TrimSpace(v)
a, err := readArguments(verb, args)
if err != nil {
return nil, err
}
need := func(keys ...string) error {
for _, k := range keys {
if str(k) == "" {
return fmt.Errorf("%s needs %q", verb, k)
argv, err := a.commandLine()
if len(a.misread) > 0 {
// The table and the command line disagree: the verb reads an argument no caller can see
// in its schema, so no caller could ever pass it.
return nil, fmt.Errorf("%s reads %s, which its schema does not declare — this build's verb "+
"table and its command lines disagree", verb, quoteAll(a.misread))
}
if err != nil {
return nil, err
}
if unused := a.unused(); len(unused) > 0 {
return nil, fmt.Errorf("%s did not use %s together with %s, and an argument a verb would pass over "+
"is refused: nothing was done", verb, quoteAll(unused), quoteAll(a.usedGiven()))
}
return argv, nil
}
// verbArguments are one call's arguments, checked against the verb's schema, and which of them the
// command line was composed from.
type verbArguments struct {
verb string
given map[string]string
used map[string]bool
declared map[string]bool
misread []string // arguments the command line read that the schema does not declare: a bug here
}
// controllerVerb is this binary's own definition of a verb: what it runs is what it declares, so the
// arguments are checked against the table compiled beside argvFor, not a row a newer or older build
// wrote.
func controllerVerb(name string) (catalogue.Verb, bool) {
for _, v := range catalogue.ControllerVerbs {
if v.Name == name {
return v, true
}
}
return catalogue.Verb{}, false
}
// declaredArguments are a schema's properties, and which of them are switches.
func declaredArguments(v catalogue.Verb) (names []string, switches map[string]bool) {
switches = map[string]bool{}
props, _ := v.Input["properties"].(map[string]any)
for name, p := range props {
names = append(names, name)
desc, _ := p.(map[string]any)
switch enum := desc["enum"].(type) {
case []string:
switches[name] = len(enum) == 2 && enum[0] == "true" && enum[1] == "false"
case []any:
switches[name] = len(enum) == 2 && enum[0] == "true" && enum[1] == "false"
}
}
sort.Strings(names)
return names, switches
}
// readArguments refuses what the verb does not take, before anything is composed.
func readArguments(verb string, args map[string]any) (*verbArguments, error) {
v, known := controllerVerb(verb)
if !known {
return nil, fmt.Errorf("%q is not a verb the %s seat serves", verb, catalogue.ControllerSeatName)
}
names, switches := declaredArguments(v)
declared := map[string]bool{}
for _, n := range names {
declared[n] = true
}
takes := "none"
if len(names) > 0 {
takes = quoteAll(names)
}
a := &verbArguments{verb: verb, given: map[string]string{}, used: map[string]bool{}, declared: declared}
keys := make([]string, 0, len(args))
for k := range args {
keys = append(keys, k)
}
sort.Strings(keys)
for _, k := range keys {
if !declared[k] {
return nil, fmt.Errorf("%s takes no argument %q — it takes %s; nothing was done", verb, k, takes)
}
var value string
switch x := args[k].(type) {
case nil:
continue
case string:
value = strings.TrimSpace(x)
case bool:
if !switches[k] {
return nil, fmt.Errorf("%s: %q is text, not true or false", verb, k)
}
value = fmt.Sprint(x)
default:
return nil, fmt.Errorf("%s: %q is text, and was given %T", verb, k, x)
}
if switches[k] {
switch value {
case "true":
case "false", "":
continue // said and off: the same as not given, and nothing passed over
default:
return nil, fmt.Errorf("%s: %q is \"true\" or \"false\", not %q", verb, k, value)
}
}
return nil
if value != "" {
a.given[k] = value
}
}
return a, nil
}
// str is one argument's value, marked as used.
func (a *verbArguments) str(key string) string {
if !a.declared[key] {
a.misread = append(a.misread, key)
}
a.used[key] = true
return a.given[key]
}
// on is a switch, marked as used.
func (a *verbArguments) on(key string) bool { return a.str(key) == "true" }
// need refuses a call missing a required argument, in the verb's own words.
func (a *verbArguments) need(keys ...string) error {
for _, k := range keys {
if a.str(k) == "" {
return fmt.Errorf("%s needs %q", a.verb, k)
}
}
return nil
}
// unused are the arguments given that the command line was not composed from.
func (a *verbArguments) unused() []string {
var out []string
for k := range a.given {
if !a.used[k] {
out = append(out, k)
}
}
sort.Strings(out)
return out
}
func (a *verbArguments) usedGiven() []string {
var out []string
for k := range a.given {
if a.used[k] {
out = append(out, k)
}
}
sort.Strings(out)
if len(out) == 0 {
return []string{"nothing"}
}
return out
}
func quoteAll(xs []string) string {
q := make([]string, len(xs))
for i, x := range xs {
if x == "nothing" {
q[i] = x
continue
}
q[i] = fmt.Sprintf("%q", x)
}
return strings.Join(q, ", ")
}
// commandLine composes the command. Every argument it reads is one it uses: a branch that reads an
// argument and then drops it would pass it over, which is what the check after it exists to refuse.
func (a *verbArguments) commandLine() ([]string, error) {
verb, str, on, need := a.verb, a.str, a.on, a.need
switch verb {
case "command":
// The generic verb: the command line as given, split as a shell would split it, with
@@ -64,7 +241,15 @@ func argvFor(verb string, args map[string]any) ([]string, error) {
if len(argv) == 0 {
return nil, errors.New("command names no command")
}
// The generic verb is no way round the hand-act log (novox/hq to-be 45 §7): a repair through
// it says why, as it would through its own verb.
if repair := repairingCommand(argv); repair != "" && !slices.ContainsFunc(argv, isWhyFlag) {
return nil, fmt.Errorf("%s is a repair done by hand, and says why: add --why <text> to the command "+
"line (recorded in the hand-act log). Nothing was done", repair)
}
return argv, nil
case "tools":
return nil, errors.New("tools is answered from the records, not by a command")
case "status":
return []string{"status", "--json"}, nil
case "nodes":
@@ -82,10 +267,14 @@ func argvFor(verb string, args map[string]any) ([]string, error) {
if id := str("log"); id != "" {
return []string{"builds", "--log", id}, nil
}
if m := str("module"); m != "" {
return []string{"builds", m}, nil
argv := []string{"builds"}
if n := str("limit"); n != "" {
argv = append(argv, "-n", n)
}
return []string{"builds"}, nil
if m := str("module"); m != "" {
argv = append(argv, m)
}
return argv, nil
case "plans":
if r := str("repository"); r != "" {
argv := []string{"plans", "--what-if", r}
@@ -97,19 +286,30 @@ func argvFor(verb string, args map[string]any) ([]string, error) {
}
return argv, nil
}
if id := str("stop"); id != "" {
return []string{"plans", "stop", id}, nil
}
if id := str("close"); id != "" {
return []string{"plans", "close", id}, nil
}
if id := str("retry"); id != "" {
return []string{"plans", "retry", id}, nil
for _, act := range []string{"stop", "close", "retry"} {
if id := str(act); id != "" {
argv := []string{"plans", act, id}
if act == "retry" {
return argv, nil
}
// Ending a plan by hand says why (novox/hq to-be 45 §7); the command refuses it without.
if w := str("why"); w != "" {
argv = append(argv, "--why", w)
}
if c := str("cause"); c != "" {
argv = append(argv, "--cause", c)
}
return argv, nil
}
}
if id := str("id"); id != "" {
return []string{"plans", id}, nil
}
return []string{"plans"}, nil
argv := []string{"plans"}
if n := str("limit"); n != "" {
argv = append(argv, "-n", n)
}
return argv, nil
// The build queue (novox/hq ADR 0219).
case "queue":
return []string{"queue"}, nil
@@ -119,7 +319,7 @@ func argvFor(verb string, args map[string]any) ([]string, error) {
}
return []string{verb, str("id")}, nil
case "clear":
if str("dead") == "true" {
if on("dead") {
return []string{"clear", "--dead"}, nil
}
return []string{"clear"}, nil
@@ -133,10 +333,10 @@ func argvFor(verb string, args map[string]any) ([]string, error) {
return nil, err
}
argv := []string{"replay", str("id")}
if str("register") == "true" {
if on("register") {
argv = append(argv, "--register")
}
if str("older") == "true" {
if on("older") {
argv = append(argv, "--older")
}
return argv, nil
@@ -149,6 +349,9 @@ func argvFor(verb string, args map[string]any) ([]string, error) {
if err := need("node"); err != nil {
return nil, err
}
if on("files") {
return []string{"plan", str("node"), "--files"}, nil
}
return []string{"plan", str("node"), "--json"}, nil
case "assign", "unassign":
if err := need("node", "module"); err != nil {
@@ -171,23 +374,185 @@ func argvFor(verb string, args map[string]any) ([]string, error) {
// Sent and not waited for: the asker reads `status` for what the machine did, which is
// what a person at a shell does too. A tool call that blocked for a push's whole apply would
// time out on every machine that takes a minute, and say nothing about the ones that did not.
if n := str("node"); n != "" {
return []string{"push", n, "--wait", "0"}, nil
// A push through the seat is a push by hand, and says why (novox/hq to-be 45 §7).
if err := need("why"); err != nil {
return nil, fmt.Errorf("%w: a push by hand is a repair, recorded in the hand-act log with why", err)
}
return []string{"push", "--behind", "--wait", "0"}, nil
why := []string{"--why", str("why")}
if c := str("cause"); c != "" {
why = append(why, "--cause", c)
}
if n := str("node"); n != "" {
// behind is not read here: given with a machine, it is refused as passed over — naming
// a machine and asking for every machine behind are two requests, and guessing one
// would push a machine nobody named, or not push one somebody did.
return append([]string{"push", n, "--wait", "0"}, why...), nil
}
// No machine: the whole mesh, whether or not behind said so. The command's answer says it
// first, so a caller who meant one machine reads that it was not one.
on("behind")
return append([]string{"push", "--behind", "--wait", "0"}, why...), nil
case "hand-act":
if err := need("what", "why", "cause"); err != nil {
return nil, err
}
argv := []string{"hand-act", "record", str("what"), "--why", str("why"), "--cause", str("cause")}
if c := str("condition"); c != "" {
argv = append(argv, "--condition", c)
}
return argv, nil
case "hand-acts":
argv := []string{"hand-acts", "--json"}
if d := str("days"); d != "" {
argv = append(argv, "--days", d)
}
return argv, nil
case "healers":
argv := []string{"healers", "--json"}
if d := str("days"); d != "" {
argv = append(argv, "--days", d)
}
return argv, nil
case "durations":
argv := []string{"durations", "--json"}
if k := str("kind"); k != "" {
argv = append(argv, "--kind", k)
}
if d := str("days"); d != "" {
argv = append(argv, "--days", d)
}
return argv, nil
case "conditions":
// One verb, four shapes, as `plans` (novox/hq to-be 45 §2): a silence, one condition, the
// history, or the open ones filtered.
if key := str("silence"); key != "" {
if err := need("for", "why"); err != nil {
return nil, err
}
argv := []string{"conditions", "silence", key, "--for", str("for"), "--why", str("why")}
if c := str("cause"); c != "" {
argv = append(argv, "--cause", c)
}
return argv, nil
}
if on("history") {
argv := []string{"conditions", "history", "--json"}
if d := str("days"); d != "" {
argv = append(argv, "--days", d)
}
if k := str("key"); k != "" {
argv = append(argv, "--key", k)
}
return argv, nil
}
if k := str("key"); k != "" {
return []string{"conditions", "show", k, "--json"}, nil
}
argv := []string{"conditions", "--json"}
for _, filter := range []string{"scope", "severity", "machine"} {
if v := str(filter); v != "" {
argv = append(argv, "--"+filter, v)
}
}
return argv, nil
case "retire":
answer := str("answer")
if answer == "" {
return []string{"retire", "--json"}, nil
}
if answer != "approve" && answer != "reject" {
return nil, fmt.Errorf("retire answers approve or reject, not %q", answer)
}
if err := need("node", "module", "why"); err != nil {
return nil, err
}
argv := []string{"retire", answer, str("node"), str("module"), "--why", str("why")}
if c := str("cause"); c != "" {
argv = append(argv, "--cause", c)
}
return argv, nil
case "cleanup":
consumer, older := str("consumer"), str("older-than")
switch {
case consumer != "" && older != "":
return nil, errors.New("cleanup deletes one consumer or those older than some days, not both")
case consumer != "":
if err := need("node", "module", "why"); err != nil {
return nil, err
}
argv := []string{"cleanup", "delete", str("node"), str("module"), consumer, "--why", str("why")}
if c := str("cause"); c != "" {
argv = append(argv, "--cause", c)
}
return argv, nil
case older != "":
if err := need("why"); err != nil {
return nil, err
}
argv := []string{"cleanup", "delete", "--older-than", older, "--why", str("why")}
if on("confirm") {
argv = append(argv, "--confirm")
}
if c := str("cause"); c != "" {
argv = append(argv, "--cause", c)
}
return argv, nil
}
return []string{"cleanup", "list", "--json"}, nil
case "data":
argv := []string{"data", "--json"}
if m := str("machine"); m != "" {
argv = append(argv, "--machine", m)
}
if on("retired") {
argv = append(argv, "--retired")
}
return argv, nil
case "doctor":
which := 0
argv := []string{"doctor"}
for _, sub := range []string{"run", "probes", "signals"} {
if on(sub) {
which++
argv = append(argv, sub)
}
}
if which > 1 {
return nil, errors.New("doctor answers one of run, probes or signals at a time")
}
return append(argv, "--json"), nil
case "rotate":
if p := str("provision"); p != "" {
argv := []string{"rotate", p}
if c := str("consumer"); c != "" {
argv = append(argv, "--consumer", c)
}
// With a provision, module narrows to one consuming module (novox/hq issue 268); node
// and secret stay the other shape's, and are refused as passed over.
if m := str("module"); m != "" {
argv = append(argv, "--module", m)
}
return argv, nil
}
if str("node") != "" && str("module") != "" && str("secret") != "" {
return []string{"secret", "rotate", str("node"), str("module"), str("secret")}, nil
_, node := a.given["node"]
_, module := a.given["module"]
_, secret := a.given["secret"]
if node || module || secret {
if err := need("node", "module", "secret"); err != nil {
return nil, fmt.Errorf("%w: a module's own secret is named by node, module and secret together", err)
}
argv := []string{"secret", "rotate", str("node"), str("module"), str("secret")}
// Why, recorded in the hand-act log (novox/hq ADR 0228); a cause only beside a why.
if w := str("why"); w != "" {
argv = append(argv, "--why", w)
if c := str("cause"); c != "" {
argv = append(argv, "--cause", c)
}
}
return argv, nil
}
// Half of either shape: the command says its usage, which names both shapes, and that is
// the answer the caller needs.
// Neither shape: the command says its usage, which names both, and that is the answer the
// caller needs.
return []string{"rotate"}, nil
case "settings":
// `settings set|clear` at a shell (novox/hq issue 198). The values travel as an argument
@@ -195,15 +560,16 @@ func argvFor(verb string, args map[string]any) ([]string, error) {
if err := need("module"); err != nil {
return nil, err
}
argv := []string{"settings", "set", str("module")}
switch {
case str("clear") == "true":
var argv []string
if on("clear") {
argv = []string{"settings", "clear", str("module")}
case str("values") != "":
argv = append(argv, str("values"))
} else {
argv = []string{"settings", "set", str("module")}
// Neither values nor clear: the command says its usage, which names both.
if v := str("values"); v != "" {
argv = append(argv, v)
}
}
// Neither values nor clear: the command says its usage, which names both, and that is the
// answer the caller needs — the same as `rotate` given half of either shape.
if n := str("node"); n != "" {
argv = append(argv, "--node", n)
}
@@ -235,11 +601,39 @@ func argvFor(verb string, args map[string]any) ([]string, error) {
}
return argv, nil
}
return nil, fmt.Errorf("%q is not a verb the %s seat serves", verb, catalogue.ControllerSeatName)
return nil, fmt.Errorf("%q is a verb of the %s seat's table that this binary has no command line for",
verb, catalogue.ControllerSeatName)
}
// jsonVerbs are the verbs whose command speaks JSON, so the answer carries it as data as well.
var jsonVerbs = map[string]bool{"status": true, "seats": true, "plan": true, "collection": true}
var jsonVerbs = map[string]bool{"status": true, "seats": true, "plan": true, "collection": true,
"hand-acts": true, "durations": true, "conditions": true, "doctor": true, "retire": true, "cleanup": true, "data": true}
// repairingCommand names a command line that repairs by hand, and so says why: a push, a plan stopped
// or closed, a consumer re-made (novox/hq to-be 45 §7). Empty for any other.
func repairingCommand(argv []string) string {
switch {
case argv[0] == "push":
return "push"
case argv[0] == "plans" && len(argv) > 1 && (argv[1] == "stop" || argv[1] == "close"):
return "plans " + argv[1]
case argv[0] == "broker" && len(argv) > 1 && argv[1] == "consumer-reset":
return "broker consumer-reset"
case argv[0] == "hand-act":
return "hand-act record"
case argv[0] == "conditions" && len(argv) > 1 && argv[1] == "silence":
return "conditions silence"
case argv[0] == "retire" && len(argv) > 1 && (argv[1] == "approve" || argv[1] == "reject"):
return "retire " + argv[1]
case argv[0] == "cleanup" && len(argv) > 1 && argv[1] == "delete":
return "cleanup delete"
}
return ""
}
func isWhyFlag(word string) bool {
return word == "--why" || word == "-why" || strings.HasPrefix(word, "--why=") || strings.HasPrefix(word, "-why=")
}
// runVerb runs this binary with the given command line and gathers what it said.
func runVerb(ctx context.Context, argv []string) (verbAnswer, error) {
@@ -251,6 +645,12 @@ func runVerb(ctx context.Context, argv []string) (verbAnswer, error) {
// The same environment: the stores' credentials, the bus, the broker — everything a command run
// from a shell in this container would have, because it is that.
cmd.Env = os.Environ()
// And who asked, so an act it does by hand is recorded as theirs (novox/hq to-be 45 §7).
caller := link.CallerIn(ctx)
if caller == "" {
caller = "a seat call whose caller the bus did not name"
}
cmd.Env = append(cmd.Env, link.CallerVar+"="+caller+", through the "+catalogue.ControllerSeatName+" seat")
// Two buffers, one answer. What the command *says* is both streams, in the order a person at
// a shell would read them; what it *answers as data* is standard output alone — `status --json`
// prints its warnings beside the document, and a JSON parsed from the two together parsed
@@ -287,8 +687,35 @@ func seatToolHandlers() (map[string]link.ToolHandler, []string, error) {
handlers := map[string]link.ToolHandler{}
for _, v := range seat.Serves {
verb := v.Name
if verb == "tools" {
handlers[verb] = func(ctx context.Context, _ json.RawMessage) (any, error) {
if inProcess[verb] {
handlers[verb] = func(ctx context.Context, raw json.RawMessage) (any, error) {
args := map[string]any{}
if len(bytes.TrimSpace(raw)) > 0 {
if err := json.Unmarshal(raw, &args); err != nil {
return nil, fmt.Errorf("the arguments are not a JSON object: %w", err)
}
}
// Refused like any verb's: what a verb does not take is not ignored.
a, err := readArguments(verb, args)
if err != nil {
return nil, err
}
if verb == "calls" {
return callsAnswer(link.Calls, a.given["call"])
}
if verb == "doctor" {
// From the serving controller, which runs the self-check and hears the signals
// (novox/hq to-be 45 §4): the last verdict at once, or a run now.
argv, err := a.commandLine()
if err != nil {
return nil, err
}
sub := ""
if len(argv) > 2 {
sub = argv[1]
}
return doctorAnswer(ctx, sub)
}
return seatTools(), nil
}
continue
@@ -327,12 +754,85 @@ func seatToolHandlers() (map[string]link.ToolHandler, []string, error) {
if err != nil {
return nil, err
}
if verb == "status" && statusFrom != nil {
// At once, from the summary the serving controller keeps (novox/hq to-be 45 Phase 0).
return statusFrom.answer(ctx)
}
if !readingVerbs[verb] && !(verb == "plans" && !actsOnAPlan(args)) {
// Whatever it did, `status` is composed again once it has.
defer statusFrom.nudge()
}
if answersFirst(argv) {
// Before anything is sent: a push sends the bus's own machine first, and a broker
// reloading its user list forgets the answer it was about to permit (novox/hq issue 265).
link.Acknowledge(ctx)
}
return runVerb(ctx, argv)
}
}
return handlers, behind, nil
}
// actsOnAPlan is `plans` asked to stop, close or retry one rather than to show them.
func actsOnAPlan(args map[string]any) bool {
for _, act := range []string{"stop", "close", "retry"} {
if v, _ := args[act].(string); strings.TrimSpace(v) != "" {
return true
}
}
return false
}
// inProcess are the verbs answered by this process rather than by a command it runs: `tools` from
// the records, `calls` from what this process served.
var inProcess = map[string]bool{"tools": true, "calls": true, "doctor": true}
// answersFirst is a command line whose caller is answered before it runs: a push, by its verb or
// through `command`. A push sends the machine holding the bus first when its user list changed, the
// broker reloads, and a reload forgets every answer the bus was about to permit — so an answer
// waiting for the push to end was refused, every time the list had changed (novox/hq issue 265).
func answersFirst(argv []string) bool {
return len(argv) > 0 && argv[0] == "push"
}
// callsAnswer is what `calls` answers: the kept calls, newest first, without their answers — or
// one call whole. Kept on the bus, so a call a controller before this one served is answered too
// (novox/hq to-be 45 §6); where the bus cannot be read, what this process served is answered and
// the reason said beside it.
func callsAnswer(log *link.CallLog, id string) (any, error) {
if id != "" {
c, ok, err := log.Get(id)
if err != nil {
return nil, fmt.Errorf("call %s is not in this controller's memory, and the calls kept on the "+
"bus could not be read: %w", id, err)
}
if !ok {
if log.IsDurable() {
return nil, fmt.Errorf("no call %s is kept: the bus keeps the last %d calls, or %s, and this "+
"is not among them — `calls` lists them", id, broker.KeptCallsDurably, broker.CallsKeptFor)
}
return nil, fmt.Errorf("no call %s is kept here: calls are kept by the controller that "+
"answered them, the last %d, and not across a restart — `calls` lists them", id, link.KeptCalls)
}
return c, nil
}
recent, err := log.Recent()
for i := range recent {
recent[i].Answer = nil
}
answer := map[string]any{"calls": recent, "note": "newest first; `calls` with a call's id gives its whole answer"}
if log.IsDurable() {
answer["kept"] = fmt.Sprintf("the last %d calls, or %s, on the bus — across a restart of the controller",
broker.KeptCallsDurably, broker.CallsKeptFor)
} else {
answer["kept"] = fmt.Sprintf("the last %d calls this controller served, in its memory only", link.KeptCalls)
}
if err != nil {
answer["unread"] = err.Error()
}
return answer, nil
}
// seatTools is what `tools` answers: every seat with a protocol, and the tools each serves, from the
// mesh's own records — no holder in the path, so it is true while a holder restarts (design 33 §5).
func seatTools() map[string]any {
@@ -353,9 +853,10 @@ func seatTools() map[string]any {
}
// sampleArguments is one of every argument a verb's schema requires, so the check at start proves the
// verb runnable rather than that it happens to want the arguments the check guessed.
// verb runnable rather than that it happens to want the arguments the check guessed — and nothing
// more, since an argument a verb does not declare is refused.
func sampleArguments(v catalogue.Verb) map[string]any {
sample := map[string]any{"node": "x", "module": "x", "repository": "x"}
sample := map[string]any{}
switch required := v.Input["required"].(type) {
case []string:
for _, k := range required {
@@ -0,0 +1,377 @@
package main
import (
"encoding/json"
"go/ast"
"go/parser"
"go/token"
"path/filepath"
"reflect"
"sort"
"strconv"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// The schemas the controller serves, verb by verb, as the console receives them: from the
// announcement, not the table — what is checked is what a caller is shown (novox/hq issue 244).
func servedSchemas(t *testing.T) map[string]catalogue.Verb {
t.Helper()
handlers, behind, err := seatToolHandlers()
if err != nil {
t.Fatal(err)
}
if len(behind) != 0 {
t.Fatalf("this build cannot run %v of its own seat's verbs", behind)
}
served := map[string]catalogue.Verb{}
for _, e := range seatAnnouncement(handlers).Endpoints {
var input map[string]any
if err := json.Unmarshal([]byte(e.Metadata["schema"]), &input); err != nil {
t.Fatalf("%s announces a schema that is not JSON: %v", e.Name, err)
}
served[e.Metadata["tool"]] = catalogue.Verb{Name: e.Metadata["tool"], Input: input}
}
if len(served) != len(catalogue.ControllerVerbs) {
t.Fatalf("%d verbs served for %d in the table", len(served), len(catalogue.ControllerVerbs))
}
return served
}
// subsetsOf is every subset of the names, the empty one included.
func subsetsOf(names []string) [][]string {
var out [][]string
for mask := 0; mask < 1<<len(names); mask++ {
var s []string
for i, n := range names {
if mask&(1<<i) != 0 {
s = append(s, n)
}
}
out = append(out, s)
}
return out
}
func argumentsFor(subset []string, switches map[string]bool) map[string]any {
args := map[string]any{}
for _, n := range subset {
if switches[n] {
args[n] = "true"
} else {
args[n] = "x-" + n
}
}
return args
}
// saidOutright are the switches that say the default aloud, so the line is the same without them —
// on purpose, and only these.
var saidOutright = map[string]string{
"push": "behind", // the whole mesh is what no machine means; behind lets a caller say they meant it
}
// **No argument a caller gives is passed over** (novox/hq issue 244). For every verb served and
// every combination of the arguments its schema declares, the verb either refuses the call, or
// composes a command line that each given argument changed: taking any one away changes the line
// or makes it refused. An argument the line is the same without is one the verb ignored — the
// shape of the push that named a machine and pushed every machine behind.
func TestNoArgumentAVerbIsGivenIsPassedOver(t *testing.T) {
for name, v := range servedSchemas(t) {
if inProcess[name] {
continue
}
names, switches := declaredArguments(v)
for _, subset := range subsetsOf(names) {
args := argumentsFor(subset, switches)
argv, err := argvFor(name, args)
if err != nil {
if strings.Contains(err.Error(), "does not declare") {
t.Errorf("%s %v: %v", name, args, err)
}
continue
}
for _, dropped := range subset {
fewer := map[string]any{}
for k, val := range args {
if k != dropped {
fewer[k] = val
}
}
without, err := argvFor(name, fewer)
if err == nil && reflect.DeepEqual(argv, without) && saidOutright[name] != dropped {
t.Errorf("%s ignores %q given with %v: %v either way", name, dropped, subset, argv)
}
}
}
}
}
// **An argument a verb does not declare is refused, naming it — never dropped.** Every verb, with
// what it requires and one argument more; and `node` in particular, for every verb whose schema
// does not take a machine.
func TestAnArgumentAVerbDoesNotDeclareIsRefused(t *testing.T) {
for name, v := range servedSchemas(t) {
names, _ := declaredArguments(v)
strangers := []string{"no-such-argument"}
if !contains(names, "node") {
strangers = append(strangers, "node")
}
for _, stranger := range strangers {
args := sampleArguments(v)
args[stranger] = "x"
_, err := argvFor(name, args)
if inProcess[name] {
_, err = readArguments(name, args)
}
if err == nil || !strings.Contains(err.Error(), strconv.Quote(stranger)) {
t.Errorf("%s took %q, which it does not declare: %v", name, stranger, err)
}
}
}
if _, err := argvFor("clear", map[string]any{"dead": "yes"}); err == nil {
t.Error("a switch took a word that is neither true nor false, and would have read it as false")
}
if _, err := argvFor("node", map[string]any{"node": 7}); err == nil {
t.Error("a number was taken as a machine's name, or as no machine")
}
if argv, err := argvFor("clear", map[string]any{"dead": true}); err != nil || strings.Join(argv, " ") != "clear --dead" {
t.Errorf("a switch given as JSON true: %v %v", argv, err)
}
}
// The push that was the cause: a machine named is that machine; none named is the whole mesh, and
// naming one beside behind is refused rather than one of the two guessed.
func TestAPushIsOneMachineOrSaysItIsTheWholeMesh(t *testing.T) {
argv, err := argvFor("push", map[string]any{"node": "g1", "why": "w"})
if err != nil || strings.Join(argv, " ") != "push g1 --wait 0 --why w" {
t.Fatalf("a named push: %v %v", argv, err)
}
for _, args := range []map[string]any{{"why": "w"}, {"behind": "true", "why": "w"}} {
argv, err := argvFor("push", args)
if err != nil || strings.Join(argv, " ") != "push --behind --wait 0 --why w" {
t.Fatalf("a push of the whole mesh %v: %v %v", args, argv, err)
}
}
if _, err := argvFor("push", map[string]any{"node": "g1", "behind": "true", "why": "w"}); err == nil ||
!strings.Contains(err.Error(), `"behind"`) {
t.Fatalf("a named push with behind was taken: %v", err)
}
if _, err := argvFor("push", map[string]any{"machine": "g1"}); err == nil || !strings.Contains(err.Error(), `"machine"`) {
t.Fatalf("a push given the machine under another name ran as a push of every machine: %v", err)
}
}
// commandFlags is every flag set this package's commands parse, by the name the set is made with,
// and the flags defined on it — read from the source, so a flag added to a command is seen here
// without anyone remembering to.
func commandFlags(t *testing.T) map[string][]string {
t.Helper()
files, err := filepath.Glob("*.go")
if err != nil {
t.Fatal(err)
}
fset := token.NewFileSet()
out := map[string][]string{}
for _, f := range files {
if strings.HasSuffix(f, "_test.go") {
continue
}
file, err := parser.ParseFile(fset, f, nil, 0)
if err != nil {
t.Fatal(err)
}
for _, decl := range file.Decls {
fn, ok := decl.(*ast.FuncDecl)
if !ok || fn.Body == nil {
continue
}
sets := map[string]string{} // variable → the set's name
ast.Inspect(fn.Body, func(n ast.Node) bool {
if assign, ok := n.(*ast.AssignStmt); ok && len(assign.Lhs) == 1 && len(assign.Rhs) == 1 {
if call, ok := assign.Rhs[0].(*ast.CallExpr); ok && isSelector(call.Fun, "flag", "NewFlagSet") {
if id, ok := assign.Lhs[0].(*ast.Ident); ok {
if name, ok := stringLit(call.Args[0]); ok {
sets[id.Name] = name
out[name] = append(out[name], []string{}...)
}
}
}
}
call, ok := n.(*ast.CallExpr)
if !ok {
return true
}
sel, ok := call.Fun.(*ast.SelectorExpr)
if !ok {
return true
}
recv, ok := sel.X.(*ast.Ident)
if !ok || sets[recv.Name] == "" {
return true
}
arg := 0
switch sel.Sel.Name {
case "Bool", "String", "Int", "Int64", "Uint", "Uint64", "Float64", "Duration", "Func", "BoolFunc", "TextVar":
case "BoolVar", "StringVar", "IntVar", "Int64Var", "UintVar", "Uint64Var", "Float64Var", "DurationVar", "Var":
arg = 1
default:
return true
}
if arg < len(call.Args) {
if flagName, ok := stringLit(call.Args[arg]); ok {
out[sets[recv.Name]] = append(out[sets[recv.Name]], flagName)
}
}
return true
})
}
}
return out
}
func isSelector(e ast.Expr, pkg, name string) bool {
sel, ok := e.(*ast.SelectorExpr)
if !ok {
return false
}
id, ok := sel.X.(*ast.Ident)
return ok && id.Name == pkg && sel.Sel.Name == name
}
func stringLit(e ast.Expr) (string, bool) {
lit, ok := e.(*ast.BasicLit)
if !ok || lit.Kind != token.STRING {
return "", false
}
s, err := strconv.Unquote(lit.Value)
return s, err == nil
}
// accountedFlags are the flags of a verb's command that are not an argument of the same name:
// carried by an argument named otherwise ("=argument"), or set by the verb itself, or withheld from
// the named verb on purpose — each with why. `command` reaches every flag of the binary regardless.
var accountedFlags = map[string]map[string]string{
"status": {"json": "set by the verb: the answer is data"},
"seats": {"json": "set by the verb: the answer is data"},
"queue": {"json": "not set: the verb answers the table a person reads"},
"plan": {"json": "set by the verb unless files is asked"},
"push": {"wait": "set by the verb to 0: a tool call cannot hold a connection for a whole apply"},
"build": {
"wait": "set by the verb to 0: the id follows the build (issue 176)",
"self": "set by the verb from the repository's form: a path on the forge, or a URL",
"dry-run": "withheld: a dry run answers only when the build ends, which a call cannot wait for; `command` reaches it",
"behind": "withheld: the named verb builds one named repository; `command` reaches the rest",
"on": "withheld: the named verb builds one named repository; `command` reaches the rest",
},
"builds": {"n": "=limit"},
"plans": {"n": "=limit", "what-if": "=repository"},
"durations": {
"json": "set by the verb: the answer is data",
"all": "withheld: every measurement of a fortnight is more than a call should carry; `command` reaches it",
},
"hand-acts": {"json": "set by the verb: the answer is data"},
"conditions": {"json": "set by the verb: the answer is data"},
"retire": {"json": "set by the verb: the answer is data"},
"cleanup": {"json": "set by the verb: the answer is data"},
"data": {"json": "set by the verb: the answer is data"},
"conditions history": {"json": "set by the verb: the answer is data"},
"conditions show": {"json": "set by the verb: the answer is data"},
"healers": {"json": "set by the verb: the answer is data"},
}
// **Every flag of the command a verb runs is in the verb's schema, or accounted for here.** Derived
// from the source, so a flag added to a command — or a verb added whose command takes flags —
// fails this until somebody decides, in writing, how a caller reaches it (novox/hq issue 244). And
// a flag accounted for that no longer exists fails too, so the table cannot rot into a list nobody
// reads.
func TestEveryFlagOfAVerbsCommandIsAnArgumentOrAccountedFor(t *testing.T) {
flags := commandFlags(t)
if len(flags["push"]) == 0 || len(flags["plan"]) == 0 {
t.Fatalf("reading the commands' flags found nothing for push or plan: %v", flags)
}
reached := map[string]map[string]bool{} // verb → flag sets its command lines reach
served := servedSchemas(t)
for name, v := range served {
if inProcess[name] || name == "command" {
continue
}
names, switches := declaredArguments(v)
for _, subset := range subsetsOf(names) {
argv, err := argvFor(name, argumentsFor(subset, switches))
if err != nil {
continue
}
// The flag set is named by the longest run of leading words that names one.
var words []string
for _, w := range argv {
if strings.HasPrefix(w, "-") {
break
}
words = append(words, w)
}
for n := len(words); n > 0; n-- {
if _, has := flags[strings.Join(words[:n], " ")]; has {
if reached[name] == nil {
reached[name] = map[string]bool{}
}
reached[name][strings.Join(words[:n], " ")] = true
break
}
}
}
}
used := map[string]map[string]bool{}
verbs := make([]string, 0, len(reached))
for verb := range reached {
verbs = append(verbs, verb)
}
sort.Strings(verbs)
for _, verb := range verbs {
declared, _ := declaredArguments(served[verb])
for set := range reached[verb] {
if used[set] == nil {
used[set] = map[string]bool{}
}
for _, flagName := range flags[set] {
why, accounted := accountedFlags[set][flagName]
switch {
case accounted && strings.HasPrefix(why, "="):
used[set][flagName] = true
if !contains(declared, strings.TrimPrefix(why, "=")) {
t.Errorf("%s: --%s of `%s` is said to be carried by %q, which the schema does not declare",
verb, flagName, set, strings.TrimPrefix(why, "="))
}
case accounted:
used[set][flagName] = true
case contains(declared, flagName):
default:
t.Errorf("%s runs `%s`, which takes --%s, and the verb's schema has no %q: declare it, "+
"or say in accountedFlags why a caller does not reach it", verb, set, flagName, flagName)
}
}
}
}
for set, fs := range accountedFlags {
for flagName := range fs {
if !used[set][flagName] {
t.Errorf("accountedFlags names --%s of `%s`, which no verb's command takes any more", flagName, set)
}
}
}
}
// Every verb's required arguments are properties of its schema: a schema that requires what it
// does not describe is the uncallable verb of issue 244 from the other side.
func TestEveryRequiredArgumentIsDescribed(t *testing.T) {
for name, v := range servedSchemas(t) {
names, _ := declaredArguments(v)
for k := range sampleArguments(v) {
if !contains(names, k) {
t.Errorf("%s requires %q and does not describe it", name, k)
}
}
}
}
+14 -28
View File
@@ -10,29 +10,6 @@ import (
"github.com/novox/mesh-controller/internal/catalogue"
)
// Every verb the mesh-controller seat declares is one this binary can run, with the arguments the
// schema names and no other (novox/hq ADR 0154, ADR 0035).
func TestEveryDeclaredVerbHasACommandLine(t *testing.T) {
for _, v := range catalogue.ControllerVerbs {
if v.Name == "tools" {
continue
}
args := map[string]any{}
props, _ := v.Input["properties"].(map[string]any)
for name := range props {
args[name] = "x"
}
argv, err := argvFor(v.Name, args)
if err != nil {
t.Errorf("%s: %v", v.Name, err)
continue
}
if argv[0] == "" {
t.Errorf("%s: empty command", v.Name)
}
}
}
// `builds` given a build's id reads that build's log from the bus rather than listing builds
// (novox/hq ADR 0157).
func TestBuildsWithAnIdReadsThatBuildsLog(t *testing.T) {
@@ -66,13 +43,22 @@ func TestRotateTakesAProvisionOrAnOwnSecret(t *testing.T) {
if strings.Join(argv, " ") != "rotate postgres-database --consumer ace" {
t.Fatalf("a pair credential: %v", argv)
}
argv, _ = argvFor("rotate", map[string]any{"provision": "postgres-database", "consumer": "ace", "module": "letta"})
if strings.Join(argv, " ") != "rotate postgres-database --consumer ace --module letta" {
t.Fatalf("one consuming module's pair credential: %v", argv)
}
argv, _ = argvFor("rotate", map[string]any{"node": "ace", "module": "nodered", "secret": "api-token"})
if strings.Join(argv, " ") != "secret rotate ace nodered api-token" {
t.Fatalf("an own secret: %v", argv)
}
argv, _ = argvFor("rotate", map[string]any{"node": "ace"})
if strings.Join(argv, " ") != "rotate" {
t.Fatalf("half an own secret falls to the command's usage: %v", argv)
if _, err := argvFor("rotate", map[string]any{"node": "ace"}); err == nil || !strings.Contains(err.Error(), `"module"`) {
t.Fatalf("half an own secret is refused, naming what it lacks: %v", err)
}
if argv, _ := argvFor("rotate", nil); strings.Join(argv, " ") != "rotate" {
t.Fatalf("neither shape falls to the command's usage: %v", argv)
}
if _, err := argvFor("rotate", map[string]any{"provision": "p", "node": "ace", "module": "m", "secret": "s"}); err == nil {
t.Fatal("both shapes at once were taken, and one of them passed over")
}
}
@@ -121,8 +107,8 @@ func TestAVerbMissingWhatItNeedsIsRefused(t *testing.T) {
// A push and a build are sent, not waited for: the asker reads status, or the build's log by its
// id, for what happened. A repository given as a forge path is said to be one (issue 176).
func TestActsDoNotBlockTheCall(t *testing.T) {
argv, _ := argvFor("push", map[string]any{"node": "one"})
if strings.Join(argv, " ") != "push one --wait 0" {
argv, _ := argvFor("push", map[string]any{"node": "one", "why": "w"})
if strings.Join(argv, " ") != "push one --wait 0 --why w" {
t.Fatalf("push waits: %v", argv)
}
argv, _ = argvFor("build", map[string]any{"repository": "novox/x", "path": "modules/x"})
+38 -5
View File
@@ -93,18 +93,31 @@ func secretCommand(ctx context.Context, args []string) error {
fmt.Printf(" run `push %s` and `push %s` to send it\n", *provider, node)
return nil
}
if err := open.inventory.AcceptSecretForModule(ctx, node, module, name, value); err != nil {
untilStart, err := open.inventory.AcceptGivenSecret(ctx, node, module, name, value)
if err != nil {
return err
}
// Not printed back, and there is nowhere it could be printed from: it is sealed to that
// machine and the mesh cannot read it again.
fmt.Printf("%s on %s now holds %q, sealed to that machine.\n", module, node, name)
fmt.Printf(" the mesh cannot read it back, and will not replace it with one of its own\n")
if untilStart {
// Accepted, and said what it is for (novox/hq ADR 0228): a value given by hand adopts
// something that already holds it, and lives only until the module has started on it.
fmt.Printf(" it lives until %s next starts well under the mesh on it, and is then replaced with a value\n"+
" the mesh makes, sealed and sent (ADR 0228): a value given by hand is for adopting something\n"+
" already running that holds it\n", module)
if record, err := open.inventory.NodeByName(ctx, node); err == nil && !record.Adopted {
fmt.Printf(" %s is not an adopted machine: if %s is installed fresh there, it needs no given value —\n"+
" the mesh makes one at the first push\n", node, module)
}
} else {
fmt.Printf(" the mesh cannot read it back, and will not replace it with one of its own\n")
}
fmt.Printf(" run `push %s` to send it\n", node)
return nil
}
const secretUsage = "secret rotate <node> <module> <name>\n" +
const secretUsage = "secret rotate <node> <module> <name> [--why <text> [--cause <word>]]\n" +
"secret accept <node> <module> <name> [--from <file>] [--provider <node> [--local <name>]]\n" +
"secret recover <node> <module> <name> --key <operator-key> [--out <file>] [--from-export <file>] [--provider <node>]\n" +
"secret export [--out <file>]"
@@ -367,9 +380,20 @@ func valueFor(node, module, name, from string) (string, error) {
// secretRotate makes a module's own secret anew and sends the machine, so the module starts again on
// the new value (novox/hq ADR 0114, issue 180). A pair credential rotates with `rotate <provision>`;
// this is the secret with one party. Said in the log with who asked and when, never the value.
//
// **A value given to the mesh rotates the same way** (novox/hq ADR 0228): what a module reads at start
// is held by nobody else, so the old value is not needed to replace it. Refused for a value an
// outside party issued, which no value of the mesh's would replace. Why it was rotated is recorded in
// the hand-act log when given — a rotation asked by a person is an act by hand, and a leak is a cause
// worth counting.
func secretRotate(ctx context.Context, args []string) error {
rest, _ := split(args)
if len(rest) != 3 {
rest, flags := split(args)
set := flag.NewFlagSet("secret rotate", flag.ContinueOnError)
why := addHandActFlags(set)
if err := set.Parse(flags); err != nil {
return err
}
if len(rest) != 3 || set.NArg() != 0 {
return errors.New(secretUsage)
}
node, module, name := rest[0], rest[1], rest[2]
@@ -378,6 +402,10 @@ func secretRotate(ctx context.Context, args []string) error {
return err
}
defer open.Close()
origin, given, err := open.inventory.OwnSecretOrigin(ctx, node, module, name)
if err != nil {
return err
}
if err := open.inventory.RotateModuleSecret(ctx, node, module, name); err != nil {
var refused inventory.ErrNotRotatable
if errors.As(err, &refused) {
@@ -385,8 +413,13 @@ func secretRotate(ctx context.Context, args []string) error {
}
return err
}
why.record(ctx, "secret rotate", []string{node, module, name})
fmt.Printf("rotated %q of %s on %s at %s, asked by %s; the value is sealed and not shown\n",
name, module, node, time.Now().UTC().Format(time.RFC3339), whoAsked())
if origin == inventory.OriginAccepted {
fmt.Printf(" it replaces the value given to the mesh on %s: the mesh made this one, so `secret rotate` "+
"replaces it again whenever asked (ADR 0228)\n", given.UTC().Format("2006-01-02"))
}
// A shared credential (ADR 0158) has as many holders as the provision has consumers, and all
// of them are sent in one act, so no machine is left reading a value the provider no longer takes.
machines, err := open.inventory.SharedHolders(ctx, node, module, name)
+25
View File
@@ -22,6 +22,11 @@ type sendable struct {
// under the node's hold just before the body is made (novox/hq 04-ISSUES/107). Zero is not sent
// at all, which a host reads as "no order claimed" — the shape of every declaration before this.
Sequence int64
// Epoch is the controller lease's epoch it was composed under (novox/hq to-be 45 §6): a machine that
// heard a later epoch refuses it. Zero is not sent at all — every machine whose node-engine has not
// said it reads one is sent none, because an older node-engine refuses a key it does not know, whole
// (link/order.go, the contract).
Epoch uint64
// Adoption is nil for a converged node, and then the body is byte for byte what it was before
// adoption existed: an older host parses the envelope strictly and would refuse the key.
Adoption *adoptionEnvelope
@@ -43,6 +48,23 @@ type sendable struct {
LeftOut []string
// leftOutWhy is why each was, for push and plan to say; never on the wire.
leftOutWhy map[string]string
// withheld is every consumer this machine's grants leave out, because its identity overflows the
// provision's bound (novox/hq ADR 0225); for push and plan to say, never on the wire.
withheld []catalogue.Overflow
// unbound is every consumer whose credential from this machine is on record and that is bound
// elsewhere (novox/hq issue 274); for push and plan to say, never on the wire.
unbound []catalogue.Unbound
// foreseen is every own secret composed with a stand-in, as `module/name`: what the next send will
// make (Foreseeing, novox/hq issue 275). Never on the wire, and a declaration that has any is never
// sent.
foreseen []string
// Builds is the build of each module this declaration carries — module to the commit its build
// was made from — recorded with the send and never on the wire (novox/hq issue 259, ADR 0221).
// Composed only on the send path; nil records that it is not known.
Builds map[string]string
// Bindings is where each consumer of a provision that keeps its data is bound in this declaration
// (novox/hq ADR 0232), recorded with the send and never on the wire. Composed only on the send path.
Bindings []inventory.Binding
}
// adoptionEnvelope is what an adopted node is told about its mode. Taken is every module taken on
@@ -63,6 +85,9 @@ func (s sendable) Body() ([]byte, error) {
if s.Sequence > 0 {
envelope["sequence"] = s.Sequence
}
if s.Epoch > 0 {
envelope["epoch"] = s.Epoch
}
if len(s.LeftOut) > 0 {
envelope["left_out"] = s.LeftOut
}
+645
View File
@@ -0,0 +1,645 @@
package main
import (
"fmt"
"sort"
"strings"
"time"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// The signals table (novox/hq to-be 45 §3, ADR 0227 rule 5), compiled in.
//
// **Every signal the core expects has a watchdog; its absence is a condition.** A row says what is
// expected, from whom, after what, within what bound, and what is raised when it does not come. One
// table, read three ways: by the watchdog loop (watchdogs.go), which runs every row's watch over the
// facts it gathered; by `doctor signals`, which says the age of every row's newest signal; and by the
// test generated from it (signals_test.go), which suppresses each signal in turn and asserts its
// condition — so a row added without a watch, or a watch that does not fire, fails the build.
//
// A row not watched yet says why, and in which phase it will be: a watchdog of a signal nothing emits
// would be a condition that can only cry wolf. Bounds marked provisional are set from what Phase 0
// measured (`durations`) and corrected in Phase 1's first live week; a corrected bound is a change to
// this table, reviewed like code.
// signalRow is one row of the table.
type signalRow struct {
Row string
Signal string
Emitter string
Trigger string
Bound string
Kind string
Severity conditions.Severity
// Phase is the phase of to-be 45 its watchdog is built in.
Phase int
// Deferred says why it is not watched yet; empty for a row that is.
Deferred string
// needs is the part of the facts the row reads: an error there is the row blind, which is
// itself said (probe-failed) rather than read as nothing wrong.
needs func(f *signalFacts) error
// watch is what is wrong now, from the facts.
watch func(f *signalFacts) []conditions.Observation
// newest is the time of the newest signal of this row, for `doctor signals`; zero when none.
newest func(f *signalFacts) time.Time
}
// The bounds, provisional where to-be 45 says so.
const (
// heartbeatEvery is the interval a node-engine or node tools that say none have always used.
heartbeatEvery = 60 * time.Second
// heartbeatsMissed is how many intervals may pass in silence (S1, S11).
heartbeatsMissed = 3
// controlNodeUrgentAfter is how long the control node may be silent before it is urgent (S1).
controlNodeUrgentAfter = 30 * time.Minute
// reportAtLeast is the least a machine is given to report a send (S2).
reportAtLeast = 2 * time.Minute
// tierAtLeast is the least a plan's tier is given (S3), the bound `status` calls a plan late at.
tierAtLeast = planWaitBound
// loopDeafAfter is how long the event loop may take nothing while its consumers hold some (S4).
loopDeafAfter = 2 * time.Minute
// askAtLeast is the least a build ask is given, and askDefault the bound while nothing is
// measured (S6): the build seat declares no timeout of its own.
askAtLeast = 20 * time.Minute
askDefault = time.Hour
// callDefault is the bound of a verb that declares none (S7); push's and build's are longer.
callDefault = 10 * time.Minute
// advisoryQuiet is how long the bus must be quiet about a thing before its advisory clears (S9).
advisoryQuiet = time.Hour
// staleRefusalsAllowed in staleRefusalsWithin are what S13 lets pass from one writer.
staleRefusalsAllowed = 5
staleRefusalsWithin = 5 * time.Minute
// leaseBound is how long the lease may go unrenewed (S12): the key's age.
leaseBound = broker.LeaseTTL
)
// callBounds are the verbs that may run longer than callDefault, and how long (S7).
var callBounds = map[string]time.Duration{
"push": 30 * time.Minute, "rotate": 30 * time.Minute, "assign": 15 * time.Minute,
"unassign": 15 * time.Minute, "command": 30 * time.Minute, "doctor": 3 * time.Minute,
}
// callBound is a verb's bound.
func callBound(verb string) time.Duration {
if b, ok := callBounds[verb]; ok {
return b
}
return callDefault
}
// signalsTable is the table, in to-be 45's order.
var signalsTable = []signalRow{
{Row: "S1", Signal: "machine heartbeat", Emitter: "node-engine", Trigger: "its interval",
Bound: "3 × the interval it says (60 s when it says none), provisional; not raised while the machine " +
"said it is asleep or shutting down (ADR 0211); urgent after 30 min for a control node",
Kind: "silent", Severity: conditions.Warning, Phase: 1,
needs: func(f *signalFacts) error { return f.machinesErr }, watch: watchHeartbeats,
newest: func(f *signalFacts) time.Time {
return newestOf(f.machines, func(m machineFacts) time.Time { return m.lastHeard })
}},
{Row: "S2", Signal: "report after a send", Emitter: "node-engine", Trigger: "each declaration sent",
Bound: "max(2 min, 3 × that machine's last apply duration), provisional; not while the machine is silent or asleep",
Kind: "sent-not-reported", Severity: conditions.Warning, Phase: 1,
needs: func(f *signalFacts) error { return f.machinesErr }, watch: watchReports,
newest: func(f *signalFacts) time.Time {
return newestOf(f.machines, func(m machineFacts) time.Time { return m.reportedAt })
}},
{Row: "S3", Signal: "plan tier progress", Emitter: "controller's plan", Trigger: "each tier entered",
Bound: "max(30 min, 3 × the p90 of that repository's measured tiers), provisional; not while the " +
"build seat is paused under it",
Kind: "stalled", Severity: conditions.Warning, Phase: 1,
needs: func(f *signalFacts) error { return f.plansErr }, watch: watchPlans,
newest: func(f *signalFacts) time.Time {
return newestOf(f.plans, func(p planFacts) time.Time { return p.entered })
}},
{Row: "S4", Signal: "the controller's event loop takes a message", Emitter: "controller",
Trigger: "while its consumers have pending messages", Bound: "2 min",
Kind: "controller-deaf", Severity: conditions.Urgent, Phase: 1,
needs: func(f *signalFacts) error { return f.loopErr }, watch: watchLoop,
newest: func(f *signalFacts) time.Time { return f.loop.took }},
{Row: "S5", Signal: "a merge announced becomes a plan, or nothing reads it", Emitter: "announcer → controller",
Trigger: "each merge", Bound: "10 min (the catch-up pass of issue 266)",
Kind: "merge-not-acted", Severity: conditions.Urgent, Phase: 1,
needs: func(f *signalFacts) error { return f.mergesErr }, watch: watchMerges,
newest: func(f *signalFacts) time.Time { return f.mergesPassed }},
{Row: "S6", Signal: "a build asked → its outcome", Emitter: "build seat", Trigger: "each ask",
Bound: "max(20 min, 3 × the p90 of measured builds), 1 h while nothing is measured, provisional — the " +
"build seat declares no timeout; and an ask the queue gave up on (dead) at once",
Kind: "ask-lost", Severity: conditions.Warning, Phase: 1,
needs: func(f *signalFacts) error { return f.asksErr }, watch: watchAsks,
newest: func(f *signalFacts) time.Time { return newestOf(f.asks, func(a askFacts) time.Time { return a.since }) }},
{Row: "S7", Signal: "a call running → finished", Emitter: "controller", Trigger: "each call",
Bound: "the verb's bound: push, rotate and command 30 min, assign and unassign 15 min, doctor 3 min, " +
"any other 10 min",
Kind: "call-hung", Severity: conditions.Warning, Phase: 1,
needs: func(*signalFacts) error { return nil }, watch: watchCalls,
newest: func(f *signalFacts) time.Time {
return newestOf(f.calls, func(c link.Call) time.Time { return c.Started })
}},
{Row: "S8", Signal: "a provider's failing word repeated", Emitter: "provider",
Trigger: "every 15 min while failing (ADR 0224)", Bound: "30 min",
Kind: kindProviderSilent, Severity: conditions.Warning, Phase: 1,
needs: func(f *signalFacts) error { return f.standingsErr }, watch: watchProviders,
newest: func(f *signalFacts) time.Time {
return newestOf(f.standings, func(c conditions.Condition) time.Time { return c.LastObserved })
}},
{Row: "S9", Signal: "bus advisories: maximum deliveries, consumer deleted; the controller's own slow " +
"consumer and refused subjects", Emitter: "bus server's advisory subjects; the controller's connection",
Trigger: "any", Bound: "any occurrence; clears after an hour without another, and a deleted consumer " +
"once it exists again or the mesh no longer expects it",
Kind: "slow-consumer, max-deliveries, refused, consumer-lost", Severity: conditions.Warning, Phase: 1,
needs: func(f *signalFacts) error { return f.advisoriesErr }, watch: watchAdvisories,
newest: func(f *signalFacts) time.Time {
return newestOf(f.advisories, func(a link.Advisory) time.Time { return a.Last })
}},
{Row: "S10", Signal: "the self-check's heartbeat", Emitter: "controller's doctor", Trigger: "every run",
Bound: "2 × its interval; watched from a second machine by mesh-watcher, and here as well",
Kind: "self-check-silent", Severity: conditions.Urgent, Phase: 1,
needs: func(*signalFacts) error { return nil }, watch: watchSelfCheck,
newest: func(f *signalFacts) time.Time { return f.selfCheck.last }},
{Row: "S11", Signal: "node tools heartbeat", Emitter: "node tools", Trigger: "its interval",
Bound: "3 × the interval it says (60 s when it says none), provisional; only where node-tools is assigned",
Kind: "tools-silent", Severity: conditions.Warning, Phase: 1,
needs: func(f *signalFacts) error { return f.machinesErr }, watch: watchTools,
newest: func(f *signalFacts) time.Time {
return newestOf(f.machines, func(m machineFacts) time.Time { return m.toolsHeard })
}},
{Row: "S12", Signal: "the controller lease renewed", Emitter: "controller", Trigger: "every 5 s",
Bound: "15 s (the key's age); a holder that lost the lease, or stopped renewing and was taken over, and a " +
"lease bucket found raised again from nothing, are said for an hour after; a controller serving without " +
"the lease, for as long as it does",
Kind: "lease-lost", Severity: conditions.Urgent, Phase: 2,
needs: func(f *signalFacts) error { return f.leaseErr }, watch: watchLease,
newest: func(f *signalFacts) time.Time { return f.lease.renewed }},
{Row: "S13", Signal: "stale refusals", Emitter: "every receiver (rule 2)", Trigger: "each refusal",
Bound: "more than 5 from one writer in 5 min: a controller epoch, a controller that claimed none, or a " +
"machine's node-engine whose accounts the controller refused",
Kind: "stale-writer", Severity: conditions.Warning, Phase: 2,
needs: func(*signalFacts) error { return nil }, watch: watchStaleRefusals,
newest: func(f *signalFacts) time.Time {
return newestOf(f.staleRefusals, func(w link.WriterRefusals) time.Time { return w.Last })
}},
{Row: "S14", Signal: "facts snapshot exported", Emitter: "controller", Trigger: "daily",
Bound: "2 days", Kind: "facts-stale", Severity: conditions.Warning, Phase: 5,
Deferred: "the facts snapshot is built in Phase 5 (to-be 45 §9): nothing exports one yet"},
{Row: "S15", Signal: "a hand act with a cause already recorded", Emitter: "hand-act log",
Trigger: "each act", Bound: "the second within 14 days; clears when fewer than two remain within 14 days",
Kind: "healer-wanted", Severity: conditions.Warning, Phase: 3,
needs: func(f *signalFacts) error { return f.handActsErr }, watch: watchHandActs,
newest: func(f *signalFacts) time.Time {
return newestOf(f.handActs, func(a link.HandAct) time.Time { return a.At })
}},
}
// newestOf is the newest time among things.
func newestOf[T any](list []T, at func(T) time.Time) time.Time {
var newest time.Time
for _, x := range list {
if t := at(x); t.After(newest) {
newest = t
}
}
return newest
}
// ago is a duration as the summaries say it.
func ago(d time.Duration) string {
if d < time.Minute {
return d.Round(time.Second).String()
}
return d.Round(time.Minute).String()
}
// heartbeatBound is a machine's S1 or S11 bound from the interval it says.
func heartbeatBound(every time.Duration) time.Duration {
if every <= 0 {
every = heartbeatEvery
}
return heartbeatsMissed * every
}
func watchHeartbeats(f *signalFacts) []conditions.Observation {
var out []conditions.Observation
for _, m := range f.machines {
if m.lastHeard.IsZero() || m.asleep() {
// Never heard is a machine that has not joined, which status says; asleep is not lost.
continue
}
bound := heartbeatBound(m.every)
silent := f.now.Sub(m.lastHeard)
if silent <= bound {
continue
}
severity := conditions.Warning
if m.control && silent > controlNodeUrgentAfter {
severity = conditions.Urgent
}
out = append(out, conditions.Observation{Scope: conditions.ScopeMachine, ID: m.name, Kind: "silent",
Machine: m.name, Severity: severity,
Summary: fmt.Sprintf("%s has not been heard from since %s (bound %s)", m.name,
m.lastHeard.UTC().Format("2006-01-02 15:04 MST"), bound),
Said: fmt.Sprintf("silent for %s", ago(silent))})
}
return out
}
func watchReports(f *signalFacts) []conditions.Observation {
var out []conditions.Observation
for _, m := range f.machines {
if m.sentAt.IsZero() || m.reportedCurrent || m.asleep() {
continue
}
if !m.lastHeard.IsZero() && f.now.Sub(m.lastHeard) > heartbeatBound(m.every) {
continue // silent: S1 says it, and a silent machine reports nothing
}
bound := max(reportAtLeast, 3*m.lastApply)
waited := f.now.Sub(m.sentAt)
if waited <= bound {
continue
}
out = append(out, conditions.Observation{Scope: conditions.ScopeMachine, ID: m.name,
Kind: "sent-not-reported", Machine: m.name, Severity: conditions.Warning,
Summary: fmt.Sprintf("%s was sent a declaration at %s and has not reported applying it (bound %s)",
m.name, m.sentAt.UTC().Format("2006-01-02 15:04 MST"), ago(bound)),
Said: fmt.Sprintf("waiting %s for the report of the declaration sent", ago(waited))})
}
return out
}
func watchPlans(f *signalFacts) []conditions.Observation {
var out []conditions.Observation
for _, p := range f.plans {
if p.paused {
continue
}
in := f.now.Sub(p.entered)
if in <= p.bound {
continue
}
out = append(out, conditions.Observation{Scope: conditions.ScopePlan, ID: p.id, Kind: "stalled",
Severity: conditions.Warning,
Summary: fmt.Sprintf("the plan for %s %s has been at tier %d of %d since %s (bound %s): %s",
p.repository, short(p.commit), p.tier+1, p.tiers, p.entered.UTC().Format("2006-01-02 15:04 MST"),
ago(p.bound), p.waiting),
Said: fmt.Sprintf("at tier %d for %s: %s", p.tier+1, ago(in), p.waiting)})
}
return out
}
func watchLoop(f *signalFacts) []conditions.Observation {
if f.loop.pending == 0 {
return nil
}
since := f.loop.took
if since.IsZero() {
since = f.started
}
if f.now.Sub(since) <= loopDeafAfter {
return nil
}
return []conditions.Observation{{Scope: conditions.ScopeCore, ID: "controller", Kind: "controller-deaf",
Token: "deaf", Machine: f.host, Severity: conditions.Urgent,
Summary: fmt.Sprintf("the controller's event loop has taken nothing for %s while its consumers hold %d "+
"message(s): reports, builds and merges are not being acted on", ago(f.now.Sub(since)), f.loop.pending),
Said: fmt.Sprintf("%d pending (%s), last taken %s", f.loop.pending, f.loop.where, since.UTC().Format(time.RFC3339))}}
}
func watchMerges(f *signalFacts) []conditions.Observation {
var out []conditions.Observation
for _, m := range f.merges {
out = append(out, conditions.Observation{Scope: conditions.ScopeMerge, ID: m.Repo + "." + short(m.Commit),
Kind: "merge-not-acted", Token: "not-acted", Severity: conditions.Urgent,
Summary: fmt.Sprintf("%s/%s merged into %s (%s) was never handed to the controller by the bus; "+
"acted on late by the catch-up", m.Owner, m.Repo, m.Base, short(m.Commit)),
Said: fmt.Sprintf("announced %s, %s behind it", m.At.UTC().Format(time.RFC3339), readableList(m.Modules))})
}
return out
}
func watchAsks(f *signalFacts) []conditions.Observation {
var out []conditions.Observation
for _, a := range f.asks {
var said string
switch {
case a.state == link.AskDead:
said = fmt.Sprintf("the %s queue handed it out as often as it may and it was never settled", a.seat)
case a.state == link.AskInFlight && f.now.Sub(a.since) > a.bound:
said = fmt.Sprintf("in flight on %s for %s (bound %s)", orSomewhere(a.on), ago(f.now.Sub(a.since)), ago(a.bound))
default:
continue
}
out = append(out, conditions.Observation{Scope: conditions.ScopeBuild, ID: a.id, Kind: "ask-lost",
Token: "lost", Machine: a.on, Severity: conditions.Warning,
Summary: fmt.Sprintf("the build %s of %s has no outcome: %s", a.id, a.what, said), Said: said})
}
return out
}
func orSomewhere(node string) string {
if node == "" {
return "a machine that did not say which"
}
return node
}
func watchCalls(f *signalFacts) []conditions.Observation {
var out []conditions.Observation
for _, c := range f.calls {
bound := callBound(c.Verb)
running := f.now.Sub(c.Started)
if running <= bound {
continue
}
out = append(out, conditions.Observation{Scope: conditions.ScopeCall, ID: c.ID, Kind: "call-hung",
Token: "hung", Machine: f.host, Severity: conditions.Warning,
Summary: fmt.Sprintf("%s.%s (call %s) has been running since %s, past its bound of %s", c.Seat, c.Verb,
c.ID, c.Started.UTC().Format("2006-01-02 15:04 MST"), ago(bound)),
Said: fmt.Sprintf("running %s, asked by %s", ago(running), orSomebody(c.Caller))})
}
return out
}
func orSomebody(caller string) string {
if caller == "" {
return "a caller the bus did not name"
}
return caller
}
func watchProviders(f *signalFacts) []conditions.Observation {
var out []conditions.Observation
for _, c := range f.standings {
quiet := f.now.Sub(c.LastObserved)
if quiet <= providerSaysAgainWithin {
continue
}
module, node, consumer, ok := providerOf(c)
if !ok {
continue
}
out = append(out, conditions.Observation{Scope: conditions.ScopeProvider, ID: module + "." + node + "." + consumer,
Token: "silent", Kind: kindProviderSilent, Machine: node, Severity: conditions.Warning,
Summary: fmt.Sprintf("%s on %s said it keeps failing %s and has said nothing since %s: it stopped "+
"saying anything, so its last word is all the mesh has", module, node, consumer,
c.LastObserved.UTC().Format("2006-01-02 15:04 MST")),
Said: fmt.Sprintf("not said again for %s", ago(quiet))})
}
return out
}
func watchAdvisories(f *signalFacts) []conditions.Observation {
var out []conditions.Observation
for _, a := range f.advisories {
if f.now.Sub(a.Last) > advisoryQuiet {
continue
}
if a.Kind == link.AdvisoryConsumerLost && !f.lostConsumers[a.Stream+"."+a.Consumer] {
continue // it exists again, or the mesh no longer expects it: a removal, not a loss
}
severity := conditions.Warning
times := ""
if a.Count > 1 {
times = fmt.Sprintf(" (%d times since %s)", a.Count, a.First.UTC().Format("15:04 MST"))
}
machine := ""
if a.ID == "controller" {
machine = f.host
}
out = append(out, conditions.Observation{Scope: conditions.ScopeBus, ID: a.ID, Kind: a.Kind,
Machine: machine, Severity: severity, Summary: a.Said + times, Said: a.Said})
}
return out
}
func watchSelfCheck(f *signalFacts) []conditions.Observation {
every := f.selfCheck.every
if every <= 0 {
every = doctorEvery
}
since := f.selfCheck.last
if since.IsZero() {
since = f.started
}
if f.now.Sub(since) <= 2*every {
return nil
}
return []conditions.Observation{{Scope: conditions.ScopeCore, ID: "doctor", Kind: "self-check-silent",
Machine: f.host, Severity: conditions.Urgent,
Summary: fmt.Sprintf("the self-check has not finished a run since %s (it runs every %s): the mesh's "+
"invariants are not being checked", since.UTC().Format("2006-01-02 15:04 MST"), every),
Said: fmt.Sprintf("no run for %s", ago(f.now.Sub(since)))}}
}
func watchTools(f *signalFacts) []conditions.Observation {
var out []conditions.Observation
for _, m := range f.machines {
if !m.tools || m.asleep() {
continue
}
bound := heartbeatBound(m.toolsEvery)
since := m.toolsHeard
if since.IsZero() {
// Not heard since this controller started: silent since then at the most.
since = f.toolsHeardFrom
}
silent := f.now.Sub(since)
if silent <= bound {
continue
}
heard := "not since this controller started at " + since.UTC().Format("2006-01-02 15:04 MST")
if !m.toolsHeard.IsZero() {
heard = "since " + m.toolsHeard.UTC().Format("2006-01-02 15:04 MST")
}
out = append(out, conditions.Observation{Scope: conditions.ScopeMachine, ID: m.name, Kind: "tools-silent",
Machine: m.name, Severity: conditions.Warning,
Summary: fmt.Sprintf("the node tools on %s have not said they are there %s (bound %s): nothing can "+
"ask that machine anything", m.name, heard, bound),
Said: fmt.Sprintf("silent for %s", ago(silent))})
}
return out
}
func watchStaleRefusals(f *signalFacts) []conditions.Observation {
var out []conditions.Observation
for _, w := range f.staleRefusals {
if w.Count <= staleRefusalsAllowed {
continue
}
scope, id, machine := conditions.ScopeCore, "controller.unnamed", ""
named := w.Writer
switch {
case w.Epoch > 0:
id = fmt.Sprintf("controller.epoch-%d", w.Epoch)
if e, ok := f.epochs[w.Epoch]; ok {
named = fmt.Sprintf("the controller of epoch %d (%s%s)", w.Epoch, e.Instance, endedWords(e))
}
case w.Writer == link.WriterNodeEngine(firstOf(w.Receivers)) || strings.HasPrefix(w.Writer, "the node-engine on "):
node := strings.TrimPrefix(w.Writer, "the node-engine on ")
scope, id, machine = conditions.ScopeMachine, node, node
}
if machine == "" && len(w.Receivers) > 0 {
machine = w.Receivers[0]
}
var also []string
for _, r := range w.Receivers {
if r != machine && r != "controller" {
also = append(also, r)
}
}
out = append(out, conditions.Observation{Scope: scope, ID: id, Kind: "stale-writer", Token: "stale-writer",
Machine: machine, Also: also, Severity: conditions.Warning,
Summary: fmt.Sprintf("%s was refused %d time(s) in %s as older than what its receivers hold (%s): a "+
"writer is sending what the mesh has moved past", named, w.Count, staleRefusalsWithin,
strings.Join(w.Receivers, ", ")),
Said: fmt.Sprintf("%d stale refusals in %s, the last at %s", w.Count, staleRefusalsWithin,
w.Last.UTC().Format(time.RFC3339))})
}
return out
}
// firstOf is a list's first, empty for none.
func firstOf(list []string) string {
if len(list) == 0 {
return ""
}
return list[0]
}
// endedWords is how an epoch ended, for a sentence naming it; nothing while it is held.
func endedWords(e inventory.Epoch) string {
if e.Ended == nil {
return ", still holding the lease"
}
return fmt.Sprintf(", %s at %s", e.How, e.Ended.UTC().Format("15:04:05 MST"))
}
// watchLease is S12: the lease held and renewed by this controller, and every holder that lost it.
func watchLease(f *signalFacts) []conditions.Observation {
var out []conditions.Observation
l := f.lease
if l.unleased != "" {
out = append(out, conditions.Observation{Scope: conditions.ScopeCore, ID: "controller.lease", Token: "unleased",
Kind: "lease-lost", Machine: f.host, Severity: conditions.Urgent,
Summary: "the controller serves WITHOUT the lease: nothing keeps a second controller from acting " +
"beside it, and its declarations carry no epoch. A bus whose user list is older than this " +
"controller does not grant it the lease's bucket: a push of the machine holding the bus sends " +
"the list that does, and the controller takes the lease within five seconds — " + l.unleased,
Said: l.unleased})
} else if l.held && !l.renewed.IsZero() && f.now.Sub(l.renewed) > leaseBound {
out = append(out, conditions.Observation{Scope: conditions.ScopeCore, ID: "controller.lease", Token: "late",
Kind: "lease-lost", Machine: f.host, Severity: conditions.Urgent,
Summary: fmt.Sprintf("the controller has not renewed its lease (epoch %d) since %s, past the key's age "+
"of %s: another controller may take it", l.epoch, l.renewed.UTC().Format(time.RFC3339), leaseBound),
Said: fmt.Sprintf("not renewed for %s", ago(f.now.Sub(l.renewed)))})
}
if !l.reset.IsZero() && f.now.Sub(l.reset) <= advisoryQuiet {
out = append(out, conditions.Observation{Scope: conditions.ScopeCore, ID: "controller.lease", Token: "reset",
Kind: "lease-lost", Machine: f.host, Severity: conditions.Urgent,
Summary: "the controller lease's bucket was raised again from nothing: " + l.resetSaid,
Said: l.resetSaid})
}
var lost []string
newest := inventory.Epoch{}
for _, e := range l.ended {
if e.Ended == nil || e.How == inventory.EpochReleased || f.now.Sub(*e.Ended) > advisoryQuiet {
continue
}
lost = append(lost, fmt.Sprintf("epoch %d (%s) %s at %s", e.Epoch, e.Instance, e.How,
e.Ended.UTC().Format("15:04:05 MST")))
if newest.Ended == nil || e.Ended.After(*newest.Ended) {
newest = e
}
}
if len(lost) > 0 {
how := "lost it: its renewal was refused or could not be made"
if newest.How == inventory.EpochExpired {
how = "stopped renewing it without giving it back, and was taken over"
}
out = append(out, conditions.Observation{Scope: conditions.ScopeCore, ID: "controller.lease", Token: "lost",
Kind: "lease-lost", Machine: newest.Host, Severity: conditions.Urgent,
Summary: fmt.Sprintf("the controller of epoch %d (%s) %s; the controller of epoch %d acts now", newest.Epoch,
newest.Instance, how, l.epoch),
Said: strings.Join(lost, "; ")})
}
return out
}
// handActsWithin is how far back a repeated cause counts (S15).
const handActsWithin = 14 * 24 * time.Hour
// personsDecision are the causes of hand acts that are a person's decision by design, never a repair a
// healer could take over: approving or rejecting a retirement, and deleting what was retired (novox/hq
// ADR 0230 — nothing is retired past the bound or deleted without a person). Repeated, they are the
// mesh working as decided, not a healer wanted.
var personsDecision = map[string]bool{kindRetireWaiting: true, kindCleanupWaiting: true}
// watchHandActs is S15: a cause recorded by hand twice within a fortnight is a healer wanted, named by
// the cause. **A heal is never a hand act** (healers.go), so a cause a healer exists for and a person
// still repaired twice says the healer is not enough — its reach or its budget — and is said so.
func watchHandActs(f *signalFacts) []conditions.Observation {
recent := make([]link.HandAct, 0, len(f.handActs))
for _, a := range f.handActs {
if personsDecision[a.Cause] {
continue
}
if f.now.Sub(a.At) <= handActsWithin {
recent = append(recent, a)
}
}
repeated := link.RepeatedCauses(recent, f.now)
causes := make([]string, 0, len(repeated))
for c := range repeated {
causes = append(causes, c)
}
sort.Strings(causes)
var out []conditions.Observation
for _, cause := range causes {
var acts []string
var newest link.HandAct
for _, a := range recent {
if a.Cause != cause {
continue
}
acts = append(acts, fmt.Sprintf("%s %s by %s: %s", a.At.UTC().Format("2006-01-02 15:04"),
strings.TrimSpace(a.Verb+" "+strings.Join(a.Args, " ")), a.By, a.Why))
if a.At.After(newest.At) {
newest = a
}
}
wanted := "a healer is wanted for it"
if h := healerNamedFor(cause); h != "" {
wanted = fmt.Sprintf("healer %s answers this cause and a person still repaired it: its reach or its "+
"budget is not enough", h)
}
out = append(out, conditions.Observation{Scope: conditions.ScopeMesh, ID: "hand-acts." + cause,
Token: "healer-wanted", Kind: "healer-wanted", Severity: conditions.Warning,
Summary: fmt.Sprintf("%q was repaired by hand %d times in %d days, the last by %s: %s", cause,
repeated[cause], int(handActsWithin.Hours()/24), newest.By, wanted),
Said: strings.Join(acts, "; ")})
}
return out
}
// watchedRows are the rows a watchdog runs for.
func watchedRows() []signalRow {
var out []signalRow
for _, r := range signalsTable {
if r.watch != nil {
out = append(out, r)
}
}
return out
}
// kindsOf is a row's condition kinds, one or several.
func kindsOf(r signalRow) []string {
var out []string
for _, k := range strings.Split(r.Kind, ",") {
out = append(out, strings.TrimSpace(k))
}
return out
}
+392
View File
@@ -0,0 +1,392 @@
package main
import (
"context"
"errors"
"slices"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// The test generated from the signals table (novox/hq to-be 45 §3, ADR 0227 rule 5, "how it is
// checked"): **every row is walked**. A watched row's signal is suppressed just inside its bound —
// nothing raised — and just past it — its condition raised, with its kind and severity — and restored,
// and the condition clears. A row that is not watched says why. A row added to the table without a
// suppression here fails, so the table cannot grow a watchdog nobody has seen fire.
// calm is a mesh whose every signal is fresh: one control node heard ten seconds ago, its last send
// reported, a plan a minute into its tier, the loop taking, the merges read, nothing asked, no call
// running, the self-check a minute old.
func calm(now time.Time) *signalFacts {
return &signalFacts{now: now, started: now.Add(-time.Hour), host: "anchor", toolsHeardFrom: now.Add(-time.Hour),
machines: []machineFacts{{name: "anchor", control: true, lastHeard: now.Add(-10 * time.Second),
every: time.Minute, sentAt: now.Add(-time.Hour), reportedCurrent: true, reportedAt: now.Add(-59 * time.Minute),
lastApply: 20 * time.Second, tools: true, toolsHeard: now.Add(-10 * time.Second), toolsEvery: time.Minute}},
plans: []planFacts{{id: "plan-1", repository: "novox/app", commit: "c0ffee00", tier: 0, tiers: 2,
entered: now.Add(-time.Minute), bound: 30 * time.Minute, waiting: "building"}},
loop: loopFacts{took: now.Add(-time.Second), pending: 1},
mergesPassed: now.Add(-time.Minute),
selfCheck: selfCheckFacts{last: now.Add(-time.Minute), every: 5 * time.Minute},
lostConsumers: map[string]bool{}, epochs: map[int64]inventory.Epoch{},
lease: leaseFacts{held: true, epoch: 57, renewed: now.Add(-2 * time.Second)},
}
}
// refusedBy is n refusals of one writer, the last a moment ago.
func refusedBy(now time.Time, epoch int64, n int) []link.WriterRefusals {
return []link.WriterRefusals{{Writer: link.WriterEpoch(epoch), Epoch: epoch, Count: n,
Receivers: []string{"anchor", "laptop"}, Last: now.Add(-time.Second)}}
}
// suppression is one row's signal held back: inside its bound, and past it.
type suppression struct{ inside, past func(f *signalFacts) }
// suppressions are every watched row's, by row.
var suppressions = map[string]suppression{
"S1": {
inside: func(f *signalFacts) { f.machines[0].lastHeard = f.now.Add(-3*time.Minute + time.Second) },
past: func(f *signalFacts) { f.machines[0].lastHeard = f.now.Add(-3*time.Minute - time.Second) },
},
"S2": {
inside: func(f *signalFacts) {
f.machines[0].sentAt, f.machines[0].reportedCurrent = f.now.Add(-110*time.Second), false
},
past: func(f *signalFacts) {
f.machines[0].sentAt, f.machines[0].reportedCurrent = f.now.Add(-121*time.Second), false
},
},
"S3": {
inside: func(f *signalFacts) { f.plans[0].entered = f.now.Add(-29 * time.Minute) },
past: func(f *signalFacts) { f.plans[0].entered = f.now.Add(-31 * time.Minute) },
},
"S4": {
inside: func(f *signalFacts) { f.loop.took = f.now.Add(-119 * time.Second) },
past: func(f *signalFacts) { f.loop.took = f.now.Add(-121 * time.Second) },
},
"S5": {
inside: func(f *signalFacts) {},
past: func(f *signalFacts) {
f.merges = []missedMerge{{Owner: "novox", Repo: "app", Base: "main", Commit: "c0ffee0011", At: f.now.Add(-11 * time.Minute)}}
},
},
"S6": {
inside: func(f *signalFacts) {
f.asks = []askFacts{{id: "build-1", seat: "node-build-agent", what: "novox/app", state: link.AskInFlight,
on: "anchor", since: f.now.Add(-59 * time.Minute), bound: time.Hour}}
},
past: func(f *signalFacts) {
f.asks = []askFacts{{id: "build-1", seat: "node-build-agent", what: "novox/app", state: link.AskInFlight,
on: "anchor", since: f.now.Add(-61 * time.Minute), bound: time.Hour}}
},
},
"S7": {
inside: func(f *signalFacts) {
f.calls = []link.Call{{ID: "call-1", Seat: "mesh-controller", Verb: "push", Started: f.now.Add(-29 * time.Minute)}}
},
past: func(f *signalFacts) {
f.calls = []link.Call{{ID: "call-1", Seat: "mesh-controller", Verb: "push", Started: f.now.Add(-31 * time.Minute)}}
},
},
"S8": {
inside: func(f *signalFacts) { f.standings = []conditions.Condition{standingSaid(f.now.Add(-29 * time.Minute))} },
past: func(f *signalFacts) { f.standings = []conditions.Condition{standingSaid(f.now.Add(-31 * time.Minute))} },
},
"S9": {
inside: func(f *signalFacts) {
f.advisories = []link.Advisory{{Kind: link.AdvisoryMaxDeliveries, ID: "EVENTS.anchor_shop", Said: "gave up",
First: f.now.Add(-2 * time.Hour), Last: f.now.Add(-61 * time.Minute), Count: 1}}
},
past: func(f *signalFacts) {
f.advisories = []link.Advisory{{Kind: link.AdvisoryMaxDeliveries, ID: "EVENTS.anchor_shop", Said: "gave up",
First: f.now.Add(-2 * time.Hour), Last: f.now.Add(-59 * time.Minute), Count: 1}}
},
},
"S10": {
inside: func(f *signalFacts) { f.selfCheck.last = f.now.Add(-9 * time.Minute) },
past: func(f *signalFacts) { f.selfCheck.last = f.now.Add(-11 * time.Minute) },
},
"S11": {
inside: func(f *signalFacts) { f.machines[0].toolsHeard = f.now.Add(-179 * time.Second) },
past: func(f *signalFacts) { f.machines[0].toolsHeard = f.now.Add(-181 * time.Second) },
},
"S12": {
inside: func(f *signalFacts) { f.lease.renewed = f.now.Add(-15 * time.Second) },
past: func(f *signalFacts) { f.lease.renewed = f.now.Add(-16 * time.Second) },
},
"S13": {
inside: func(f *signalFacts) { f.staleRefusals = refusedBy(f.now, 41, 5) },
past: func(f *signalFacts) { f.staleRefusals = refusedBy(f.now, 41, 6) },
},
// Twice by hand within a fortnight is a healer wanted; once, or the first of two a day too old, is not.
"S15": {
inside: func(f *signalFacts) {
f.handActs = []link.HandAct{actByHand(f.now.Add(-15*24*time.Hour), "consumer-behind"),
actByHand(f.now.Add(-time.Hour), "consumer-behind")}
},
past: func(f *signalFacts) {
f.handActs = []link.HandAct{actByHand(f.now.Add(-13*24*time.Hour), "consumer-behind"),
actByHand(f.now.Add(-time.Hour), "consumer-behind")}
},
},
}
// actByHand is one entry in the hand-act log, with its cause.
func actByHand(at time.Time, cause string) link.HandAct {
return link.HandAct{ID: "act-" + at.Format("150405"), At: at, By: "jochen at a shell on the laptop",
Verb: "broker consumer-reset", Args: []string{"EVENTS", "controller"}, Why: "it replayed a week", Cause: cause}
}
// **S15 names the cause and, where a healer answers it, that the healer was not enough.**
func TestARepeatedHandActNamesItsCauseAndItsHealer(t *testing.T) {
now := time.Date(2026, 10, 6, 12, 0, 0, 0, time.UTC)
f := calm(now)
f.handActs = []link.HandAct{actByHand(now.Add(-2*time.Hour), "consumer-behind"),
actByHand(now.Add(-time.Hour), "consumer-behind"), actByHand(now.Add(-time.Hour), "restarted the proxy"),
actByHand(now.Add(-time.Minute), "restarted the proxy")}
got := watchHandActs(f)
if len(got) != 2 {
t.Fatalf("%+v", got)
}
if got[0].Key() != "mesh.hand-acts.consumer-behind.healer-wanted" || !strings.Contains(got[0].Summary, "healer H4") {
t.Errorf("a cause a healer answers: %s — %s", got[0].Key(), got[0].Summary)
}
if got[1].Key() != "mesh.hand-acts.restarted_the_proxy.healer-wanted" ||
!strings.Contains(got[1].Summary, "a healer is wanted") {
t.Errorf("a cause no healer answers: %s — %s", got[1].Key(), got[1].Summary)
}
}
// **Approving a retirement and deleting what was retired are a person's decision by design** (ADR
// 0230): repeated, they are not a healer wanted.
func TestARetirementDecisionRepeatedWantsNoHealer(t *testing.T) {
now := time.Date(2026, 10, 6, 12, 0, 0, 0, time.UTC)
f := calm(now)
f.handActs = []link.HandAct{actByHand(now.Add(-2*time.Hour), kindRetireWaiting),
actByHand(now.Add(-time.Hour), kindRetireWaiting), actByHand(now.Add(-time.Hour), kindCleanupWaiting),
actByHand(now.Add(-time.Minute), kindCleanupWaiting)}
if got := watchHandActs(f); len(got) != 0 {
t.Fatalf("a person's decision asked for a healer: %+v", got)
}
}
// standingSaid is a provider's failing word last said at a moment.
func standingSaid(at time.Time) conditions.Condition {
return conditions.Condition{Key: "provider.idp.anchor.app.failing", Kind: kindProviderFailing, LastObserved: at}
}
func TestEveryRowOfTheSignalsTableIsWatchedRaisedAndCleared(t *testing.T) {
now := time.Date(2026, 10, 6, 12, 0, 0, 0, time.UTC)
seen := map[string]bool{}
for _, row := range signalsTable {
t.Run(row.Row, func(t *testing.T) {
if seen[row.Row] {
t.Fatalf("%s is in the table twice", row.Row)
}
seen[row.Row] = true
if row.Signal == "" || row.Emitter == "" || row.Trigger == "" || row.Bound == "" || row.Kind == "" ||
(row.Severity != conditions.Urgent && row.Severity != conditions.Warning) {
t.Fatalf("%s does not say what it expects, from whom, within what, and what it raises: %+v", row.Row, row)
}
if row.Deferred != "" {
if row.watch != nil || row.Phase <= 1 {
t.Fatalf("%s is deferred and watched, or deferred out of Phase 1's own rows: %+v", row.Row, row)
}
if _, has := suppressions[row.Row]; has {
t.Fatalf("%s is deferred and has a suppression: one of the two is stale", row.Row)
}
return
}
if row.watch == nil || row.needs == nil || row.newest == nil {
t.Fatalf("%s is watched and lacks its watch, its needs or its newest", row.Row)
}
s, ok := suppressions[row.Row]
if !ok {
t.Fatalf("%s has no suppression in this test: a watchdog nobody has seen fire", row.Row)
}
if got := row.watch(calm(now)); len(got) != 0 {
t.Fatalf("%s raised on a calm mesh: %+v", row.Row, got)
}
inside := calm(now)
s.inside(inside)
if got := row.watch(inside); len(got) != 0 {
t.Fatalf("%s raised inside its bound: %+v", row.Row, got)
}
past := calm(now)
s.past(past)
got := row.watch(past)
if len(got) == 0 {
t.Fatalf("%s raised nothing past its bound", row.Row)
}
for _, o := range got {
if !slices.Contains(kindsOf(row), o.Kind) {
t.Errorf("%s raised %q, which is not its kind %q", row.Row, o.Kind, row.Kind)
}
if o.Severity != row.Severity {
t.Errorf("%s raised %s, the table says %s", row.Row, o.Severity, row.Severity)
}
if strings.TrimSpace(o.Summary) == "" {
t.Errorf("%s raised a condition that says nothing", row.Row)
}
}
// Through the store: raised past the bound, cleared when the signal returns.
store := conditions.NewInMemory()
told := &conditions.Told{}
k := conditions.NewKeeper(t.Context(), conditions.Options{Store: store, History: store, Teller: told,
Now: func() time.Time { return now }})
defer k.Close(context.Background())
w := &watchdogs{keeper: k, started: now.Add(-time.Hour)}
w.see(t.Context(), past)
open, err := k.Open(t.Context())
if err != nil || len(open) != len(got) {
t.Fatalf("%s past its bound left %d open (%v), want %d", row.Row, len(open), err, len(got))
}
w.see(t.Context(), calm(now))
if open, _ := k.Open(t.Context()); len(open) != 0 {
t.Fatalf("%s's condition stayed open after the signal returned: %+v", row.Row, open)
}
})
}
for name := range suppressions {
if !seen[name] {
t.Errorf("a suppression for %s, which the table does not have", name)
}
}
}
// **The control node silent for half an hour is urgent** (S1); any other machine stays a warning.
func TestTheControlNodeSilentIsUrgentAfterHalfAnHour(t *testing.T) {
now := time.Now()
f := calm(now)
f.machines[0].lastHeard = now.Add(-31 * time.Minute)
f.machines = append(f.machines, machineFacts{name: "laptop", lastHeard: now.Add(-31 * time.Minute)})
got := watchHeartbeats(f)
if len(got) != 2 || got[0].Severity != conditions.Urgent || got[1].Severity != conditions.Warning {
t.Fatalf("%+v", got)
}
}
// **A machine that said it sleeps is not silent** (ADR 0211), nor late to report; one that woke is.
func TestAMachineThatSaidItSleepsIsNotSilent(t *testing.T) {
now := time.Now()
f := calm(now)
f.machines[0].lastHeard = now.Add(-2 * time.Hour)
f.machines[0].sentAt, f.machines[0].reportedCurrent = now.Add(-time.Hour), false
f.machines[0].power = link.PowerState{State: "sleeping", At: now.Add(-2 * time.Hour)}
if got := append(watchHeartbeats(f), append(watchReports(f), watchTools(f)...)...); len(got) != 0 {
t.Fatalf("a sleeping machine raised %+v", got)
}
f.machines[0].power = link.PowerState{State: "woke", At: now.Add(-time.Hour)}
if got := watchHeartbeats(f); len(got) != 1 {
t.Fatalf("a woken machine silent past its bound raised %+v", got)
}
}
// **A watchdog that cannot see says so, and clears nothing it raised** (ADR 0227 rule 4): the store
// unreadable is a probe-failed of its own, and the machine's silence stays open until it can see again.
func TestABlindWatchdogSaysSoAndClearsNothing(t *testing.T) {
now := time.Now()
store := conditions.NewInMemory()
k := conditions.NewKeeper(t.Context(), conditions.Options{Store: store, History: store})
defer k.Close(context.Background())
w := &watchdogs{keeper: k, started: now.Add(-time.Hour)}
silent := calm(now)
silent.machines[0].lastHeard = now.Add(-10 * time.Minute)
w.see(t.Context(), silent)
blind := calm(now)
blind.machines, blind.machinesErr = nil, errors.New("the store is away")
w.see(t.Context(), blind)
open, err := k.Open(t.Context())
if err != nil {
t.Fatal(err)
}
var keys []string
for _, c := range open {
keys = append(keys, c.Key)
}
for _, want := range []string{"machine.anchor.silent", "probe.S1.failed", "probe.S2.failed", "probe.S11.failed"} {
if !slices.Contains(keys, want) {
t.Errorf("%s is not open while the machines cannot be read: %v", want, keys)
}
}
w.see(t.Context(), calm(now))
if open, _ := k.Open(t.Context()); len(open) != 0 {
t.Fatalf("seeing again left open %+v", open)
}
}
// **A controller standing by sees nothing and says nothing**: it hears no heartbeat, and would call
// every machine silent.
func TestAControllerStandingBySaysNothing(t *testing.T) {
store := conditions.NewInMemory()
k := conditions.NewKeeper(t.Context(), conditions.Options{Store: store, History: store})
defer k.Close(context.Background())
w := &watchdogs{keeper: k, started: time.Now(), acting: func() bool { return false }}
w.tick(t.Context())
if w.lastTick().IsZero() {
t.Fatal("a tick standing by was not counted")
}
if open, _ := k.Open(t.Context()); len(open) != 0 {
t.Fatalf("%+v", open)
}
}
// **A stale writer is named** (novox/hq to-be 45 §3, S13): by the controller instance that held the epoch
// its refused declarations claimed, and how that epoch ended — the question issue 204 could not answer.
func TestAStaleWriterIsNamedByItsEpoch(t *testing.T) {
now := time.Now()
f := calm(now)
ended := now.Add(-time.Minute)
f.staleRefusals = refusedBy(now, 41, 6)
f.epochs[41] = inventory.Epoch{Epoch: 41, Instance: "controller@anchor pid 7 since 2026-10-06T10:00:00Z",
Host: "anchor", Ended: &ended, How: inventory.EpochExpired}
got := watchStaleRefusals(f)
if len(got) != 1 || got[0].Key() != "core.controller.epoch-41.stale-writer" ||
!strings.Contains(got[0].Summary, "pid 7") || !strings.Contains(got[0].Summary, "expired") ||
got[0].Machine != "anchor" || !slices.Equal(got[0].Also, []string{"laptop"}) {
t.Fatalf("the stale writer is not named: %+v", got)
}
// An account the controller refused names the machine whose node-engine sent it.
f.staleRefusals = []link.WriterRefusals{{Writer: link.WriterNodeEngine("laptop"), Count: 6,
Receivers: []string{"controller"}, Last: now}}
got = watchStaleRefusals(f)
if len(got) != 1 || got[0].Key() != "machine.laptop.stale-writer" || got[0].Machine != "laptop" {
t.Fatalf("a node-engine sending older accounts is not named: %+v", got)
}
}
// **The lease lost is said for an hour, and serving without it for as long as it lasts** (S12).
func TestALeaseLostOrMissingIsSaid(t *testing.T) {
now := time.Now()
f := calm(now)
expired := now.Add(-59 * time.Minute)
f.lease.ended = []inventory.Epoch{{Epoch: 41, Instance: "controller@anchor pid 7", Host: "anchor",
Ended: &expired, How: inventory.EpochExpired}}
got := watchLease(f)
if len(got) != 1 || got[0].Key() != "core.controller.lease.lost" || !strings.Contains(got[0].Summary, "epoch 41") ||
got[0].Severity != conditions.Urgent {
t.Fatalf("a holder that stopped renewing was not said: %+v", got)
}
long := now.Add(-61 * time.Minute)
f.lease.ended[0].Ended = &long
if got := watchLease(f); len(got) != 0 {
t.Fatalf("a loss an hour old is still said: %+v", got)
}
f.lease.ended[0].How, f.lease.ended[0].Ended = inventory.EpochReleased, &expired
if got := watchLease(f); len(got) != 0 {
t.Fatalf("a lease given back is said as lost: %+v", got)
}
f.lease = leaseFacts{held: true, epoch: 501, renewed: now, reset: now.Add(-time.Minute), resetSaid: "moved past 500"}
if got := watchLease(f); len(got) != 1 || got[0].Key() != "core.controller.lease.reset" {
t.Fatalf("a lease bucket raised again from nothing was not said: %+v", got)
}
f.lease = leaseFacts{unleased: "the bus refused the key"}
if got := watchLease(f); len(got) != 1 || got[0].Key() != "core.controller.lease.unleased" {
t.Fatalf("serving without the lease was not said: %+v", got)
}
}
+13 -3
View File
@@ -3,6 +3,7 @@ package main
import (
"context"
"fmt"
"os"
"strconv"
"strings"
@@ -134,19 +135,28 @@ func seatBase(world catalogue.World, seatName string) (string, error) {
// seatBases is the clone base of every seat a recipe's context may name, for a build request
// (novox/hq ADR 0155). A seat nobody holds is left out rather than refused here: the build may not
// name it at all, and if it does the builder refuses with the seat's name.
//
// **What cannot be read is said, not passed over as no seats** (novox/hq to-be 45 Phase 2, the
// empty-on-error lint): the build still goes ahead — one that names no seat needs none — and one that
// does is refused naming it, but the reason is the store, and that is said here where it is known.
func seatBases(ctx context.Context) map[string]string {
unread := func(what string, err error) map[string]string {
fmt.Fprintf(os.Stderr, "could not read %s, so a build naming a seat's clone base will be told that "+
"seat is not held: %v\n", what, err)
return nil
}
open, err := openStores(ctx)
if err != nil {
return nil
return unread("the mesh's store", err)
}
defer open.Close()
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
return nil
return unread("the catalogue", err)
}
world, err := theRestOfTheMesh(ctx, open.inventory, shelf, "")
if err != nil {
return nil
return unread("who holds which seat", err)
}
bases := map[string]string{}
for _, seatName := range []string{gitSeat} {
+189
View File
@@ -0,0 +1,189 @@
package main
import (
"context"
"errors"
"fmt"
"strings"
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A provider that keeps failing a consumer is a problem the controller reports (novox/hq ADR 0224) —
// **the condition store's first kind** (to-be 45 §2, ADR 0227).
//
// On 2026-10-05 the identity provider's provisioner failed every consumer from shortly after midnight
// until it was fixed by hand that night — 31,000 refused logins after its database was moved and its
// admin kept an older password — and `status` called the mesh well all day (novox/hq issue 179). A
// provider announces a consumer it has failed for minutes; the controller keeps it until the provider
// says it recovered; and `status`, its JSON and `node show` name it, breaking "all well". Unchanged in
// what it says and when; kept as a condition, `provider.<module>.<node>.<consumer>.failing`, rather
// than a row of its own, so it is said outward like every other fault and silenced like one.
// Kinds of the provider standing.
const (
kindProviderFailing = "provider-failing"
kindProviderSilent = "provider-silent"
// sourceProvisioner is what raised a standing: the provider's own event.
sourceProvisioner = "provisioner.failing"
)
// providerSaysAgainWithin is how long a failing word stays current without being said again: twice
// the quarter of an hour a provider repeats it at (ADR 0224). Past it, S8.
const providerSaysAgainWithin = 30 * time.Minute
// standings keeps what providers say, as conditions.
type standings struct {
keeper func() *conditions.Keeper
}
// standingObservation is a provider's failing word as an observation: the provider, its machine and
// the consumer name it, so the same consumer failed again is the same condition.
func standingObservation(st link.Standing) conditions.Observation {
whom := st.Consumer
if st.Node != "" {
whom += " on " + st.Node
}
summary := fmt.Sprintf("%s on %s keeps failing %s: %s, %d attempt(s) since %s", st.Module, st.ProviderNode,
whom, orUnclassed(st.Class), st.Attempts, st.Since.UTC().Format("2006-01-02 15:04 MST"))
said := orUnclassed(st.Class)
if e := firstLine(st.Error); e != "" {
said += ": " + e
}
if st.Provider != "" {
said += fmt.Sprintf(" (provision %s, %d attempts)", st.Provider, st.Attempts)
}
return conditions.Observation{Scope: conditions.ScopeProvider,
ID: st.Module + "." + st.ProviderNode + "." + st.Consumer,
Token: "failing", Kind: kindProviderFailing, Machine: st.ProviderNode, Also: alsoOn(st.Node, st.ProviderNode),
Severity: conditions.Warning, Summary: summary, Said: said, Source: sourceProvisioner}
}
// Stood keeps a provider's newest word: failing raises or observes its condition, recovered clears
// it. An error is the store away, and the link holds the message to be asked again — a recovery is
// said once, and dropping it would leave a consumer named failing that is fine.
func (s standings) Stood(ctx context.Context, st link.Standing) (bool, error) {
k := s.keeper()
if k == nil {
return false, fmt.Errorf("the condition store is not open in this controller: %w", link.ErrTryAgain)
}
o := standingObservation(st)
if !st.Failing {
why := "the provider says it recovered"
if st.Why != "" {
why += ": " + st.Why
}
cleared, err := k.Clear(ctx, o.Key(), why)
return cleared, storeAway(err)
}
_, err := k.Observe(ctx, o)
return false, storeAway(err)
}
// storeAway reads the condition store failing as the bus being away for the moment: the link holds the
// message and asks again, as it does for a store restarting (ADR 0083), rather than taking it unkept.
func storeAway(err error) error {
if err == nil {
return nil
}
return fmt.Errorf("%v: %w", err, link.ErrTryAgain)
}
// providerStandings is every open provider-failing condition, from what is open.
func providerStandings(open []conditions.Condition) []conditions.Condition {
var out []conditions.Condition
for _, c := range open {
if c.Kind == kindProviderFailing {
out = append(out, c)
}
}
return out
}
// providerConditions is every open condition a provider's word raised: a consumer failing (ADR 0224),
// and a retirement waiting for a person, kept by a rejection, or cleanup waiting (ADR 0230).
func providerConditions(open []conditions.Condition) []conditions.Condition {
var out []conditions.Condition
for _, c := range open {
switch c.Kind {
case kindProviderFailing, kindRetireWaiting, kindRetireRejected, kindCleanupWaiting:
out = append(out, c)
}
}
return out
}
// providerOf reads a provider condition's module and machine back from its key: a standing's
// `provider.<module>.<node>.<consumer>.failing`, and a retirement's `provider.<module>.<node>.<token>`,
// which names no consumer.
func providerOf(c conditions.Condition) (module, node, consumer string, ok bool) {
parts := strings.Split(c.Key, ".")
if parts[0] != conditions.ScopeProvider {
return "", "", "", false
}
switch len(parts) {
case 5:
return parts[1], parts[2], parts[3], true
case 4:
return parts[1], parts[2], "", true
}
return "", "", "", false
}
// unassignedProviders clears the standing of every provider no longer assigned where it ran — and its
// retirement and cleanup conditions with it (ADR 0230): nothing runs there to retire or delete anything.
//
// **A provider no longer assigned is not asked about** (ADR 0224 §4): nothing runs there to fail
// anybody, and nothing there will ever say it recovered. The observation that resolves it is the
// assignment. Assigned again, its first failure raises it again.
func unassignedProviders(ctx context.Context, inv *inventory.Inventory, k *conditions.Keeper,
open []conditions.Condition) error {
assigned := map[string]map[string]bool{}
for _, c := range providerConditions(open) {
module, node, _, ok := providerOf(c)
if !ok {
continue
}
on, asked := assigned[node]
if !asked {
modules, err := inv.Assigned(ctx, node)
if err != nil {
if errors.Is(err, inventory.ErrNoSuchNode) {
modules = nil // a machine the mesh no longer knows runs nothing
} else {
return fmt.Errorf("what %s is assigned cannot be read: %w", node, err)
}
}
on = map[string]bool{}
for _, m := range modules {
on[m] = true
}
assigned[node] = on
}
if !on[module] {
if _, err := k.Clear(ctx, c.Key, module+" is no longer assigned to "+node+
": nothing runs there to fail anybody"); err != nil {
return err
}
}
}
return nil
}
func orUnclassed(class string) string {
if class == "" {
return "failing"
}
return class
}
// alsoOn is a consumer's machine, when it is not the provider's.
func alsoOn(consumerNode, providerNode string) []string {
if consumerNode == "" || consumerNode == providerNode {
return nil
}
return []string{consumerNode}
}
+118
View File
@@ -0,0 +1,118 @@
package main
import (
"encoding/json"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
)
// A provider that keeps failing a consumer is a problem `status` names (novox/hq ADR 0224), kept as
// the condition store's first kind (to-be 45 §2). On 2026-10-05 the identity provider refused every
// consumer for a day and status called the mesh well (04-ISSUES/179): this is that day, told to the
// controller the way the provider now tells it.
func TestAProviderFailingAConsumerBreaksAllWellUntilItRecovers(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, catalogue.Manifest{Module: "idp", Version: "1",
Receives: map[string]string{"oidc-client": "/var/lib/mesh/idp/mesh.json"}})
if _, err := assign(ctx, open, "anchor", "idp"); err != nil {
t.Fatal(err)
}
kept := standings{keeper: func() *conditions.Keeper { return conditionsFrom }}
since := time.Now().Add(-23 * time.Hour)
failing := link.Standing{Module: "idp", Failing: true, Provider: "oidc-client", ProviderNode: "anchor",
Consumer: "mesh_laptop_dashboard", Node: "laptop", Class: "credentials-rejected",
Error: `Keycloak token request failed: 401 {"error":"invalid_grant"}`, Since: since, Attempts: 31000}
if _, err := kept.Stood(ctx, failing); err != nil {
t.Fatal(err)
}
asked, err := theThreeQuestions(ctx, open)
if err != nil {
t.Fatal(err)
}
if asked.well() {
t.Fatal("a mesh whose identity provider fails a consumer reads as well")
}
said := printed(t, func() error { return printStatus(asked) })
for _, want := range []string{"1 open condition(s)", "provider.idp.anchor.mesh_laptop_dashboard.failing",
"idp on anchor keeps failing mesh_laptop_dashboard on laptop", "credentials-rejected", "31000 attempt(s)"} {
if !strings.Contains(said, want) {
t.Fatalf("status does not say %q:\n%s", want, said)
}
}
if strings.Contains(said, "all doing what they were told") {
t.Fatalf("status said all well beside a failing provider:\n%s", said)
}
if !strings.HasPrefix(said, "1 open condition(s)") {
t.Fatalf("status does not lead with what is open:\n%s", said)
}
body, err := statusAsJSON(asked)
if err != nil {
t.Fatal(err)
}
var doc struct {
Conditions []conditions.Condition `json:"conditions"`
Failing []conditions.Condition `json:"failing"`
}
if err := json.Unmarshal(body, &doc); err != nil || len(doc.Failing) != 1 || len(doc.Conditions) != 1 ||
!strings.Contains(doc.Failing[0].Evidence[0].Said, "invalid_grant") {
t.Fatalf("the document does not carry it: %v\n%s", err, body)
}
// Both machines' `node show` name it: where the provider runs, and where the consumer is.
for _, node := range []string{"anchor", "laptop"} {
shown := printed(t, func() error { return showNode(ctx, open.inventory, node) })
if !strings.Contains(shown, "open condition(s) about this machine") || !strings.Contains(shown, "mesh_laptop_dashboard") {
t.Fatalf("node show %s does not name it:\n%s", node, shown)
}
}
// Recovered: gone, and the mesh may be well again as far as this is concerned.
failing.Failing = false
if cleared, err := kept.Stood(ctx, failing); err != nil || !cleared {
t.Fatalf("%v %v", cleared, err)
}
asked, err = theThreeQuestions(ctx, open)
if err != nil {
t.Fatal(err)
}
if len(asked.conditions) != 0 {
t.Fatalf("a recovered consumer is still named: %+v", asked.conditions)
}
}
// A provider no longer assigned where it ran has nothing running to fail anybody: its last word is
// cleared on the next look, said as resolved by the assignment.
func TestAnUnassignedProvidersLastWordIsCleared(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
kept := standings{keeper: func() *conditions.Keeper { return conditionsFrom }}
if _, err := kept.Stood(ctx, link.Standing{Module: "gone", Failing: true,
ProviderNode: "anchor", Consumer: "x", Since: time.Now()}); err != nil {
t.Fatal(err)
}
all, err := conditionsFrom.Open(ctx)
if err != nil || len(all) != 1 {
t.Fatalf("%+v %v", all, err)
}
if err := unassignedProviders(ctx, open.inventory, conditionsFrom, all); err != nil {
t.Fatal(err)
}
if all, _ := conditionsFrom.Open(ctx); len(all) != 0 {
t.Fatalf("%+v", all)
}
}
// **The store away holds a recovery** (ADR 0224 §3): a standing that cannot be kept is asked again.
func TestAStandingTheStoreCannotKeepIsAskedAgain(t *testing.T) {
kept := standings{keeper: func() *conditions.Keeper { return nil }}
if _, err := kept.Stood(t.Context(), link.Standing{Module: "idp", ProviderNode: "anchor", Consumer: "x"}); err == nil ||
!strings.Contains(err.Error(), link.ErrTryAgain.Error()) {
t.Fatalf("answered %v", err)
}
}
+74 -6
View File
@@ -8,6 +8,7 @@ import (
"strings"
"time"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/overlay"
)
@@ -81,8 +82,12 @@ func printStatus(asked answers) error {
wrong, nodes, quiet := asked.wrong, asked.nodes, asked.quiet
behind, sources := asked.behind, asked.sources
// **What is wrong leads** (novox/hq to-be 45 §2): every open condition, urgent first, oldest
// first, silenced ones with when their silence ends.
printConditions(asked.conditions, asked.conditionsUnread, time.Now())
if len(asked.refused) > 0 {
// First, above everything else. A machine that cannot be worked out is not running an old
// First of what follows. A machine that cannot be worked out is not running an old
// declaration — it has no declaration, and nothing below this line is about it.
var names []string
for name := range asked.refused {
@@ -298,6 +303,38 @@ func printStatus(asked answers) error {
fmt.Printf("\n `assign <node> <holder>` meets it; reported until every machine has its holders, then refused\n\n")
}
if len(asked.overflowing) > 0 {
// **Reported, and the provider still pushed** (novox/hq ADR 0225, issue 263). Each is left
// out of its provider's grants, so the module holds a login nothing created; the provider's
// machine is sent everything else rather than refused for one consumer elsewhere.
fmt.Printf("%d module(s) identified too long for a provision they require, and not granted it:\n",
len(asked.overflowing))
for _, o := range asked.overflowing {
fmt.Printf(" %-12s %-20s %-22s %q is %d, %s keeps %d\n", o.Consumer, o.Module, o.Provision,
o.Identity, len(o.Identity), o.Bound.In, o.Bound.Max)
}
fmt.Printf("\n a shorter `slug` in the module's definition fits it; `module check` refuses one before merge\n\n")
}
// Repairs done by hand this week (novox/hq to-be 45 §7). Not a fault, so it does not break "all
// well"; each is a healer the mesh does not have yet, and the count is how that is watched.
switch {
case asked.handActsUnread != "":
fmt.Printf("the hand-act log could not be read, so how much was done by hand this week is not known: %s\n\n",
asked.handActsUnread)
case asked.handActs != nil && *asked.handActs > 0:
fmt.Printf("%d act(s) done by hand in the last seven days — `hand-acts` lists them, and why\n\n", *asked.handActs)
}
// And what the mesh repaired by itself (novox/hq to-be 45 §7): seen, not only done.
switch {
case asked.healsUnread != "":
fmt.Printf("what the healers did could not be read: %s\n\n", asked.healsUnread)
case asked.heals != nil && (asked.heals.Acts > 0 || asked.heals.Escalated > 0):
fmt.Printf("%d repair(s) made by the healers in the last seven days, %d handed to the operator — `healers` "+
"lists them, and each is in its condition's tried\n\n", asked.heals.Acts, asked.heals.Escalated)
}
if adopted := adoptedNodes(nodes); len(adopted) > 0 {
// Said, because nothing forces the flip: a node left adopted is visible here rather than
// read as converged (novox/hq ADR 0100). Not a fault, so it does not break "all well".
@@ -307,8 +344,9 @@ func printStatus(asked answers) error {
if asked.well() {
// Said plainly. "Nothing to report" and "nothing was checked" must never look the same,
// and getting here means every question was asked and answered.
fmt.Printf("%d machine(s), all doing what they were told, all heard from, running what "+
// and getting here means every question was asked and answered. **No open conditions first**
// (novox/hq to-be 45 §2): it is what the sentence means now, silenced ones included.
fmt.Printf("no open conditions; %d machine(s), all doing what they were told, all heard from, running what "+
"the mesh would send them, and every module current with its source\n", len(nodes))
// **And what that sentence does not cover**, because for eleven hours it was true of a mesh
// in which no module could reach another (novox/hq 04-ISSUES/145). Every question above is
@@ -406,15 +444,27 @@ func theThreeQuestions(ctx context.Context, open *stores) (answers, error) {
// ADR 0207). Each machine resolved again rather than threaded through whoResolves, whose answer
// the private network is built from and should say nothing else; a machine that does not
// resolve is already in refused, and is passed over here.
plans := map[string]planned{}
for _, n := range out.nodes {
plan, _, err := planFor(ctx, open, n.Name)
plan, settings, err := planFor(ctx, open, n.Name)
if err != nil {
if unresolvable(err) {
continue
}
return answers{}, err
}
plans[n.Name] = planned{plan, settings}
out.unheld = append(out.unheld, plan.Unheld...)
// And which of its modules a provider leaves out of its grants, for an identity too long
// for what the provision keeps (novox/hq ADR 0225) — judged from the consumer's own
// resolution, as the provider's composition judges it.
out.overflowing = append(out.overflowing, plan.Overflowing()...)
}
// And every open condition (novox/hq to-be 45 §2): what the watchdogs, the self-check and the
// providers' own words say is wrong — a provider failing a consumer among them (ADR 0224). Kept on
// the bus; a process that cannot read them says so, and the mesh is then not called well.
if out.conditions, err = openConditions(ctx); err != nil {
out.conditionsUnread = err.Error()
}
out.plans, err = inv.RecentPlans(ctx, 5)
if err != nil {
@@ -422,6 +472,23 @@ func theThreeQuestions(ctx context.Context, open *stores) (answers, error) {
}
// Whether the build seat takes work, for a plan waiting on it (novox/hq ADR 0219).
out.paused = buildSeatPause(ctx, inv, out.plans)
// And how many repairs were done by hand this week (novox/hq to-be 45 §7) — where there is a bus
// to read the log from; a process with none has no log to count.
if _, onBus := broker.BusAddress(); onBus == nil {
n, unread := handActsThisWeek(ctx)
if unread != "" {
out.handActsUnread = unread
} else {
out.handActs = &n
}
}
// And what the healers did this week (novox/hq to-be 45 §7).
if heals, err := inv.HealsSince(ctx, time.Now().Add(-7*24*time.Hour)); err != nil {
out.healsUnread = err.Error()
} else {
out.heals = countHeals(heals)
}
// And which machines are not running what the mesh would send them. The same question as a
// module being behind its source, one level down: that one says the catalogue is out of date,
@@ -433,7 +500,7 @@ func theThreeQuestions(ctx context.Context, open *stores) (answers, error) {
// said nothing at all and `status --json` emitted prose to stderr and no JSON anywhere. The
// reason is kept and reported as data; every question that does not depend on it is still
// answered.
would, err := wouldSend(ctx, open, out.nodes)
would, err := wouldSendFrom(ctx, open, out.nodes, plans)
if err != nil {
out.network = err.Error()
would = map[string]string{}
@@ -528,7 +595,8 @@ func untakenModules(ctx context.Context, inv *inventory.Inventory, nodes []inven
func (a answers) well() bool {
return len(a.wrong) == 0 && len(a.quiet) == 0 && len(a.behind) == 0 &&
len(a.waiting) == 0 && len(a.refused) == 0 && a.network == "" && len(a.untaken) == 0 &&
len(a.filtered) == 0 && len(a.unheld) == 0
len(a.filtered) == 0 && len(a.unheld) == 0 && len(a.overflowing) == 0 &&
len(a.conditions) == 0 && a.conditionsUnread == ""
}
// hostSplit is which machines report which host version, for every version more than one machine
+208
View File
@@ -0,0 +1,208 @@
package main
import (
"context"
"encoding/json"
"fmt"
"sync"
"time"
"github.com/novox/mesh-controller/internal/link"
)
// `status` answered from a summary the serving controller keeps current (novox/hq to-be 45 Phase 0,
// §4's D9 and §8's health of the controller).
//
// **Asked, it was composed: every machine resolved, twice, while its caller waited.** On 2026-10-06
// the verb took eighteen seconds on a mesh of four machines, so its caller read "still running" and
// had to ask `calls` for the answer to "is the mesh alright" — the one question that must answer at
// once, and the one a self-check and a rollout gate will ask every few minutes. So the serving
// controller composes it in the background — at its start, after anything that changes what it says
// (a machine's report, a build, a verb that acts), and every minute regardless — and the verb answers
// the last composition at once, saying when it was composed and how long that took. A caller who
// needs it newer than that reads the time and asks again; nothing is answered as current that is not.
// statusEvery is how often the summary is composed with nothing having nudged it; statusSettle how
// long a nudge waits for the next, so a push answered by four machines is composed once.
var (
statusEvery = time.Minute
statusSettle = 2 * time.Second
// statusComposeWithin bounds one composition, so a store that hangs cannot stop the summary for
// good; the attempt is said as failed, and the last summary stands with its age.
statusComposeWithin = 2 * time.Minute
)
// statusSummary is the last composed `status --json`, and when.
type statusSummary struct {
compose func(context.Context) ([]byte, error)
// every, settle and within are the clocks above, read once when it is made.
every, settle, within time.Duration
mu sync.Mutex
body []byte
composedAt time.Time
took time.Duration
failed string
failedAt time.Time
started time.Time
first chan struct{} // closed when the first attempt ends, either way
nudged chan struct{}
}
func newStatusSummary(compose func(context.Context) ([]byte, error)) *statusSummary {
return &statusSummary{compose: compose, started: time.Now(), first: make(chan struct{}),
nudged: make(chan struct{}, 1), every: statusEvery, settle: statusSettle, within: statusComposeWithin}
}
// statusFrom is the serving controller's summary; nil in any other process, where `status` is
// composed when asked, as at a shell.
var statusFrom *statusSummary
// nudge asks for a composition soon. Never blocks: one pending is as good as many.
func (s *statusSummary) nudge() {
if s == nil {
return
}
select {
case s.nudged <- struct{}{}:
default:
}
}
// keep composes until ctx ends: now, on a nudge once things settle, and every statusEvery.
func (s *statusSummary) keep(ctx context.Context) {
once := sync.Once{}
for {
s.composeOnce(ctx)
once.Do(func() { close(s.first) })
timer := time.NewTimer(s.every)
select {
case <-ctx.Done():
timer.Stop()
return
case <-timer.C:
case <-s.nudged:
timer.Stop()
// Let what else is arriving arrive, then compose once for all of it.
select {
case <-ctx.Done():
return
case <-time.After(s.settle):
}
select {
case <-s.nudged:
default:
}
}
}
}
func (s *statusSummary) composeOnce(ctx context.Context) {
start := time.Now()
asking, cancel := context.WithTimeout(ctx, s.within)
body, err := s.compose(asking)
cancel()
took := time.Since(start)
s.mu.Lock()
defer s.mu.Unlock()
if err != nil {
s.failed, s.failedAt = err.Error(), time.Now()
fmt.Printf("status could not be composed (after %s): %v — `status` answers the last summary, "+
"with its age\n", took.Round(time.Millisecond), err)
return
}
s.body, s.composedAt, s.took, s.failed = body, start, took, ""
}
// answer is what the `status` verb answers: the last summary at once, the same document `status
// --json` prints, with when it was composed. Before the first composition has ended it waits for it,
// but never past the caller's window; a controller that has none says so and why, rather than
// answering an empty mesh as a well one.
func (s *statusSummary) answer(ctx context.Context) (any, error) {
wait := time.NewTimer(link.AnswerWithin - time.Second)
defer wait.Stop()
select {
case <-s.first:
case <-wait.C:
case <-ctx.Done():
}
s.mu.Lock()
defer s.mu.Unlock()
if s.body == nil {
why := "its first composition has not finished"
if s.failed != "" {
why = "it could not be composed: " + s.failed
}
return nil, fmt.Errorf("this controller started %s ago and has no status to answer yet — %s. "+
"Ask again shortly", time.Since(s.started).Round(time.Second), why)
}
var parsed any
_ = json.Unmarshal(s.body, &parsed)
out := map[string]any{
"output": string(s.body), "ok": true, "answer": parsed,
"composed": s.composedAt.UTC().Format(time.RFC3339),
"age": time.Since(s.composedAt).Round(time.Second).String(),
"composedIn": s.took.Round(time.Millisecond).String(),
"note": "composed by the serving controller at its start, after each report, build or act, and " +
"every minute; answered at once from the last composition",
}
if s.failed != "" && s.failedAt.After(s.composedAt) {
out["lastAttemptFailed"] = fmt.Sprintf("%s: %s — this summary is the last that could be composed",
s.failedAt.UTC().Format(time.RFC3339), s.failed)
}
return out, nil
}
// composeStatus is `status --json`, composed in this process against its stores.
func composeStatus(open *stores) func(context.Context) ([]byte, error) {
return func(ctx context.Context) ([]byte, error) {
asked, err := theThreeQuestions(ctx, open)
if err != nil {
return nil, err
}
return statusAsJSON(asked)
}
}
// readingVerbs are the verbs that only read; after any other, what `status` says may have changed, so the summary is
// composed again; a verb that only reads leaves it alone, or a console polling `nodes` would keep the
// controller composing for ever.
var readingVerbs = map[string]bool{
"tools": true, "calls": true, "status": true, "nodes": true, "node": true, "modules": true,
"seats": true, "builds": true, "plan": true, "queue": true, "durations": true, "hand-acts": true,
"doctor": true, "conditions": true, "healers": true,
}
// nudgingListener is the enrolment, nudging the summary when a machine said something new.
type nudgingListener struct {
link.Enrolment
summary *statusSummary
// open is the serving controller's stores, for replacing a given value after a module's first
// good start (novox/hq ADR 0228).
open *stores
}
func (l nudgingListener) Heard(ctx context.Context, report link.Report) (bool, error) {
// A declaration refused as older than the one the machine holds is counted by its writer — the
// controller epoch it claimed (novox/hq to-be 45 §6, S13) — and so is what the machine's own count
// says it refused beyond the refusals heard.
now := time.Now()
if report.StaleRefusalOf() {
link.StaleRefusals.Refused(link.Refusal{Writer: link.WriterEpoch(report.Epoch), Epoch: report.Epoch,
Receiver: report.Node, At: now})
}
if report.Ordered() {
link.StaleRefusals.Lifetime(report.Node, report.RefusedOlder, now)
}
news, err := l.Enrolment.Heard(ctx, report)
if news {
l.summary.nudge()
}
if err == nil && l.open != nil && startedWell(report) {
// Off the report's path: replacing a given value sends the machine, and a report waits for
// nothing it caused (novox/hq ADR 0228).
go replaceGiven(context.WithoutCancel(ctx), l.open, report)
}
return news, err
}
+152
View File
@@ -0,0 +1,152 @@
package main
import (
"context"
"encoding/json"
"errors"
"strings"
"sync/atomic"
"testing"
"time"
"github.com/novox/mesh-controller/internal/link"
)
// quickly shortens the summary's clocks for one test.
func quickly(t *testing.T) {
t.Helper()
every, settle := statusEvery, statusSettle
statusEvery, statusSettle = time.Hour, 10*time.Millisecond
t.Cleanup(func() { statusEvery, statusSettle = every, settle })
}
// **`status` answers in full within ten seconds, five times in a row** (novox/hq to-be 45 Phase 0,
// D9) — however long composing it takes. On 2026-10-06 composing took eighteen seconds and the
// verb answered "still running"; from the summary it answers at once, in full, saying when.
func TestStatusAnswersAtOnceHoweverLongComposingTakes(t *testing.T) {
quickly(t)
var composed atomic.Int32
slow := make(chan struct{})
s := newStatusSummary(func(ctx context.Context) ([]byte, error) {
if composed.Add(1) > 1 {
<-slow // every composition after the first outlasts any caller
}
return []byte(`{"wrong":[],"machines":4}`), nil
})
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
defer close(slow)
go s.keep(ctx)
for i := 0; i < 5; i++ {
s.nudge() // a composition is under way and does not finish
start := time.Now()
got, err := s.answer(ctx)
if err != nil {
t.Fatal(err)
}
if took := time.Since(start); took > time.Second {
t.Fatalf("answer %d took %s", i+1, took)
}
m := got.(map[string]any)
if m["answer"].(map[string]any)["machines"] != float64(4) || m["composed"] == "" || m["ok"] != true {
t.Fatalf("answer %d was not in full: %v", i+1, m)
}
}
}
// The first composition is waited for, never past the caller's window; a controller with none yet
// says so rather than answering an empty mesh as a well one.
func TestStatusBeforeItsFirstCompositionSaysSo(t *testing.T) {
quickly(t)
was := link.AnswerWithin
link.AnswerWithin = 1100 * time.Millisecond
t.Cleanup(func() { link.AnswerWithin = was })
never := make(chan struct{})
defer close(never)
s := newStatusSummary(func(context.Context) ([]byte, error) { <-never; return nil, nil })
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go s.keep(ctx)
start := time.Now()
_, err := s.answer(ctx)
if err == nil || !strings.Contains(err.Error(), "has no status to answer yet") {
t.Fatalf("answered %v", err)
}
if took := time.Since(start); took > link.AnswerWithin {
t.Fatalf("waited %s, past the caller's window", took)
}
}
// A nudge composes it again, once for several close together; a failed composition leaves the last
// summary standing and says it is the last that could be composed.
func TestANudgeComposesAgainAndAFailureKeepsTheLastSummary(t *testing.T) {
quickly(t)
var composed atomic.Int32
fail := atomic.Bool{}
s := newStatusSummary(func(context.Context) ([]byte, error) {
n := composed.Add(1)
if fail.Load() {
return nil, errors.New("the store did not answer")
}
body, _ := json.Marshal(map[string]any{"n": n})
return body, nil
})
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go s.keep(ctx)
if _, err := s.answer(ctx); err != nil {
t.Fatal(err)
}
s.nudge()
s.nudge()
s.nudge()
waitFor(t, func() bool { return composed.Load() == 2 })
time.Sleep(50 * time.Millisecond)
if n := composed.Load(); n != 2 {
t.Fatalf("three nudges together composed %d times after the first", n-1)
}
fail.Store(true)
s.nudge()
waitFor(t, func() bool { return composed.Load() == 3 })
waitFor(t, func() bool {
got, err := s.answer(ctx)
if err != nil {
t.Fatal(err)
}
m := got.(map[string]any)
return m["lastAttemptFailed"] != nil && m["answer"].(map[string]any)["n"] == float64(2)
})
}
// The seat's `status` answers from the summary when this process keeps one.
func TestTheStatusVerbAnswersFromTheSummary(t *testing.T) {
quickly(t)
s := newStatusSummary(func(context.Context) ([]byte, error) { return []byte(`{"from":"summary"}`), nil })
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go s.keep(ctx)
statusFrom = s
t.Cleanup(func() { statusFrom = nil })
handlers, _, err := seatToolHandlers()
if err != nil {
t.Fatal(err)
}
got, err := handlers["status"](ctx, json.RawMessage(`{}`))
if err != nil {
t.Fatal(err)
}
if got.(map[string]any)["answer"].(map[string]any)["from"] != "summary" {
t.Fatalf("answered %v", got)
}
}
func waitFor(t *testing.T, ok func() bool) {
t.Helper()
deadline := time.Now().Add(3 * time.Second)
for !ok() {
if time.Now().After(deadline) {
t.Fatal("never happened")
}
time.Sleep(5 * time.Millisecond)
}
}
+18
View File
@@ -73,6 +73,22 @@ func migrate(ctx context.Context) error {
}
fmt.Printf("provided %s\n", m.Module)
}
// And what an earlier release shipped and this one does not goes (novox/hq ADR 0226) — unless a
// machine still has it, which is said rather than overridden.
var shipped []string
for _, m := range provided {
shipped = append(shipped, m.Module)
}
retired, kept, err := inv.RetireUnshipped(ctx, shipped)
if err != nil {
return err
}
for _, name := range retired {
fmt.Printf("retired %s: the control plane no longer ships it\n", name)
}
for name, why := range kept {
fmt.Printf("kept %s, which the control plane no longer ships: %s\n", name, why)
}
// The seats the mesh ships with, into the table that now holds the set (novox/hq ADR 0122).
// Idempotent: fills an empty table on first boot, adds a seat a release ships, and leaves an
// operator's changes in the table as they are.
@@ -94,6 +110,8 @@ func openInventory(ctx context.Context) (*inventory.Inventory, error) {
inv.Close()
return nil, err
}
// A plan is written only under the lease, carrying its epoch (novox/hq to-be 45 §6).
inv.ActsUnder(theLease.epoch)
// Load the seat set from the store, so the control plane reads the set as data rather than as
// the slice it was compiled with (novox/hq ADR 0122). A store not yet seeded — or one whose
// seat table a migration has not reached — returns nothing, and UseSeats leaves the compiled
+10 -6
View File
@@ -72,24 +72,28 @@ func TestAPersonClosesAStuckPlan(t *testing.T) {
stuck := inventory.Plan{ID: "plan-97b1b2b", Repository: "novox/mesh-catalog", Commit: "97b1b2b",
Created: time.Now().UTC(), State: inventory.PlanRolling, Tier: 1, Tiers: [][]string{{"a"}, {"b"}},
Modules: map[string]*inventory.PlanModule{"a": {State: "built"}, "b": {}}}
if err := open.inventory.SavePlan(ctx, stuck); err != nil {
if err := open.inventory.SavePlan(ctx, &stuck); err != nil {
t.Fatal(err)
}
if err := plansCommand(ctx, []string{"close", stuck.ID}); err != nil {
if err := plansCommand(ctx, []string{"close", stuck.ID}); err == nil || !strings.Contains(err.Error(), "--why") {
t.Fatalf("a plan was closed by hand without saying why: %v", err)
}
if err := plansCommand(ctx, []string{"close", stuck.ID, "--why", "its report will not come"}); err != nil {
t.Fatal(err)
}
closed, err := open.inventory.PlanByID(ctx, stuck.ID)
if err != nil {
t.Fatal(err)
}
if closed.State != inventory.PlanFailed || !strings.Contains(closed.Note, "closed by hand") {
if closed.State != inventory.PlanFailed || !strings.Contains(closed.Note, "closed by hand") ||
!strings.Contains(closed.Note, "its report will not come") {
t.Fatalf("the plan was left %s: %q", closed.State, closed.Note)
}
if err := plansCommand(ctx, []string{"close", stuck.ID}); err == nil {
if err := plansCommand(ctx, []string{"close", stuck.ID, "--why", "again"}); err == nil {
t.Fatal("a plan already closed was closed again")
}
if argv, err := argvFor("plans", map[string]any{"close": stuck.ID}); err != nil ||
!reflect.DeepEqual(argv, []string{"plans", "close", stuck.ID}) {
if argv, err := argvFor("plans", map[string]any{"close": stuck.ID, "why": "w"}); err != nil ||
!reflect.DeepEqual(argv, []string{"plans", "close", stuck.ID, "--why", "w"}) {
t.Fatalf("the seat's verb does not close a plan: %v %v", argv, err)
}
}
+61 -31
View File
@@ -8,6 +8,7 @@ import (
"path"
"regexp"
"strings"
"sync"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
@@ -266,6 +267,11 @@ func notNow(err error) error {
// (novox/hq 04-ISSUES/131). Nothing is pushed here: what a finished build does to the machines
// running the module is the upgrade's decision, taken when the catalogue announces it.
func (f following) SourceMoved(ctx context.Context, m link.SourceMoved) error {
// One merge acted on at a time, whoever hands it over: the bus, or the catch-up that reads back
// what the bus did not hand over (novox/hq issue 266). Each judges against what the other wrote.
actingOnMerges.Lock()
defer actingOnMerges.Unlock()
inv := f.open.inventory
entries, err := inv.Catalogued(ctx)
if err != nil {
@@ -276,34 +282,7 @@ func (f following) SourceMoved(ctx context.Context, m link.SourceMoved) error {
return notNow(err)
}
// Two kinds of module are affected by one merge, and they are affected differently.
//
// A module **built from** this repository and branch has moved: the mesh records the new commit
// as what its source now has, and only what the merge actually changed is rebuilt. A module that
// only **packages source from** it has not moved — its own source is somewhere else, at the
// commit it already records — so it is rebuilt and its record left alone. Writing this commit as
// its source would make it permanently behind a repository its manifest does not come from.
var from, packaging []inventory.Entry
already := 0
for _, e := range entries {
switch {
case sourceIs(e.Source, m):
if e.Source.BuiltFrom == m.Commit {
already++
continue
}
// **A merge older than the last look at the source is history, not a move.** The forge
// announces what it finds merged, and an old merge surfacing late would otherwise move
// the recorded head backwards and rebuild everything built from that repository, once
// per old merge (2026-09-28).
if isHistory(m.MergedAt, e.Source.Seen) {
continue
}
from = append(from, e)
case readsFrom(read[e.Manifest.Module], m):
packaging = append(packaging, e)
}
}
from, packaging, already := mergeCandidates(m, entries, read)
if len(from) == 0 && len(packaging) == 0 {
// "Already built from it" and "nothing reads it" are different facts, and reading the first
// as the second sends somebody looking for a broken trigger when the mesh is up to date.
@@ -404,13 +383,13 @@ func (f following) SourceMoved(ctx context.Context, m link.SourceMoved) error {
fmt.Printf(" the last tier depends on itself: %s — built together, in no order\n",
strings.Join(plan.Tiers[len(plan.Tiers)-1], ", "))
}
if err := inv.SavePlan(ctx, plan); err != nil {
if err := inv.SavePlan(ctx, &plan); err != nil {
return notNow(err)
}
// Closed after the newer plan is kept, never before: a controller replaced between the two leaves
// both open, which the next merge settles, rather than neither.
for _, old := range superseded {
if err := inv.SavePlan(ctx, old); err != nil {
if err := inv.SavePlan(ctx, &old); err != nil {
return notNow(err)
}
fmt.Printf(" %s (%s at %s) is %s\n", old.ID, old.Repository, short(old.Commit), old.Note)
@@ -432,12 +411,63 @@ func (f following) SourceMoved(ctx context.Context, m link.SourceMoved) error {
if err := askTier(ctx, inv, &plan); err != nil {
return notNow(err)
}
if err := inv.SavePlan(ctx, plan); err != nil {
if err := inv.SavePlan(ctx, &plan); err != nil {
return notNow(err)
}
return nil
}
// actingOnMerges keeps one merge acted on at a time (novox/hq issue 266).
var actingOnMerges sync.Mutex
// mergeCandidates is what one merge could move, judged against what the catalogue holds.
//
// Two kinds of module are affected by one merge, and they are affected differently.
//
// A module **built from** this repository and branch has moved: the mesh records the new commit as
// what its source now has, and only what the merge actually changed is rebuilt. A module that only
// **packages source from** it has not moved — its own source is somewhere else, at the commit it
// already records — so it is rebuilt and its record left alone. Writing this commit as its source
// would make it permanently behind a repository its manifest does not come from. `already` counts
// the modules built from this repository that are already built from this very commit.
func mergeCandidates(m link.SourceMoved, entries []inventory.Entry,
read map[string][]inventory.ReadRepository) (from, packaging []inventory.Entry, already int) {
for _, e := range entries {
switch {
case sourceIs(e.Source, m):
if e.Source.BuiltFrom == m.Commit {
already++
continue
}
// **A merge older than the last look at the source is history, not a move.** The forge
// announces what it finds merged, and an old merge surfacing late would otherwise move
// the recorded head backwards and rebuild everything built from that repository, once
// per old merge (2026-09-28).
if isHistory(m.MergedAt, e.Source.Seen) {
continue
}
from = append(from, e)
case readsFrom(read[e.Manifest.Module], m):
packaging = append(packaging, e)
}
}
return from, packaging, already
}
// wouldMove is the modules built from the merged repository that acting on this merge would mark as
// moved and rebuild — SourceMoved's judgement, made without acting (novox/hq issue 266). Empty for a
// merge already acted on: acting marks each of them as looked at, so the merge then reads as history.
//
// **Only the modules built from it, never the ones that merely package source from it.** Acting
// records nothing about those, so a merge acted on would go on reading as unacted for them, and be
// acted on again on every look. A merge that moves both is caught by the first kind, and acting on it
// rebuilds the second as well.
func wouldMove(m link.SourceMoved, entries []inventory.Entry,
read map[string][]inventory.ReadRepository) []inventory.Entry {
from, _, _ := mergeCandidates(m, entries, read)
return whatTheMergeTouched(from, entries, m)
}
// sourceIs is whether a recorded source is the repository and branch a merge announced. A source on
// the git seat is recorded as its path on the forge; one elsewhere as the URL it was cloned from.
// An empty recorded ref is the repository's default branch, which is what a merge into the base
+620
View File
@@ -0,0 +1,620 @@
package main
import (
"context"
"errors"
"fmt"
"os"
"slices"
"sort"
"sync"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// The watchdogs (novox/hq to-be 45 §3): every row of the signals table, run over what the serving
// controller knows, every half minute.
//
// **One loop, one gathering, every row.** The facts are gathered once a tick — the store's machines,
// plans and durations, the bus's queue and consumers, what this process heard — and each row's watch
// is a pure reading of them, which is what lets the test generated from the table suppress a signal by
// changing a fact. A part that cannot be gathered makes the rows that read it blind: each says so as
// a condition of its own (`probe.<row>.failed`) and keeps what it raised before, because a watchdog
// that cannot see must not read as one that sees nothing wrong (ADR 0227 rule 4).
//
// **The watchdogs are themselves watched.** Each tick is recorded; the self-check (doctor.go) fails
// its probe DW when the ticks stop, and the watchdogs raise S10 when the self-check stops — two loops
// watching each other, and mesh-watcher on a second machine watching the self-check's heartbeat for
// the case both stop with the process.
// watchEvery is how often the watchdogs run.
var watchEvery = 30 * time.Second
// signalFacts is what one tick of the watchdogs reads.
type signalFacts struct {
now time.Time
started time.Time
// host is the machine this controller runs on, as the mesh names it, for a condition about itself.
host string
machines []machineFacts
machinesErr error
// toolsHeardFrom is when this process began hearing node tools: one not heard since is silent
// since then at the most.
toolsHeardFrom time.Time
plans []planFacts
plansErr error
loop loopFacts
loopErr error
merges []missedMerge
mergesPassed time.Time
mergesErr error
asks []askFacts
asksErr error
calls []link.Call
standings []conditions.Condition
standingsErr error
// providerWords are every open condition a provider's word raised — failing, waiting for a person
// to approve a retirement, kept by a rejection, cleanup waiting (ADR 0224, 0230) — so a provider no
// longer assigned has them all cleared.
providerWords []conditions.Condition
advisories []link.Advisory
lostConsumers map[string]bool
advisoriesErr error
selfCheck selfCheckFacts
// staleRefusals are the writers refused as older lately, and epochs the mesh's record of each epoch
// they name (novox/hq to-be 45 §6, S13).
staleRefusals []link.WriterRefusals
epochs map[int64]inventory.Epoch
// handActs are the acts done by hand within the fortnight S15 counts.
handActs []link.HandAct
handActsErr error
// lease is this controller's standing to the lease, and the epochs that ended lately (S12).
lease leaseFacts
leaseErr error
}
type leaseFacts struct {
held bool
epoch uint64
renewed time.Time
unleased string
ended []inventory.Epoch
// reset is when the lease bucket was found raised again from nothing; resetSaid what of it.
reset time.Time
resetSaid string
}
type machineFacts struct {
name string
control bool
// lastHeard is the store's last word from it, any word; zero when never.
lastHeard time.Time
// every is the heartbeat interval it said; zero when it said none.
every time.Duration
power link.PowerState
// sentAt is when it was last sent a declaration; reportedCurrent whether it has reported that one.
sentAt time.Time
reportedCurrent bool
reportedAt time.Time
// lastApply is its newest measured apply.
lastApply time.Duration
// tools is whether node-tools is assigned there; toolsHeard and toolsEvery its heartbeat.
tools bool
toolsHeard time.Time
toolsEvery time.Duration
}
// asleep says the machine said it would be away and has not said it is back (ADR 0211).
func (m machineFacts) asleep() bool { return m.power.Away() }
type planFacts struct {
id, repository, commit string
tier, tiers int
entered time.Time
bound time.Duration
waiting string
paused bool
}
type loopFacts struct {
took time.Time
pending uint64
where string
}
type askFacts struct {
id, seat, what, state, on string
since time.Time
bound time.Duration
}
type selfCheckFacts struct {
last time.Time
every time.Duration
}
// watchdogs is the loop and what it needs.
type watchdogs struct {
open *stores
server *link.Server
js *broker.JetStream
keeper *conditions.Keeper
doctor *doctor
started time.Time
// acting says this controller is the one acting, not one standing by (link.Holding): a controller
// standing by hears no heartbeat and would call every machine silent.
acting func() bool
mu sync.Mutex
ticked time.Time
last *signalFacts
failed string
}
// lastTick is when the watchdogs last finished a tick.
func (w *watchdogs) lastTick() time.Time {
w.mu.Lock()
defer w.mu.Unlock()
return w.ticked
}
// lastFacts is the facts of the newest tick, for `doctor signals`; nil before the first.
func (w *watchdogs) lastFacts() *signalFacts {
w.mu.Lock()
defer w.mu.Unlock()
return w.last
}
// keep runs the watchdogs until ctx ends.
func (w *watchdogs) keep(ctx context.Context) {
tick := time.NewTicker(watchEvery)
defer tick.Stop()
for {
w.tick(ctx)
select {
case <-ctx.Done():
return
case <-tick.C:
}
}
}
// tick is one run of every row.
func (w *watchdogs) tick(ctx context.Context) {
if w.acting != nil && !w.acting() {
// Standing by: nothing seen, nothing said, and the tick counted — this process's self-check
// is not the one that matters while another acts.
w.mu.Lock()
w.ticked = time.Now()
w.mu.Unlock()
return
}
running, cancel := context.WithTimeout(ctx, watchEvery)
defer cancel()
w.see(running, w.gather(running))
}
// see runs every watched row over one gathering, keeps what each found, and records the tick.
func (w *watchdogs) see(running context.Context, f *signalFacts) {
var problems []string
var blind []conditions.Observation
for _, row := range watchedRows() {
if err := row.needs(f); err != nil {
blind = append(blind, blindRow(row, err))
continue
}
if err := w.keeper.Reconcile(running, row.Row, row.watch(f)); err != nil {
problems = append(problems, row.Row+": "+err.Error())
}
}
// The rows that could not see, said; the ones that see again, cleared.
if err := w.keeper.Reconcile(running, sourceWatchdogs, blind); err != nil {
problems = append(problems, err.Error())
}
// A provider no longer assigned where it ran: its standing is resolved by the assignment.
if f.standingsErr == nil && w.open != nil {
if err := unassignedProviders(running, w.open.inventory, w.keeper, f.providerWords); err != nil {
problems = append(problems, "S8: "+err.Error())
}
}
if err := w.keeper.EndSilences(running); err != nil {
problems = append(problems, err.Error())
}
failed := ""
if len(problems) > 0 {
failed = fmt.Sprintf("%v", problems)
}
w.mu.Lock()
said := w.failed
w.ticked, w.last, w.failed = time.Now(), f, failed
w.mu.Unlock()
if failed != said {
if failed != "" {
fmt.Printf("the watchdogs could not keep what they saw: %s\n", failed)
} else if said != "" {
fmt.Println("the watchdogs keep what they see again")
}
}
}
// sourceWatchdogs is what raises a blind row's condition.
const sourceWatchdogs = "watchdogs"
// blindRow is a row whose facts could not be gathered, as a condition of its own.
func blindRow(row signalRow, err error) conditions.Observation {
return conditions.Observation{Scope: conditions.ScopeProbe, ID: row.Row, Kind: "probe-failed", Token: "failed",
Severity: conditions.Warning,
Summary: fmt.Sprintf("the watchdog of %s (%s) cannot see: what it reads could not be read, so nothing "+
"it would raise can be — and nothing it raised before is cleared", row.Row, row.Signal),
Said: firstLine(err.Error())}
}
// gather reads every fact a tick needs. Each part's failure is kept beside it, never an empty part.
func (w *watchdogs) gather(ctx context.Context) *signalFacts {
now := time.Now()
f := &signalFacts{now: now, started: w.started, toolsHeardFrom: link.ToolsBeats.Started(), calls: link.Calls.Running(),
staleRefusals: link.StaleRefusals.Within(now.Add(-staleRefusalsWithin)), lostConsumers: map[string]bool{},
epochs: map[int64]inventory.Epoch{}}
if w.doctor != nil {
f.selfCheck = selfCheckFacts{last: w.doctor.lastRunEnded(), every: doctorEvery}
}
inv := w.open.inventory
f.host = controlHost(ctx, inv)
f.lease, f.leaseErr = gatherLease(ctx, inv, now)
for _, r := range f.staleRefusals {
if r.Epoch <= 0 {
continue
}
// Named where the record has it; a writer the record cannot name is still said by its epoch.
if e, found, err := inv.EpochOf(ctx, uint64(r.Epoch)); err == nil && found {
f.epochs[r.Epoch] = e
}
}
f.machines, f.machinesErr = w.gatherMachines(ctx, inv, now)
f.plans, f.plansErr = gatherPlans(ctx, inv, now)
f.loop, f.loopErr = w.gatherLoop()
f.mergesPassed, f.merges, f.mergesErr = watchedMerges.last()
if f.mergesErr == nil && !f.mergesPassed.IsZero() && now.Sub(f.mergesPassed) > 3*mergeCatchUpEvery {
f.mergesErr = fmt.Errorf("the catch-up of merges has not passed since %s", f.mergesPassed.UTC().Format(time.RFC3339))
}
f.asks, f.asksErr = w.gatherAsks(ctx, inv)
if open, err := w.keeper.Open(ctx); err != nil {
f.standingsErr = err
} else {
f.standings = providerStandings(open)
f.providerWords = providerConditions(open)
}
f.advisories = link.Advisories.Since(now.Add(-advisoryQuiet))
f.lostConsumers, f.advisoriesErr = w.lostConsumers(ctx, f.advisories)
f.handActs, f.handActsErr = w.gatherHandActs(ctx, now)
return f
}
// gatherLease is this controller's standing to the lease and the epochs that ended within the hour.
func gatherLease(ctx context.Context, inv *inventory.Inventory, now time.Time) (leaseFacts, error) {
st := theLease.standing()
f := leaseFacts{held: st.Held, epoch: st.Epoch, renewed: st.Renewed, unleased: st.Unleased, reset: st.Reset,
resetSaid: st.ResetSaid}
ended, err := inv.EpochsSince(ctx, now.Add(-advisoryQuiet))
if err != nil {
return f, fmt.Errorf("the epochs the mesh issued cannot be read: %w", err)
}
f.ended = ended
return f, nil
}
// controlHost is the machine running the controller, as the mesh names it: the one the controller
// module is assigned to, or this process's host name where that is not one machine.
func controlHost(ctx context.Context, inv *inventory.Inventory) string {
if on, err := inv.Running(ctx, "mesh-controller"); err == nil && len(on) == 1 {
return on[0]
}
host, _ := os.Hostname()
return host
}
func (w *watchdogs) gatherMachines(ctx context.Context, inv *inventory.Inventory, now time.Time) ([]machineFacts, error) {
nodes, err := inv.Nodes(ctx)
if err != nil {
return nil, fmt.Errorf("the machines cannot be read: %w", err)
}
reports, err := inv.LastReports(ctx)
if err != nil {
return nil, fmt.Errorf("what each machine last reported cannot be read: %w", err)
}
reported := map[string]inventory.Reported{}
for _, r := range reports {
reported[r.Node] = r
}
control, err := inv.Running(ctx, "mesh-controller")
if err != nil {
return nil, fmt.Errorf("where the controller runs cannot be read: %w", err)
}
tooled, err := inv.Running(ctx, broker.RuntimeModule)
if err != nil {
return nil, fmt.Errorf("where the node tools run cannot be read: %w", err)
}
applies, err := inv.Durations(ctx, inventory.DurationApply, now.Add(-7*24*time.Hour))
if err != nil {
return nil, fmt.Errorf("the measured applies cannot be read: %w", err)
}
lastApply := map[string]time.Duration{}
for _, d := range applies { // oldest first: the last one read is the newest
lastApply[d.Node] = d.Took
}
var out []machineFacts
anyPast := false
for _, n := range nodes {
m := machineFacts{name: n.Name, control: slices.Contains(control, n.Name), lastHeard: n.LastSeen,
lastApply: lastApply[n.Name], tools: slices.Contains(tooled, n.Name)}
if beat, ok := link.HostBeats.Of(n.Name); ok {
m.every = beat.Every
}
if beat, ok := link.ToolsBeats.Of(n.Name); ok {
m.toolsHeard, m.toolsEvery = beat.At, beat.Every
}
if r, ok := reported[n.Name]; ok {
if r.Sent != nil {
m.sentAt = *r.Sent
}
if r.At != nil {
m.reportedAt = *r.At
}
m.reportedCurrent = r.Current
}
if !m.lastHeard.IsZero() && now.Sub(m.lastHeard) > heartbeatBound(m.every) {
anyPast = true
}
if m.tools && now.Sub(later(m.toolsHeard, link.ToolsBeats.Started())) > heartbeatBound(m.toolsEvery) {
anyPast = true
}
out = append(out, m)
}
// What a machine past its bound last said of its power, read only then: a machine that said it
// is asleep is not lost (ADR 0211). Unreadable is said: a sleeping laptop is then called silent,
// which is the louder mistake and the right one to make.
if anyPast && w.server != nil {
states, err := w.server.PowerStates(ctx, now.Add(-7*24*time.Hour))
if err != nil {
fmt.Printf("what machines said of their power cannot be read, so a sleeping one is called silent: %v\n", err)
}
for i := range out {
out[i].power = states[out[i].name]
}
}
return out, nil
}
// gatherPlans is every open plan, its tier's bound from what was measured, and what it waits on.
func gatherPlans(ctx context.Context, inv *inventory.Inventory, now time.Time) ([]planFacts, error) {
plans, err := inv.OpenPlans(ctx)
if err != nil {
return nil, fmt.Errorf("the open plans cannot be read: %w", err)
}
if len(plans) == 0 {
return nil, nil
}
tiers, err := inv.Durations(ctx, inventory.DurationPlanTier, now.Add(-14*24*time.Hour))
if err != nil {
return nil, fmt.Errorf("the measured plan tiers cannot be read: %w", err)
}
measured := map[string][]time.Duration{}
for _, d := range tiers {
measured[d.Subject] = append(measured[d.Subject], d.Took)
}
pause := buildSeatPause(ctx, inv, plans)
var out []planFacts
for _, p := range plans {
_, paused := pausedWaiting(p, pause, now)
out = append(out, planFacts{id: p.ID, repository: p.Repository, commit: p.Commit, tier: p.Tier,
tiers: len(p.Tiers), entered: p.TierEntered, bound: max(tierAtLeast, 3*p90(measured[p.Repository])),
waiting: planLineWith(p, now, pause), paused: paused})
}
return out, nil
}
// p90 is the ninetieth percentile of measurements; zero for none.
func p90(took []time.Duration) time.Duration {
if len(took) == 0 {
return 0
}
sorted := append([]time.Duration(nil), took...)
sort.Slice(sorted, func(i, j int) bool { return sorted[i] < sorted[j] })
return sorted[int(0.9*float64(len(sorted)-1))]
}
// gatherLoop is when the event loop last took a message and what its consumers hold.
func (w *watchdogs) gatherLoop() (loopFacts, error) {
f := loopFacts{took: link.Loop.Last()}
if w.js == nil {
return f, errors.New("this controller is not on the bus")
}
var where []string
for _, c := range broker.MeshConsumers() {
info, err := w.js.Context().ConsumerInfo(c.Stream, c.Name)
if err != nil {
return f, fmt.Errorf("the controller's consumer on %s cannot be read: %w", c.Stream, err)
}
if held := info.NumPending + uint64(info.NumAckPending); held > 0 {
f.pending += held
where = append(where, fmt.Sprintf("%d on %s", held, c.Stream))
}
}
f.where = fmt.Sprint(where)
return f, nil
}
// gatherAsks is every ask in the build seat's queue that is in flight or dead, with its bound.
func (w *watchdogs) gatherAsks(ctx context.Context, inv *inventory.Inventory) ([]askFacts, error) {
if w.js == nil {
return nil, errors.New("this controller is not on the bus")
}
entries, err := inv.Catalogued(ctx)
if err != nil {
return nil, fmt.Errorf("the catalogue cannot be read: %w", err)
}
seat := buildSeatAmong(entries)
q, err := link.ReadQueue(ctx, w.js, seat)
if err != nil {
return nil, err
}
builds, err := inv.Durations(ctx, inventory.DurationBuild, time.Now().Add(-14*24*time.Hour))
if err != nil {
return nil, fmt.Errorf("the measured builds cannot be read: %w", err)
}
var took []time.Duration
for _, d := range builds {
took = append(took, d.Took)
}
bound := askDefault
if len(took) > 0 {
bound = max(askAtLeast, 3*p90(took))
}
var out []askFacts
for _, a := range q.Asks {
if a.State != link.AskInFlight && a.State != link.AskDead {
continue
}
since := a.Started
if since.IsZero() {
since = a.AskedAt
}
what := a.Repository
if a.Path != "" {
what += " " + a.Path
}
out = append(out, askFacts{id: a.ID, seat: seat, what: what, state: a.State, on: a.On, since: since, bound: bound})
}
return out, nil
}
// lostConsumers is, of the consumers the bus said were deleted, each that is still missing and that
// the mesh expects: a consumer removed because its module was unassigned is not lost.
func (w *watchdogs) lostConsumers(ctx context.Context, heard []link.Advisory) (map[string]bool, error) {
out := map[string]bool{}
var deleted []link.Advisory
for _, a := range heard {
if a.Kind == link.AdvisoryConsumerLost {
deleted = append(deleted, a)
}
}
if len(deleted) == 0 {
return out, nil
}
if w.js == nil {
return nil, errors.New("this controller is not on the bus")
}
_, expected, err := expectedBusObjects(ctx, w.open.inventory)
if err != nil {
return nil, err
}
want := map[string]bool{}
for _, c := range expected {
want[c.Stream+"."+c.Name] = true
}
for _, a := range deleted {
_, err := w.js.Context().ConsumerInfo(a.Stream, a.Consumer)
switch {
case errors.Is(err, nats.ErrConsumerNotFound):
out[a.Stream+"."+a.Consumer] = want[a.Stream+"."+a.Consumer]
case err != nil:
return nil, fmt.Errorf("whether %s exists cannot be read: %w", link.ConsumerInWords(a.Stream, a.Consumer), err)
}
}
return out, nil
}
// gatherHandActs is the hand-act log's fortnight (S15), read from the bus.
func (w *watchdogs) gatherHandActs(ctx context.Context, now time.Time) ([]link.HandAct, error) {
if w.js == nil {
return nil, errors.New("this controller is not on the bus")
}
reading, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
acts, err := link.HandActs(reading, w.js.Conn(), now.Add(-handActsWithin))
if err != nil {
return nil, fmt.Errorf("the hand-act log cannot be read: %w", err)
}
return acts, nil
}
// later is the later of two moments.
func later(a, b time.Time) time.Time {
if a.After(b) {
return a
}
return b
}
// watchTheMesh opens the condition store and starts the watchdogs, the bus's advisories and the
// self-check, for the serving controller; the returned function stops them. Nil when the store could
// not be opened, which is said.
func watchTheMesh(ctx context.Context, open *stores, server *link.Server, bus link.OverNATS) func() {
keeper, err := keeperOn(ctx, bus.Conn)
if err != nil {
fmt.Printf("the condition store could NOT be opened, so nothing that goes wrong is kept or said, and "+
"status says the conditions cannot be read: %v\n", err)
return nil
}
conditionsFrom = keeper
logf := func(format string, args ...any) { fmt.Printf(format+"\n", args...) }
stopHearing, err := server.HearAdvisories(logf)
if err != nil {
fmt.Printf("what the bus says about itself cannot be heard (S9 is blind): %v\n", err)
stopHearing = func() {}
}
host := controlHost(ctx, open.inventory)
w := &watchdogs{open: open, server: server, js: server.JetStream(), keeper: keeper, started: time.Now(),
acting: link.Holding}
d := &doctor{open: open, js: server.JetStream(), keeper: keeper, teller: bus, watchdogs: w, host: host}
w.doctor = d
doctorFrom = d
watching, stop := context.WithCancel(ctx)
go w.keep(watching)
go d.keep(watching)
// And the healers (novox/hq to-be 45 §7): what is open that a registered healer answers, repaired
// under the lease and the brake, every act said.
healers := newHealing(open, keeper, bus, server.JetStream())
go healers.keep(watching)
go forgettingOldHeals(watching, open.inventory)
fmt.Printf("watching the mesh: %d signal(s) every %s, %d probe(s) every %s; what is wrong is kept in %s "+
"and said as %s events\n", len(watchedRows()), watchEvery, len(runnableProbes()), doctorEvery,
broker.ConditionsBucket, conditions.Seat)
return func() {
stop()
stopHearing()
flushing, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
keeper.Close(flushing)
}
}
// runnableProbes are the probes the registry runs.
func runnableProbes() []probe {
var out []probe
for _, p := range probeRegistry {
if p.run != nil {
out = append(out, p)
}
}
return out
}
+10 -1
View File
@@ -5,7 +5,9 @@ go 1.26.0
require (
github.com/jackc/pgx/v5 v5.10.0
github.com/nats-io/nats.go v1.54.0
github.com/novox/mesh-host v0.0.0
golang.org/x/crypto v0.57.0
golang.org/x/net v0.58.0
)
require (
@@ -15,8 +17,15 @@ require (
github.com/klauspost/compress v1.20.0 // indirect
github.com/nats-io/nkeys v0.4.16 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
golang.org/x/net v0.58.0 // indirect
golang.org/x/sync v0.23.0 // indirect
golang.org/x/sys v0.48.0 // indirect
golang.org/x/text v0.42.0 // indirect
)
// The node-engine's own validator (mesh-host/validate, novox/hq to-be 45 D1): one validator, the host's.
// The host's module path names no forge a build can fetch from, so the module is read from the one
// that holds it, at the host's commit — **once, by whoever moves the pin, into vendor/**, which is
// committed. Every build (the build agent's `go build`, the Dockerfile) compiles from vendor/ and
// fetches nothing; go refuses to build when vendor/ and this file disagree, so a pin moved without
// `go mod vendor` fails loudly, at once, everywhere.
replace github.com/novox/mesh-host => git.novox.be/novox/mesh-host v0.0.0-20261006095519-3e80b7ae325e
+2
View File
@@ -1,3 +1,5 @@
git.novox.be/novox/mesh-host v0.0.0-20261006095519-3e80b7ae325e h1:g9h4QRaAMg5yaJLwqtb0FoOs23DVGUYpW6qvnQ3oY5A=
git.novox.be/novox/mesh-host v0.0.0-20261006095519-3e80b7ae325e/go.mod h1:VlilMCRZ5yyNXg7SNigNBLr0Gt32jrGw5KSNq5JAVYs=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+174
View File
@@ -0,0 +1,174 @@
package broker
import (
"context"
"fmt"
"time"
"github.com/nats-io/nats.go/jetstream"
)
// The controller's own key-value buckets (novox/hq to-be 45 §1, §6, §7).
//
// **What the controller must remember across its own restart, it keeps on the bus.** A call's
// outcome lived in the memory of the process that served it (novox/hq issue 265), so a controller
// replaced while a push ran answered "no such call" for the one thing its caller had been told to
// ask about. The bus already outlives the controller and is the shape ADR 0201 gives a module's
// current state: one value per key, written by one owner, read by anybody granted it. These are the
// controller's, written by it alone — the writers table of to-be 45 §1 — and asserted on every start
// like the streams, so a bus raised from nothing has them before the first call is served.
// CallsBucket keeps every call of the mesh's own verbs and what came of it; HandActsBucket every act
// a person did by hand, with why; ConditionsBucket every condition open now (to-be 45 §2), one key
// each, and ConditionHistoryBucket every transition of one — raised, changed, silenced, cleared —
// for ninety days.
//
// **The history is a bucket of its own** because its keys expire and an open condition's must not:
// a bucket has one age for every key, and a condition open longer than the history is kept would
// otherwise vanish from the store while still true.
var (
CallsBucket = BucketName(ControllerSeat, "calls")
HandActsBucket = BucketName(ControllerSeat, "hand-acts")
ConditionsBucket = BucketName(ControllerSeat, "conditions")
ConditionHistoryBucket = BucketName(ControllerSeat, "condition-history")
// LeaseBucket holds the controller's lease (to-be 45 §6): one key, `holder`, which the instance
// allowed to act writes by compare-and-set and renews; its revision when taken is the epoch.
LeaseBucket = BucketName(ControllerSeat, "lease")
)
// LeaseTTL is how long the lease's key lives unrenewed (to-be 45 §6): fifteen seconds, renewed
// every five. The bucket's age, so the bus forgets a holder that stopped renewing.
const LeaseTTL = 15 * time.Second
// The bounds to-be 45 §6 sets for calls: the last thousand, or fourteen days, whichever is fewer.
// A call is two keys — its record, and its answer apart so a listing does not read every answer —
// so the stream holds twice as many messages as it keeps calls.
const (
KeptCallsDurably = 1000
CallsKeptFor = 14 * 24 * time.Hour
// CallAnswerBytes is the most of one answer kept: a whole declaration is far smaller, and an
// answer larger is cut and says so.
CallAnswerBytes = 64 << 10
// HandActsKeptFor is as long as a condition's history (to-be 45 §2): an act by hand is read
// back beside what it addressed.
HandActsKeptFor = 90 * 24 * time.Hour
// ConditionHistoryKeptFor is how long a condition's transitions are kept (to-be 45 §2).
ConditionHistoryKeptFor = 90 * 24 * time.Hour
)
// IsControllerBucket says a bucket is the controller's own, not a module's state nothing declares.
func IsControllerBucket(bucket string) bool {
return bucket == CallsBucket || bucket == HandActsBucket || bucket == ConditionsBucket ||
bucket == ConditionHistoryBucket || bucket == LeaseBucket
}
// ControllerBuckets are the controller's own buckets, in the order they are asserted.
func ControllerBuckets() []string {
return []string{LeaseBucket, CallsBucket, HandActsBucket, ConditionsBucket, ConditionHistoryBucket}
}
// ControllerBucketsAsserter is what raising the controller's buckets needs of a connection.
type ControllerBucketsAsserter interface {
EnsureControllerBuckets() error
}
// EnsureControllerBuckets creates the controller's buckets if absent and brings their options to
// match. An update, never a delete: what they hold is the record of what the mesh was asked.
func (j *JetStream) EnsureControllerBuckets() error {
js, err := jetstream.New(j.conn)
if err != nil {
return err
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
if err := EnsureLeaseBucket(ctx, js); err != nil {
return err
}
if _, err := js.CreateOrUpdateKeyValue(ctx, jetstream.KeyValueConfig{
Bucket: CallsBucket,
Description: "the calls of the mesh's own verbs and what came of each (novox/hq to-be 45 §6, issue " +
"265): written by the controller alone, read through `calls`; the last thousand, or fourteen days",
History: 1,
TTL: CallsKeptFor,
MaxValueSize: CallAnswerBytes + 4<<10,
MaxBytes: 2 * KeptCallsDurably * (CallAnswerBytes + 4<<10),
Storage: jetstream.FileStorage,
}); err != nil {
return fmt.Errorf("asserting bucket %s: %w", CallsBucket, err)
}
// **The count, on the stream under the bucket.** A bucket has an age and a size and no count;
// the stream it is made of does, and with one value per key the oldest message is the oldest
// call. Asserted after the bucket, every time, because asserting the bucket writes the stream's
// configuration whole and puts the count back to none.
stream, err := js.Stream(ctx, "KV_"+CallsBucket)
if err != nil {
return fmt.Errorf("reading the stream under %s: %w", CallsBucket, err)
}
cfg := stream.CachedInfo().Config
if cfg.MaxMsgs != 2*KeptCallsDurably {
cfg.MaxMsgs = 2 * KeptCallsDurably
cfg.Discard = jetstream.DiscardOld
if _, err := js.UpdateStream(ctx, cfg); err != nil {
return fmt.Errorf("bounding %s to the last %d calls: %w", CallsBucket, KeptCallsDurably, err)
}
}
if _, err := js.CreateOrUpdateKeyValue(ctx, jetstream.KeyValueConfig{
Bucket: HandActsBucket,
Description: "every act a person did by hand, with why (novox/hq to-be 45 §7): written by the " +
"controller's repairing verbs and `hand-act record`, read through `hand-acts`",
History: 1,
TTL: HandActsKeptFor,
MaxValueSize: 16 << 10,
MaxBytes: 64 << 20,
Storage: jetstream.FileStorage,
}); err != nil {
return fmt.Errorf("asserting bucket %s: %w", HandActsBucket, err)
}
// **No age on the open conditions.** A condition is removed when observation clears it and at no
// other moment: one that expired would be a fault the store forgot while it was still true.
if _, err := js.CreateOrUpdateKeyValue(ctx, jetstream.KeyValueConfig{
Bucket: ConditionsBucket,
Description: "every condition open now, one key each (novox/hq to-be 45 §2): written by the " +
"controller alone, raised and cleared by observation, read through `conditions`",
History: 1,
MaxValueSize: 64 << 10,
MaxBytes: 64 << 20,
Storage: jetstream.FileStorage,
}); err != nil {
return fmt.Errorf("asserting bucket %s: %w", ConditionsBucket, err)
}
if _, err := js.CreateOrUpdateKeyValue(ctx, jetstream.KeyValueConfig{
Bucket: ConditionHistoryBucket,
Description: "every transition of a condition — raised, changed, silenced, cleared — kept ninety " +
"days (novox/hq to-be 45 §2): written by the controller alone, read through `conditions history`",
History: 1,
TTL: ConditionHistoryKeptFor,
MaxValueSize: 64 << 10,
MaxBytes: 256 << 20,
Storage: jetstream.FileStorage,
}); err != nil {
return fmt.Errorf("asserting bucket %s: %w", ConditionHistoryBucket, err)
}
return nil
}
// EnsureLeaseBucket creates the lease's bucket if absent and brings its options to match (to-be 45
// §6). **Before the lease is taken, by any candidate**: it is the lease's own precondition, and asserting
// a bucket that exists changes nothing. File storage, so its revisions — the epochs — outlive a restart of
// the bus; one raised again from nothing is moved past the highest epoch issued when the lease is taken.
func EnsureLeaseBucket(ctx context.Context, js jetstream.JetStream) error {
if _, err := js.CreateOrUpdateKeyValue(ctx, jetstream.KeyValueConfig{
Bucket: LeaseBucket,
Description: "the controller's lease (novox/hq to-be 45 §6): the key `holder`, written by compare-and-set " +
"by the one controller instance that may act and renewed every five seconds; its revision when taken " +
"is the epoch every declaration and plan write carries",
History: 1,
TTL: LeaseTTL,
MaxValueSize: 4 << 10,
MaxBytes: 1 << 20,
Storage: jetstream.FileStorage,
}); err != nil {
return fmt.Errorf("asserting bucket %s: %w", LeaseBucket, err)
}
return nil
}
@@ -0,0 +1,61 @@
package broker
import (
"slices"
"strings"
"testing"
)
// **The controller may write every bucket it writes** (novox/hq to-be 45 §1, issue 269). Writing a
// key is a publish to the bucket's own subject, which the management interface's grant does not
// cover: the cancelled sets' writes timed out for want of this, and the controller's own buckets
// would have.
func TestTheControllerMayWriteEveryBucketItWrites(t *testing.T) {
p, err := PermissionsFor(Principal{Kind: KindController, PasswordHash: "x"})
if err != nil {
t.Fatal(err)
}
want := []string{"$KV." + CallsBucket + ".>", "$KV." + HandActsBucket + ".>"}
for _, seat := range seatsTheControllerAsks {
if hasCancelledSet(seat) {
want = append(want, "$KV."+CancelledSetName(seat)+".>")
}
}
for _, subject := range want {
if !slices.Contains(p.Publish, subject) {
t.Errorf("the controller may not publish %s, so it cannot write that bucket", subject)
}
}
if slices.Contains(p.Publish, "$KV.>") {
t.Error("the controller may write any bucket, a module's state included")
}
}
// **A machine's node tools may say they are there, as that machine and no other** (novox/hq to-be 45
// S11), and the controller may hear the bus's advisories and ask who answers — read-only, named.
func TestTheWatchedSignalsMayBeSaidAndHeard(t *testing.T) {
tools, err := PermissionsFor(Principal{Kind: KindNodeTools, Node: "anchor", Module: RuntimeModule, PasswordHash: "x"})
if err != nil {
t.Fatal(err)
}
if !slices.Contains(tools.Publish, "mesh.control.anchor.tools-alive") {
t.Error("the node tools may not say they are there")
}
for _, s := range tools.Publish {
if strings.Contains(s, "tools-alive") && s != "mesh.control.anchor.tools-alive" {
t.Errorf("the node tools may say %s", s)
}
}
controller, err := PermissionsFor(Principal{Kind: KindController, PasswordHash: "x"})
if err != nil {
t.Fatal(err)
}
for _, s := range BusAdvisories {
if !slices.Contains(controller.Subscribe, s) {
t.Errorf("the controller may not hear %s", s)
}
}
if !slices.Contains(controller.Publish, "$SRV.INFO") || slices.Contains(controller.Subscribe, "$JS.EVENT.>") {
t.Error("the controller may not ask who answers, or hears every API call")
}
}
+5
View File
@@ -54,6 +54,11 @@ type Consumer struct {
// that exists keeps where it is, whatever this says; only its making is decided here.
FromNow bool
Why string
// Resettable says why this consumer may be re-made to deliver from now by the mesh itself, with
// nobody asked (novox/hq to-be 45 §7, healer H4): what a reset drops, something else catches up.
// Empty for every consumer where nothing would — a module's, whose events would be lost to it.
// Not a property of the consumer on the bus: nothing here is sent to the server.
Resettable string
}
// seatStreamName is the stream holding a seat's inbound work. Named after the seat rather than
+115 -5
View File
@@ -18,6 +18,7 @@ import (
"regexp"
"sort"
"strings"
"time"
)
// A Kind is what a principal is, which decides the shape of its authority rather than its
@@ -109,6 +110,11 @@ type Principal struct {
State []string
Reads []string
// SnapshotsTheBus is the bus's own module, the one holding mesh-broker (novox/hq ADR 0235). Its
// whole authority is BusSnapshotGrants: it copies the streams for the night's backup and nothing
// else — not its declarations, which the node's tool runtime serves for it.
SnapshotsTheBus bool
// PasswordHash is the bcrypt hash the mesh minted. The plaintext is sealed to the principal
// and never appears here: this file is written to a node's disk and read by a server, and a
// secret that can be read from a configuration file is a secret with a wider blast radius
@@ -124,6 +130,20 @@ type Principal struct {
// goes with the retired seat row.
var seatsTheControllerAsks = []string{"node-build-agent", "mesh-build-machine"}
// SeatVerb is one verb of one seat, on every machine holding it.
type SeatVerb struct{ Seat, Verb string }
// VerbsTheSelfCheckAsks are the seat verbs the controller's self-check and watchdogs call (novox/hq
// to-be 45 §4): D8 reads every machine's ban list. **Named one by one, and the test that holds them
// to the probe registry is the reason they cannot drift** — a probe that calls a verb its grant does
// not name is refused by the bus on every run (found live on 2026-10-06: D8 timed out on each
// machine, refused). Asked of any machine (`.*`), read-only verbs, nothing else of the seat.
//
// D13 reads every machine's backup holder: what it measured of the data declared there (novox/hq ADR
// 0233).
var VerbsTheSelfCheckAsks = []SeatVerb{{Seat: "node-intrusion-prevention", Verb: "banned"},
{Seat: "node-backup", Verb: "backed-up"}}
// perMachineEvents are a node-scoped seat's events about the holder itself, whose last token is the
// holder's machine (novox/hq ADR 0219): `paused.<node>`, the build agent saying whether it takes work.
var perMachineEvents = map[string]bool{"paused.*": true}
@@ -168,6 +188,10 @@ func (p Principal) Username() string {
return ""
}
// AskReportSubject is where the mesh asks one machine's node-engine to say again what it last applied
// (novox/hq to-be 45 §6): its answer is an ordinary report, on its own report subject.
func AskReportSubject(node string) string { return "mesh.node." + node + ".ask.report" }
// inbox is a principal's own reply space. No user is ever granted a bare `_INBOX.>` (design 25
// §4): with one account, inbox privacy is the permission list or it is nothing, so each user's
// inbox is derived from its own identity and its permissions name that prefix and no other.
@@ -189,6 +213,38 @@ type Permissions struct {
AllowResponses bool
}
// ResponseTTL is how long the bus lets a principal answer a request it received. Its one answer has
// to come inside this, and a seat's holder answers within link.AnswerWithin — inside it by design.
// **A broker reloading its user list forgets every answer it was about to permit**, whatever this
// says (novox/hq issue 265): a call that is still running when the list reloads has its answer
// refused, which is why a holder answers before it does what can reload it.
const ResponseTTL = time.Minute
// BusSnapshotGrants is the whole authority of the bus's own module (novox/hq ADR 0235): the
// snapshot API and what it needs, and nothing that changes a stream.
//
// **Read-only, and the writers table proves it**: none of these overlaps a subject a write of the
// mesh's state is a publish to — not a stream's definition, not a bucket, not a message. A snapshot is
// the server reading its own blocks out to the asker, while it goes on taking writes; it neither
// pauses nor reconfigures the stream. Named one by one rather than `$JS.API.>`, which would be the
// controller's authority over every stream:
//
// - the stream names, and one stream's information (whether it is on disk at all — a memory stream
// cannot be snapshotted and is said as skipped);
// - the snapshot request itself, for any stream: the archive comes to the asker's own inbox;
// - the acknowledgement the server waits for past its window: a publish to the subject the server
// put on each chunk, `$JS.SNAPSHOT.ACK.<stream>.<id>.<size>.<index>`, which only the server's
// own internal subscription hears.
//
// Restoring is not here: a restore creates a stream, which is the controller's to define, and the
// mesh restores into a new store beside the live one, swapped in by a person (to-be 43).
var BusSnapshotGrants = struct{ Publish []string }{Publish: []string{
"$JS.API.STREAM.NAMES",
"$JS.API.STREAM.INFO.*",
"$JS.API.STREAM.SNAPSHOT.*",
"$JS.SNAPSHOT.ACK.>",
}}
// PermissionsFor derives a principal's authority. Pure, and the only place authority is decided:
// a permission that cannot be derived from a declaration is a permission nobody can explain.
func PermissionsFor(p Principal) (Permissions, error) {
@@ -204,6 +260,17 @@ func PermissionsFor(p Principal) (Permissions, error) {
}
}
if p.Kind == KindModule && p.SnapshotsTheBus {
pub := append([]string(nil), BusSnapshotGrants.Publish...)
sort.Strings(pub)
if err := CheckWriters(p, pub); err != nil {
return Permissions{}, err
}
// Its own inbox for the answers and the archive's chunks, and nothing else: nothing is asked
// of it, so it answers nothing.
return Permissions{Publish: pub, Subscribe: []string{p.inbox()}}, nil
}
var pub, sub []string
switch p.Kind {
case KindController:
@@ -211,7 +278,12 @@ func PermissionsFor(p Principal) (Permissions, error) {
// stream definitions (design 25 §3), so it alone reaches the JetStream API — and it alone
// issues memberships (novox/hq ADR 0160), which it publishes into the assignments stream
// after each push; refused by the server on 2026-10-01 until this line named them.
pub = []string{"mesh.control.>", "mesh.node.>", "mesh.assignment.>", "$JS.API.>"}
//
// **Not what the machines say** (novox/hq to-be 45 §1): `mesh.control.>` is subscribed, never
// published — a machine's report has one writer, its node-engine, and the controller granted
// that subject would be a second (the writers table refuses it). It was granted from the first
// composition and nothing ever published under it.
pub = []string{"mesh.node.>", "mesh.assignment.>", "$JS.API.>"}
// **And where its consumers deliver.** A push consumer delivers on `_DELIVER.<its name>`,
// and a client bound to it subscribes exactly that; the server refused it for every
// principal the first time one bound a consumer (2026-09-28). Each kind below is granted
@@ -229,6 +301,12 @@ func PermissionsFor(p Principal) (Permissions, error) {
// building, kill it, pause it, resume it. The queue is the controller's to show and to
// change, and what one machine is doing with an ask it took only that machine can say.
pub = append(pub, "mesh.seat."+seat+".tool.>")
// **And its cancelled set** (novox/hq ADR 0219, issue 269): a cancel writes the ask's id
// there before it deletes the ask, and a write is a publish to the bucket's subject, which
// `$JS.API.>` does not cover — so every cancel timed out, refused by this list.
if hasCancelledSet(seat) {
pub = append(pub, "$KV."+CancelledSetName(seat)+".>")
}
}
// **And what the mesh says it did** (novox/hq ADR 0134). The control plane states its own
// facts under the seat it holds, because a role's events belong to the role and keep their
@@ -252,11 +330,19 @@ func PermissionsFor(p Principal) (Permissions, error) {
sub = append(sub, "mesh.seat."+ControllerSeat+".tool.>")
// And says so (novox/hq ADR 0197): it answers discovery for the seat it serves.
sub = append(sub, announcing(ControllerSeat)...)
// And the verbs the self-check reads with, of any machine's holder (novox/hq to-be 45 §4).
for _, v := range VerbsTheSelfCheckAsks {
pub = append(pub, "mesh.seat."+v.Seat+".tool."+v.Verb+".*")
}
// And asks who answers (novox/hq to-be 45 §4, D3): the self-check finds every seat's holder by
// the same discovery the console reads. The question only; the answers come to its own inbox.
pub = append(pub, "$SRV.INFO")
// The two events it reacts to, and its ack subject on the stream they arrive from
// The events it reacts to, and its ack subject on the stream they arrive from
// (streams.go). **Each named, not a pattern**: `mesh.mod.*.event.>` would make the
// controller a subscriber to every event in the mesh, and its permission list would stop
// saying what it is for. The ack grant below is scoped per stream because the controller's
// saying what it is for. The one wildcard is the emitter of a provider's standing (ADR
// 0224) — still two named events, from whichever module provides. The ack grant below is scoped per stream because the controller's
// consumer name is the same on both and `$JS.ACK.CONTROL.controller.>` does not cover a
// delivery from EVENTS — a consumer that cannot ack has every message redelivered for
// ever, refused by the list it already has.
@@ -279,6 +365,18 @@ func PermissionsFor(p Principal) (Permissions, error) {
// enrolments and can reach nothing else.
pub = append(pub, "_INBOX."+enrolmentPrefix+".>")
// **And its own buckets** (novox/hq to-be 45 §1): the calls it served and the acts done by
// hand, which it alone writes. A put is a publish to the bucket's subject, which `$JS.API.>`
// does not cover; each bucket named, not `$KV.>`, which would let it write any module's state.
for _, bucket := range ControllerBuckets() {
pub = append(pub, "$KV."+bucket+".>")
}
// **And what the bus says about itself, read-only** (novox/hq to-be 45 §3, S9): a durable
// consumer that gave up on a message, or one that was deleted. The server already publishes
// both in the mesh's own account; the controller says each as a condition in the mesh's words.
// Named, not `$JS.EVENT.>`: the other advisories are every API call the mesh makes.
sub = append(sub, BusAdvisories...)
case KindPerson:
// Tools, and nothing else. Every subject a person may publish is a tool call; a person
// who could publish an event would be able to claim a module said something.
@@ -329,7 +427,11 @@ func PermissionsFor(p Principal) (Permissions, error) {
// next assertion moves it, and a permission that only allowed the new shape would refuse
// every node in the mesh for exactly as long as that took.
sub = []string{"mesh.node." + p.Node + ".declare",
"_DELIVER." + p.Node, "_DELIVER." + p.Node + ".>"}
"_DELIVER." + p.Node, "_DELIVER." + p.Node + ".>",
// And the mesh asking it to say again what it last applied (novox/hq to-be 45 §6, the
// `report` verb healer H1 asks): its own machine's, on core NATS and off any stream. It
// answers through its report, the one thing it already says — no reply to anybody's inbox.
AskReportSubject(p.Node)}
case KindModule:
// 1. Its own namespace: it publishes its events there and serves its tools there. Nothing
@@ -490,6 +592,9 @@ func PermissionsFor(p Principal) (Permissions, error) {
// that varies is the module, so the pattern is the machine's own assignments.
sub = append(sub, "mesh.assignment."+p.Node+".*")
pub = append(pub, "$JS.API.DIRECT.GET."+AssignmentsStream+".mesh.assignment."+p.Node+".*")
// And that it is there (novox/hq to-be 45 §3, S11): its own heartbeat, under its machine's
// name and no other's, on core NATS like the host's.
pub = append(pub, "mesh.control."+p.Node+".tools-alive")
// And every tool on the mesh (ADR 0175, decision 5): any node may call any tool on any
// node, as the console already could — the runtime is the console's serving mode.
invoked, err := invokedSubjects([]string{"*"})
@@ -557,6 +662,11 @@ func PermissionsFor(p Principal) (Permissions, error) {
sort.Strings(pub)
sort.Strings(sub)
// One writer per piece of state (novox/hq to-be 45 §1): a grant that would make a second is
// refused here, at composition, naming the state and its writer.
if err := CheckWriters(p, pub); err != nil {
return Permissions{}, err
}
return Permissions{
Publish: pub,
Subscribe: sub,
@@ -777,7 +887,7 @@ func ComposeAccounts(principals []Principal) (string, error) {
fmt.Fprintf(&b, " publish: { allow: [%s] }\n", quoted(perms.Publish))
fmt.Fprintf(&b, " subscribe: { allow: [%s] }\n", quoted(perms.Subscribe))
if perms.AllowResponses {
b.WriteString(" allow_responses: { max: 1, ttl: \"1m\" }\n")
fmt.Fprintf(&b, " allow_responses: { max: 1, ttl: \"%dm\" }\n", int(ResponseTTL/time.Minute))
}
b.WriteString(" } }\n")
}
+3
View File
@@ -28,6 +28,9 @@ func TestTheComposedConfigMatchesTheGolden(t *testing.T) {
Emits: []string{"order.placed"}, PasswordHash: "$2a$11$ssssssssssssssssssssss"},
{Kind: KindModule, Node: "two", Module: "audit",
Consumes: []string{"shop.order.placed"}, PasswordHash: "$2a$11$aaaaaaaaaaaaaaaaaaaaaa"},
// The bus's own module: the snapshot API and its inbox, nothing else (novox/hq ADR 0235).
{Kind: KindModule, Node: "one", Module: "nats", SnapshotsTheBus: true,
Serves: []string{"nats_streams"}, PasswordHash: "$2a$11$bbbbbbbbbbbbbbbbbbbbbb"},
})
if err != nil {
t.Fatal(err)
+8 -8
View File
@@ -5,16 +5,16 @@ import "testing"
// A node-scoped seat's tool carries the node (novox/hq ADR 0132, design 33 §4): two nodes holding one
// node-scoped seat derive two addresses, and a user of the seat may publish any node's.
func TestTwoNodesHoldingOneNodeSeatDeriveTwoToolAddresses(t *testing.T) {
seat := Seat{Name: "node-dns-resolver", Scope: "node", Serves: []string{"lookup"}}
one, _ := PermissionsFor(Principal{Kind: KindModule, Node: "one", Module: "dnsmasq", Holds: []Seat{seat}, PasswordHash: "x"})
two, _ := PermissionsFor(Principal{Kind: KindModule, Node: "two", Module: "dnsmasq", Holds: []Seat{seat}, PasswordHash: "x"})
has(t, one.Subscribe, "mesh.seat.node-dns-resolver.tool.lookup.one")
has(t, two.Subscribe, "mesh.seat.node-dns-resolver.tool.lookup.two")
hasNot(t, one.Subscribe, "mesh.seat.node-dns-resolver.tool.lookup")
hasNot(t, one.Subscribe, "mesh.seat.node-dns-resolver.tool.lookup.two")
seat := Seat{Name: "node-hostname", Scope: "node", Serves: []string{"entries"}}
one, _ := PermissionsFor(Principal{Kind: KindModule, Node: "one", Module: "hostname", Holds: []Seat{seat}, PasswordHash: "x"})
two, _ := PermissionsFor(Principal{Kind: KindModule, Node: "two", Module: "hostname", Holds: []Seat{seat}, PasswordHash: "x"})
has(t, one.Subscribe, "mesh.seat.node-hostname.tool.entries.one")
has(t, two.Subscribe, "mesh.seat.node-hostname.tool.entries.two")
hasNot(t, one.Subscribe, "mesh.seat.node-hostname.tool.entries")
hasNot(t, one.Subscribe, "mesh.seat.node-hostname.tool.entries.two")
user, _ := PermissionsFor(Principal{Kind: KindModule, Node: "three", Module: "asker", Uses: []Seat{seat}, PasswordHash: "x"})
has(t, user.Publish, "mesh.seat.node-dns-resolver.tool.lookup.*")
has(t, user.Publish, "mesh.seat.node-hostname.tool.entries.*")
}
// A mesh-scoped seat's tool stays flat: nothing about it changes.
+183
View File
@@ -0,0 +1,183 @@
package broker
import (
"encoding/json"
"fmt"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
"time"
"github.com/nats-io/nats.go"
"golang.org/x/crypto/bcrypt"
)
// The bus's own module's composed authority, against the bus's release in a throwaway container
// (novox/hq ADR 0235): it can take a whole snapshot of a stream through the server's chunked
// protocol, and every write — to a stream's definition, its messages, a bucket — is refused.
//
// MESH_TEST_DOCKER=1 go test ./internal/broker/ -run TestTheComposedSnapshotUser
//
// The program that takes the night's snapshot lives in the nats module (mesh-catalog
// modules/nats/snapshot) and is tested there against these same grants; this proves the grants are
// what the controller composes, by composing them.
func TestTheComposedSnapshotUserCanSnapshotAndCannotWrite(t *testing.T) {
if os.Getenv("MESH_TEST_DOCKER") != "1" {
t.Skip("MESH_TEST_DOCKER is not 1: this starts a throwaway nats container")
}
hash := func(pw string) string {
h, err := bcrypt.GenerateFromPassword([]byte(pw), bcrypt.MinCost)
if err != nil {
t.Fatal(err)
}
return string(h)
}
accounts, err := ComposeAccounts([]Principal{
{Kind: KindController, PasswordHash: hash("controller")},
{Kind: KindModule, Node: "anchor", Module: "nats", SnapshotsTheBus: true, PasswordHash: hash("snap")},
})
if err != nil {
t.Fatal(err)
}
conf, data := t.TempDir(), t.TempDir()
_ = os.WriteFile(filepath.Join(conf, "accounts.conf"), []byte(accounts), 0o644)
_ = os.WriteFile(filepath.Join(conf, "nats.conf"), []byte("port: 4222\njetstream { store_dir: \"/data\" }\ninclude accounts.conf\n"), 0o644)
name := fmt.Sprintf("mesh-controller-snapshot-test-%d", time.Now().UnixNano())
run := exec.Command("docker", "run", "-d", "--rm", "--name", name, "--user", fmt.Sprintf("%d:%d", os.Getuid(), os.Getgid()),
"-p", "127.0.0.1::4222", "-v", conf+":/etc/nats:ro", "-v", data+":/data", "nats:2.11-alpine", "-c", "/etc/nats/nats.conf")
if out, err := run.CombinedOutput(); err != nil {
t.Fatalf("%v\n%s", err, out)
}
t.Cleanup(func() { _ = exec.Command("docker", "rm", "-f", name).Run() })
out, err := exec.Command("docker", "port", name, "4222/tcp").Output()
if err != nil {
t.Fatal(err)
}
_, port, _ := strings.Cut(strings.TrimSpace(strings.Split(string(out), "\n")[0]), ":")
url := "nats://127.0.0.1:" + port
dial := func(user, pw string, errs chan error) *nats.Conn {
var nc *nats.Conn
var err error
for deadline := time.Now().Add(15 * time.Second); ; time.Sleep(200 * time.Millisecond) {
nc, err = nats.Connect(url, nats.UserInfo(user, pw), nats.CustomInboxPrefix("_INBOX."+user),
nats.ErrorHandler(func(_ *nats.Conn, _ *nats.Subscription, err error) {
if errs != nil {
select {
case errs <- err:
default:
}
}
}))
if err == nil || time.Now().After(deadline) {
break
}
}
if err != nil {
t.Fatal(err)
}
t.Cleanup(nc.Close)
return nc
}
// The controller defines the streams and fills them, as it does.
controller := dial("controller", "controller", nil)
js, _ := controller.JetStream()
if _, err := js.AddStream(&nats.StreamConfig{Name: "EVENTS", Subjects: []string{"mesh.mod.*.event.>"}}); err != nil {
t.Fatal(err)
}
kv, err := js.CreateKeyValue(&nats.KeyValueConfig{Bucket: HandActsBucket})
if err != nil {
t.Fatal(err)
}
if _, err := kv.Put("act", []byte("done by hand")); err != nil {
t.Fatal(err)
}
errs := make(chan error, 32)
snap := dial("anchor.nats", "snap", errs)
// What it may do: list, read a stream's information, and take a whole snapshot.
names, err := snap.Request("$JS.API.STREAM.NAMES", nil, 5*time.Second)
if err != nil || !strings.Contains(string(names.Data), "KV_"+HandActsBucket) {
t.Fatalf("it cannot list the streams: %v", err)
}
deliver := snap.NewRespInbox()
done := make(chan string, 1)
var bytes int
sub, err := snap.Subscribe(deliver, func(m *nats.Msg) {
if len(m.Data) == 0 {
done <- m.Header.Get("Status")
return
}
bytes += len(m.Data)
if m.Reply != "" {
_ = snap.Publish(m.Reply, nil)
}
})
if err != nil {
t.Fatal(err)
}
defer sub.Unsubscribe()
req, _ := json.Marshal(map[string]any{"deliver_subject": deliver})
answer, err := snap.Request("$JS.API.STREAM.SNAPSHOT.KV_"+HandActsBucket, req, 5*time.Second)
if err != nil || strings.Contains(string(answer.Data), `"error"`) {
t.Fatalf("the snapshot was not granted: %v", err)
}
select {
case status := <-done:
if status != "" && status != "204" {
t.Fatalf("the snapshot ended with %s", status)
}
case <-time.After(10 * time.Second):
t.Fatal("the snapshot never finished")
}
if bytes == 0 {
t.Fatal("the snapshot delivered nothing")
}
select {
case e := <-errs:
t.Fatalf("taking a snapshot met a refusal: %v", e)
default:
}
// What it may not: every write, and every subscription beyond its own inbox.
for _, subject := range []string{"$JS.API.STREAM.CREATE.NEW", "$JS.API.STREAM.UPDATE.EVENTS",
"$JS.API.STREAM.DELETE.EVENTS", "$JS.API.STREAM.PURGE.EVENTS", "$JS.API.STREAM.MSG.DELETE.EVENTS",
"$JS.API.STREAM.RESTORE.NEW", "$JS.API.CONSUMER.CREATE.EVENTS", "$KV." + HandActsBucket + ".act",
"mesh.mod.nats.event.anything", "mesh.node.anchor.declare"} {
if _, err := snap.Request(subject, []byte(`{}`), 300*time.Millisecond); err == nil {
t.Errorf("%s was answered", subject)
}
select {
case e := <-errs:
if !strings.Contains(strings.ToLower(e.Error()), "permissions violation") {
t.Errorf("%s: %v", subject, e)
}
case <-time.After(2 * time.Second):
t.Errorf("the bus did not refuse %s", subject)
}
}
for _, subject := range []string{"mesh.>", "_INBOX.controller.>", "$KV.>"} {
if _, err := snap.SubscribeSync(subject); err != nil {
t.Fatal(err)
}
_ = snap.Flush()
select {
case e := <-errs:
if !strings.Contains(strings.ToLower(e.Error()), "permissions violation") {
t.Errorf("subscribing %s: %v", subject, e)
}
case <-time.After(2 * time.Second):
t.Errorf("the bus let it subscribe %s", subject)
}
}
if info, err := js.StreamInfo("EVENTS"); err != nil || info == nil {
t.Fatalf("the stream is gone: %v", err)
}
if e, err := kv.Get("act"); err != nil || string(e.Value()) != "done by hand" {
t.Fatalf("the bucket changed: %v", err)
}
}
+75
View File
@@ -0,0 +1,75 @@
package broker
import (
"slices"
"testing"
)
// The bus's own module copies the bus and does nothing else on it (novox/hq ADR 0235): its whole
// authority is the snapshot API and its own inbox, whatever else it declared — its tools are the
// node's runtime's to serve.
func TestTheBussOwnModuleMaySnapshotAndNothingElse(t *testing.T) {
p := Principal{Kind: KindModule, Node: "anchor", Module: "nats", SnapshotsTheBus: true,
Serves: []string{"nats_streams"}, PasswordHash: "x"}
perms, err := PermissionsFor(p)
if err != nil {
t.Fatal(err)
}
want := []string{"$JS.API.STREAM.INFO.*", "$JS.API.STREAM.NAMES", "$JS.API.STREAM.SNAPSHOT.*", "$JS.SNAPSHOT.ACK.>"}
if !slices.Equal(perms.Publish, want) {
t.Errorf("it may publish %v, want exactly %v", perms.Publish, want)
}
if !slices.Equal(perms.Subscribe, []string{"_INBOX.anchor.nats.>"}) {
t.Errorf("it may subscribe %v, want its own inbox alone", perms.Subscribe)
}
if perms.AllowResponses {
t.Error("nothing is asked of it, and it may answer")
}
}
// Read-only, by the writers table: no grant of it overlaps a subject a write of the mesh's state is a
// publish to — a stream's definition, a bucket, a message.
func TestTheSnapshotGrantsWriteNothing(t *testing.T) {
p := Principal{Kind: KindModule, Node: "anchor", Module: "nats"}
if err := CheckWriters(p, BusSnapshotGrants.Publish); err != nil {
t.Fatal(err)
}
for _, grant := range BusSnapshotGrants.Publish {
for _, write := range []string{"$JS.API.STREAM.CREATE.X", "$JS.API.STREAM.UPDATE.X", "$JS.API.STREAM.DELETE.X",
"$JS.API.STREAM.PURGE.X", "$JS.API.STREAM.MSG.DELETE.X", "$JS.API.STREAM.RESTORE.X",
"$JS.API.CONSUMER.CREATE.X", "$JS.API.CONSUMER.DURABLE.CREATE.X.y", "$KV.b.k", "mesh.mod.m.event.e"} {
if SubjectsOverlap(grant, write) {
t.Errorf("%s would let the bus's own module publish %s", grant, write)
}
}
}
}
// A module that is not the bus gets nothing of it, and the flag travels from the records to the user.
func TestOnlyTheBussOwnModuleIsGrantedTheSnapshot(t *testing.T) {
users, err := Users(Records{Nodes: []string{"anchor"}, Assigned: map[string][]Declared{"anchor": {
{Module: "nats", SnapshotsTheBus: true},
{Module: "shop", Emits: []string{"order.placed"}},
}}})
if err != nil {
t.Fatal(err)
}
for _, u := range users {
perms, err := PermissionsFor(u)
if err != nil {
t.Fatal(err)
}
snapshots := slices.Contains(perms.Publish, "$JS.API.STREAM.SNAPSHOT.*")
switch u.Username() {
case "anchor.nats":
if !snapshots {
t.Error("the bus's own module is not granted the snapshot")
}
case "controller":
default:
if snapshots {
t.Errorf("%s may snapshot the bus", u.Username())
}
}
}
}
+3 -2
View File
@@ -160,8 +160,9 @@ func RaiseBuckets(a BucketAsserter, buckets []Bucket) (undeclared []string, err
return nil, fmt.Errorf("listing the bus's state: %w", err)
}
for _, n := range names {
// A seat's cancelled set is the mesh's own (novox/hq ADR 0219), not a module's state.
if !declared[n] && !IsCancelledSet(n) {
// A seat's cancelled set is the mesh's own (novox/hq ADR 0219), not a module's state; so are
// the controller's own buckets (novox/hq to-be 45 §1).
if !declared[n] && !IsCancelledSet(n) && !IsControllerBucket(n) {
undeclared = append(undeclared, n)
}
}
+10 -2
View File
@@ -6,6 +6,7 @@ import (
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
)
@@ -20,12 +21,19 @@ func TestTheFactsTheGrantPermitsAreTheFactsTheMeshStates(t *testing.T) {
t.Fatalf("the grant is written for the %q seat and the mesh states its facts under %q",
broker.ControllerSeat, link.MeshControllerSeat)
}
for _, event := range []string{link.KeyApplied, link.KeyRefused, link.KeyBuiltBefore} {
// And what is wrong, as it changes, and the self-check's heartbeat (novox/hq to-be 45 §2, §4).
states := append([]string{link.KeyApplied, link.KeyRefused, link.KeyBuiltBefore}, conditions.Events...)
states = append(states, conditions.HeartbeatEvent)
// And a value given by hand, replaced after its module's first good start (novox/hq ADR 0228).
states = append(states, link.KeySecretReplaced)
// And every act a healer takes (novox/hq to-be 45 §7).
states = append(states, link.KeyHealerActed)
for _, event := range states {
if !slices.Contains(broker.ControllerStates, event) {
t.Errorf("the mesh states %q and its account may not publish it", event)
}
}
if len(broker.ControllerStates) != 3 {
if len(broker.ControllerStates) != len(states) {
t.Errorf("the grant permits %v, which is more than the mesh states", broker.ControllerStates)
}
// **And the seat says it.** A seat carries the protocol of its role (novox/hq ADR 0129), so the
+50 -2
View File
@@ -201,7 +201,28 @@ const ControllerName = "controller"
// something to say.
const ControllerSeat = "mesh-controller"
var ControllerStates = []string{"applied", "refused", "built-before"}
var ControllerStates = []string{"applied", "refused", "built-before",
// What is wrong, said as it changes (novox/hq to-be 45 §2): a condition raised, changed in
// severity, resolver or silence, and cleared. The operator-channel's holder and any other surface
// consume them; the controller tells nobody itself.
"condition-raised", "condition-changed", "condition-cleared",
// And the self-check's heartbeat, at the end of every run (to-be 45 §4, S10): watched from a
// machine that is not the control node, so the controller going quiet is itself said.
"doctor-heartbeat",
// And a value given by hand, replaced after its module's first good start (novox/hq ADR 0228).
"secret-replaced",
// And every act a healer takes on a condition (novox/hq to-be 45 §7, Phase 3): a repair the mesh
// made by itself is said like one a person made, never quietly.
"healer-acted"}
// BusAdvisories are what the bus server says about the mesh's own account that the controller
// reads (novox/hq to-be 45 §3, S9): a durable consumer that handed a message over as often as it
// may and gave up on it, and one that was deleted. Read-only: an advisory is the server's to
// publish, and the controller's subscription changes nothing on the bus.
var BusAdvisories = []string{
"$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>",
"$JS.EVENT.ADVISORY.CONSUMER.DELETED.>",
}
// ControllerFollows are the events the controller reacts to: the catalogue saying a module's
// current version moved, and a catalogue that has just started saying it may have missed builds.
@@ -226,8 +247,28 @@ var ControllerFollows = []string{
// registers build-agent itself comes from there. Appended, for the same reason as above; goes
// with the retired seat row.
seatEventSubject("mesh-build-machine", "built"),
// **Every provider's standing** (novox/hq ADR 0224): a consumer it has failed for minutes, and
// that consumer recovered. The one pattern on this list, and a narrow one — two named events,
// from whichever module provides — because the rule is about every provider, and a list of
// providers here would be a list somebody forgets to extend. On 2026-10-05 the identity provider
// failed every consumer for a day and only its journal said so (issue 179). Appended, because
// the index is a name.
moduleEventSubject("*", ProvisionerFailing),
moduleEventSubject("*", ProvisionerRecovered),
// **What becomes of a consumer the mesh stopped asking for** (novox/hq ADR 0230): retired, waiting
// for a person, re-enabled, deleted — a provider's third word, from whichever module provides.
// Appended, because the index is a name.
moduleEventSubject("*", ProvisionerRetirement),
}
// The provider standing events, by their local names. Written here as well as in the catalogue
// (catalogue.ProvisionerEvents), which this package cannot import; a test keeps them agreeing.
const (
ProvisionerFailing = "provisioner.failing"
ProvisionerRecovered = "provisioner.recovered"
ProvisionerRetirement = "provisioner.retirement"
)
// moduleEventSubject is where one module's event lands. The same derivation PermissionsFor uses, so
// what the controller subscribes and what the emitter is permitted to publish cannot drift apart.
func moduleEventSubject(module, event string) string {
@@ -271,7 +312,14 @@ func MeshConsumers() []Consumer {
// client and come back to be acted on again.
MaxAckPending: 1,
FromNow: true,
Why: "the two events the mesh's own controller reacts to, one at a time; after " +
// **The one consumer the mesh resets by itself** (healer H4, issue 248): it fell a week
// behind once and held every merge after it. What a reset drops is caught up elsewhere —
// a merge by the catch-up pass that reads the forge (issue 266), a build's outcome from the
// build records a plan settles from (issue 214), a provider's failing word by the provider
// saying it again every quarter of an hour (ADR 0224).
Resettable: "what it drops is caught up: merges by the catch-up pass (issue 266), build outcomes " +
"from the build records (issue 214), a provider's failing word said again (ADR 0224)",
Why: "the events the mesh's own controller reacts to, one at a time; after " +
"max-deliver it dead-letters, because an announcement it cannot act on will not " +
"become actionable",
},
+7 -3
View File
@@ -24,8 +24,8 @@ accounts {
jetstream: enabled
users = [
{ user: "controller", password: "$2a$11$cccccccccccccccccccccc", permissions: {
publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "_INBOX.enrol.>", "mesh.assignment.>", "mesh.control.>", "mesh.mod.*.tool.>", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>", "mesh.seat.mesh-build-machine.tool.>", "mesh.seat.mesh-controller.event.applied", "mesh.seat.mesh-controller.event.built-before", "mesh.seat.mesh-controller.event.refused", "mesh.seat.node-build-agent.accept.>", "mesh.seat.node-build-agent.tool.>"] }
subscribe: { allow: ["$JS.API.>", "$SRV.INFO", "$SRV.INFO.mesh-controller", "$SRV.INFO.mesh-controller.>", "$SRV.PING", "$SRV.PING.mesh-controller", "$SRV.PING.mesh-controller.>", "$SRV.STATS", "$SRV.STATS.mesh-controller", "$SRV.STATS.mesh-controller.>", "_DELIVER.controller", "_DELIVER.controller.>", "_INBOX.controller.>", "mesh.control.>", "mesh.mod.gitea.event.pull.merged", "mesh.mod.mesh-catalog.event.catching-up", "mesh.mod.mesh-catalog.event.upgraded", "mesh.seat.mesh-build-machine.event.built", "mesh.seat.mesh-controller.tool.>", "mesh.seat.node-build-agent.event.built"] }
publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>", "$KV.SEAT_NODE_BUILD_AGENT_cancelled.>", "$KV.mesh-controller_calls.>", "$KV.mesh-controller_condition-history.>", "$KV.mesh-controller_conditions.>", "$KV.mesh-controller_hand-acts.>", "$KV.mesh-controller_lease.>", "$SRV.INFO", "_INBOX.enrol.>", "mesh.assignment.>", "mesh.mod.*.tool.>", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>", "mesh.seat.mesh-build-machine.tool.>", "mesh.seat.mesh-controller.event.applied", "mesh.seat.mesh-controller.event.built-before", "mesh.seat.mesh-controller.event.condition-changed", "mesh.seat.mesh-controller.event.condition-cleared", "mesh.seat.mesh-controller.event.condition-raised", "mesh.seat.mesh-controller.event.doctor-heartbeat", "mesh.seat.mesh-controller.event.healer-acted", "mesh.seat.mesh-controller.event.refused", "mesh.seat.mesh-controller.event.secret-replaced", "mesh.seat.node-backup.tool.backed-up.*", "mesh.seat.node-build-agent.accept.>", "mesh.seat.node-build-agent.tool.>", "mesh.seat.node-intrusion-prevention.tool.banned.*"] }
subscribe: { allow: ["$JS.API.>", "$JS.EVENT.ADVISORY.CONSUMER.DELETED.>", "$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>", "$SRV.INFO", "$SRV.INFO.mesh-controller", "$SRV.INFO.mesh-controller.>", "$SRV.PING", "$SRV.PING.mesh-controller", "$SRV.PING.mesh-controller.>", "$SRV.STATS", "$SRV.STATS.mesh-controller", "$SRV.STATS.mesh-controller.>", "_DELIVER.controller", "_DELIVER.controller.>", "_INBOX.controller.>", "mesh.control.>", "mesh.mod.*.event.provisioner.failing", "mesh.mod.*.event.provisioner.recovered", "mesh.mod.*.event.provisioner.retirement", "mesh.mod.gitea.event.pull.merged", "mesh.mod.mesh-catalog.event.catching-up", "mesh.mod.mesh-catalog.event.upgraded", "mesh.seat.mesh-build-machine.event.built", "mesh.seat.mesh-controller.tool.>", "mesh.seat.node-build-agent.event.built"] }
allow_responses: { max: 1, ttl: "1m" }
} }
{ user: "enrol.one", password: "$2a$11$eeeeeeeeeeeeeeeeeeeeee", permissions: {
@@ -34,7 +34,11 @@ accounts {
} }
{ user: "node.one", password: "$2a$11$nnnnnnnnnnnnnnnnnnnnnn", permissions: {
publish: { allow: ["$JS.ACK.NODES.one.>", "$JS.API.CONSUMER.INFO.NODES.one", "mesh.control.one.>"] }
subscribe: { allow: ["_DELIVER.one", "_DELIVER.one.>", "_INBOX.node.one.>", "mesh.node.one.declare"] }
subscribe: { allow: ["_DELIVER.one", "_DELIVER.one.>", "_INBOX.node.one.>", "mesh.node.one.ask.report", "mesh.node.one.declare"] }
} }
{ user: "one.nats", password: "$2a$11$bbbbbbbbbbbbbbbbbbbbbb", permissions: {
publish: { allow: ["$JS.API.STREAM.INFO.*", "$JS.API.STREAM.NAMES", "$JS.API.STREAM.SNAPSHOT.*", "$JS.SNAPSHOT.ACK.>"] }
subscribe: { allow: ["_INBOX.one.nats.>"] }
} }
{ user: "one.telegram", password: "$2a$11$tttttttttttttttttttttt", permissions: {
publish: { allow: ["$JS.ACK.EVENTS.one_telegram.>", "$JS.ACK.SEAT_TELEGRAM_SENDER.SEAT_TELEGRAM_SENDER_worker.>", "$JS.API.CONSUMER.INFO.EVENTS.one_telegram", "$JS.API.CONSUMER.INFO.SEAT_TELEGRAM_SENDER.SEAT_TELEGRAM_SENDER_worker", "$JS.API.CONSUMER.MSG.NEXT.EVENTS.one_telegram", "$JS.API.CONSUMER.MSG.NEXT.SEAT_TELEGRAM_SENDER.SEAT_TELEGRAM_SENDER_worker", "$JS.API.DIRECT.GET.ASSIGNMENTS.mesh.assignment.one.telegram", "mesh.seat.telegram-sender.event.delivered", "mesh.seat.telegram-sender.event.failed"] }
+4 -1
View File
@@ -41,6 +41,9 @@ type Declared struct {
// delivered to it and nothing can connect as it (novox/hq issue 195). Said in the negative so a
// record that does not say is composed as it always was.
NoAccount bool
// SnapshotsTheBus says the module holds mesh-broker — it is the bus — and so is the one module
// granted the snapshot API, to copy the bus's streams for the night's backup (novox/hq ADR 0235).
SnapshotsTheBus bool
}
// Records is what composing a user list needs to know about the mesh, and nothing more.
@@ -98,7 +101,7 @@ func Users(r Records) ([]Principal, error) {
Kind: KindModule, Node: node, Module: d.Module,
Emits: d.Emits, Consumes: d.Consumes, Serves: d.Serves,
Holds: d.Holds, Uses: d.Uses, Watches: d.Watches, Invokes: d.Invokes,
State: stateNames(d.State), Reads: d.Reads,
State: stateNames(d.State), Reads: d.Reads, SnapshotsTheBus: d.SnapshotsTheBus,
})
}
if runtimeHere {
+172
View File
@@ -0,0 +1,172 @@
package broker
import (
"fmt"
"slices"
"strings"
)
// The writers table (novox/hq to-be 45 §1, ADR 0227 rule 1), compiled in.
//
// **Every kind of state the core keeps has one writer; anyone else asks it.** The table is the design's,
// row for row, with what each row writes on the bus where it writes there. Two things read it: the
// composition of every principal's grants (PermissionsFor), which **refuses a grant that lets a
// principal publish on a subject another writes** — so a second writer cannot be granted by accident,
// and the composition says which state and whose — and the test beside it, which walks the rows
// against the design's list and the composition of a whole mesh. Changing a writer is a change to
// this table, through a decision.
// WriterRow is one row of the writers table.
type WriterRow struct {
State string
Writer string
KeptIn string
Others string
// Subjects are the bus subjects a write of this state is a publish to; none for state kept off the
// bus (the controller's store, a module's own) or not built yet.
Subjects []string
// Writes says a principal granted a publish pattern overlapping Subjects is this state's writer —
// given the pattern, because a machine writes its own report and no other's.
Writes func(p Principal, pattern string) bool
// Shared says why more than one principal may publish here; empty for one writer.
Shared string
}
// isController, and the other writers' tests, are who a row's writer is as a principal.
func isController(p Principal, _ string) bool { return p.Kind == KindController }
// ownMachine is a node writing a subject whose third token is its own name, and no wildcard there.
func ownMachine(p Principal, pattern string) bool {
tokens := strings.Split(pattern, ".")
return p.Kind == KindNode && len(tokens) > 2 && tokens[2] == p.Node
}
// holdsSeatOf is a principal holding the seat a `mesh.seat.<seat>.…` pattern names: its module's own
// principal, or the node tools carrying a module that holds it (ADR 0175, the runtime is its modules).
func holdsSeatOf(p Principal, pattern string) bool {
tokens := strings.Split(pattern, ".")
if len(tokens) < 3 {
return false
}
seat := tokens[2]
holds := func(seats []Seat) bool {
return slices.ContainsFunc(seats, func(s Seat) bool { return s.Name == seat })
}
switch p.Kind {
case KindModule:
return holds(p.Holds)
case KindNodeTools:
return slices.ContainsFunc(p.Carries, func(d Declared) bool { return holds(d.Holds) })
}
return false
}
// ownModule is a module publishing under its own name, or the node tools carrying it.
func ownModule(p Principal, pattern string) bool {
tokens := strings.Split(pattern, ".")
if len(tokens) < 3 {
return false
}
module := tokens[2]
switch p.Kind {
case KindModule:
return p.Module == module
case KindNodeTools:
return slices.ContainsFunc(p.Carries, func(d Declared) bool { return d.Module == module })
}
return false
}
// kvOf is a bucket's write subjects.
func kvOf(bucket string) []string { return []string{"$KV." + bucket + ".>"} }
// WritersTable is to-be 45 §1, in its order.
var WritersTable = []WriterRow{
{State: "a machine's declaration", Writer: "controller (lease holder)", KeptIn: "the bus, last per subject",
Others: "read", Subjects: []string{"mesh.node.*.declare"}, Writes: isController},
{State: "a machine's applied state and its report", Writer: "the node-engine's apply queue",
KeptIn: "the machine; the report on the bus", Others: "the reconcile and a delivery enqueue, never apply",
Subjects: []string{"mesh.control.*.report"}, Writes: ownMachine},
{State: "the controller lease", Writer: "the controller instance holding it", KeptIn: "key-value " + LeaseBucket,
Others: "a candidate waits", Subjects: kvOf(LeaseBucket), Writes: isController},
{State: "plans and their tiers", Writer: "controller (lease holder), compare-and-set on the plan's revision",
KeptIn: "the controller's store", Others: "read through plans"},
{State: "conditions", Writer: "controller", KeptIn: "key-value " + ConditionsBucket + " (and its history, " +
ConditionHistoryBucket + ")", Others: "raise or clear only through observations the controller reads",
Subjects: append(kvOf(ConditionsBucket), kvOf(ConditionHistoryBucket)...), Writes: isController},
{State: "calls and their outcomes", Writer: "controller", KeptIn: "key-value " + CallsBucket,
Others: "read by id", Subjects: kvOf(CallsBucket), Writes: isController},
{State: "the hand-act log", Writer: "controller, through the verbs that act", KeptIn: "key-value " + HandActsBucket,
Others: "—", Subjects: kvOf(HandActsBucket), Writes: isController},
// What the healers did (novox/hq to-be 45 §7, Phase 3): the controller's alone, in its store — the
// budgets and the mesh-wide brake are counted from it, so a controller restarting cannot reset them.
{State: "the healers' acts and their brake", Writer: "controller (lease holder), each act begun before it is made",
KeptIn: "the controller's store", Others: "read through healers; each act said as healer-acted and in its condition's tried"},
{State: "stream definitions and bus permissions", Writer: "controller", KeptIn: "the bus", Others: "—",
// A stream's definition, and a durable consumer's by the API that names it so. Not every
// consumer create: a module watching its own bucket makes and deletes an ordered consumer on the
// bucket's stream (ADR 0201), which defines nothing the mesh keeps.
Subjects: []string{"$JS.API.STREAM.CREATE.>", "$JS.API.STREAM.UPDATE.>", "$JS.API.STREAM.DELETE.>",
"$JS.API.CONSUMER.DURABLE.CREATE.>"},
Writes: isController},
{State: "builds and their outcomes", Writer: "the build seat's holder", KeptIn: "its own state",
Others: "the controller asks",
Subjects: []string{"mesh.seat.node-build-agent.event.built", "mesh.seat.mesh-build-machine.event.built"},
Writes: holdsSeatOf,
Shared: "every machine holding the build seat answers the asks it took; each outcome names its ask"},
{State: "a merge announced", Writer: "one announcer per forge (the hook, or the poll when the hook is absent — never both)",
KeptIn: "the bus", Others: "—", Subjects: []string{"mesh.mod.*.event.pull.merged"}, Writes: ownModule},
{State: "a provider's standing", Writer: "the provider", KeptIn: "the provider's events",
Others: "the controller keeps the newest word as a condition",
Subjects: []string{"mesh.mod.*.event.provisioner.failing", "mesh.mod.*.event.provisioner.recovered",
"mesh.mod.*.event.provisioner.retirement"},
Writes: ownModule},
{State: "the operator-channel's open messages", Writer: "the seat's holder", KeptIn: "its own key-value state",
Others: "—"},
{State: "the facts snapshot", Writer: "controller", KeptIn: "the artifact store, facts/latest",
Others: "the build seat reads"},
}
// CheckWriters refuses a grant that lets a principal publish on a subject the writers table gives
// another writer (to-be 45 §1): which state, whose, and the pattern that would make a second writer.
func CheckWriters(p Principal, publish []string) error {
var problems []string
for _, pattern := range publish {
for _, row := range WritersTable {
if row.Writes == nil {
continue
}
for _, subject := range row.Subjects {
if !SubjectsOverlap(pattern, subject) || row.Writes(p, pattern) {
continue
}
problems = append(problems, fmt.Sprintf("%s may publish %s, which writes %s (%s), whose writer is %s",
p.Username(), pattern, row.State, subject, row.Writer))
}
}
}
if len(problems) == 0 {
return nil
}
return fmt.Errorf("a second writer would be granted (novox/hq to-be 45 §1, one writer per piece of state):\n %s",
strings.Join(problems, "\n "))
}
// SubjectsOverlap says some subject matches both patterns: `*` is one token, `>` one or more to the end.
func SubjectsOverlap(a, b string) bool {
x, y := strings.Split(a, "."), strings.Split(b, ".")
for i := 0; ; i++ {
switch {
case i == len(x) && i == len(y):
return true
case i == len(x) || i == len(y):
return false
case x[i] == ">" || y[i] == ">":
return true
case x[i] == "*" || y[i] == "*" || x[i] == y[i]:
continue
default:
return false
}
}
}
+144
View File
@@ -0,0 +1,144 @@
package broker
import (
"slices"
"strings"
"testing"
)
// The writers table, checked (novox/hq to-be 45 §1, ADR 0227 rule 1, "how it is checked"): it is the
// design's table row for row; every row that writes on the bus names its writer; a whole mesh composes
// with one writer per piece of state; and a grant that would make a second writer is refused, naming
// the state and whose it is.
// designRows are the states of to-be 45 §1, in its order. A row added to the design is added here and
// to the table; one dropped from either fails.
var designRows = []string{
"a machine's declaration",
"a machine's applied state and its report",
"the controller lease",
"plans and their tiers",
"conditions",
"calls and their outcomes",
"the hand-act log",
"the healers' acts and their brake",
"stream definitions and bus permissions",
"builds and their outcomes",
"a merge announced",
"a provider's standing",
"the operator-channel's open messages",
"the facts snapshot",
}
func TestTheWritersTableIsTheDesigns(t *testing.T) {
var states []string
for _, row := range WritersTable {
states = append(states, row.State)
if row.Writer == "" || row.KeptIn == "" || row.Others == "" {
t.Errorf("%q does not say who writes it, where it is kept and what others do", row.State)
}
if len(row.Subjects) > 0 && row.Writes == nil {
t.Errorf("%q is written on the bus and names no writer among the principals", row.State)
}
}
if !slices.Equal(states, designRows) {
t.Fatalf("the writers table is not to-be 45 §1's:\n have %q\n want %q", states, designRows)
}
}
// A mesh of every kind of principal composes: nobody is granted a second writer's subject.
func TestAWholeMeshComposesWithOneWriterPerState(t *testing.T) {
builder := Seat{Name: "node-build-agent", Scope: "node", Accepts: []string{"build"}, Emits: []string{"built", "started"}}
principals := []Principal{
{Kind: KindController, PasswordHash: "x"},
{Kind: KindNode, Node: "one", PasswordHash: "x"},
{Kind: KindEnrolment, Node: "two", PasswordHash: "x"},
{Kind: KindPerson, Module: "jochen", Invokes: []string{"*"}, PasswordHash: "x"},
{Kind: KindModule, Node: "one", Module: "gitea", Emits: []string{"pull.merged"}, PasswordHash: "x"},
{Kind: KindModule, Node: "one", Module: "postgres", Emits: []string{"provisioner.failing", "provisioner.recovered", "provisioner.retirement"},
PasswordHash: "x"},
{Kind: KindModule, Node: "one", Module: "build-agent", Holds: []Seat{builder}, PasswordHash: "x"},
{Kind: KindNodeTools, Node: "one", Module: RuntimeModule, PasswordHash: "x", Carries: []Declared{
{Module: "gitea", Emits: []string{"pull.merged"}},
{Module: "build-agent", Holds: []Seat{builder}}}},
}
for _, p := range principals {
if _, err := PermissionsFor(p); err != nil {
t.Errorf("%s does not compose: %v", p.Username(), err)
}
}
if _, err := ComposeAccounts(principals); err != nil {
t.Fatalf("the mesh does not compose: %v", err)
}
}
func TestASecondWriterIsRefusedAtComposition(t *testing.T) {
for _, c := range []struct {
name string
p Principal
publish []string
state string
}{
{"the controller publishing what machines say", Principal{Kind: KindController},
[]string{"mesh.control.>"}, "a machine's applied state and its report"},
{"a machine publishing another's report", Principal{Kind: KindNode, Node: "one"},
[]string{"mesh.control.two.report"}, "a machine's applied state and its report"},
{"a machine publishing every machine's", Principal{Kind: KindNode, Node: "one"},
[]string{"mesh.control.*.>"}, "a machine's applied state and its report"},
{"a module writing the lease", Principal{Kind: KindModule, Module: "shop"},
[]string{"$KV.mesh-controller_lease.>"}, "the controller lease"},
{"a module sending a declaration", Principal{Kind: KindModule, Module: "shop"},
[]string{"mesh.node.one.declare"}, "a machine's declaration"},
{"a module defining a stream", Principal{Kind: KindModule, Module: "shop"},
[]string{"$JS.API.>"}, "stream definitions and bus permissions"},
{"a module announcing another forge's merge", Principal{Kind: KindModule, Module: "shop"},
[]string{"mesh.mod.gitea.event.pull.merged"}, "a merge announced"},
{"a module saying a build it did not do", Principal{Kind: KindModule, Module: "shop"},
[]string{"mesh.seat.node-build-agent.event.built"}, "builds and their outcomes"},
} {
t.Run(c.name, func(t *testing.T) {
err := CheckWriters(c.p, c.publish)
if err == nil {
t.Fatalf("%v granted to %s was not refused", c.publish, c.p.Username())
}
if !strings.Contains(err.Error(), c.state) {
t.Fatalf("the refusal does not name %q: %v", c.state, err)
}
})
}
// And the writers themselves are not refused.
for _, ok := range []struct {
p Principal
publish []string
}{
{Principal{Kind: KindNode, Node: "one"}, []string{"mesh.control.one.>"}},
{Principal{Kind: KindController}, []string{"mesh.node.>", "$JS.API.>", "$KV.mesh-controller_lease.>"}},
{Principal{Kind: KindModule, Module: "gitea"}, []string{"mesh.mod.gitea.event.pull.merged"}},
{Principal{Kind: KindModule, Module: "shop"}, []string{"$JS.API.CONSUMER.CREATE.KV_shop_carts.>"}},
} {
if err := CheckWriters(ok.p, ok.publish); err != nil {
t.Errorf("a writer was refused its own: %v", err)
}
}
}
func TestSubjectsOverlap(t *testing.T) {
for _, c := range []struct {
a, b string
want bool
}{
{"mesh.control.>", "mesh.control.*.report", true},
{"mesh.control.one.>", "mesh.control.*.report", true},
{"mesh.control.one.alive", "mesh.control.*.report", false},
{"mesh.control.*", "mesh.control.*.report", false},
{"$JS.API.>", "$JS.API.STREAM.CREATE.>", true},
{"$JS.API.CONSUMER.CREATE.KV_x.>", "$JS.API.STREAM.CREATE.>", false},
{"a.b", "a.b", true},
{"a.b", "a.b.c", false},
{"a.>", "a", false},
} {
if got := SubjectsOverlap(c.a, c.b); got != c.want || SubjectsOverlap(c.b, c.a) != c.want {
t.Errorf("%s ~ %s: %v, want %v", c.a, c.b, got, c.want)
}
}
}
+8 -33
View File
@@ -5,35 +5,10 @@ import (
"testing"
)
// A store provider says how its data is backed up (novox/hq ADR 0214); a provider of something that
// holds nothing does not have to.
func TestAStoreProviderWithoutABackupIsRefusedByTheCheck(t *testing.T) {
bare, err := ParseManifest([]byte(`{"module":"pg","version":"1",
"provides":[{"name":"postgres-database","scope":"mesh"}]}`))
if err != nil {
t.Fatalf("parsing refused it, and the rule is the check's: %v", err)
}
if problems := CheckBackup(bare); len(problems) != 1 || !strings.Contains(problems[0], "node-backup") {
t.Fatalf("a store with no backup passed the check: %v", problems)
}
backed, err := ParseManifest([]byte(`{"module":"pg","version":"1",
"provides":[{"name":"postgres-database","scope":"mesh"}],
"resources":[{"id":"dumps","type":"directory","mode":"0700"}],
"contributions":[{"seat":"node-backup","kind":"backup","content":"path ${dir:dumps}"}]}`))
if err != nil {
t.Fatal(err)
}
if problems := CheckBackup(backed); len(problems) != 0 {
t.Fatalf("a store that contributes a backup was refused: %v", problems)
}
route, _ := ParseManifest([]byte(`{"module":"r","version":"1","provides":[{"name":"route","scope":"mesh"}]}`))
if problems := CheckBackup(route); len(problems) != 0 {
t.Fatalf("a route was asked for a backup: %v", problems)
}
}
// A backup contribution names its module's own directories; one it does not declare is refused
// where it is written rather than reaching the holder as the literal text.
// A backup contribution written by hand still names its module's own directories; one it does not
// declare is refused where it is written rather than reaching the holder as the literal text. (The
// catalogue check refuses a hand-written backup line at all since ADR 0233; parsing still reads one,
// because a module on the shelf was written before.)
func TestABackupNamingAnUndeclaredDirectoryIsRefused(t *testing.T) {
_, err := ParseManifest([]byte(`{"module":"pg","version":"1",
"contributions":[{"seat":"node-backup","kind":"backup","content":"path ${dir:nowhere}"}]}`))
@@ -42,9 +17,9 @@ func TestABackupNamingAnUndeclaredDirectoryIsRefused(t *testing.T) {
}
}
// The holder receives every module's backup lines with each module's own directories filled, each
// module's under a comment naming it — and a kind written in a shell's grammar is left untouched.
func TestBackupContributionsArePlacedWithTheirModulesDirectories(t *testing.T) {
// A module on the shelf from before ADR 0233, which writes its backup lines by hand and declares no
// data, is still backed up: its lines are placed as written, each module's under a comment naming it.
func TestHandWrittenBackupLinesOfAnOlderModuleArePlaced(t *testing.T) {
pg, err := ParseManifest([]byte(`{"module":"pg","version":"1",
"resources":[{"id":"dumps","type":"directory","path":"/srv/pg/dumps","mode":"0700"}],
"contributions":[{"seat":"node-backup","kind":"backup","content":"run pg-dump-all\npath ${dir:dumps}"}]}`))
@@ -57,7 +32,7 @@ func TestBackupContributionsArePlacedWithTheirModulesDirectories(t *testing.T) {
if err != nil {
t.Fatal(err)
}
placed, err := seatContributions([]Manifest{pg, mail}, BackupSeat, "backup", Rendering{})
placed, err := seatContributions([]Manifest{pg, mail}, BackupSeat, "backup", Rendering{}, nil)
if err != nil {
t.Fatal(err)
}
+183
View File
@@ -0,0 +1,183 @@
package catalogue
import (
"fmt"
"slices"
"strings"
)
// A consumer of a provision that keeps its data stays bound where its data is (novox/hq ADR 0232).
//
// **What a resolution chooses is not where a consumer's data is.** Resolving answers "which provider
// would I pick now", from the seats' holders, the pins and what is assigned where, and every one of
// those can change under a consumer without anybody meaning to move it. For a resolver that is the
// point: any provider answers alike. For a database it is the consumer's whole state: on 2026-10-05
// one change to how a seat's holder answers (issue 258) re-bound five database consumers on one
// machine to the store on another, each was given a fresh, empty database there, and nothing warned
// for twenty hours (issue 273). Nothing was lost only because the old provider kept everything.
//
// So the mesh records where each such consumer was last sent (the store's `binding` table), and a
// resolution that would answer it from anywhere else keeps the recorded provider instead and says so.
// **Only a pin moves it**, because a pin is a person: the one act that says "answer this machine's
// consumers from there", taken knowing the data must go first.
// KeptBinding is one consumer this resolution would have moved, and did not.
type KeptBinding struct {
// Machine is where the consumer runs, and Consumer the module there that is bound.
Machine string
Consumer string
// Provision is what it is bound for.
Provision string
// Bound is the provider it was recorded at, and still is; Would is the one the resolution chose.
Bound Chosen
Would Chosen
}
// String is the condition's sentence: the move, where the data is, and the act that confirms it.
func (k KeptBinding) String() string {
return fmt.Sprintf("would move %s's %s from %s to %s — its data is on %s; kept there. "+
"`pin %s %s %s %s` to confirm a move (and move the data first)",
k.Consumer, k.Provision, k.Bound, k.Would, k.Bound, k.Machine, k.Provision, k.Would.Node, k.Would.Module)
}
// keepBound holds every need for a provision that keeps its consumers' data at the provider its
// consumer was bound to, where the resolution chose another.
//
// A need with no record is a binding being made, and is left as resolved: it is recorded when it is
// first sent. A pin naming the provider the resolution chose is a person moving it, and is left too.
// Otherwise the recorded provider answers, if it still provides the provision — beside the consumer,
// or offered from elsewhere — and the move is returned as kept. **One that no longer does is refused,
// never answered by the provider chosen**: answering it there is exactly the silent move this exists
// to stop, and the consumer's data is still wherever it was.
func keepBound(needs []Needed, catalogue map[string]Manifest, node Node, world World,
keeps map[string]bool, here func(string) bool) ([]Needed, []KeptBinding, []string) {
var kept []KeptBinding
var problems []string
refused := map[[2]string]bool{}
out := make([]Needed, 0, len(needs))
for _, n := range needs {
if n.ByRecord || !keeps[n.Name] {
out = append(out, n)
continue
}
n.KeepsData = true
bound, recorded := world.Bound[n.For][n.Name]
chose := Chosen{Node: n.From, Module: n.Module}
if !recorded || sameProvider(bound, chose) {
out = append(out, n)
continue
}
if pin, pinned := world.Pinned[n.Name]; pinned && pin.matches(Provider{Node: chose.Node, Module: chose.Module}) {
out = append(out, n)
continue
}
held, ok, why := boundNeed(n, bound, catalogue, node, world, here)
if !ok {
if key := [2]string{n.For, n.Name}; !refused[key] {
refused[key] = true
problems = append(problems, fmt.Sprintf(
"%s on %s is bound to %s for %q, which keeps its data, and %s — it would move to %s, "+
"which holds none of it. Move its data and say so with `pin %s %s %s %s`, or give "+
"%s back what it provided",
n.For, node.Name, bound, n.Name, why, chose, node.Name, n.Name, chose.Node, chose.Module,
bound.Node))
}
continue
}
out = append(out, held)
kept = append(kept, KeptBinding{Machine: node.Name, Consumer: n.For, Provision: n.Name, Bound: bound, Would: chose})
}
return out, kept, problems
}
// sameProvider is whether a recorded provider is the one chosen. A record naming no module (none
// is written without one, but a person's hand might) matches any module on its node.
func sameProvider(bound, chose Chosen) bool {
return bound.Node == chose.Node && (bound.Module == "" || bound.Module == chose.Module)
}
// boundNeed is the need answered by the recorded provider, or why it cannot be.
func boundNeed(n Needed, bound Chosen, catalogue map[string]Manifest, node Node, world World,
here func(string) bool) (Needed, bool, string) {
held := Needed{Name: n.Name, For: n.For, Local: n.Local, KeepsData: true}
if bound.Node == node.Name {
m, known := catalogue[bound.Module]
if !known || !here(bound.Module) || !providesAt(m, n.Name, ScopeMesh) {
return Needed{}, false, fmt.Sprintf("%s no longer runs on %s", bound.Module, node.Name)
}
at := node.At
if at == "" {
at = "127.0.0.1"
}
held.From, held.At, held.Module = node.Name, at, m.Module
held.Serves, held.SharedOwn, held.Identity = servedByOne(m, n.Name), sharedByOne(m, n.Name), m.IdentityBoundOf(n.Name)
return held, true, ""
}
matching := bound.among(world.Offered[n.Name])
if len(matching) != 1 {
return Needed{}, false, fmt.Sprintf("%s no longer provides it", bound)
}
p := matching[0]
if node.At == "" || p.At == "" {
return Needed{}, false, fmt.Sprintf("%s and %s are not both on the private network", node.Name, p.Node)
}
identity := DefaultIdentityBound
if pm, known := catalogue[p.Module]; known {
held.SharedOwn, _ = pm.SharedCredentialOf(n.Name)
identity = pm.IdentityBoundOf(n.Name)
}
held.From, held.At, held.Module, held.Serves, held.Identity = p.Node, p.At, p.Module, p.Serves, identity
return held, true, ""
}
// Unbound is one consumer that still asks for a provision and whose own resolution binds it to
// another provider than the one a pair credential on record was made with (novox/hq issue 274).
//
// **A provider is granted exactly the consumers bound to it.** The credential from the old provider
// stays on record — it is the key to a login that provider keeps, disabled, with the consumer's data,
// until a person deletes it with `cleanup delete` (ADR 0230), and a pin back must find it — but it is
// no longer granted, so the provider stops being asked for it and retires it. Said on every plan and
// push of the provider, so a credential the mesh keeps and does not use is never kept silently.
type Unbound struct {
// Provision is what was required, Provider the machine the credential on record is from.
Provision string `json:"provision"`
Provider string `json:"provider"`
// Consumer is the machine, Module the module on it that requires it, Local the credential's
// name inside it where it keeps several (ADR 0094).
Consumer string `json:"consumer"`
Module string `json:"module"`
Local string `json:"local,omitempty"`
// BoundTo is every provider the consumer's resolution binds this credential to now; empty when
// it binds it nowhere — the module asks for the provision under other local names.
BoundTo []string `json:"bound_to,omitempty"`
}
func (u Unbound) String() string {
who := u.Module
if u.Local != "" {
who += " (as " + u.Local + ")"
}
now := "is bound to no provider under that name"
if len(u.BoundTo) > 0 {
now = "is bound to " + strings.Join(u.BoundTo, ", ")
}
return fmt.Sprintf("%s on %s %s for %s, not to %s — %s no longer grants it, so it retires that "+
"login and keeps its data until `cleanup delete` (ADR 0230); the credential from %s stays on "+
"record while that login does", who, u.Consumer, now, u.Provision, u.Provider, u.Provider, u.Provider)
}
// BindsFrom is the providers this resolution binds a pair credential's need to — the provision, the
// module that requires it and the credential's local name — answered by a machine rather than by a
// record. None means this machine states no such binding.
func (r Resolution) BindsFrom(provision, module, local string) []string {
var from []string
for _, n := range r.Needs {
if n.ByRecord || n.Name != provision || n.For != module || n.Local != local {
continue
}
if !slices.Contains(from, n.From) {
from = append(from, n.From)
}
}
return from
}
+257
View File
@@ -0,0 +1,257 @@
package catalogue
import (
"encoding/json"
"strings"
"testing"
)
// A consumer of a provision that keeps its data stays bound where its data is (novox/hq ADR 0232,
// issue 273).
//
// The incident, exactly: a machine runs its own store and five consumers of it; the mesh's store seat
// is held by the store on another machine. Issue 258's rule — the seat's holder elsewhere answers
// before this machine's own provider — re-bound all five to the holder, each was made a fresh, empty
// database there, and nothing said so.
var incidentConsumers = []string{"board", "listings", "workflows", "agents", "game"}
func storeMesh() map[string]Manifest {
shelf := map[string]Manifest{
"store": {Module: "store", Version: "1",
Provides: []Offer{{Name: "postgres-database", Scope: ScopeMesh}},
Claims: []Claim{{Name: "mesh-store", Scope: ScopeMesh}},
Serves: map[string]map[string]any{"postgres-database": {"port": 5432}},
Grants: map[string]string{"postgres-database": "/grants"}},
"resolver": {Module: "resolver", Version: "1",
Provides: []Offer{{Name: "wildcard-resolution", Scope: ScopeMesh}},
Claims: []Claim{{Name: "mesh-dns-resolver", Scope: ScopeMesh}}},
"network": {Module: "network", Version: "1", Requires: []string{"wildcard-resolution"}},
}
for _, c := range incidentConsumers {
shelf[c] = Manifest{Module: c, Version: "1", Requires: []string{"postgres-database"}}
}
return shelf
}
// storeWorld is the rest of the mesh as the home server's plan sees it: the anchor runs a store and a
// resolver and holds both seats; the home server runs its own of each.
func storeWorld() World {
w := World{Offered: map[string][]Provider{
"postgres-database": {
{Node: "anchor", At: "anchor.internal", Module: "store", Serves: map[string]any{"port": 5432}},
{Node: "home", At: "home.internal", Module: "store", Serves: map[string]any{"port": 5434}},
},
"wildcard-resolution": {
{Node: "anchor", At: "anchor.internal", Module: "resolver"},
{Node: "home", At: "home.internal", Module: "resolver"},
},
}}
w.Held = []Held{
{Claim: "mesh-store", Scope: ScopeMesh, Node: "anchor", Module: "store"},
{Claim: "mesh-dns-resolver", Scope: ScopeMesh, Node: "anchor", Module: "resolver"},
}
w.Holdings = w.Held
return w
}
var home = Node{Name: "home", At: "home.internal"}
func homeAssigned() []string {
return append([]string{"store", "resolver", "network"}, incidentConsumers...)
}
func boundFrom(t *testing.T, r Resolution, consumer, provision string) Needed {
t.Helper()
for _, n := range r.Needs {
if n.For == consumer && n.Name == provision {
return n
}
}
t.Fatalf("no binding of %s for %s: %+v", consumer, provision, r.Needs)
return Needed{}
}
func TestTheIncidentAMachinesOwnStoreKeepsItsConsumersWhenTheSeatIsHeldElsewhere(t *testing.T) {
got, err := Resolve(storeMesh(), homeAssigned(), home, storeWorld())
if err != nil {
t.Fatal(err)
}
for _, c := range incidentConsumers {
if n := boundFrom(t, got, c, "postgres-database"); n.From != "home" || n.Module != "store" {
t.Errorf("%s bound to %s/%s; its data is on the store beside it (issue 273)", c, n.From, n.Module)
}
}
// And the resolver, which keeps nothing of anybody's, still answers from the seat's holder (258).
if n := boundFrom(t, got, "network", "wildcard-resolution"); n.From != "anchor" {
t.Errorf("the resolver bound to %s; the seat is held on anchor (issue 258)", n.From)
}
if len(got.Kept) != 0 {
t.Errorf("nothing was moved, and %d kept: %+v", len(got.Kept), got.Kept)
}
}
func TestTheIncidentWithEveryConsumerOnRecordStillMovesNothing(t *testing.T) {
w := storeWorld()
w.Bound = map[string]map[string]Chosen{}
for _, c := range incidentConsumers {
w.Bound[c] = map[string]Chosen{"postgres-database": {Node: "home", Module: "store"}}
}
got, err := Resolve(storeMesh(), homeAssigned(), home, w)
if err != nil {
t.Fatal(err)
}
for _, c := range incidentConsumers {
if n := boundFrom(t, got, c, "postgres-database"); n.From != "home" {
t.Errorf("%s bound to %s", c, n.From)
}
}
if len(got.Kept) != 0 {
t.Errorf("kept %+v", got.Kept)
}
}
func TestAPinElsewhereStillMovesAStoresConsumers(t *testing.T) {
w := storeWorld()
w.Pinned = map[string]Chosen{"postgres-database": {Node: "anchor", Module: "store"}}
w.Bound = map[string]map[string]Chosen{"board": {"postgres-database": {Node: "home", Module: "store"}}}
got, err := Resolve(storeMesh(), homeAssigned(), home, w)
if err != nil {
t.Fatal(err)
}
if n := boundFrom(t, got, "board", "postgres-database"); n.From != "anchor" || n.Module != "store" {
t.Errorf("bound to %s/%s; a person pinned it to anchor", n.From, n.Module)
}
if len(got.Kept) != 0 {
t.Errorf("a pin is a person's move, not one to keep: %+v", got.Kept)
}
}
// A consumer on a machine with no store of its own, bound to one store, when the seat moves to
// another: the holder would answer, and the binding stays.
func laptopWorld(holder string) World {
w := storeWorld()
w.Held = []Held{{Claim: "mesh-store", Scope: ScopeMesh, Node: holder, Module: "store"}}
w.Holdings = w.Held
return w
}
var laptop = Node{Name: "laptop", At: "laptop.internal"}
func TestABoundConsumerStaysWhenTheSeatsHolderChanges(t *testing.T) {
w := laptopWorld("home")
w.Bound = map[string]map[string]Chosen{"board": {"postgres-database": {Node: "anchor", Module: "store"}}}
got, err := Resolve(storeMesh(), []string{"board"}, laptop, w)
if err != nil {
t.Fatal(err)
}
n := boundFrom(t, got, "board", "postgres-database")
if n.From != "anchor" || n.At != "anchor.internal" || n.Serves["port"] != 5432 {
t.Fatalf("bound to %s at %s %v; its data is on anchor", n.From, n.At, n.Serves)
}
if len(got.Kept) != 1 {
t.Fatalf("the move was not said: %+v", got.Kept)
}
k := got.Kept[0]
if k.Bound != (Chosen{"anchor", "store"}) || k.Would != (Chosen{"home", "store"}) || k.Consumer != "board" {
t.Fatalf("kept %+v", k)
}
for _, want := range []string{"would move board's postgres-database from anchor/store to home/store",
"its data is on anchor/store", "pin laptop postgres-database home store", "move the data first"} {
if !strings.Contains(k.String(), want) {
t.Errorf("%q does not say %q", k.String(), want)
}
}
// Without a record it is a binding being made, and the holder answers it as before.
w.Bound = nil
got, err = Resolve(storeMesh(), []string{"board"}, laptop, w)
if err != nil {
t.Fatal(err)
}
if n := boundFrom(t, got, "board", "postgres-database"); n.From != "home" {
t.Errorf("a new consumer bound to %s; the seat is held on home", n.From)
}
}
func TestABoundConsumerWhoseProviderIsGoneIsRefusedNotMoved(t *testing.T) {
w := laptopWorld("anchor")
w.Offered["postgres-database"] = w.Offered["postgres-database"][:1] // only anchor's store is left
w.Bound = map[string]map[string]Chosen{"board": {"postgres-database": {Node: "home", Module: "store"}}}
_, err := Resolve(storeMesh(), []string{"board"}, laptop, w)
if err == nil {
t.Fatal("bound to home's store, which is gone, and answered by anchor's — the silent move")
}
for _, want := range []string{"board on laptop is bound to home/store", "home/store no longer provides it",
"pin laptop postgres-database anchor store"} {
if !strings.Contains(err.Error(), want) {
t.Errorf("%q does not say %q", err, want)
}
}
}
func TestAMachinesOwnStoreUnassignedRefusesItsBoundConsumers(t *testing.T) {
w := storeWorld()
w.Bound = map[string]map[string]Chosen{"board": {"postgres-database": {Node: "home", Module: "store"}}}
_, err := Resolve(storeMesh(), []string{"board"}, home, w)
if err == nil || !strings.Contains(err.Error(), "store no longer runs on home") {
t.Fatalf("got %v", err)
}
}
// The first pass asks only what a machine offers, and is never refused by a binding.
func TestTheFirstPassKeepsNoBinding(t *testing.T) {
w := storeWorld()
w.Unchecked = true
w.Bound = map[string]map[string]Chosen{"board": {"postgres-database": {Node: "gone", Module: "store"}}}
if _, err := Resolve(storeMesh(), []string{"board"}, laptop, w); err != nil {
t.Fatal(err)
}
}
func TestAnOfferSaysWhetherItKeepsConsumerData(t *testing.T) {
var o Offer
if err := json.Unmarshal([]byte(`{"name":"wildcard-resolution","scope":"mesh","keeps-consumer-data":false}`), &o); err != nil {
t.Fatal(err)
}
if o.KeepsConsumerData == nil || *o.KeepsConsumerData {
t.Fatalf("read %+v", o)
}
out, err := json.Marshal(o)
if err != nil || !strings.Contains(string(out), `"keeps-consumer-data":false`) {
t.Fatalf("wrote %s, %v", out, err)
}
yes, no := true, false
granting := Manifest{Module: "g", Provides: []Offer{{Name: "p", Scope: ScopeMesh}}, Grants: map[string]string{"p": "/g"}}
if !granting.KeepsConsumerData("p") {
t.Error("a provider granting each consumer a credential keeps what it writes, unsaid")
}
if (Manifest{Module: "r", Provides: []Offer{{Name: "p", Scope: ScopeMesh}}}).KeepsConsumerData("p") {
t.Error("a provider granting nothing keeps nothing, unsaid")
}
granting.Provides[0].KeepsConsumerData = &no
if granting.KeepsConsumerData("p") {
t.Error("what an offer says, it gets")
}
said := Manifest{Module: "s", Provides: []Offer{{Name: "p", Scope: ScopeMesh, KeepsConsumerData: &yes}}}
if !said.KeepsConsumerData("p") || !KeepsConsumerData(map[string]Manifest{"s": said, "r": {Module: "r",
Provides: []Offer{{Name: "p", Scope: ScopeMesh, KeepsConsumerData: &no}}}}, "p") {
t.Error("a name any provider says keeps data keeps data")
}
}
// An offer saying it keeps nothing is answered by the seat's holder as the resolver is, even where it
// grants a credential.
func TestAStoreSayingItKeepsNothingFollowsTheSeat(t *testing.T) {
shelf := storeMesh()
no := false
s := shelf["store"]
s.Provides = []Offer{{Name: "postgres-database", Scope: ScopeMesh, KeepsConsumerData: &no}}
shelf["store"] = s
got, err := Resolve(shelf, homeAssigned(), home, storeWorld())
if err != nil {
t.Fatal(err)
}
if n := boundFrom(t, got, "board", "postgres-database"); n.From != "anchor" {
t.Errorf("bound to %s; it keeps nothing, and the seat is held on anchor", n.From)
}
}
+67
View File
@@ -0,0 +1,67 @@
package catalogue
import (
"os"
"strings"
"testing"
)
// The module holding mesh-broker is the bus, and its account is granted the bus's snapshot API and
// nothing else (novox/hq ADR 0235). Anything it declared to say or hear on the bus would be granted
// nothing, so it is refused at the parser rather than silently dropped.
func TestTheBussAccountSaysNothingOnTheBus(t *testing.T) {
raw := []byte(`{"module":"bus","version":"1","provides":[{"name":"mesh-bus","scope":"mesh"}],
"claims":[{"name":"mesh-broker","scope":"mesh"}],"own-secrets":{"broker":"/run/broker"},
"emits":["something.happened"]}`)
_, err := ParseManifest(raw)
if err == nil || !strings.Contains(err.Error(), "granted the bus's snapshot API and nothing else") {
t.Fatalf("a bus module declaring what it emits was not refused, or not for the reason: %v", err)
}
quiet := []byte(`{"module":"bus","version":"1","provides":[{"name":"mesh-bus","scope":"mesh"}],
"claims":[{"name":"mesh-broker","scope":"mesh"}],"own-secrets":{"broker":"/run/broker"},
"tools":["bus_streams"]}`)
m, err := ParseManifest(quiet)
if err != nil {
t.Fatalf("a bus module with an account and tools was refused: %v", err)
}
if !m.ClaimsSeat(BrokerSeat) {
t.Fatal("it does not read as holding the broker seat")
}
}
// The catalogue's bus protects its streams by the snapshot, not as live files: its streams' item is a
// dump into its snapshots, it has the account the dump runs as, and the dump runs the snapshot
// program in the bus's own container.
func TestTheCataloguesBusIsBackedUpBySnapshot(t *testing.T) {
raw, err := os.ReadFile("../../../mesh-catalog/modules/nats/module.json")
if err != nil {
t.Skip("the catalogue is not checked out beside this repository")
}
m, err := ParseManifest(raw)
if err != nil {
t.Fatal(err)
}
if _, account := m.OwnSecrets["broker"]; !account {
t.Fatal("the bus declares no account, so its snapshot could not reach it")
}
var found bool
if m.Data == nil {
t.Fatal("the bus declares no data")
}
for _, it := range m.Data.Own {
if it.ID != "jetstream" {
continue
}
found = true
if !it.Backup.IsDump() || it.Backup.Into != "snapshots" {
t.Fatalf("the bus's streams are protected by %+v, not a snapshot into its snapshots", it.Backup)
}
if !strings.Contains(it.Backup.Dump, "mesh-nats-snapshot snapshot") {
t.Fatalf("the dump does not run the snapshot program: %s", it.Backup.Dump)
}
}
if !found {
t.Fatal("the bus declares no item for its streams")
}
}
+53 -14
View File
@@ -87,15 +87,18 @@ func TestNoCatalogueManifestNamesAnInstallation(t *testing.T) {
}
}
// TestEveryCatalogueStoreSaysHowItIsBackedUp is ADR 0214's check over the real catalogue: a module
// providing a store contributes a backup to node-backup, so a store added is a store backed up.
func TestEveryCatalogueStoreSaysHowItIsBackedUp(t *testing.T) {
// TestEveryCatalogueModuleDeclaresItsData is ADR 0233's check over the real catalogue (it replaced ADR
// 0214's store list): every provider that grants says what it keeps for its consumers, nothing writes a
// backup line by hand, every directory a container writes is declared, and every irreplaceable item is
// backed up — so a store added is a store backed up, without a list of stores to keep in step.
func TestEveryCatalogueModuleDeclaresItsData(t *testing.T) {
root := catalogueRoot(t)
found, err := filepath.Glob(filepath.Join(root, "modules", "*", "module.json"))
if err != nil || len(found) == 0 {
t.Fatalf("no manifests under %s: %v", root, err)
}
stores := 0
shelf := Shelf{}
granting := 0
for _, p := range found {
raw, err := os.ReadFile(p)
if err != nil {
@@ -105,17 +108,53 @@ func TestEveryCatalogueStoreSaysHowItIsBackedUp(t *testing.T) {
if err != nil {
continue // TestEveryCatalogueManifestParses says why
}
for _, o := range m.Provides {
if storeProvisions[o.Name] {
stores++
break
}
}
for _, problem := range CheckBackup(m) {
t.Error(problem)
if len(m.Grants) > 0 {
granting++
}
shelf[m.Module] = m
}
if stores == 0 {
t.Fatal("no module in the catalogue provides a store, so this proved nothing")
for _, problem := range DataProblems(shelf) {
t.Error(problem)
}
if granting == 0 {
t.Fatal("no module in the catalogue grants a provision, so this proved nothing")
}
}
// TestEveryCatalogueIdentityFitsWhatItRequires is ADR 0225's check over the real catalogue: every
// module's identity, on the longest machine name, fits the bound of every provision it wants. An
// overflow fails here, in the pull request that introduces it, rather than on the provider's machine
// the first time a real machine's name meets the module's (issue 263).
func TestEveryCatalogueIdentityFitsWhatItRequires(t *testing.T) {
root := catalogueRoot(t)
found, err := filepath.Glob(filepath.Join(root, "modules", "*", "module.json"))
if err != nil || len(found) == 0 {
t.Fatalf("no manifests under %s: %v", root, err)
}
shelf := Shelf{}
for _, p := range found {
raw, err := os.ReadFile(p)
if err != nil {
t.Fatalf("%s: %v", p, err)
}
m, err := ParseManifest(raw)
if err != nil {
t.Fatalf("%s: %v", p, err)
}
shelf[m.Module] = m
}
for _, p := range IdentityProblems(shelf, DefaultLongestMachine) {
t.Error(p)
}
// The night it was found: the resolver provision bounds nothing, and the object store still 20.
if dns, ok := shelf["dnsmasq"]; ok {
if b := dns.IdentityBoundOf("wildcard-resolution"); b.Bounded() {
t.Errorf("the resolver provision bounds its consumers' identities: %+v", b)
}
}
if store, ok := shelf["minio"]; ok {
if b := store.IdentityBoundOf("s3-bucket"); b.Max != 20 {
t.Errorf("the object store's access key is not bounded at 20: %+v", b)
}
}
}
+2 -1
View File
@@ -159,7 +159,8 @@ func TestTwoWaysToBeOnAPrivateNetworkRefuseAndNameBoth(t *testing.T) {
}
}
// reloading answers the runtime's trust as the networking module does (novox/hq ADR 0102): a file
// reloading answers the runtime's trust as the networking module once did (novox/hq ADR 0102; it no
// longer does, ADR 0222 — and beside the runtime's module it is refused, generated_collision_test): a file
// written into, and the runtime reloaded on it.
type reloading struct{}
+23 -1
View File
@@ -114,7 +114,9 @@ func consumerInto(value, as string) (string, error) {
// asDNSLabel writes a minted identity as a DNS label.
//
// The mesh's identities are already lower-case letters, digits and `_` (ConsumerIdentity), and
// already short enough for the tightest backend they reach (CheckIdentity, twenty characters). So
// already inside the bound of the provision serving them: a provider that serves one as a DNS label
// bounds it at 63 or less (CheckServes, novox/hq ADR 0225), and one that serves it at all without
// saying is held to twenty (IdentityBoundOf). So
// this is the separator and nothing else — no lower-casing of what is already lower case, no
// truncation to a limit the identity is already inside, no padding of a name that is already long
// enough. Each of those would be the mesh guessing at a rule it has not been given.
@@ -141,11 +143,31 @@ func CheckServes(m Manifest) []string {
problems = append(problems, fmt.Sprintf(
"%s serves %s, and the value it serves as %q %s", m.Module, provision, key, err))
}
// A label longer than DNS keeps is not truncated here (asDNSLabel), so the offer's own
// bound has to keep the identity inside one (ADR 0225).
if strings.Contains(text, "${consumer:as:dns}") {
if b := m.IdentityBoundOf(provision); !b.Bounded() || b.Max > dnsLabelLimit {
problems = append(problems, fmt.Sprintf(
"%s serves %s's consumers their identity as a DNS label in %q, and bounds "+
"that identity at %s: a label keeps %d — state an `identity` of at most %d",
m.Module, provision, key, boundWords(b), dnsLabelLimit, dnsLabelLimit))
}
}
}
}
return problems
}
// dnsLabelLimit is the longest DNS label (RFC 1035 §2.3.4).
const dnsLabelLimit = 63
func boundWords(b IdentityBound) string {
if !b.Bounded() {
return "nothing"
}
return fmt.Sprintf("%d", b.Max)
}
func sortedServes(serves map[string]map[string]any) []string {
out := make([]string, 0, len(serves))
for k := range serves {
+791
View File
@@ -0,0 +1,791 @@
package catalogue
import (
"bytes"
"encoding/json"
"fmt"
"regexp"
"sort"
"strconv"
"strings"
"time"
)
// A module declares the data it holds, and the mesh protects and watches it from that declaration
// (novox/hq ADR 0233).
//
// **One section, `data`, for every kind of data a module keeps:** its own, by directory — a store's
// files, a mail spool, an application's uploads — or by an operator's path it was given; what it keeps
// for its consumers, by the provision they reach it through; and what of its own lives with a
// provider, by the provision it requires. Each entry has a class — how precious it is — and, for its
// own data, how it is protected; everything the mesh does is derived from those, never written per
// module:
//
// - a binding to a consumer's data does not move (ADR 0232) where the provision's class keeps data;
// - the backup holder's lines are composed from it — a module no longer writes them by hand;
// - what an unassignment leaves behind of an irreplaceable or valuable item is retired, listed by
// `cleanup list` and deleted only by `cleanup delete` (ADR 0230);
// - the self-check measures every item and says when one shrinks, disappears, stops being written,
// goes without its backup, sits on a degraded array, or is replaced by an empty copy of itself —
// urgent for what is irreplaceable, a warning for what is valuable.
// The classes: how precious the data is. A fixed vocabulary, ranked by the operator (2026-10-06): a
// class is what the protections are derived from, so a new one is a decision, not a manifest's choice.
const (
// ClassIrreplaceable is what must never be lost: the operator names it (the media library, the
// photo sites' storage). Protected by a backup or by a declared redundancy — one is required —
// retired rather than removed, and every alert about it is urgent.
ClassIrreplaceable = "irreplaceable"
// ClassValuable is anybody's work that would be painful to lose: in the nightly backup by default,
// retired rather than removed, and every alert about it a warning.
ClassValuable = "valuable"
// ClassRebuildable can be made again from something kept elsewhere — a clone, an index, a
// download, a night's dump — at a cost in time, not in data. In the nightly backup by default;
// forgotten when its module goes, and never alerted on.
ClassRebuildable = "rebuildable"
// ClassCache may be emptied at any moment with nothing lost but speed: never backed up, never
// measured, never alerted on.
ClassCache = "cache"
// ClassNone is a provision that keeps nothing of its consumers' (consumers only): a resolver, a
// certificate authority, an artifact store.
ClassNone = "none"
)
// classRank orders the classes by how precious they are, so the stricter of two is chosen.
var classRank = map[string]int{ClassNone: 0, ClassCache: 1, ClassRebuildable: 2, ClassValuable: 3, ClassIrreplaceable: 4}
// StricterClass is the more precious of two classes.
func StricterClass(a, b string) string {
if classRank[b] > classRank[a] {
return b
}
return a
}
// Retires is whether a class's data is retired, not forgotten, when its machine no longer declares it.
func Retires(class string) bool { return class == ClassIrreplaceable || class == ClassValuable }
// Watched is whether a class's data raises conditions: irreplaceable and valuable.
func Watched(class string) bool { return Retires(class) }
// DefaultBackupWithin is how old the last good backup of an item may be before the self-check says so
// (novox/hq ADR 0214: a machine with data and no good backup in 48 hours).
const DefaultBackupWithin = 48 * time.Hour
// Data is a manifest's `data` section.
type Data struct {
// Own is the data this module keeps itself.
Own []DataItem `json:"own,omitempty"`
// Consumers is what it keeps for its consumers, per provision it grants.
Consumers map[string]ConsumerData `json:"consumers,omitempty"`
// KeptBy is what of its own lives with the provider of a provision it requires — its rows, its
// objects — and how precious that is. The provider's protections follow the stricter of its own
// class for its consumers and this.
KeptBy map[string]KeptData `json:"kept-by,omitempty"`
}
// DataItem is one piece of a module's own data.
type DataItem struct {
// ID names it within the module: what `data`, `cleanup` and a condition call it.
ID string `json:"id"`
// Path is one of the module's directories, `${dir:<id>}`, or an operator's path it was given,
// `${access:<id>}`, or a path beneath either. Never a machine path (novox/hq ADR 0112).
Path string `json:"path"`
Class string `json:"class"`
// Backup is how it is copied: "copy" (the holder reads it as it stands), "none", or a dump — a
// command that writes a consistent copy into another item, which is copied. Unsaid, anything but a
// cache is copied, unless it says it is protected by redundancy instead.
Backup *DataBackup `json:"backup,omitempty"`
// Redundancy says it is protected by the redundancy of the storage it lives on rather than by a
// copy, and why that is enough: the media library on an array there is no room to copy. The
// backup holder then watches that array, and a degraded one is said.
Redundancy string `json:"redundancy,omitempty"`
// Within is how old its last good backup may be; unsaid, DefaultBackupWithin.
Within string `json:"within,omitempty"`
// Measure is how the backup holder measures it: `walk` (the default — every file, at most daily and
// bounded, for small items), `dataset` (a ZFS dataset's own counters, hourly, nothing walked) or
// `shallow` (its top-level entries only, no size). A large item never says walk.
Measure string `json:"measure,omitempty"`
// Active is how long it may go unwritten before that is a fault — for data something is
// expected to write all the time. Unsaid, a quiet item is not a fault.
Active string `json:"active,omitempty"`
// Why is a line for the reviewer: why this class.
Why string `json:"why,omitempty"`
}
// ConsumerData is what a provider keeps for the consumers of one provision.
type ConsumerData struct {
Class string `json:"class"`
// In is where it lives: one of the module's own items (whose protection covers it), or a
// provision this module requires (whose provider keeps it, as its consumer). Unsaid only for none
// and cache.
In string `json:"in,omitempty"`
Why string `json:"why,omitempty"`
}
// KeptData is how precious what a module keeps with a provider is.
type KeptData struct {
Class string `json:"class"`
Why string `json:"why,omitempty"`
}
// DataBackup is how an item is copied.
type DataBackup struct {
Copy bool
None bool
// Dump is the command writing a consistent copy into the item Into.
Dump string
Into string
}
// UnmarshalJSON reads "copy", "none" or {"dump": "...", "into": "<item>"}.
func (b *DataBackup) UnmarshalJSON(raw []byte) error {
var word string
if err := json.Unmarshal(raw, &word); err == nil {
switch word {
case "copy":
*b = DataBackup{Copy: true}
case "none":
*b = DataBackup{None: true}
default:
return fmt.Errorf("a data item's backup is \"copy\", \"none\" or {dump, into}, not %q", word)
}
return nil
}
var full struct {
Dump string `json:"dump"`
Into string `json:"into"`
}
dec := json.NewDecoder(bytes.NewReader(raw))
dec.DisallowUnknownFields()
if err := dec.Decode(&full); err != nil {
return fmt.Errorf("a data item's backup is \"copy\", \"none\" or {dump, into}: %w", err)
}
*b = DataBackup{Dump: full.Dump, Into: full.Into}
return nil
}
// MarshalJSON writes it back in the form it was written.
func (b DataBackup) MarshalJSON() ([]byte, error) {
switch {
case b.Copy:
return json.Marshal("copy")
case b.None:
return json.Marshal("none")
}
return json.Marshal(struct {
Dump string `json:"dump"`
Into string `json:"into"`
}{b.Dump, b.Into})
}
// IsDump is whether the item is copied by a dump.
func (b *DataBackup) IsDump() bool { return b != nil && b.Dump != "" }
// BackedUp is whether the holder keeps restore points of this item: anything but a cache by default —
// the nightly backup is the standard plan — unless it says "none", or says it is protected by
// redundancy and says nothing of a backup.
func (it DataItem) BackedUp() bool {
switch {
case it.Backup == nil:
return it.Class != ClassCache && it.Redundancy == ""
case it.Backup.None:
return false
}
return true
}
// Protection is how the item is protected, in one word: "backup", "redundancy", both joined by "+",
// or "none".
func (it DataItem) Protection() string {
var by []string
if it.BackedUp() {
by = append(by, "backup")
}
if it.Redundancy != "" {
by = append(by, "redundancy")
}
if len(by) == 0 {
return "none"
}
return strings.Join(by, "+")
}
// The ways an item is measured.
const (
MeasureWalk = "walk"
MeasureDataset = "dataset"
MeasureShallow = "shallow"
)
// MeasuredBy is how the item is measured, with the default applied.
func (it DataItem) MeasuredBy() string {
if it.Measure == "" {
return MeasureWalk
}
return it.Measure
}
// OwnedByModule is whether the item is in one of the module's own directories — the mesh's to retire —
// rather than an operator's path it was given, which the mesh never retires and never deletes.
func (it DataItem) OwnedByModule() bool { return strings.HasPrefix(it.Path, "${dir:") }
// BackupWithin is the item's bound on its last good backup.
func (it DataItem) BackupWithin() time.Duration {
if d, err := ParseDataDuration(it.Within); err == nil && d > 0 {
return d
}
return DefaultBackupWithin
}
// ActiveWithin is how long the item may go unwritten; zero when quiet is not a fault.
func (it DataItem) ActiveWithin() time.Duration {
d, _ := ParseDataDuration(it.Active)
return d
}
// ParseDataDuration reads "48h", "90m" or "7d"; empty is zero.
func ParseDataDuration(s string) (time.Duration, error) {
s = strings.TrimSpace(s)
if s == "" {
return 0, nil
}
if days, ok := strings.CutSuffix(s, "d"); ok {
n, err := strconv.Atoi(days)
if err != nil || n <= 0 {
return 0, fmt.Errorf("%q is not a number of days", s)
}
return time.Duration(n) * 24 * time.Hour, nil
}
d, err := time.ParseDuration(s)
if err != nil || d <= 0 {
return 0, fmt.Errorf("%q is not a duration (48h, 90m, 7d)", s)
}
return d, nil
}
// DataItems is the module's own data, in the order declared.
func (m Manifest) DataItems() []DataItem {
if m.Data == nil {
return nil
}
return m.Data.Own
}
// DataItem is one own item by id.
func (m Manifest) DataItem(id string) (DataItem, bool) {
for _, it := range m.DataItems() {
if it.ID == id {
return it, true
}
}
return DataItem{}, false
}
// ConsumerDataOf is what this module says it keeps for a provision's consumers, and whether it says.
func (m Manifest) ConsumerDataOf(provision string) (ConsumerData, bool) {
if m.Data == nil {
return ConsumerData{}, false
}
c, ok := m.Data.Consumers[provision]
return c, ok
}
// KeptByOf is how precious what this module keeps with a provision's provider is, and whether it says.
func (m Manifest) KeptByOf(provision string) (KeptData, bool) {
if m.Data == nil {
return KeptData{}, false
}
k, ok := m.Data.KeptBy[provision]
return k, ok
}
// keepsByClass is whether a class keeps something a binding must not move away from.
func keepsByClass(class string) bool {
return class == ClassIrreplaceable || class == ClassValuable || class == ClassRebuildable
}
// dataID is what an item's id may be: it is a token in a condition's key and a word on a line.
var dataID = regexp.MustCompile(`^[a-z0-9][a-z0-9-]*$`)
// dataPathRef is an item's path: one of the module's directories or accesses, and optionally a path
// beneath it.
var dataPathRef = regexp.MustCompile(`^\$\{(dir|access):([a-z0-9][a-z0-9-]*)\}(/[^\s$]*)?$`)
// dataProblems is what is wrong with the `data` section itself, from the manifest alone: judged at
// registration as every other per-manifest problem is. Whether a module declares what it should is
// the catalogue check's (DataProblems), because a module already running was written before it.
func (m Manifest) dataProblems() []string {
if m.Data == nil {
return nil
}
var problems []string
say := func(format string, args ...any) {
problems = append(problems, fmt.Sprintf("%s's data: ", m.Module)+fmt.Sprintf(format, args...))
}
dirs := map[string]bool{}
for _, r := range m.Resources {
if fmt.Sprint(r["type"]) == "directory" {
dirs[fmt.Sprint(r["id"])] = true
}
}
accesses := map[string]bool{}
for _, a := range m.Accesses {
if a.ID != "" {
accesses[a.ID] = true
}
}
ids := map[string]DataItem{}
paths := map[string]string{}
for i, it := range m.Data.Own {
label := it.ID
if label == "" {
label = fmt.Sprintf("item %d", i+1)
}
if !dataID.MatchString(it.ID) {
say("%s has no usable id: lower-case letters, digits and dashes", label)
} else if _, twice := ids[it.ID]; twice {
say("%s is declared twice", it.ID)
}
ids[it.ID] = it
ref := dataPathRef.FindStringSubmatch(it.Path)
switch {
case ref == nil:
say("%s's path %q is not one of the module's directories or accesses: ${dir:<id>} or ${access:<id>}, "+
"or a path beneath one (a definition names no machine path, novox/hq ADR 0112)", label, it.Path)
case ref[1] == "dir" && !dirs[ref[2]]:
say("%s's path names ${dir:%s}, and %s declares no directory %q", label, ref[2], m.Module, ref[2])
case ref[1] == "access" && !accesses[ref[2]]:
say("%s's path names ${access:%s}, and %s declares no access %q", label, ref[2], m.Module, ref[2])
case strings.Contains(it.Path, ".."):
say("%s's path %q climbs out of its directory", label, it.Path)
}
if other, twice := paths[it.Path]; twice && it.Path != "" {
say("%s and %s name the same path %s", other, label, it.Path)
}
paths[it.Path] = label
switch it.Class {
case ClassIrreplaceable, ClassValuable, ClassRebuildable, ClassCache:
case ClassNone:
say("%s is class none, which only a provision keeping nothing of its consumers' is; own data "+
"that is disposable is cache", label)
default:
say("%s's class %q is not one the mesh protects by: irreplaceable, valuable, rebuildable or cache",
label, it.Class)
}
if it.Class == ClassIrreplaceable && !it.BackedUp() && strings.TrimSpace(it.Redundancy) == "" {
say("%s is irreplaceable and is neither backed up nor said to be protected by redundancy; the only "+
"copy of something is protected one way or the other (novox/hq ADR 0233) — copy it, dump it into "+
"another item, or say `redundancy` with why that is enough", label)
}
if it.Class == ClassCache && it.Backup != nil && !it.Backup.None {
say("%s is a cache and asks to be backed up; a cache is disposable, or it is not a cache", label)
}
if it.Class == ClassCache && it.Redundancy != "" {
say("%s is a cache and says it is protected by redundancy; a cache is not protected", label)
}
switch it.Measure {
case "", MeasureWalk, MeasureDataset, MeasureShallow:
default:
say("%s's measure %q is walk, dataset or shallow", label, it.Measure)
}
if _, err := ParseDataDuration(it.Within); err != nil {
say("%s's within: %v", label, err)
}
if _, err := ParseDataDuration(it.Active); err != nil {
say("%s's active: %v", label, err)
}
if it.Within != "" && !it.BackedUp() {
say("%s states within, and is not backed up", label)
}
if it.Active != "" && !Watched(it.Class) {
say("%s is %s and expects writes; only irreplaceable and valuable data is watched", label, it.Class)
}
}
// Dumps go into an item of the same module, declared, and not into themselves.
for _, it := range m.Data.Own {
if it.Backup == nil || it.Backup.Copy || it.Backup.None {
continue
}
switch into, ok := ids[it.Backup.Into]; {
case strings.TrimSpace(it.Backup.Dump) == "":
say("%s's backup names no dump command", it.ID)
case it.Backup.Into == "":
say("%s's dump says no item it writes into", it.ID)
case !ok:
say("%s's dump writes into %q, which is not one of the module's data items", it.ID, it.Backup.Into)
case into.ID == it.ID:
say("%s's dump writes into itself; a dump is copied from where it lands", it.ID)
case into.Class == ClassCache:
say("%s's dump writes into %s, a cache; what is copied must not be disposable", it.ID, into.ID)
}
if it.Backup.Dump != "" {
declared := map[string]string{}
for d := range dirs {
declared[d] = ""
}
if _, err := dirFill(it.Backup.Dump, declared, m.Module); err != nil {
say("%s's dump: %v", it.ID, err)
}
}
}
provided := map[string]bool{}
for _, o := range m.Provides {
provided[o.Name] = true
}
wants := map[string]bool{}
for _, w := range m.Wants() {
wants[w] = true
}
for _, provision := range sortedKeys(m.Data.Consumers) {
c := m.Data.Consumers[provision]
if !provided[provision] {
say("says what it keeps for the consumers of %q, which it does not provide", provision)
}
switch c.Class {
case ClassIrreplaceable, ClassValuable, ClassRebuildable, ClassCache, ClassNone:
default:
say("its consumers' %s is class %q; one of irreplaceable, valuable, rebuildable, cache or none", provision, c.Class)
}
switch {
case c.Class == ClassNone && c.In != "":
say("its consumers' %s keeps nothing and says where it is kept (%s)", provision, c.In)
case keepsByClass(c.Class) && c.In == "":
say("its consumers' %s is %s and says nowhere it lives: `in` names one of the module's items, or a "+
"provision it requires", provision, c.Class)
case c.In != "":
if own, ok := ids[c.In]; ok {
if c.Class == ClassIrreplaceable && !own.BackedUp() && own.Redundancy == "" {
say("its consumers' %s is irreplaceable and lives in %s, which is not protected", provision, c.In)
}
if classRank[own.Class] < classRank[c.Class] {
say("its consumers' %s is %s and lives in %s, which is only %s", provision, c.Class, c.In, own.Class)
}
} else if !wants[c.In] {
say("its consumers' %s lives in %q, which is neither one of its data items nor a provision it "+
"requires", provision, c.In)
}
}
}
for _, provision := range sortedKeys(m.Data.KeptBy) {
k := m.Data.KeptBy[provision]
if !wants[provision] {
say("says it keeps data with the provider of %q, which it does not require", provision)
}
switch k.Class {
case ClassIrreplaceable, ClassValuable, ClassRebuildable, ClassCache:
default:
say("what it keeps with %s is class %q; one of irreplaceable, valuable, rebuildable or cache", provision, k.Class)
}
}
return problems
}
// KeepsConsumerData is whether this module, providing a provision, keeps what each consumer writes
// there (novox/hq ADR 0232, ADR 0233): whether a consumer bound to it is bound to its data.
//
// **What the data section says, it gets**: irreplaceable, valuable or rebuildable keeps; cache and none
// do not — a cache lost in a move costs speed, not data. Before the section, an offer said it with
// `keeps-consumer-data`, which is still read; unsaid in both, a provider that grants each consumer a
// credential of its own keeps that consumer's data (ADR 0232 §1). The catalogue check refuses the
// unsaid case for a provider that grants (DataProblems), so the inference is what an older definition
// on the shelf gets, never a new one.
func (m Manifest) KeepsConsumerData(provision string) bool {
if c, ok := m.ConsumerDataOf(provision); ok {
return keepsByClass(c.Class)
}
for _, o := range m.Provides {
if o.Name == provision && o.KeepsConsumerData != nil {
return *o.KeepsConsumerData
}
}
_, grants := m.Grants[provision]
return grants
}
// NeedsBackupHolder is whether a module's data needs the machine's backup holder to be there: it keeps
// something irreplaceable — backed up by the holder, or protected by an array the holder watches. A
// module keeping only valuable or rebuildable data is backed up where a holder is, and refused nowhere
// for want of one: the standard plan, not a requirement.
func (m Manifest) NeedsBackupHolder() bool {
for _, it := range m.DataItems() {
if it.Class == ClassIrreplaceable {
return true
}
}
return false
}
// --- derived: the backup holder's lines ---------------------------------------------------------
// The node-backup seat's kinds (novox/hq to-be 43, ADR 0233): `backup` is what to run and which paths
// to keep, `data` every item with its class and protection, for the holder to measure and watch.
const (
BackupKindBackup = "backup"
BackupKindData = "data"
)
// derivedContributions is what a module's data section gives the backup holder: its backup lines and
// its items. Every item is listed, so a valuable one on a machine is measured whether or not it is
// copied; a cache is listed and not measured.
func (m Manifest) derivedContributions() []SeatContribution {
items := m.DataItems()
if len(items) == 0 {
return nil
}
var lines []string
kept := map[string]bool{}
keep := func(path string) {
if !kept[path] {
kept[path] = true
lines = append(lines, "path "+path)
}
}
for _, it := range items {
if !it.BackedUp() {
continue
}
if it.Backup.IsDump() {
lines = append(lines, "run "+strings.TrimSpace(it.Backup.Dump))
if into, ok := m.DataItem(it.Backup.Into); ok {
keep(into.Path)
}
continue
}
keep(it.Path)
}
var described []string
for _, it := range items {
covered := "-"
switch {
case it.Backup.IsDump():
if into, ok := m.DataItem(it.Backup.Into); ok {
covered = into.Path
}
case it.BackedUp():
covered = it.Path
}
described = append(described, fmt.Sprintf("item %s %s %s %s %s %s", it.ID, it.Class, it.Path, covered,
it.Protection(), it.MeasuredBy()))
}
var out []SeatContribution
if len(lines) > 0 {
out = append(out, SeatContribution{Seat: BackupSeat, Kind: BackupKindBackup, Content: strings.Join(lines, "\n") + "\n"})
}
return append(out, SeatContribution{Seat: BackupSeat, Kind: BackupKindData, Content: strings.Join(described, "\n") + "\n"})
}
// allContributions is every contribution a module makes to a seat's holder: what it wrote, and what
// its data section derives. **One list**: a module that declares its data has its backup lines
// composed from it, and a backup line it still writes by hand is not placed — the catalogue check
// refuses it — so the holder never copies two lists that disagree.
func (m Manifest) allContributions() []SeatContribution {
if m.Data == nil {
return m.Contributions
}
derived := m.derivedContributions()
out := make([]SeatContribution, 0, len(m.Contributions)+len(derived))
for _, c := range m.Contributions {
if s, known := SeatNamed(c.Seat); known && s.Name == BackupSeat {
continue
}
out = append(out, c)
}
return append(out, derived...)
}
// --- the catalogue check ------------------------------------------------------------------------
// DataProblems is what the catalogue check refuses about the data a module declares (novox/hq ADR
// 0233), judged over the manifests given together:
//
// - a provider that grants a provision says what it keeps for that provision's consumers;
// - nobody says it on the offer any more (`keeps-consumer-data`): the data section is the one place;
// - nobody writes a backup line by hand: it is derived from the data section;
// - a directory a container writes, mounted whole, is a data item of some class;
// - consumer data said to live in a required provision lives where that provision's provider keeps
// its consumers' data, as preciously, when the provider is given;
// - data a consumer keeps with a provider as irreplaceable is protected there, when the provider is
// given.
//
// **The check's, not registration's**, as ADR 0214's backup rule was: a module already on the shelf
// was written before the rule, and refusing it there would refuse the very providers whose data the
// rule protects. Each module meets it where it is written.
func DataProblems(shelf Shelf) []string {
var problems []string
for _, name := range shelfOrder(shelf) {
m := shelf[name]
for _, provision := range sortedKeys(m.Grants) {
if _, said := m.ConsumerDataOf(provision); !said {
problems = append(problems, fmt.Sprintf(
"%s grants %s and does not say what it keeps for its consumers: data.consumers.%s with a "+
"class — irreplaceable, valuable, rebuildable, cache, or none (novox/hq ADR 0233)", name, provision, provision))
}
}
for _, o := range m.Provides {
if o.KeepsConsumerData != nil {
problems = append(problems, fmt.Sprintf(
"%s says keeps-consumer-data on its offer of %s; it is said once, in data.consumers.%s, with a "+
"class (novox/hq ADR 0233)", name, o.Name, o.Name))
}
}
for i, c := range m.Contributions {
if s, known := SeatNamed(c.Seat); known && s.Name == BackupSeat {
problems = append(problems, fmt.Sprintf(
"%s's contribution %d is a backup line written by hand; backups are derived from the data "+
"section — declare the data, with its class and how it is protected (novox/hq ADR 0233)", name, i+1))
}
}
for _, dir := range writtenDirectories(m) {
if !coversDirectory(m, dir) {
problems = append(problems, fmt.Sprintf(
"%s mounts its directory %q into a container to be written, and declares no data in it: "+
"data.own with a class — cache if it is disposable (novox/hq ADR 0233)", name, dir))
}
}
if m.Data == nil {
continue
}
for _, provision := range sortedKeys(m.Data.Consumers) {
c := m.Data.Consumers[provision]
if c.In == "" {
continue
}
if _, own := m.DataItem(c.In); own {
continue
}
for _, pname := range shelfOrder(shelf) {
p := shelf[pname]
pc, said := p.ConsumerDataOf(c.In)
if !said || !providesName(p, c.In) {
continue
}
if classRank[pc.Class] < classRank[c.Class] {
problems = append(problems, fmt.Sprintf(
"%s keeps its consumers' %s, %s, in %s — and %s keeps its consumers' %s as %s, "+
"so it is not protected as %s", name, provision, c.Class, c.In, pname, c.In, pc.Class, c.Class))
}
}
}
for _, provision := range sortedKeys(m.Data.KeptBy) {
k := m.Data.KeptBy[provision]
if k.Class != ClassIrreplaceable {
continue
}
for _, pname := range shelfOrder(shelf) {
p := shelf[pname]
pc, said := p.ConsumerDataOf(provision)
if !said || !providesName(p, provision) {
continue
}
if !keepsByClass(pc.Class) {
problems = append(problems, fmt.Sprintf(
"%s keeps irreplaceable data with %s, and %s keeps its consumers' %s as %s — nothing of it is "+
"protected there", name, provision, pname, provision, pc.Class))
continue
}
if in, own := p.DataItem(pc.In); own && !in.BackedUp() && in.Redundancy == "" {
problems = append(problems, fmt.Sprintf(
"%s keeps irreplaceable data with %s, and %s keeps it in %s, which is neither backed up nor "+
"on declared redundancy", name, provision, pname, pc.In))
}
}
}
}
return problems
}
func providesName(m Manifest, provision string) bool {
for _, o := range m.Provides {
if o.Name == provision {
return true
}
}
return false
}
// writtenDirectories are the module's directories a container mounts whole and may write: a volume
// `${dir:<id>}:<target>` without `:ro`, or a directory stated by an absolute path mounted the same
// way. A file beneath a directory, or a read-only mount, is configuration the mesh wrote.
func writtenDirectories(m Manifest) []string {
absolute := map[string]string{}
for _, r := range m.Resources {
if fmt.Sprint(r["type"]) != "directory" {
continue
}
if p, ok := r["path"].(string); ok && strings.HasPrefix(p, "/") {
absolute[strings.TrimRight(p, "/")] = fmt.Sprint(r["id"])
}
}
seen := map[string]bool{}
for _, r := range m.Resources {
if fmt.Sprint(r["type"]) != "container" {
continue
}
vols, _ := r["volumes"].([]any)
for _, v := range vols {
s, _ := v.(string)
mount := volumeMount.FindStringSubmatch(s)
if mount == nil || volumeReadOnly(mount[3]) {
continue
}
src := strings.TrimRight(mount[1], "/")
if ref := dirPlain.FindStringSubmatch(src); ref != nil {
seen[ref[1]] = true
} else if id, ok := absolute[src]; ok {
seen[id] = true
}
}
}
out := make([]string, 0, len(seen))
for id := range seen {
out = append(out, id)
}
sort.Strings(out)
return out
}
// volumeMount is a container's volume, `<source>:<target>[:<options>]`, its source possibly a
// `${dir:<id>}` — whose own colon is not the separator.
var volumeMount = regexp.MustCompile(`^(\$\{(?:dir|access):[a-z0-9][a-z0-9-]*\}[^:]*|[^:]+):([^:]+)(?::(.*))?$`)
// volumeReadOnly is whether a volume's options mount it read-only.
func volumeReadOnly(options string) bool {
for _, o := range strings.Split(options, ",") {
if strings.TrimSpace(o) == "ro" {
return true
}
}
return false
}
// dirPlain is a whole directory, `${dir:<id>}` and nothing after it.
var dirPlain = regexp.MustCompile(`^\$\{dir:([a-z0-9][a-z0-9-]*)\}$`)
// coversDirectory is whether a data item names the directory, a path within it, or a directory it
// is placed beneath.
func coversDirectory(m Manifest, dir string) bool {
parents := map[string]string{}
for _, r := range m.Resources {
if fmt.Sprint(r["type"]) != "directory" {
continue
}
if p, ok := r["path"].(string); ok {
if ref := dirRef.FindStringSubmatch(p); ref != nil && strings.HasPrefix(p, "${dir:") {
parents[fmt.Sprint(r["id"])] = ref[1]
}
}
}
for _, it := range m.DataItems() {
ref := dataPathRef.FindStringSubmatch(it.Path)
if ref == nil || ref[1] != "dir" {
continue
}
for d, hops := dir, 0; d != "" && hops < 4; d, hops = parents[d], hops+1 {
if ref[2] == d {
return true
}
}
}
return false
}

Some files were not shown because too many files have changed in this diff Show More