Compare commits

...
Author SHA1 Message Date
jschoubben a81717e53c Seal the seeded account in the adapter-input test, the second fixture of its kind (issue 1945, PR #467)
mesh/delivery merged: source.main moved to novox/mesh-controller@fda4859a95b8
mesh/delivery-group group fix/1945-memberships merged: every member is merged
mesh/check checking a81717e53cf4 on 7cd4d18bd54e | novox/mesh-controller#468 in group fix/1945-memberships
The kind: a test that seeds a bus account without sealing it, which an account counted as issued only once sealed
(ADR 0543 §2) no longer counts. The whole integration level, run locally, found two: the addresses test (fixed
before) and TestTheAdaptersInputCarriesTheOwnSecretsTheBoxesTheLicencesAndTheBus. Test only.
2026-10-25 23:30:20 +01:00
jschoubben d01839c29c Grant the grants part reading every membership; seal the seeded account in the addresses test (ADR 0547 §5, review; issue 1945)
mesh/delivery-group group fix/1945-memberships rejected: composed together: the checker's verdict at the members' heads (2253ba8bbe9b): integration: github.com…
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: integration: github.com/novox/mesh-controller/cmd/mesh-controller.TestTheAdaptersInputCarriesTheOwnSecretsTheBoxesTheLicencesAndThe…
The grants part reads each module's last membership before it publishes, so it is granted the direct get on every
membership subject beside the publish (review of mesh-grants, B1).

The test TestTheControlPlaneIsToldWhereTheNodePutTheStoreAndTheBroker failed on main since PR #467: it seeds
anchor.mesh-controller with no seal, and an account counts as issued only once sealed (ADR 0543 §2). The rule is right;
the fixture is fixed, sealing the seeded credential the controller already uses. Test only.
2026-10-25 22:51:25 +01:00
jschoubben f67d15536b Grant the grants part publishing the modules' memberships, and nothing more (ADR 0547 §5, issue 1945)
No membership is published since release 6 (issue 1945 notes 42, 47), so a seat-holding module placed now cannot
serve its seat or pull asks (mesh-build-go on ace, 2026-10-25). The operator decided at 22:31 that the grants part
publishes them from the first milestone: its principal, module mesh-grants, may publish mesh.assignment.*.* and no
stream definition. The SDK's copy of this composer gets the same line, so the two stay byte-equal.

Research 072 (the controller, NATS-native): this moves the memberships of its Bus row to the grants part.
2026-10-25 22:32:35 +01:00
jschoubben 7cd4d18bd5 Merge pull request 'A bus account counts as issued only once sealed, stale accounts are forgotten, and grants name sealed accounts only (issue 1945 note 35)' (#467) from fix/1945-credential-when-row-without-seal into main
Reviewed-on: #467
2026-10-25 19:31:42 +00:00
jschoubben fbf14b68db Forget the accounts of modules no longer on their nodes at the start's look, and grant sealed accounts alone (issue 1945, ADR 0543, review)
mesh/delivery held for a person: merged without a passing check: only a person decides what follows
mesh/delivery-group group fix/1945-credential-when-row-without-seal held: a member is held: novox/mesh-controller@fbf14b68dbab (merged without a passing check:…
mesh/check checking fbf14b68dbab on 40034be479f8
Migration 0108 marks every standing account sealed, ace's stale mesh-build-go
rows from 17:19 among them, and the forgetting ran only at a later unassign:
the next assignment to ace would have issued nothing again.

- The start's look first forgets, from each node's own resolution (assigned
  and required), the accounts of every module that no longer resolves on its
  node, and says each; only from a reading that holds assignments, and a node
  whose resolution is unknown is left alone.
- Unassign forgets a module's accounts only once it no longer resolves on the
  node (a module another requires keeps running), and never the node's own
  provisioner account; the forgetting moved from the inventory to the act.
- Forgetting holds each account's lock across its delete, so an issue in flight
  never writes a row back after it.
- A module whose accounts are not all sealed is granted its sealed accounts
  alone, and its grant is dropped when none is: an older grant naming ace no
  longer lets the assigner want it.
- An issue's locked section has a deadline of two minutes, since it holds a
  connection of the store's pool across the vault and the bus.
Tests: cases in the existing tables.
Research 072: the grants piece (memberships and credentials heard from placement).
2026-10-25 20:27:30 +01:00
jschoubben 79ac62ffbc Issue an account whose credential is not sealed, and forget an unassigned module's accounts (issue 1945, ADR 0543)
Met a second time live on 2026-10-25 at 20:04 (note 35): mesh-build-go was
assigned to ace again, and its account rows, minted at 17:19, had outlived the
unassignment. The issue minted only when no row stood, so it minted and sealed
nothing; the node had no credential and the bus no user, and the module
crash-looped 20 times. The watch still said "the grants written".

- A bus account now says when its credential was sealed to its node (migration
  0108; every account standing before is taken as sealed). An issue under the
  assign act, the assignments' watch or a registration mints only an account
  that is not issued, a row with no seal counting as none, under the store's
  lock on the account across its look, mint and seal, and marks it sealed once
  sealed. A seal that fails leaves it unsealed for the next issue.
- Unassigning a module forgets its bus accounts and their credentials on that
  node, so an assignment again issues them afresh.
- Only sealed accounts count as minted for a composition and are named in a
  grant, so the assigner, which now wants a node only once the grant names its
  account (mesh-delivery, same branch), holds the node until it is.
- The assignment's log line says which accounts were issued and sealed, or that
  they stood issued and sealed.

The process-only issue drops the atomic insert of the review of 9ff47f6a for
the lock, which also covers the seal. Tests: cases in the existing tables.
Research 072: the grants piece (memberships and credentials heard from placement).
2026-10-25 20:17:46 +01:00
jschoubben 40034be479 Merge pull request 'Kept compositions follow config changes from a durable consumer, so going back to an earlier build keeps today's config (issue 1945, ADR 0545)' (#466) from fix/1945-kept-compositions-follow-config into main
Reviewed-on: #466
2026-10-25 18:37:57 +00:00
jschoubben 045f25bd13 Merge pull request 'Take new builds in from a durable queue on the bus, register on the fingerprint, and issue a node's bus accounts when its assignment stands (issue 1945, ADR 0543, ADR 0544)' (#465) from fix/1945-take-in-consumer into main
Reviewed-on: #465
2026-10-25 18:06:47 +00:00
jschoubben ff3cc47027 Let the kept compositions' pass end at the stop (issue 1945, ADR 0545, review 3)
mesh/delivery held for a person: merged without a passing check: only a person decides what follows
mesh/delivery-group group fix/1945-kept-compositions-follow-config held: a member is held: novox/mesh-controller@ff3cc47027e0 (merged without a passing check: …
mesh/check failed: integration: github.com/novox/mesh-controller/cmd/mesh-controller.TestTheDeclarationsStreamGoesOnceEveryNodeRunsTheNewEngine: with …
followKept went on composing every kept key, and the config consumer every
touched node, after the controller was told to stop. Both now check the stop
before each key and each node; the writes of an unfinished batch are refused
at once, so the next controller takes them.
Research 072: milestone 2, the composer reacting to the change of an input.
2026-10-25 18:43:38 +01:00
jschoubben 31a2bcdb3c Read each resolved config key once when the consumer is made, so values set before this release reach the kept compositions (issue 1945, ADR 0545, review 3)
The config consumer delivered new writes only, and mesh-config never writes an
unchanged value again: g14's mesh-build-go capacity of 2, set before this
release, would never reach its kept compositions carrying 0. A newly made
durable now reads each key's last value once, through the same gather,
per-write lock and acknowledgement; its deliver policy is set before its first
delivery. Acknowledgements are synchronous and sent past the stop, and one the
bus refuses is said by its key.
Research 072: milestone 2, the composer reacting to the change of an input.
2026-10-25 18:38:45 +01:00
jschoubben 2df47d7b1c Merge branch 'fix/1945-take-in-consumer' into fix/1945-kept-compositions-follow-config 2026-10-25 18:30:53 +01:00
jschoubben 666d1b682e Register on the fingerprint, not the commit; count the take-in's attempts as settled (issue 1945, ADR 0544, review 3)
mesh/delivery held for a person: merged without a passing check: only a person decides what follows
mesh/check checking 666d1b682e7f on 045f25bd1353
registeredAt compared commits, so a catalogue merge that moved only the commit
(2df05030 at 18:02, a85d9037 before it) took every catalogue module in again
and composed it: about 200 take-ins, some 25 minutes, on every mesh-catalog
merge, which held nextcloud for an hour. A module whose trunk fingerprint is
the one it is registered at now has the new commit recorded (built_from and
source_head, as a registration moves them) and composes nothing unless its
composition or grant is missing. Readers of the registered commit: the
fingerprints read the commit's build or, with none, the indexer's value at it,
which names the same fingerprint; status, modules, the board and the facts
read it as the source's commit, which it is; the upgrade path counts it as
built from that commit, which it is.

The stall watch counted only settles the bus accepted, so each refused
acknowledgement at a first start left work held for good: it now counts the
loop's every attempt. The acknowledgement and refusals at the stop are sent
past the stop, and the connection closes only after the readers and the loop
have ended.
Research 072: the take-ins on a bounded pull consumer.
2026-10-25 18:30:07 +01:00
jschoubben e40b1e70db Follow config from a durable consumer, compose again only what changed since, and hold the node only for a write (issue 1945, ADR 0545, review)
The start composed every kept composition of every module on every node, each a
whole node composed in memory under the node's lock: 10 to 50 minutes at a
start, holding take-ins back. Now the writes of mesh-config_resolved are read
from the durable consumer mesh-controller-config-follow, acknowledged once the
change is composed, so a restart composes again only what was written since
the last change handled, and a first start nothing it did not hear. The node's
lock is taken for each write alone. A consumer that cannot be made or read
raises config-not-followed and is tried again a minute later, cleared once it
reads. The controller's account is granted that consumer's acknowledgements
where mesh-config serves, as its verb is.
Research 072: milestone 2, the composer reacting to the change of an input.
2026-10-25 18:27:43 +01:00
jschoubben 6b116586fc Merge branch 'fix/1945-take-in-consumer' into fix/1945-kept-compositions-follow-config 2026-10-25 18:21:58 +01:00
jschoubben bf20d9d8cf Judge the take-in's stall by keys settled with work to do, and clear its alarms after a restart (issue 1945, ADR 0544, review 2)
A refused key stays unacknowledged during its delay, so one waiting key on a
quiet mesh raised take-in-stalled, cleared at the next ack and raised again.
The watch now counts the keys the loop settled (acknowledged or refused, as
the bus took them) and calls it a stall only with work to do, keys not
delivered yet or a batch handed and not settled, and nothing settled for 5
minutes: a hung loop stays loud. Its first look always says what it found, so
a stall raised before a restart is cleared once this controller moves. A
consumer not made raises take-in-not-consumed under its own id per bucket,
cleared when that consumer is made. Acks are synchronous, so only those the
bus took count; on stop the iterator is drained and what it held, and a batch
cut by the stop, are refused with no delay.
Research 072: the take-ins on a bounded pull consumer.
2026-10-25 18:21:52 +01:00
jschoubben 81e335e42d Say why a node with no compositions answers none to compose again (ADR 0545, the empty-on-error rule) 2026-10-25 18:15:22 +01:00
jschoubben 2407bfb9ff Compose kept compositions again when a module's config changes (issue 1945, ADR 0545)
The controller composed a module only at the fingerprint it is registered at,
so each of the up to five compositions kept for going back carried the config
of the moment it was made: a want back to an earlier mesh-build-go on g14 put
its capacity back to 0. Nothing composed a node when mesh-config's resolved
values changed, either.

Now a write of mesh-config_resolved (<node>.<module>.<key>), heard from that
bucket's watch and gathered for a second, drops the controller's 5 s reading
of the values, composes the node, and composes every kept fingerprint of the
touched modules on that node again from its own build's manifest with today's
inputs, writing only the compositions whose digest differs. A starting
controller does it once for every node, for a change no controller heard.
The two replaced-controller tests are one table test with the kept cases.
Research 072: milestone 2, the composer reacting to the change of an input.
2026-10-25 18:14:03 +01:00
jschoubben 850f4b4d26 Keep refused keys from holding the take-in's slots, and judge its stall by acknowledgements (issue 1945, ADR 0544, review)
A refused key stays unacknowledged until its delay ends, so with 8 a consumer
a handful refused for long (a manifest word this controller does not read
yet, a node that keeps failing) stopped that consumer for good, the build that
reads the word among what it held back. Each consumer now leaves up to 1000
unacknowledged; the store's bound is the one loop's batch of 16. A refusal
waits 30 s, doubling each time up to 10 minutes.

The stall watch judged movement by deliveries, which rise with every
redelivery: it now counts the keys this process acknowledged. A consumer that
cannot be made raises take-in-stalled instead of a log line. On stop the
gathered batch and a key in hand are refused with no delay, so the next
controller has them at once. A trunk value with no fingerprint of the new form
is said once, as every wait for a write.
Research 072: the take-ins on a bounded pull consumer.
2026-10-25 18:10:35 +01:00
jschoubben e7673e3bd3 Merge branch 'fix/1945-credential-on-assignment' into fix/1945-take-in-consumer 2026-10-25 18:07:30 +01:00
jschoubben 94bb74966f Bound the take-in's two consumers to 16 keys in flight together, 8 each (ADR 0544 §1, its review)
Two consumers of 16 each allowed 32 take-ins on the store at once, twice
issue 1945's bound. A batch takes up to 16. Research 072: the take-ins on a
bounded pull consumer.
2026-10-25 18:01:18 +01:00
jschoubben bfb271a91b Take done keys in from a durable pull consumer, a batch at a time (issue 1945, ADR 0544)
The take-in watched the builder's and the indexer's buckets, started a follower
per module that retried on a 30 s timer, and at start took every catalogued
module in alphabetically: on 2026-10-25 a restart took in about 2 modules a
minute, about 190 followers queued behind 16 slots, and every take-in composed
two or three times even with nothing new, so a new build waited 30 minutes and
more for its credential.

Now two durable consumers named mesh-controller-take-in read the indexer's
modules bucket (every module's last trunk value when the consumer is made, then
each write) and the builder's builds bucket (each write), at most 16 keys
unacknowledged each. The bus keeps what is not taken in: the start's look is the
consumers' own catch-up, and a restart costs only what was left unacknowledged.
A batch reads the catalogue once, takes each module in once, and asks the
composer twice: once for its modules' nodes, once for the bus's node with the
grants. A module registered at its trunk whose compositions and grant exist, or
that this process took in at the same trunk, composes nothing; a key delivered
again is taken in whole. A wait for the builder's or indexer's next write is
acknowledged, a wait on a read or a failure refused with a 30 s delay. The
backlog is in status, and a backlog standing still for 5 minutes raises
take-in-stalled. The controller's account is granted the two consumers' acks.

The follower tests go with the followers: twelve tests become three.
Research 072: the take-ins on a bounded pull consumer (milestone 4, done keys
alone) and the start's look composing only what changed (milestone 2).
2026-10-25 17:59:20 +01:00
36 changed files with 2227 additions and 926 deletions
+18 -2
View File
@@ -292,6 +292,22 @@ func unassign(ctx context.Context, open *stores, node string, modules ...string)
answer += "\n " + strings.TrimSpace(line)
}
}
// Its bus accounts on the node go with it once it no longer resolves there (a module another requires keeps running
// and keeps them), so an assignment again issues them afresh (issue 1945). The node's own provisioner account stays.
if after.err == nil {
for _, module := range modules {
if slices.Contains(after.modules, module) {
continue
}
forgot, err := open.inventory.ForgetBusAccountsOf(ctx, node, module, false)
switch {
case err != nil:
answer += fmt.Sprintf("\n its bus accounts on %s are not forgotten: %v", node, err)
case len(forgot) > 0:
answer += fmt.Sprintf("\n its bus accounts on %s forgotten: %s", node, strings.Join(forgot, ", "))
}
}
}
// What it leaves behind that is irreplaceable is kept and retired, never removed (novox/hq ADR 0233).
for _, line := range keptOnUnassign(ctx, open.inventory, node, modules) {
answer += "\n " + line
@@ -421,9 +437,9 @@ func issueOnAssign(ctx context.Context, open *stores, node, module string) strin
// A module running the provisioner client alone is minted the client's account (issue 917).
user = broker.Principal{Kind: broker.KindProvisioner, Node: node, Module: module}.Username()
}
if _, minted, err := inv.BusUserHash(ctx, user); err != nil {
if _, sealed, err := inv.BusAccountIssued(ctx, user); err != nil {
return ""
} else if minted {
} else if sealed { // a row whose credential is not sealed to the node is not issued (issue 1945)
// Its own account stands and is kept; a process it gained is not issued yet (issue 1226).
return processAccountsOnAssign(ctx, inv, m, node)
}
@@ -111,6 +111,11 @@ func TestTheControlPlaneIsToldWhereTheNodePutTheStoreAndTheBroker(t *testing.T)
Kind: inventory.BusController, Node: "anchor", Module: "mesh-controller"}, "bootstrap"); err != nil {
t.Fatal(err)
}
// Seeded with the credential the controller already uses, so it is sealed: an account counts as issued only once
// sealed (ADR 0543 §2, PR #467), and genesis's seed is the credential in use.
if err := open.inventory.MarkBusAccountSealed(ctx, "anchor.mesh-controller"); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "anchor", "mesh-controller"); err != nil {
t.Fatal(err)
}
+111 -9
View File
@@ -5,6 +5,8 @@ import (
"encoding/json"
"errors"
"fmt"
"maps"
"slices"
"strings"
"sync"
"time"
@@ -39,6 +41,9 @@ import (
type assignmentAccountSteps struct {
// holds says this controller holds the lease: only its holder issues.
holds func(ctx context.Context) bool
// forgetStale forgets every module's accounts on a node where the module no longer resolves, and answers a line for
// each (issue 1945: rows of an unassignment before this release, marked sealed by migration 0108).
forgetStale func(ctx context.Context) ([]string, error)
// catalogue reads the catalogue once, for every assignment a look issues for.
catalogue func(ctx context.Context) (map[string]catalogue.Manifest, error)
// issue issues the module's accounts on the node that are not minted, from the catalogue read; issued says it
@@ -85,6 +90,17 @@ func (s assignmentAccountSteps) settle(ctx context.Context, node, module string)
// it issued (an assignment that came to stand while no controller heard it may have its accounts and still lack its
// place in the bus's user list or its grant). It answers what it did.
func (s assignmentAccountSteps) look(ctx context.Context, standing []chain.Assignment) (string, error) {
var forgot []string
if len(standing) > 0 && s.forgetStale != nil {
// Only from a reading of the assignments that holds some: an empty or failed one would forget every account.
if !s.holds(ctx) {
return "", errNotHolding
}
var err error
if forgot, err = s.forgetStale(ctx); err != nil {
return "", fmt.Errorf("the accounts of modules no longer on their nodes are not forgotten: %w", err)
}
}
var looked []chain.Assignment
for _, a := range standing {
if a.Module != controllerModule {
@@ -92,7 +108,7 @@ func (s assignmentAccountSteps) look(ctx context.Context, standing []chain.Assig
}
}
if len(looked) == 0 {
return "", nil
return strings.Join(forgot, "\n"), nil
}
if !s.holds(ctx) {
return "", errNotHolding
@@ -113,7 +129,7 @@ func (s assignmentAccountSteps) look(ctx context.Context, standing []chain.Assig
}
}
said, err := s.cover(ctx)
lines = append(lines, fmt.Sprintf("%d standing assignment(s) looked at; %s", len(looked), said))
lines = append(append(forgot, lines...), fmt.Sprintf("%d standing assignment(s) looked at; %s", len(looked), said))
return strings.Join(lines, "\n"), err
}
@@ -195,22 +211,27 @@ func issueAssignmentAccounts(ctx context.Context, inv *inventory.Inventory, shel
if !known || mayIssue(m) != nil {
return false, "", nil
}
if user, own := ownAccountOf(m, node); own {
_, minted, err := inv.BusUserHash(ctx, user)
if _, own := ownAccountOf(m, node); own {
minted, err := inv.BusMinted(ctx) // sealed accounts alone: a row with no sealed credential is not issued
if err != nil {
return false, "", fmt.Errorf("whether %s is minted cannot be read: %w", user, err)
return false, "", fmt.Errorf("the issued accounts cannot be read: %w", err)
}
if !minted {
if missingAccounts(m, node, minted) {
busAddress, err := broker.BusAddress()
if err != nil {
return false, "", err
}
// Its processes' accounts too: issueOnTheNewBus issues every one of them with the module's own.
// Its processes' and provisioner client's accounts too: issueOnTheNewBus issues each that is not issued.
ctx, got := withIssued(ctx)
if err := issueOnTheNewBus(ctx, inv, m, node, busAddress); err != nil {
return false, "", err
}
return true, "its bus credential issued as " + user, nil
if issued := got.list(); len(issued) > 0 {
return true, "issued and sealed to " + node + ": " + strings.Join(issued, ", "), nil
}
return false, "", nil
}
return false, "its bus accounts on " + node + " stood issued and sealed", nil
}
line, err := issueMissingProcessAccounts(ctx, inv, m, node)
if err != nil {
@@ -219,6 +240,18 @@ func issueAssignmentAccounts(ctx context.Context, inv *inventory.Inventory, shel
return line != "", line, nil
}
// missingAccounts says some account of m on node is not issued: its own, a process's, or its provisioner client's.
func missingAccounts(m catalogue.Manifest, node string, minted []string) bool {
if user, own := ownAccountOf(m, node); own && !slices.Contains(minted, user) {
return true
}
if runsTheProvisionerClient(m) &&
!slices.Contains(minted, broker.Principal{Kind: broker.KindProvisioner, Node: node, Module: m.Module}.Username()) {
return true
}
return len(processAccountsMissing(m, node, minted)) > 0
}
// liveAssignmentAccountSteps are the steps on the controller's own stores and the bus: the composer's passes, as the
// done keys' take-ins use them, so a node asked for by both is composed once a pass.
func liveAssignmentAccountSteps(open *stores) assignmentAccountSteps {
@@ -229,7 +262,8 @@ func liveAssignmentAccountSteps(open *stores) assignmentAccountSteps {
_, err := theLease.epoch(ctx)
return err == nil
},
catalogue: func(ctx context.Context) (map[string]catalogue.Manifest, error) { return inv.Catalogue(ctx) },
forgetStale: func(ctx context.Context) ([]string, error) { return forgetStaleAccounts(ctx, open) },
catalogue: func(ctx context.Context) (map[string]catalogue.Manifest, error) { return inv.Catalogue(ctx) },
issue: func(ctx context.Context, shelf map[string]catalogue.Manifest, node, module string) (bool, string, error) {
return issueAssignmentAccounts(ctx, inv, shelf, node, module)
},
@@ -385,3 +419,71 @@ func watchAssignmentsOnce(ctx context.Context, watch func(context.Context) (<-ch
}
}
}
// staleAccounts is, by node, each module some account of which stands on a node where the module no longer resolves:
// the accounts of a module's own, its processes' and its provisioner client's. A node whose resolution is unknown is
// left alone.
func staleAccounts(users map[string]inventory.BusUser, resolved map[string]map[string]bool) map[string][]string {
seen := map[string]map[string]bool{}
for _, u := range users {
switch u.Kind {
case inventory.BusModule, inventory.BusNodeTools, inventory.BusProcess, inventory.BusProvisioner:
default:
continue
}
on, known := resolved[u.Node]
if u.Node == "" || u.Module == "" || !known || on[u.Module] {
continue
}
if seen[u.Node] == nil {
seen[u.Node] = map[string]bool{}
}
seen[u.Node][u.Module] = true
}
out := map[string][]string{}
for node, modules := range seen {
out[node] = slices.Sorted(maps.Keys(modules))
}
return out
}
// forgetStaleAccounts forgets the accounts of every module on a node where it no longer resolves (assigned or
// required), from each node's own resolution; a node that does not resolve is left alone.
func forgetStaleAccounts(ctx context.Context, open *stores) ([]string, error) {
inv := open.inventory
users, err := inv.BusUsers(ctx)
if err != nil {
return nil, err
}
ctx = withInputCache(ctx) // one reading of the mesh for every node resolved
resolved := map[string]map[string]bool{}
for _, u := range users {
if _, done := resolved[u.Node]; done || u.Node == "" {
continue
}
plan, _, err := resolveNode(ctx, open, u.Node)
if err != nil {
continue // unknown: nothing of this node is forgotten
}
on := map[string]bool{}
for _, m := range plan.Modules {
on[m.Module] = true
}
resolved[u.Node] = on
}
var lines []string
stale := staleAccounts(users, resolved)
for _, node := range slices.Sorted(maps.Keys(stale)) {
for _, module := range stale[node] {
forgot, err := inv.ForgetBusAccountsOf(ctx, node, module, true)
if err != nil {
return lines, err
}
if len(forgot) > 0 {
lines = append(lines, fmt.Sprintf("%s no longer runs on %s: its bus accounts forgotten: %s", module, node,
strings.Join(forgot, ", ")))
}
}
}
return lines, nil
}
@@ -5,6 +5,7 @@ import (
"encoding/json"
"errors"
"fmt"
"reflect"
"strings"
"testing"
"time"
@@ -12,6 +13,7 @@ import (
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/chain"
"github.com/novox/mesh-controller/internal/inventory"
)
// An assignment that comes to stand issues its node's bus accounts and the module's grant (ADR 0543, issue 1945): on
@@ -193,6 +195,7 @@ func TestSettlingAndTheStartsLook(t *testing.T) {
wantComposed []string
wantGrants bool
wantErr string
stale []string // what forgetting the accounts of modules no longer on their nodes says
}{
{name: "an account not minted", module: "mesh-build-node", issued: true, holder: "novox", wantIssues: 1,
wantComposed: []string{"novox"}, wantGrants: true},
@@ -213,13 +216,21 @@ func TestSettlingAndTheStartsLook(t *testing.T) {
look: []chain.Assignment{ace("mesh-build-node"), ace("mesh-build-go")}, holder: "novox", wantIssues: 2,
wantComposed: []string{"novox"}, wantGrants: true},
{name: "the start's look at nothing does nothing", look: []chain.Assignment{ace(controllerModule)}, holder: "novox"},
{name: "the start's look forgets the accounts of modules no longer on their nodes first",
look: []chain.Assignment{ace("mesh-build-node")}, holder: "novox", wantIssues: 1, wantComposed: []string{"novox"},
wantGrants: true, stale: []string{"mesh-build-go no longer runs on ace: its bus accounts forgotten: ace.mesh-build-go"}},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
var composed []string
issues, catalogues, grants := 0, 0, 0
forgets := 0
steps := assignmentAccountSteps{
holds: func(context.Context) bool { return !c.notHolding },
forgetStale: func(context.Context) ([]string, error) {
forgets++
return c.stale, nil
},
catalogue: func(context.Context) (map[string]catalogue.Manifest, error) {
catalogues++
return map[string]catalogue.Manifest{}, nil
@@ -261,6 +272,14 @@ func TestSettlingAndTheStartsLook(t *testing.T) {
if want := map[bool]int{true: 1}[c.wantGrants]; grants != want {
t.Fatalf("grants written %d time(s), want %d", grants, want)
}
if want := map[bool]int{true: 1}[len(c.look) > 0 && !c.notHolding]; forgets != want {
t.Fatalf("forgot stale accounts %d time(s), want %d (only from a reading that holds assignments)", forgets, want)
}
for _, line := range c.stale {
if !strings.Contains(said, line) {
t.Fatalf("said %q, want it to say %q", said, line)
}
}
if c.wantGrants && c.wantErr == "" && !strings.Contains(said, "the grants written") {
t.Fatalf("said %q, want it to name the grants", said)
}
@@ -324,6 +343,68 @@ func TestTheAccountsAModuleSpeaksWithOnANode(t *testing.T) {
}
})
// An issue mints and seals an account only when it is not issued: no row, or a row with no sealed credential (issue
// 1945, live on 2026-10-25 at 20:04: ace's mesh-build-go had a row from 17:19 and no credential), under the lock on
// the account; a seal that fails leaves it unsealed for the next issue. A rotation always mints.
for _, c := range []struct {
name string
only bool
minted, sealed bool
sealErr error
want string
}{
{"no row: minted unsealed, sealed, marked", true, false, false, nil, "lock mint-unsealed seal mark release"},
{"a row sealed: nothing", true, true, true, nil, "lock release"},
{"a row with no sealed credential: minted and sealed again", true, true, false, nil, "lock mint-unsealed seal mark release"},
{"a seal that fails: not marked", true, false, false, errors.New("the vault did not take it"), "lock mint-unsealed seal release"},
{"a rotation mints whatever stands", false, true, true, nil, "mint seal"},
} {
t.Run("issuing an account: "+c.name, func(t *testing.T) {
acts := &anAccountStore{minted: c.minted, sealed: c.sealed}
ctx, got := withIssued(t.Context())
if c.only {
ctx = onlyIfAbsent(ctx)
}
err := issueAccount(ctx, acts, inventory.BusUser{Username: "ace.mesh-build-go", Kind: inventory.BusModule},
func(password string) error {
acts.did("seal")
if password == "" {
t.Error("sealed no password")
}
return c.sealErr
})
if (err != nil) != (c.sealErr != nil) {
t.Fatalf("error %v, want %v", err, c.sealErr)
}
if strings.Join(acts.acts, " ") != c.want {
t.Fatalf("acts %q, want %q", strings.Join(acts.acts, " "), c.want)
}
if issued := len(got.list()) == 1; issued != (c.sealErr == nil && strings.Contains(c.want, "seal")) {
t.Fatalf("said issued %v", got.list())
}
})
}
// The accounts of a module that no longer resolves on its node are stale, its provisioner client's among them; a
// module still resolved (assigned or required), an account of another kind, and a node whose resolution is unknown
// are left alone (issue 1945: ace's mesh-build-go rows from 17:19, marked sealed by migration 0108).
t.Run("stale accounts", func(t *testing.T) {
u := func(kind, node, module string) inventory.BusUser {
return inventory.BusUser{Username: node + "." + module + "." + kind, Kind: kind, Node: node, Module: module}
}
users := map[string]inventory.BusUser{}
for _, b := range []inventory.BusUser{u(inventory.BusModule, "ace", "mesh-build-go"),
u(inventory.BusProcess, "ace", "mesh-build-go"), u(inventory.BusProvisioner, "ace", "mesh-provisioner"),
u(inventory.BusModule, "ace", "mesh-build-node"), u(inventory.BusNode, "ace", ""),
u(inventory.BusModule, "g14", "mesh-build-go")} {
users[b.Username] = b
}
got := staleAccounts(users, map[string]map[string]bool{"ace": {"mesh-build-node": true}})
if want := map[string][]string{"ace": {"mesh-build-go", "mesh-provisioner"}}; !reflect.DeepEqual(got, want) {
t.Fatalf("stale %v, want %v", got, want)
}
})
t.Run("a process may be issued only with its own broker secret", func(t *testing.T) {
// mesh-mail's real shape: no `broker` of its own, its process declaring bus (review round 1 of issue 1181).
if err := mayIssue(mailSenderWithProcess()); err != nil {
@@ -342,3 +423,30 @@ func TestTheAccountsAModuleSpeaksWithOnANode(t *testing.T) {
}
})
}
// anAccountStore is an accountIssuer that writes down what was asked of it.
type anAccountStore struct {
minted, sealed bool
acts []string
}
func (a *anAccountStore) did(act string) { a.acts = append(a.acts, act) }
func (a *anAccountStore) LockBusAccount(context.Context, string) (func(), error) {
a.did("lock")
return func() { a.did("release") }, nil
}
func (a *anAccountStore) BusAccountIssued(context.Context, string) (bool, bool, error) {
return a.minted, a.sealed, nil
}
func (a *anAccountStore) MintBusPassword(context.Context, inventory.BusUser) (string, error) {
a.did("mint")
return "a-password", nil
}
func (a *anAccountStore) MintBusPasswordUnsealed(context.Context, inventory.BusUser) (string, error) {
a.did("mint-unsealed")
return "a-password", nil
}
func (a *anAccountStore) MarkBusAccountSealed(context.Context, string) error {
a.did("mark")
return nil
}
+3
View File
@@ -273,4 +273,7 @@ type answers struct {
// applied, is not well: an assignment somebody made is not made yet, or will not be.
pending []inventory.PendingAssignment
pendingUnread string
// takeIn is the take-in's backlog of done keys on the bus (novox/hq ADR 0544 §6), nil where it cannot be read
// (a process with no bus, or a consumer not made yet).
takeIn *takeInBacklog
}
@@ -217,6 +217,11 @@ func TestTheAdaptersInputCarriesTheOwnSecretsTheBoxesTheLicencesAndTheBus(t *tes
Module: "objects"}, "a-password"); err != nil {
t.Fatal(err)
}
// Minted and sealed to its node: an account counts as issued, and is named among the minted, only once sealed
// (ADR 0543 §2, PR #467).
if err := inv.MarkBusAccountSealed(ctx, "anchor.objects"); err != nil {
t.Fatal(err)
}
if err := inv.PutBusMembership(ctx, "laptop", "sealed-membership"); err != nil {
t.Fatal(err)
}
@@ -102,8 +102,10 @@ func TestTheDeclarationsStreamGoesOnceEveryNodeRunsTheNewEngine(t *testing.T) {
// The new node-engine registered at its fingerprint, from its done key.
n6 := fpOf('6')
keys, trunk := aKeyFor("mesh-host", "6", n6, "novox/mesh-host", "", "f00d0123456789ab")
if _, err := liveDoneKeySteps(m.open, trunkKeys{keyBuckets: keys, trunk: trunk}).take(ctx, "mesh-host"); err != nil {
t.Fatal(err)
settled, _ := liveTakeInSteps(m.open, trunkKeys{keyBuckets: keys, trunk: trunk}, m.js).batch(ctx,
[]delivered{{module: "mesh-host", again: true}}, map[string]string{})
if len(settled) != 1 || settled[0].err != nil || settled[0].verdict != acknowledged {
t.Fatalf("the take-in of mesh-host: %+v", settled)
}
if _, err := m.js.CreateOrUpdateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: sdkdelivery.HaveBucket, History: sdkdelivery.HaveHistory}); err != nil {
t.Fatal(err)
+314 -310
View File
@@ -4,6 +4,8 @@ import (
"context"
"errors"
"fmt"
"maps"
"slices"
"sort"
"strings"
"sync"
@@ -12,212 +14,285 @@ import (
"git.novox.be/novox/mesh-sdk/go/catalogue"
sdkdelivery "git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/builderkeys"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
)
// A new fingerprint of a registered module, from its done key to its compositions (novox/hq ADR 0539 clauses 5 and 7,
// ADR 0501 §2). With no delivery plan left to take a module's done key in, a write of the builder's or the indexer's
// keys does it: the controller takes in the done key of the fingerprint the indexer's trunk value names (registers the
// module at it, as a plan's take-in did), composes every node the module is assigned to (which makes what a module's
// new manifest needs, a credential among it, as a send made it), composes the node holding the bus (its user list now
// carries the module's permissions at the new fingerprint: a new revision of the bus module's composition alone,
// written into its want at once by the assigner), and writes `grants.<module>.<fp>`. The assigner's sixth and seventh
// rules then let a want name the fingerprint. **No timer**: woken by the keys, and a starting controller looks at every
// module once (doneKeysAtStart).
// ADR 0501 §2, ADR 0544). With no delivery plan left to take a module's done key in, a write of the builder's or the
// indexer's keys does it: the controller takes in the done key of the fingerprint the indexer's trunk value names
// (registers the module at it, as a plan's take-in did), composes every node the module is assigned to (which makes what
// a module's new manifest needs, a credential among it), composes the node holding the bus (its user list now carries
// the module's permissions at the new fingerprint), and writes `grants.<module>.<fp>`. The assigner's sixth and seventh
// rules then let a want name the fingerprint.
//
// **A durable pull consumer, not a watch** (ADR 0544; research 072, milestones 2 and 4). The writes are read from two
// durable consumers (take_in_consumer.go): the bus keeps what is not taken in yet, so a restart takes in only what the
// controller before it left unacknowledged, and the start's look is the consumers' own catch-up. A batch of keys asks the
// composer twice whatever its size, and a key with nothing new composes nothing.
// doneKeySteps are what taking a module's done key in does, each a step a test may stand in for.
type doneKeySteps struct {
// held says the catalogue holds the module (a new one is the new-module take-in's, new_modules.go).
held func(ctx context.Context, module string) (bool, error)
// takeInSteps are what taking a batch of done keys in does, each a step a test may stand in for.
type takeInSteps struct {
// catalogue is every module the catalogue holds, with the nodes it is assigned to: read once a batch. A module it
// does not hold is the new-module take-in's (new_modules.go).
catalogue func(ctx context.Context) (map[string][]string, error)
// trunk is the indexer's trunk value of the module.
trunk func(ctx context.Context, module string) (sdkdelivery.Module, error)
// registeredAt says the module is registered at the commit.
registeredAt func(ctx context.Context, module, commit string) (bool, error)
// takeIn registers the module from the builder's done key of the trunk's fingerprint (takeInKey); errKeyWaits when
// the key is not done yet.
// registered says whether the module is registered at the trunk's commit, and whether at its fingerprint: a commit
// that moves while the fingerprint stays (a catalogue merge that leaves the module's sources alone) is no new build.
registered func(ctx context.Context, module string, trunk sdkdelivery.Module) (atCommit, atFP bool, err error)
// recordCommit records the module's registration at the trunk's commit, its fingerprint unchanged: nothing composed.
recordCommit func(ctx context.Context, module, commit string) error
// takeIn registers the module from the builder's done key of the trunk's fingerprint (takeInKey).
takeIn func(ctx context.Context, module string, trunk sdkdelivery.Module) error
// nodesOf are the nodes the module is assigned to; busHolder the node holding the bus, "" for none.
nodesOf func(ctx context.Context, module string) ([]string, error)
// taken says the module's compositions at fp exist for every node and its grant is written: nothing to compose.
taken func(ctx context.Context, module, fp string, nodes []string) (bool, error)
// busHolder is the node holding the bus, "" for none.
busHolder func(ctx context.Context) (string, error)
// compose writes the nodes' compositions; grants writes the grants issued.
compose func(ctx context.Context, nodes []string) error
grants func(ctx context.Context) error
// compose composes the nodes, and writes the grants after when grants: each node's error, and the grants'.
compose func(ctx context.Context, nodes []string, grants bool) (map[string]error, error, error)
}
// take takes module's done key in and composes what it changed. It answers what it did in a line, "" when there was
// nothing to do.
func (s doneKeySteps) take(ctx context.Context, module string) (string, error) {
held, err := s.held(ctx, module)
if err != nil || !held {
return "", err
}
trunk, err := s.trunk(ctx, module)
if errors.Is(err, builderkeys.ErrNoValue) {
return "", nil
}
if err != nil {
return "", err
}
if !strings.HasPrefix(trunk.FP2, sdkdelivery.SourcePrefix2) {
return "", nil
}
at, err := s.registeredAt(ctx, module, trunk.Commit)
if err != nil {
return "", err
}
said := ""
if !at {
if err := s.takeIn(ctx, module, trunk); err != nil {
return "", err
}
said = fmt.Sprintf("%s registered from the builder's done key %s (%s)", module, short(trunk.FP2), short(trunk.Commit))
}
nodes, err := s.nodesOf(ctx, module)
if err != nil {
return said, err
}
if len(nodes) == 0 {
return said, nil
}
// The module's nodes first: composing them makes what the new manifest needs, its credentials among it.
if err := s.compose(ctx, nodes); err != nil {
return said, err
}
// Then the bus's user list, which carries the module's permissions at its new fingerprint.
holder, err := s.busHolder(ctx)
if err != nil {
return said, err
}
if holder != "" && !inList(nodes, holder) {
if err := s.compose(ctx, []string{holder}); err != nil {
return said, err
}
}
// Then the grant, which with the composition lets a want name the fingerprint (rules 6 and 7).
if err := s.grants(ctx); err != nil {
return said, err
}
if said == "" {
said = fmt.Sprintf("%s at %s", module, short(trunk.FP2))
}
return said + "; composed for " + strings.Join(nodes, ", ") + ", its grant written", nil
// delivered is one key a consumer delivered: its module, and whether it was delivered before (a take-in that may have
// stopped half-way, so it is taken in whole).
type delivered struct {
module string
again bool
// deliveries is how often the bus delivered it, this time included: what a refusal's delay grows with.
deliveries uint64
}
func inList(list []string, s string) bool {
for _, v := range list {
if v == s {
return true
}
}
return false
}
// verdict is what becomes of a module's delivered keys.
type verdict int
// doneKeyTakeOnce is one take-in; a variable so a test may stand in for it. doneKeysSay says a line of the take-ins.
var (
doneKeyTakeOnce = takeDoneKeyOnce
doneKeysSay = func(s string) { fmt.Println(s) }
const (
// acknowledged: taken in, nothing new, a wait a later write ends, or a build key that failed.
acknowledged verdict = iota
// refused: delivered again by the bus after doneKeyRetry (a wait on a read, or a failure).
refused
)
// doneKeysBeforeLetGo is called just before a follower lets go of its module; a variable so a test can wake the module
// at that moment. Nil does nothing.
var doneKeysBeforeLetGo func(module string)
// doneKeysLookAgain marks a module woken while a take-in of it ran: that take-in goes round once more.
var doneKeysLookAgain sync.Map
// doneKeysMu keeps one take-in of a module's done key at a time in this process, per module.
var doneKeysMu sync.Map
// takeDoneKey is a woken module's take-in on the live stores and buckets, said. A take-in that fails (its compositions
// or its grant not written) is tried again every doneKeyRetry while it fails, loudly: printed each time, and an urgent
// condition raised by the first until it has worked, as the chain's own moves are (layBuildsOutSoon). A key not done
// yet is no failure: it is said once for each reason it waits for and looked at again every doneKeyRetry.
func takeDoneKey(ctx context.Context, open *stores, module string) {
mu, _ := doneKeysMu.LoadOrStore(module, &sync.Mutex{})
for {
if !mu.(*sync.Mutex).TryLock() {
// A take-in of the module already follows it: it is told to look again once it is done, so a newer key
// written meanwhile is never lost (the review of issue 1945).
doneKeysLookAgain.Store(module, true)
return
}
followDoneKey(ctx, open, module)
if doneKeysBeforeLetGo != nil {
doneKeysBeforeLetGo(module)
}
mu.(*sync.Mutex).Unlock()
// A wake between this follower's last look and its letting go failed its TryLock and only marked the module:
// looked at again now that it is let go, and taken once more (the review of issue 1945).
if _, again := doneKeysLookAgain.LoadAndDelete(module); !again || ctx.Err() != nil {
return
}
}
// settledKey is what a batch did with one module.
type settledKey struct {
module string
verdict verdict
// taken: registered, composed and granted now (not nothing new); its condition is cleared.
taken bool
// nothingNew: registered and composed before, so nothing was done.
nothingNew bool
// waits is why it waits, "" when it does not.
waits string
// err is a failure: raised as the module's condition.
err error
// said is the line said of it, "" for none.
said string
}
// followDoneKey takes a module's done key in, under the module's lock, until it is taken in or ctx ends: a failure or a
// wait is tried again every doneKeyRetry.
func followDoneKey(ctx context.Context, open *stores, module string) {
failed := false
saidWait := ""
for {
doneKeysLookAgain.Delete(module) // what this look reads covers every wake before it
err := takeOnceInSlot(ctx, open, module)
o := doneKeyObservation(module)
if err == nil {
// Cleared on every success, whoever raised it: a controller before a restart or another follower (issue
// 1945). A condition that is not open costs one read.
if keeper := conditionsFrom; keeper != nil {
_, _ = keeper.Clear(ctx, o.Key(), "composed and granted")
}
failed = false
if _, again := doneKeysLookAgain.LoadAndDelete(module); again {
continue // a wake came while this one ran: once more, for the key written meanwhile
}
return
// batch takes the delivered keys in, each module once: it answers one settledKey per module in the order first
// delivered, and the number of composer passes it asked. memo is, per module, the trunk (fingerprint and commit) this
// process took in last; a module at the same trunk delivered for the first time has nothing new.
func (s takeInSteps) batch(ctx context.Context, keys []delivered, memo map[string]string) ([]settledKey, int) {
var order []string
again := map[string]bool{}
for _, k := range keys {
if _, seen := again[k.module]; !seen {
order = append(order, k.module)
}
if errors.Is(err, errKeyWaits) {
// A take-in that waits is said once for each reason, and tried again every doneKeyRetry: the key write that
// would wake it may never come (issue 1945, live on 2026-10-25: mesh-host's take-in waited unsaid).
if why := err.Error(); why != saidWait {
doneKeysSay(fmt.Sprintf("done keys: %s waits: %s; looked at again every %s", module,
strings.TrimPrefix(why, errKeyWaits.Error()+": "), doneKeyRetry))
saidWait = why
}
} else {
doneKeysSay(fmt.Sprintf("done keys: %s: %v; tried again in %s", module, err, doneKeyRetry))
again[k.module] = again[k.module] || k.again
}
out := make([]settledKey, 0, len(order))
all := func(v verdict, err error) ([]settledKey, int) {
for _, m := range order {
out = append(out, settledKey{module: m, verdict: v, err: err})
}
if errors.Is(err, errKeyWaits) {
select {
case <-ctx.Done():
return
case <-time.After(doneKeyRetry):
}
return out, 0
}
held, err := s.catalogue(ctx)
if err != nil {
return all(refused, fmt.Errorf("the catalogue cannot be read: %w", err))
}
type toCompose struct {
at int
nodes []string
mark string
said string
}
var need []toCompose
for _, m := range order {
r := settledKey{module: m}
nodes, isHeld := held[m]
if !isHeld {
out = append(out, r)
continue
}
if ctx.Err() != nil {
return // stopped: a take-in cut by the stop is no fault of the build, and the next controller takes it in
trunk, err := s.trunk(ctx, m)
if errors.Is(err, builderkeys.ErrNoValue) {
out = append(out, r)
continue
}
if keeper := conditionsFrom; keeper != nil && !failed && !errors.Is(err, context.Canceled) {
// A take-in whose own step was cancelled is tried again, and raises nothing: a cancellation is no fault
// of the build (issue 1945: 30 urgent conditions live, each a take-in cut by a restart).
o.Summary += err.Error()
if _, oerr := keeper.Observe(ctx, o); oerr == nil {
failed = true
if err == nil && !strings.HasPrefix(trunk.FP2, sdkdelivery.SourcePrefix2) {
// The indexer's next write names one: said once, as every wait for a write (ADR 0544 §4).
why := trunk.None2
if why == "" {
why = "the indexer wrote no fingerprint of the new form"
}
r.waits = fmt.Sprintf("%v: %s at %.8s has no fingerprint to take a build by: %s", errWaitsForAWrite, m,
trunk.Commit, why)
out = append(out, r)
continue
}
if err != nil {
r.verdict, r.err = refused, err
out = append(out, r)
continue
}
mark := trunk.FP2 + "@" + trunk.Commit
if !again[m] && memo[m] == mark {
r.nothingNew = true
out = append(out, r)
continue
}
at, atFP, err := s.registered(ctx, m, trunk)
if err == nil && !at && atFP {
// Only the commit moved (issue 1945: every catalogue merge took about 200 modules in again, 25 minutes): the
// new commit is recorded, and the module is judged as one registered, composed only if something is missing.
if err = s.recordCommit(ctx, m, trunk.Commit); err == nil {
at = true
}
}
select {
case <-ctx.Done():
return
case <-time.After(doneKeyRetry):
if err != nil {
r.verdict, r.err = refused, err
out = append(out, r)
continue
}
said := ""
if !at {
if err := s.takeIn(ctx, m, trunk); err != nil {
switch {
case errors.Is(err, errWaitsForAWrite):
r.waits = err.Error()
case errors.Is(err, errKeyWaits):
r.verdict, r.waits = refused, err.Error()
case errors.Is(err, errKeyFailed):
r.err = err // acknowledged: only a new build ends it, and its key is delivered then
default:
r.verdict, r.err = refused, err
}
out = append(out, r)
continue
}
said = fmt.Sprintf("%s registered from the builder's done key %s (%s)", m, short(trunk.FP2), short(trunk.Commit))
} else if !again[m] {
// Registered at its trunk already: composed and granted by a take-in before this one, or at this
// controller's start, unless that take-in stopped half-way, which leaves a composition or the grant missing.
if done, err := s.taken(ctx, m, trunk.FP2, nodes); err == nil && done {
memo[m] = mark
r.nothingNew = true
out = append(out, r)
continue
}
}
if len(nodes) == 0 {
memo[m] = mark
r.taken, r.said = true, said
out = append(out, r)
continue
}
if said == "" {
said = fmt.Sprintf("%s at %s", m, short(trunk.FP2))
}
need = append(need, toCompose{at: len(out), nodes: nodes, mark: mark, said: said})
out = append(out, r)
}
if len(need) == 0 {
return out, 0
}
refuseNeed := func(err error) ([]settledKey, int) {
for _, n := range need {
out[n.at].verdict, out[n.at].err = refused, err
}
return out, 0
}
holder, err := s.busHolder(ctx)
if err != nil {
return refuseNeed(fmt.Errorf("the node holding the bus cannot be read: %w", err))
}
// The modules' nodes first, in one pass: composing them makes what their new manifests need, credentials among it.
var union []string
for _, n := range need {
for _, node := range n.nodes {
if node != holder && !slices.Contains(union, node) {
union = append(union, node)
}
}
}
sort.Strings(union)
passes := 0
failed := map[string]error{}
if len(union) > 0 {
f, _, err := s.compose(ctx, union, false)
passes++
if err != nil {
return refuseNeed(err)
}
maps.Copy(failed, f)
}
// Then the bus's user list, which carries the modules' permissions at their new fingerprints, and the grants, which
// with the compositions let a want name each fingerprint (rules 6 and 7).
var bus []string
if holder != "" {
bus = []string{holder}
}
f, grantsErr, err := s.compose(ctx, bus, true)
passes++
if err != nil {
return refuseNeed(err)
}
maps.Copy(failed, f)
for _, n := range need {
var errs []error
for _, node := range append(slices.Clone(n.nodes), bus...) {
if e := failed[node]; e != nil && !slices.ContainsFunc(errs, func(x error) bool { return x == e }) {
errs = append(errs, e)
}
}
if grantsErr != nil {
errs = append(errs, grantsErr)
}
r := &out[n.at]
if len(errs) > 0 {
r.verdict, r.err = refused, errors.Join(errs...)
continue
}
memo[r.module] = n.mark
r.taken = true
r.said = n.said + "; composed for " + strings.Join(n.nodes, ", ") + ", its grant written"
}
return out, passes
}
// doneKeyRetry is how long a failed take-in of a done key waits before it tries again.
// doneKeysSay says a line of the take-ins; a variable so a test may hear it.
var doneKeysSay = func(s string) { fmt.Println(s) }
// doneKeyRetry is how long the bus waits before it delivers again a key whose take-in was refused the first time; each
// refusal after doubles it, up to doneKeyRetryMost, so a key refused for long (a manifest word this controller does not
// read yet, a node whose composition keeps failing) costs a take-in every few minutes, not every half minute.
var doneKeyRetry = 30 * time.Second
// doneKeyRetryMost is the longest a refused key waits before it is delivered again.
const doneKeyRetryMost = 10 * time.Minute
// refusalDelay is how long the bus waits before it delivers again a key refused on its deliveries-th delivery.
func refusalDelay(deliveries uint64) time.Duration {
d := doneKeyRetry
for i := uint64(1); i < deliveries && d < doneKeyRetryMost; i++ {
d *= 2
}
return min(d, doneKeyRetryMost)
}
// kindNotComposed is the condition of a module's new fingerprint whose compositions or grant are not written: until
// they are, no want can name it.
const kindNotComposed = "fingerprint-not-composed"
@@ -228,40 +303,37 @@ func doneKeyObservation(module string) conditions.Observation {
Summary: module + "'s new fingerprint is not composed or granted, so no want can name it: "}
}
// takeDoneKeyOnce is one take-in of a module's done key.
func takeDoneKeyOnce(ctx context.Context, open *stores, module string) error {
keys, closeKeys, err := openBuilderKeys()
if err != nil {
return fmt.Errorf("the builder's keys cannot be read: %w", err)
}
defer closeKeys()
said, err := liveDoneKeySteps(open, keys).take(ctx, module)
if err == nil && said != "" {
fmt.Printf("done keys: %s\n", said)
}
return err
}
// liveDoneKeySteps are the steps on the controller's own stores and the bus.
func liveDoneKeySteps(open *stores, keys registerKeys) doneKeySteps {
// liveTakeInSteps are the steps on the controller's own stores and the bus.
func liveTakeInSteps(open *stores, keys registerKeys, js jetstream.JetStream) takeInSteps {
inv := open.inventory
return doneKeySteps{
held: func(ctx context.Context, module string) (bool, error) {
_, err := inv.SourceOf(ctx, module)
if errors.Is(err, inventory.ErrNoSuchModule) {
return false, nil
return takeInSteps{
catalogue: func(ctx context.Context) (map[string][]string, error) {
entries, err := inv.Catalogued(ctx)
if err != nil {
return nil, err
}
return err == nil, err
out := make(map[string][]string, len(entries))
for _, e := range entries {
out[e.Manifest.Module] = e.On
}
return out, nil
},
trunk: func(ctx context.Context, module string) (sdkdelivery.Module, error) {
return keys.ModuleAt(ctx, module, "")
},
registeredAt: func(ctx context.Context, module, commit string) (bool, error) {
registered: func(ctx context.Context, module string, trunk sdkdelivery.Module) (bool, bool, error) {
src, err := inv.SourceOf(ctx, module)
if err != nil {
return false, err
return false, false, err
}
return sameCommitID(src.BuiltFrom, commit), nil
if sameCommitID(src.BuiltFrom, trunk.Commit) {
return true, true, nil
}
fp, err := registeredFingerprint(ctx, inv, keys, module, src.BuiltFrom)
return false, err == nil && fp != "" && fp == trunk.FP2, nil
},
recordCommit: func(ctx context.Context, module, commit string) error {
return inv.RecordBuiltFrom(ctx, module, commit)
},
takeIn: func(ctx context.Context, module string, trunk sdkdelivery.Module) error {
_, _, err := takeInKey(ctx, inv, keys, keyTake{Module: module, Trunk: trunk, Ask: registrationAsk{Merge: trunk.Commit}})
@@ -270,17 +342,8 @@ func liveDoneKeySteps(open *stores, keys registerKeys) doneKeySteps {
}
return err
},
nodesOf: func(ctx context.Context, module string) ([]string, error) {
entries, err := inv.Catalogued(ctx)
if err != nil {
return nil, err
}
for _, e := range entries {
if e.Manifest.Module == module {
return e.On, nil
}
}
return nil, nil
taken: func(ctx context.Context, module, fp string, nodes []string) (bool, error) {
return composedAndGranted(ctx, js, module, fp, nodes)
},
busHolder: func(ctx context.Context) (string, error) {
holders, err := seatHolders(ctx, inv)
@@ -291,123 +354,64 @@ func liveDoneKeySteps(open *stores, keys registerKeys) doneKeySteps {
},
// The composer's, never the take-in's own: one pass at a time composes each node asked for once, and writes the
// grants once (issue 1945, composer.go).
compose: func(ctx context.Context, nodes []string) error {
return composerFor(open).compose(ctx, nodes, false)
},
grants: func(ctx context.Context) error {
return composerFor(open).compose(ctx, nil, true)
compose: func(ctx context.Context, nodes []string, grants bool) (map[string]error, error, error) {
return composerFor(open).composeEach(ctx, nodes, grants)
},
}
}
// watchDoneKeys wakes the take-in of a module's done key for each module the builder or the indexer writes, until ctx
// ends: never on the watch's own goroutine.
func watchDoneKeys(ctx context.Context, open *stores) {
keys, closeKeys, err := openBuilderKeys()
if err != nil {
fmt.Printf("done keys: the builder's keys are not watched: %v\n", err)
return
// registeredFingerprint is the fingerprint a module is registered at, as currentFingerprints reads it: its commit's
// newest build's, or the indexer's value at the commit when the commit has no build of its own; "" when neither says.
func registeredFingerprint(ctx context.Context, inv *inventory.Inventory, keys registerKeys, module, commit string) (string, error) {
if commit == "" {
return "", nil
}
if err := builderkeys.Watch(ctx, keys.JS, func(module string) {
inBackground(ctx, func() { takeDoneKey(ctx, open, module) })
}); err != nil {
closeKeys()
fmt.Printf("done keys: the builder's keys are not watched: %v\n", err)
return
}
goServing(ctx, func() {
<-ctx.Done()
closeKeys()
})
// And every module once now, after the watch is open so no write falls between the two.
inBackground(ctx, func() {
doneKeysAtStart(ctx, catalogedModules(open), func(module string) { takeDoneKeyAtStart(ctx, open, module) })
})
}
// doneKeysAtStart takes in the done key of every catalogued module once, one after another: the start's look the watch of
// new writes cannot give (it hears updates only). A done key written while no controller served, or whose take-in the
// controller this one replaced was stopped in, is taken here; without it the module stays in the node's own composition
// with no credential until its keys are written again (issue 1945, live on 2026-10-25 at 02:34). A take-in with nothing
// to do does nothing; a catalogue that cannot be read is said.
func doneKeysAtStart(ctx context.Context, modules func(context.Context) ([]string, error), take func(module string)) {
names, err := modules(ctx)
if err != nil {
fmt.Printf("done keys: the modules are not looked at once at the start: %v\n", err)
return
}
sort.Strings(names)
// Side by side, a few at a time: their compositions meet in the composer's passes, each node composed once per pass
// (issue 1945), so the start's look is not one module's composition after another.
sem := make(chan struct{}, startTakeIns)
var wg sync.WaitGroup
for _, m := range names {
if ctx.Err() != nil {
break
if id, err := inv.BuildOf(ctx, module, commit); err == nil && id != "" {
if b, found, err := inv.BuildByID(ctx, id); err == nil && found && b.SourceFingerprint != "" {
return b.SourceFingerprint, nil
}
sem <- struct{}{}
wg.Add(1)
go func() {
defer func() { <-sem; wg.Done() }()
take(m)
}()
}
wg.Wait()
at, err := keys.ModuleAt(ctx, module, commit)
if err != nil {
return "", err
}
return at.FP2, nil
}
// startTakeIns is how many take-ins the start's look runs at once, and how many run at once in all.
const startTakeIns = 16
// takeInSlots are the take-ins that may run at once in this process, the watch's and the start's look's together: a
// burst of done keys (about 120 at 16:07:36 on 2026-10-25) once started a take-in each, all at once, and their reads
// filled the store until it refused every connection (issues 1945 and 1277).
var takeInSlots = make(chan struct{}, startTakeIns)
// takeOnceInSlot is one take-in once a slot is free; a take-in that waits for its next try holds none.
func takeOnceInSlot(ctx context.Context, open *stores, module string) error {
select {
case takeInSlots <- struct{}{}:
case <-ctx.Done():
return ctx.Err()
// composedAndGranted says the bus holds module's grant at fp and its composition at fp for every node: a few reads of
// keys, never a composition.
func composedAndGranted(ctx context.Context, js jetstream.JetStream, module, fp string, nodes []string) (bool, error) {
grants, err := js.KeyValue(ctx, broker.GrantsBucket)
if err != nil {
return false, err
}
defer func() { <-takeInSlots }()
return doneKeyTakeOnce(ctx, open, module)
}
// takeDoneKeyAtStart is one take-in of the start's look, under the module's lock; one that fails is handed to takeDoneKey,
// which tries again every doneKeyRetry, loudly, so a failing module never holds up the look at the others.
func takeDoneKeyAtStart(ctx context.Context, open *stores, module string) {
mu, _ := doneKeysMu.LoadOrStore(module, &sync.Mutex{})
if !mu.(*sync.Mutex).TryLock() {
// A take-in already follows the module: told to look again, it takes what the start would have (issue 1945).
doneKeysLookAgain.Store(module, true)
return
key, err := sdkdelivery.ControllerGrantKey(module, fp)
if err != nil {
return false, err
}
doneKeysLookAgain.Delete(module) // what this look reads covers every wake before it
err := takeOnceInSlot(ctx, open, module)
mu.(*sync.Mutex).Unlock()
// Read after letting go, as every holder of a module's lock does: a wake during this look only marked the module
// (the review of issue 1945).
_, again := doneKeysLookAgain.LoadAndDelete(module)
if err != nil || again {
// A failure, a wait or a wake meanwhile is followed by takeDoneKey, which says it and looks again.
inBackground(ctx, func() { takeDoneKey(ctx, open, module) })
if _, err := grants.Get(ctx, key); err != nil {
if errors.Is(err, jetstream.ErrKeyNotFound) {
return false, nil
}
return false, err
}
}
// catalogedModules are the names of every module the catalogue holds.
func catalogedModules(open *stores) func(context.Context) ([]string, error) {
return func(ctx context.Context) ([]string, error) {
entries, err := open.inventory.Catalogued(ctx)
digests, err := js.KeyValue(ctx, sdkdelivery.ControllerComposedBucket)
if err != nil {
return false, err
}
for _, node := range nodes {
key, err := sdkdelivery.ComposedKey(node, module, fp)
if err != nil {
return nil, err
return false, err
}
out := make([]string, 0, len(entries))
for _, e := range entries {
out = append(out, e.Manifest.Module)
if _, err := digests.Get(ctx, key); err != nil {
if errors.Is(err, jetstream.ErrKeyNotFound) {
return false, nil
}
return false, err
}
return out, nil
}
return true, nil
}
// doneKeysComposers are the composers of the done keys' take-ins in this process, one for each set of stores (a test
@@ -1,83 +0,0 @@
package main
import (
"context"
"fmt"
"testing"
"time"
"github.com/novox/mesh-controller/internal/conditions"
)
// A take-in cut by a cancellation raises no condition (issue 1945: on 2026-10-25 thirty urgent fingerprint-not-composed
// conditions were live, each a take-in a stop or a restart cut with "context canceled"): when the controller stops, the
// next one takes it in; when a step of its own was cancelled, it is tried again.
func TestATakeInCutByACancellationRaisesNothing(t *testing.T) {
for _, tc := range []struct {
name string
stop bool // the controller's context ends, else only the step's error is a cancellation
}{{"the controller stops", true}, {"a step was cancelled", false}} {
t.Run(tc.name, func(t *testing.T) {
was, wasRetry, wasSay, wasKeeper := doneKeyTakeOnce, doneKeyRetry, doneKeysSay, conditionsFrom
t.Cleanup(func() {
doneKeyTakeOnce, doneKeyRetry, doneKeysSay, conditionsFrom = was, wasRetry, wasSay, wasKeeper
})
store := conditions.NewInMemory()
k := conditions.NewKeeper(t.Context(), conditions.Options{Store: store, History: store,
Teller: &conditions.Told{}})
t.Cleanup(func() { k.Close(context.Background()) })
conditionsFrom = k
doneKeyRetry = time.Millisecond
doneKeysSay = func(string) {}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
calls := 0
doneKeyTakeOnce = func(context.Context, *stores, string) error {
calls++
if tc.stop {
cancel() // stopped mid-way
return fmt.Errorf("the module cannot be composed: %w", context.Canceled)
}
if calls == 1 {
return fmt.Errorf("the module cannot be composed: %w", context.Canceled)
}
return nil
}
done := make(chan struct{})
go func() { takeDoneKey(ctx, nil, "cut-module"); close(done) }()
select {
case <-done:
case <-time.After(5 * time.Second):
t.Fatal("the take-in did not end")
}
if !tc.stop && calls != 2 {
t.Errorf("a take-in whose step was cancelled was tried %d time(s), want 2", calls)
}
if c, found, err := conditions.ReadOne(t.Context(), store, doneKeyObservation("cut-module").Key()); err != nil || found {
t.Fatalf("a take-in cut by a cancellation raised %+v (found %v, %v)", c, found, err)
}
})
}
}
// A take-in that succeeds clears the module's fingerprint-not-composed condition whoever raised it: a controller before a
// restart, or another follower (issue 1945: the operator read "a build needs a look" for take-ins that had since worked).
func TestASucceedingTakeInClearsAConditionAnotherRaised(t *testing.T) {
was, wasRetry, wasSay, wasKeeper := doneKeyTakeOnce, doneKeyRetry, doneKeysSay, conditionsFrom
t.Cleanup(func() { doneKeyTakeOnce, doneKeyRetry, doneKeysSay, conditionsFrom = was, wasRetry, wasSay, wasKeeper })
store := conditions.NewInMemory()
k := conditions.NewKeeper(t.Context(), conditions.Options{Store: store, History: store, Teller: &conditions.Told{}})
t.Cleanup(func() { k.Close(context.Background()) })
conditionsFrom = k
doneKeysSay = func(string) {}
o := doneKeyObservation("raised-before")
o.Summary += "the store refused the connection"
if _, err := k.Observe(t.Context(), o); err != nil {
t.Fatal(err)
}
doneKeyTakeOnce = func(context.Context, *stores, string) error { return nil }
takeDoneKey(t.Context(), nil, "raised-before")
if c, found, err := conditions.ReadOne(t.Context(), store, o.Key()); err != nil || found {
t.Fatalf("a take-in that succeeded left the condition another raised: %+v (found %v, %v)", c, found, err)
}
}
@@ -1,58 +0,0 @@
package main
import (
"context"
"fmt"
"sync"
"sync/atomic"
"testing"
"time"
)
// A burst of done keys runs at most startTakeIns take-ins at once, the watch's and the start's look together: at
// 16:07:36 on 2026-10-25 the builder rewrote about 120 done keys at once, each woke its own take-in, and each opened
// pools of its own until novox's store refused every connection (issues 1945 and 1277).
func TestABurstOfDoneKeysRunsAtMostSixteenTakeInsAtOnce(t *testing.T) {
was := doneKeyTakeOnce
t.Cleanup(func() { doneKeyTakeOnce = was })
var running, most atomic.Int64
var taken atomic.Int64
doneKeyTakeOnce = func(context.Context, *stores, string) error {
now := running.Add(1)
for {
m := most.Load()
if now <= m || most.CompareAndSwap(m, now) {
break
}
}
time.Sleep(2 * time.Millisecond)
running.Add(-1)
taken.Add(1)
return nil
}
ctx := context.Background()
var wg sync.WaitGroup
// The watch's wakes: one take-in each, in the background, as watchDoneKeys starts them.
for i := range 120 {
wg.Add(1)
go func() { defer wg.Done(); takeDoneKey(ctx, nil, fmt.Sprintf("burst-%03d", i)) }()
}
// And the start's look at the same moment, over other modules.
wg.Add(1)
go func() {
defer wg.Done()
var start []string
for i := range 40 {
start = append(start, fmt.Sprintf("start-%03d", i))
}
doneKeysAtStart(ctx, func(context.Context) ([]string, error) { return start, nil },
func(m string) { takeDoneKeyAtStart(ctx, nil, m) })
}()
wg.Wait()
if got := taken.Load(); got != 160 {
t.Fatalf("took in %d modules, want 160", got)
}
if m := most.Load(); m > startTakeIns {
t.Fatalf("ran %d take-ins at once, want at most %d", m, startTakeIns)
}
}
@@ -1,34 +0,0 @@
package main
import (
"context"
"errors"
"reflect"
"sort"
"sync"
"testing"
)
// A starting controller takes in the done key of every catalogued module once, beside the watch of new writes: a done key
// written while no controller served, or whose take-in the controller it replaced was stopped in, is otherwise never
// taken, and its module stays in the node's own composition with no credential (issue 1945, live on 2026-10-25 at 02:34:
// the controller replaced mid take-in left mesh-provisioner-client, gitea, postgres and many more unregistered at their
// new fingerprints, and the new one, watching updates only, never looked).
func TestAStartingControllerTakesEveryModulesDoneKeyOnce(t *testing.T) {
var took []string
var mu sync.Mutex
doneKeysAtStart(context.Background(), func(context.Context) ([]string, error) {
return []string{"mesh-provisioner-client", "gitea", "postgres"}, nil
}, func(module string) { mu.Lock(); took = append(took, module); mu.Unlock() })
sort.Strings(took)
if want := []string{"gitea", "mesh-provisioner-client", "postgres"}; !reflect.DeepEqual(took, want) {
t.Fatalf("took %v, want %v", took, want)
}
// A catalogue that cannot be read takes nothing and is said, never fatal.
took = nil
doneKeysAtStart(context.Background(), func(context.Context) ([]string, error) { return nil, errors.New("down") },
func(module string) { took = append(took, module) })
if len(took) != 0 {
t.Fatalf("took %v from a catalogue that could not be read", took)
}
}
+277 -68
View File
@@ -4,84 +4,305 @@ import (
"context"
"encoding/json"
"errors"
"fmt"
"slices"
"strings"
"testing"
"time"
sdkdelivery "git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/builderkeys"
)
// stepsFor are doneKeySteps over a module registered at registered, whose trunk value names fp at commit, assigned to
// nodes, the bus held on holder; every act is written to acts in order.
func stepsFor(acts *[]string, registered, commit, fp string, nodes []string, holder string) doneKeySteps {
return doneKeySteps{
held: func(_ context.Context, module string) (bool, error) { return module == "mesh-mail", nil },
trunk: func(context.Context, string) (sdkdelivery.Module, error) {
if fp == "" {
// aTakeIn is a mesh for the take-in's steps: per module its registered commit, its trunk (commit, fingerprint), its
// nodes, whether its compositions and grant exist, and what its take-in answers; every act is written to acts.
type aTakeIn struct {
registered map[string]string
sameFP map[string]bool // registered at another commit, at the trunk's fingerprint
trunks map[string]string // module -> commit; no entry: no trunk value
nodes map[string][]string
taken map[string]bool
takeInErr map[string]error
composeErr map[string]error
grantsErr error
holder string
acts []string
}
func (m *aTakeIn) steps() takeInSteps {
fp := func(module string) string { return sdkdelivery.SourcePrefix2 + strings.Repeat(module[:1], 64) }
return takeInSteps{
catalogue: func(context.Context) (map[string][]string, error) {
m.acts = append(m.acts, "catalogue")
out := map[string][]string{}
for k, v := range m.nodes {
out[k] = v
}
return out, nil
},
trunk: func(_ context.Context, module string) (sdkdelivery.Module, error) {
commit, ok := m.trunks[module]
if !ok {
return sdkdelivery.Module{}, builderkeys.ErrNoValue
}
return sdkdelivery.Module{Commit: commit, FP2: fp}, nil
return sdkdelivery.Module{Commit: commit, FP2: fp(module)}, nil
},
registered: func(_ context.Context, module string, trunk sdkdelivery.Module) (bool, bool, error) {
if m.registered[module] == trunk.Commit {
return true, true, nil
}
return false, m.sameFP[module], nil
},
recordCommit: func(_ context.Context, module, commit string) error {
m.acts = append(m.acts, "record "+module+" at "+commit)
return nil
},
registeredAt: func(_ context.Context, _ string, c string) (bool, error) { return c == registered, nil },
takeIn: func(_ context.Context, module string, trunk sdkdelivery.Module) error {
*acts = append(*acts, "take in "+module+" at "+trunk.Commit)
return nil
m.acts = append(m.acts, "take in "+module+" at "+trunk.Commit)
return m.takeInErr[module]
},
nodesOf: func(context.Context, string) ([]string, error) { return nodes, nil },
busHolder: func(context.Context) (string, error) { return holder, nil },
compose: func(_ context.Context, ns []string) error {
*acts = append(*acts, "compose "+strings.Join(ns, ","))
return nil
taken: func(_ context.Context, module, _ string, _ []string) (bool, error) { return m.taken[module], nil },
busHolder: func(context.Context) (string, error) { return m.holder, nil },
compose: func(_ context.Context, nodes []string, grants bool) (map[string]error, error, error) {
act := "compose " + strings.Join(nodes, ",")
if grants {
act += " and grants"
}
m.acts = append(m.acts, act)
failed := map[string]error{}
for _, n := range nodes {
if e := m.composeErr[n]; e != nil {
failed[n] = e
}
}
if grants {
return failed, m.grantsErr, nil
}
return failed, nil, nil
},
grants: func(context.Context) error { *acts = append(*acts, "grants"); return nil },
}
}
// 1945/AC3 (unit half): a done key of a new fingerprint of an assigned module is taken in, then the module's nodes are
// composed, then the node holding the bus (its user list), then the grants: the grant and the composition exist
// before the assigner's rules let a want name the fingerprint. A module registered at the trunk's commit already is
// composed and granted without a second take-in; one assigned nowhere, unregistered, or with no trunk value is left
// alone.
func TestADoneKeyIsTakenInComposedAndGranted(t *testing.T) {
fp := sdkdelivery.SourcePrefix2 + strings.Repeat("c", 64)
var acts []string
said, err := stepsFor(&acts, "old", "new", fp, []string{"ace", "g14"}, "novox").take(context.Background(), "mesh-mail")
if err != nil {
t.Fatal(err)
// A batch of done keys takes each module in once and asks the composer at most twice, whatever its size: once for the
// nodes of every module with something new, then once for the node holding the bus with the grants (ADR 0544 §5). A
// module registered at its trunk whose compositions and grant exist, or that this process took in at the same trunk,
// composes nothing (§3): the old take-in composed it three times over, every key and every start (issue 1945, 17:36 on
// 2026-10-25). A key delivered again is taken in whole. A wait for a write is acknowledged, a wait on a read and a
// failure refused, a failed build key acknowledged with its condition (§4).
func TestATakeInBatch(t *testing.T) {
waitsForAWrite := fmt.Errorf("%w: the key is building", errWaitsForAWrite)
waitsOnARead := fmt.Errorf("%w: the store is away", errKeyWaits)
keyFailed := fmt.Errorf("%w: the build failed", errKeyFailed)
type want struct {
verdict verdict
taken, nothingNew bool
waits, err bool
}
if got := strings.Join(acts, "; "); got != "take in mesh-mail at new; compose ace,g14; compose novox; grants" {
t.Errorf("the acts were %q", got)
cases := []struct {
name string
mesh aTakeIn
keys []delivered
memo map[string]string
acts []string
passes int
want map[string]want
}{
{name: "a new fingerprint: taken in, its nodes, then the bus's node and the grants",
mesh: aTakeIn{registered: map[string]string{"mail": "old"}, trunks: map[string]string{"mail": "new"},
nodes: map[string][]string{"mail": {"ace", "g14", "novox"}}, holder: "novox"},
keys: []delivered{{module: "mail"}},
acts: []string{"catalogue", "take in mail at new", "compose ace,g14", "compose novox and grants"}, passes: 2,
want: map[string]want{"mail": {taken: true}}},
{name: "registered, composed and granted: nothing composed (fails on the old take-in)",
mesh: aTakeIn{registered: map[string]string{"mail": "new", "nats": "n"}, trunks: map[string]string{"mail": "new", "nats": "n"},
nodes: map[string][]string{"mail": {"ace"}, "nats": {"novox"}}, taken: map[string]bool{"mail": true, "nats": true},
holder: "novox"},
keys: []delivered{{module: "mail"}, {module: "nats"}, {module: "mail"}},
acts: []string{"catalogue"},
want: map[string]want{"mail": {nothingNew: true}, "nats": {nothingNew: true}}},
{name: "a catalogue merge that moved the commit and not the fingerprint: recorded, nothing composed",
mesh: aTakeIn{registered: map[string]string{"mail": "a85d9037"}, sameFP: map[string]bool{"mail": true},
trunks: map[string]string{"mail": "2df05030"}, nodes: map[string][]string{"mail": {"ace"}},
taken: map[string]bool{"mail": true}, holder: "novox"},
keys: []delivered{{module: "mail"}},
acts: []string{"catalogue", "record mail at 2df05030"}, want: map[string]want{"mail": {nothingNew: true}}},
{name: "taken in by this process at the same trunk: nothing new, not even a look at its keys",
mesh: aTakeIn{registered: map[string]string{"mail": "new"}, trunks: map[string]string{"mail": "new"},
nodes: map[string][]string{"mail": {"ace"}}, holder: "novox"},
keys: []delivered{{module: "mail"}},
memo: map[string]string{"mail": sdkdelivery.SourcePrefix2 + strings.Repeat("m", 64) + "@new"},
acts: []string{"catalogue"}, want: map[string]want{"mail": {nothingNew: true}}},
{name: "delivered again: taken in whole, even when it looks done",
mesh: aTakeIn{registered: map[string]string{"mail": "new"}, trunks: map[string]string{"mail": "new"},
nodes: map[string][]string{"mail": {"ace"}}, taken: map[string]bool{"mail": true}, holder: "novox"},
keys: []delivered{{module: "mail", again: true}},
acts: []string{"catalogue", "compose ace", "compose novox and grants"}, passes: 2,
want: map[string]want{"mail": {taken: true}}},
{name: "a take-in that stopped half-way: composed",
mesh: aTakeIn{registered: map[string]string{"mail": "new"}, trunks: map[string]string{"mail": "new"},
nodes: map[string][]string{"mail": {"ace"}}, holder: "novox"},
keys: []delivered{{module: "mail"}},
acts: []string{"catalogue", "compose ace", "compose novox and grants"}, passes: 2,
want: map[string]want{"mail": {taken: true}}},
{name: "three modules, one pass for their nodes and one for the bus",
mesh: aTakeIn{registered: map[string]string{}, trunks: map[string]string{"mail": "1", "web": "2", "bus": "3"},
nodes: map[string][]string{"mail": {"ace", "novox"}, "web": {"g14", "ace"}, "bus": {"novox"}}, holder: "novox"},
keys: []delivered{{module: "mail"}, {module: "web"}, {module: "bus"}},
acts: []string{"catalogue", "take in mail at 1", "take in web at 2", "take in bus at 3", "compose ace,g14",
"compose novox and grants"}, passes: 2,
want: map[string]want{"mail": {taken: true}, "web": {taken: true}, "bus": {taken: true}}},
{name: "assigned nowhere, held by no catalogue, no trunk value: nothing composed",
mesh: aTakeIn{registered: map[string]string{}, trunks: map[string]string{"mail": "1", "odd": "2"},
nodes: map[string][]string{"mail": nil, "none": {"ace"}}, holder: "novox"},
keys: []delivered{{module: "mail"}, {module: "odd"}, {module: "none"}},
acts: []string{"catalogue", "take in mail at 1"},
want: map[string]want{"mail": {taken: true}, "odd": {}, "none": {}}},
{name: "a grant not written refuses the key, never swallowed",
mesh: aTakeIn{registered: map[string]string{}, trunks: map[string]string{"mail": "1"},
nodes: map[string][]string{"mail": {"ace"}}, holder: "novox", grantsErr: errors.New("the bucket mesh-controller_grants: not found")},
keys: []delivered{{module: "mail"}},
acts: []string{"catalogue", "take in mail at 1", "compose ace", "compose novox and grants"}, passes: 2,
want: map[string]want{"mail": {verdict: refused, err: true}}},
{name: "a node that fails refuses only its modules",
mesh: aTakeIn{registered: map[string]string{}, trunks: map[string]string{"mail": "1", "web": "2"},
nodes: map[string][]string{"mail": {"ace"}, "web": {"g14"}}, holder: "novox",
composeErr: map[string]error{"ace": errors.New("store refused")}},
keys: []delivered{{module: "mail"}, {module: "web"}},
acts: []string{"catalogue", "take in mail at 1", "take in web at 2", "compose ace,g14", "compose novox and grants"},
passes: 2, want: map[string]want{"mail": {verdict: refused, err: true}, "web": {taken: true}}},
{name: "waits and failures: a write's wait acknowledged, a read's refused, a failed key acknowledged",
mesh: aTakeIn{registered: map[string]string{}, trunks: map[string]string{"a": "1", "b": "2", "c": "3"},
nodes: map[string][]string{"a": {"ace"}, "b": {"ace"}, "c": {"ace"}}, holder: "novox",
takeInErr: map[string]error{"a": waitsForAWrite, "b": waitsOnARead, "c": keyFailed}},
keys: []delivered{{module: "a"}, {module: "b"}, {module: "c"}},
acts: []string{"catalogue", "take in a at 1", "take in b at 2", "take in c at 3"},
want: map[string]want{"a": {waits: true}, "b": {verdict: refused, waits: true}, "c": {err: true}}},
}
if !strings.Contains(said, "registered from the builder's done key") {
t.Errorf("said %q", said)
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
memo := c.memo
if memo == nil {
memo = map[string]string{}
}
settled, passes := c.mesh.steps().batch(t.Context(), c.keys, memo)
if !slices.Equal(c.mesh.acts, c.acts) {
t.Fatalf("the acts were\n %q\nwant\n %q", c.mesh.acts, c.acts)
}
if passes != c.passes {
t.Fatalf("%d composer pass(es), want %d", passes, c.passes)
}
if len(settled) != len(c.want) {
t.Fatalf("settled %d module(s), want %d: %+v", len(settled), len(c.want), settled)
}
for _, r := range settled {
w := c.want[r.module]
got := want{verdict: r.verdict, taken: r.taken, nothingNew: r.nothingNew, waits: r.waits != "", err: r.err != nil}
if got != w {
t.Errorf("%s: %+v, want %+v (%v)", r.module, got, w, r.err)
}
if r.taken && memo[r.module] == "" {
t.Errorf("%s was taken in and not remembered", r.module)
}
}
})
}
acts = nil
if _, err := stepsFor(&acts, "new", "new", fp, []string{"novox"}, "novox").take(context.Background(), "mesh-mail"); err != nil {
t.Fatal(err)
}
if got := strings.Join(acts, "; "); got != "compose novox; grants" {
t.Errorf("a module registered at its trunk already: %q", got)
}
// And each delivered key is acknowledged or refused by its module's verdict, the refused with the retry's delay.
t.Run("acknowledged and refused", func(t *testing.T) {
mesh := aTakeIn{registered: map[string]string{}, trunks: map[string]string{"a": "1", "b": "2"},
nodes: map[string][]string{"a": {"ace"}, "b": {"ace"}}, takeInErr: map[string]error{"b": waitsOnARead}}
var acked, naked []string
msg := func(module string) takeInMsg {
return takeInMsg{delivered: delivered{module: module, deliveries: 3}, ack: func() { acked = append(acked, module) },
nak: func(d time.Duration) {
if d != refusalDelay(3) {
t.Errorf("refused with a delay of %s", d)
}
naked = append(naked, module)
}}
}
state := &takeInState{memo: map[string]string{}, raised: map[string]bool{}, saidWait: map[string]string{}}
said := state.take(t.Context(), mesh.steps(), []takeInMsg{msg("a"), msg("b"), msg("a")})
if !slices.Equal(acked, []string{"a", "a"}) || !slices.Equal(naked, []string{"b"}) {
t.Fatalf("acknowledged %v and refused %v", acked, naked)
}
if !strings.Contains(said, "1 taken in") || !strings.Contains(said, "1 waiting") {
t.Fatalf("said %q", said)
}
})
for name, s := range map[string]doneKeySteps{
"assigned nowhere": stepsFor(&acts, "old", "new", fp, nil, "novox"),
"no trunk value": stepsFor(&acts, "old", "new", "", []string{"ace"}, "novox"),
// A key refused for long holds no slot the others need: each consumer leaves far more keys unacknowledged than a
// batch takes, so 8 keys refused on each stop no ninth (the review of ADR 0544: with 8 a consumer, they stopped it
// for good), and a refusal waits longer each time, up to ten minutes.
t.Run("refused keys stop no other", func(t *testing.T) {
for _, bucket := range []string{sdkdelivery.ModulesBucket, sdkdelivery.BuilderBuildsBucket} {
if c := takeInConfig(bucket); c.MaxAckPending < 8+startTakeIns || c.AckPolicy != jetstream.AckExplicitPolicy {
t.Errorf("%s: %d unacknowledged at most (%v): 8 refused keys and a batch do not fit", bucket,
c.MaxAckPending, c.AckPolicy)
}
}
was := time.Duration(0)
for n := uint64(1); n <= 8; n++ {
d := refusalDelay(n)
if d < was || d > doneKeyRetryMost || (n == 1 && d != doneKeyRetry) {
t.Fatalf("the %d. refusal waits %s (the one before %s)", n, d, was)
}
was = d
}
if was != doneKeyRetryMost {
t.Fatalf("the 8. refusal waits %s, want %s", was, doneKeyRetryMost)
}
})
}
// The take-in stalls when it holds work (keys not delivered yet, or a batch handed and not settled) and settles none
// for five minutes; refused keys waiting for their delay are no work, so a quiet mesh is no stall. The first look
// always says what it found, so a stall raised before a restart is cleared once this controller moves (ADR 0544 §6
// and its second review).
func TestATakeInStallsWhenItHoldsWorkAndSettlesNone(t *testing.T) {
at := time.Date(2026, 10, 25, 17, 36, 0, 0, time.UTC)
type reading struct {
after time.Duration
pending, taking uint64
handed, settled uint64
stalled, changed bool
}
for _, c := range []struct {
name string
readings []reading
}{
{"keys not delivered and nothing settled", []reading{
{0, 190, 8, 50, 50, false, true}, // the first look always says what it found
{4 * time.Minute, 190, 8, 50, 50, false, false},
{5 * time.Minute, 190, 8, 50, 50, true, true},
{6 * time.Minute, 190, 8, 50, 50, true, false},
{7 * time.Minute, 180, 8, 60, 60, false, true},
{20 * time.Minute, 0, 0, 60, 60, false, false},
}},
{"refused keys waiting alone are no stall", []reading{
{0, 0, 3, 40, 40, false, true},
{30 * time.Minute, 0, 3, 40, 40, false, false},
}},
{"every key settled, one of them refused by the bus, nothing pending: no stall", []reading{
{0, 0, 1, 3, 3, false, true},
{6 * time.Minute, 0, 1, 3, 3, false, false},
}},
{"a batch handed and never settled is a stall", []reading{
{0, 0, 16, 56, 40, false, true},
{5 * time.Minute, 0, 16, 56, 40, true, true},
}},
} {
acts = nil
if _, err := s.take(context.Background(), "mesh-mail"); err != nil {
t.Fatal(err)
}
if name == "assigned nowhere" && strings.Join(acts, "; ") != "take in mesh-mail at new" {
t.Errorf("%s: %v", name, acts)
}
if name == "no trunk value" && len(acts) != 0 {
t.Errorf("%s: %v", name, acts)
}
}
acts = nil
if _, err := stepsFor(&acts, "old", "new", fp, []string{"ace"}, "novox").take(context.Background(), "unknown"); err != nil || len(acts) != 0 {
t.Errorf("a module the catalogue does not hold: %v %v", acts, err)
t.Run(c.name, func(t *testing.T) {
w := &stallWatch{}
for i, r := range c.readings {
stalled, changed := w.look(takeInBacklog{Pending: r.pending, Taking: r.taking}, r.handed, r.settled,
at.Add(r.after))
if stalled != r.stalled || changed != r.changed {
t.Fatalf("reading %d: stalled %v changed %v, want %v %v", i, stalled, changed, r.stalled, r.changed)
}
}
})
}
}
@@ -115,15 +336,3 @@ func TestAChangedUserListIsANewRevisionOfTheBusModulesCompositionAlone(t *testin
}
}
}
// 1945 (the review's D1): a grant that is not written is the take-in's error, never swallowed: with no send left,
// nothing else would write it, and no want could name the fingerprint.
func TestAGrantNotWrittenIsTheTakeInsError(t *testing.T) {
fp := sdkdelivery.SourcePrefix2 + strings.Repeat("c", 64)
var acts []string
s := stepsFor(&acts, "old", "new", fp, []string{"ace"}, "novox")
s.grants = func(context.Context) error { return errors.New("the bucket mesh-controller_grants: not found") }
if _, err := s.take(context.Background(), "mesh-mail"); err == nil || !strings.Contains(err.Error(), "grants") {
t.Errorf("a grant not written: %v", err)
}
}
@@ -1,184 +0,0 @@
package main
import (
"context"
"fmt"
"strings"
"sync"
"testing"
"time"
)
// A take-in that waits is said once for each reason it waits for, and tried again every doneKeyRetry until it is taken
// in: never left for a key write that may not come (issue 1945, live on 2026-10-25: mesh-host's take-in at 26bd waited,
// said nothing, and nothing woke it until another write).
func TestAWaitingTakeInIsSaidOnceAndTriedAgain(t *testing.T) {
was, wasRetry, wasSay := doneKeyTakeOnce, doneKeyRetry, doneKeysSay
t.Cleanup(func() { doneKeyTakeOnce, doneKeyRetry, doneKeysSay = was, wasRetry, wasSay })
doneKeyRetry = 5 * time.Millisecond
var mu sync.Mutex
var said []string
doneKeysSay = func(s string) { mu.Lock(); said = append(said, s); mu.Unlock() }
calls := 0
doneKeyTakeOnce = func(context.Context, *stores, string) error {
calls++
switch {
case calls <= 2:
return fmt.Errorf("%w: the artifact store could not hold mesh-host's artifact", errKeyWaits)
case calls == 3:
return fmt.Errorf("%w: mesh-host's key cannot be read", errKeyWaits)
}
return nil
}
done := make(chan struct{})
go func() { takeDoneKey(context.Background(), nil, "mesh-host"); close(done) }()
select {
case <-done:
case <-time.After(5 * time.Second):
t.Fatal("a waiting take-in was not tried again until it was taken in")
}
if calls != 4 {
t.Fatalf("tried %d times, want 4", calls)
}
mu.Lock()
defer mu.Unlock()
waits := 0
for _, s := range said {
if strings.Contains(s, "mesh-host waits") {
waits++
}
}
if waits != 2 {
t.Fatalf("said %q: want each of the two reasons once", said)
}
}
// A wake while a take-in of the module already follows it adds no second one: the follower looks again itself.
func TestAWakeWhileATakeInFollowsAddsNone(t *testing.T) {
was, wasRetry, wasSay := doneKeyTakeOnce, doneKeyRetry, doneKeysSay
t.Cleanup(func() { doneKeyTakeOnce, doneKeyRetry, doneKeysSay = was, wasRetry, wasSay })
doneKeyRetry = 20 * time.Millisecond
doneKeysSay = func(string) {}
var mu sync.Mutex
calls := 0
release := make(chan struct{})
doneKeyTakeOnce = func(context.Context, *stores, string) error {
mu.Lock()
calls++
mu.Unlock()
select {
case <-release:
return nil
default:
return fmt.Errorf("%w: the key says building", errKeyWaits)
}
}
follower := make(chan struct{})
go func() { takeDoneKey(context.Background(), nil, "board"); close(follower) }()
time.Sleep(5 * time.Millisecond)
returned := make(chan struct{})
go func() { takeDoneKey(context.Background(), nil, "board"); close(returned) }()
select {
case <-returned:
case <-time.After(time.Second):
t.Fatal("a second wake waited for the follower")
}
close(release)
<-follower
}
// A wake while a take-in of the module follows it is not lost: the follower goes round once more after it succeeds, so
// a newer build's key written meanwhile is taken in (the review of issue 1945).
func TestAWakeDuringATakeInIsTakenAfterIt(t *testing.T) {
was, wasRetry, wasSay := doneKeyTakeOnce, doneKeyRetry, doneKeysSay
t.Cleanup(func() { doneKeyTakeOnce, doneKeyRetry, doneKeysSay = was, wasRetry, wasSay })
doneKeyRetry = 20 * time.Millisecond
doneKeysSay = func(string) {}
var mu sync.Mutex
calls := 0
inFirst := make(chan struct{})
release := make(chan struct{})
doneKeyTakeOnce = func(context.Context, *stores, string) error {
mu.Lock()
calls++
n := calls
mu.Unlock()
if n == 1 {
close(inFirst)
<-release
}
return nil
}
follower := make(chan struct{})
go func() { takeDoneKey(context.Background(), nil, "mesh-host"); close(follower) }()
<-inFirst
takeDoneKey(context.Background(), nil, "mesh-host") // the newer key's wake, while the first runs
close(release)
<-follower
mu.Lock()
defer mu.Unlock()
if calls != 2 {
t.Fatalf("taken %d times: the wake during the take-in was lost", calls)
}
}
// A wake that comes after the follower's last look but before it lets go of the module is not lost: the follower looks
// at the mark again once it has let go, and takes the module once more (the review of issue 1945).
func TestAWakeJustBeforeTheFollowerLetsGoIsTaken(t *testing.T) {
was, wasRetry, wasSay, wasHook := doneKeyTakeOnce, doneKeyRetry, doneKeysSay, doneKeysBeforeLetGo
t.Cleanup(func() {
doneKeyTakeOnce, doneKeyRetry, doneKeysSay, doneKeysBeforeLetGo = was, wasRetry, wasSay, wasHook
})
doneKeyRetry = 10 * time.Millisecond
doneKeysSay = func(string) {}
var mu sync.Mutex
calls := 0
doneKeyTakeOnce = func(context.Context, *stores, string) error {
mu.Lock()
calls++
mu.Unlock()
return nil
}
woke := false
doneKeysBeforeLetGo = func(module string) {
if !woke {
woke = true
takeDoneKey(context.Background(), nil, module) // fails its TryLock and marks the module
}
}
takeDoneKey(context.Background(), nil, "mesh-host")
mu.Lock()
defer mu.Unlock()
if calls != 2 {
t.Fatalf("taken %d times: the wake just before the follower let go was lost", calls)
}
}
// A wake during the start's take-in of a module is not lost: the start pass reads the mark once it lets go of the module,
// and hands the module to takeDoneKey, which takes it once more (the review of issue 1945: the third dropped wake of its
// kind).
func TestAWakeDuringTheStartsTakeInIsTaken(t *testing.T) {
was, wasRetry, wasSay := doneKeyTakeOnce, doneKeyRetry, doneKeysSay
t.Cleanup(func() { doneKeyTakeOnce, doneKeyRetry, doneKeysSay = was, wasRetry, wasSay })
doneKeyRetry = 10 * time.Millisecond
doneKeysSay = func(string) {}
var mu sync.Mutex
calls := 0
doneKeyTakeOnce = func(context.Context, *stores, string) error {
mu.Lock()
calls++
n := calls
mu.Unlock()
if n == 1 {
takeDoneKey(context.Background(), nil, "mesh-host") // the newer key's wake, during the start's take-in
}
return nil
}
takeDoneKeyAtStart(context.Background(), nil, "mesh-host")
background.Wait()
mu.Lock()
defer mu.Unlock()
if calls != 2 {
t.Fatalf("taken %d times: the wake during the start's take-in was lost", calls)
}
}
+10 -4
View File
@@ -316,20 +316,26 @@ func composeReplacedController(ctx context.Context, open *stores) error {
// replacedComposition is the controller's composition on r's node at r's fingerprint: in composed with r's manifest
// in place of the controller's registered one.
func replacedComposition(in compose.Input, r replacedController) (sdkdelivery.Composed, error) {
return keptComposition(in, r.node, controllerModule, r.fp, r.manifest)
}
// keptComposition is module's composition on node at fp: in composed with manifest, that fingerprint's own, in place of
// the module's registered one, in the reading mode (nothing made). in is not changed.
func keptComposition(in compose.Input, node, module, fp string, manifest catalogue.Manifest) (sdkdelivery.Composed, error) {
shelf := make(map[string]catalogue.Manifest, len(in.Catalogue))
for k, v := range in.Catalogue {
shelf[k] = v
}
shelf[controllerModule] = r.manifest
shelf[module] = manifest
in.Catalogue = shelf
all, _, err := compose.Compositions(in, r.node, map[string]string{controllerModule: r.fp}, compose.Reading)
all, _, err := compose.Compositions(in, node, map[string]string{module: fp}, compose.Reading)
if err != nil {
return sdkdelivery.Composed{}, err
}
key, _ := sdkdelivery.ComposedKey(r.node, controllerModule, r.fp)
key, _ := sdkdelivery.ComposedKey(node, module, fp)
c, ok := all[key]
if !ok {
return sdkdelivery.Composed{}, fmt.Errorf("%s composes no %s with the replaced build's manifest", r.node, controllerModule)
return sdkdelivery.Composed{}, fmt.Errorf("%s composes no %s with the manifest of %s", node, module, short(fp))
}
return c, nil
}
+91 -44
View File
@@ -4,6 +4,7 @@ import (
"context"
"encoding/json"
"fmt"
"slices"
"strings"
"testing"
@@ -46,51 +47,97 @@ func TestTheFirstStartDeletesThePlansBucketsAndSaysEachOnce(t *testing.T) {
}
}
// 1945/AC6 (unit half): the controller this one replaced is composed on its node from its own build's manifest at its
// own fingerprint: the composition holds what that manifest declares, not what the controller registered now declares.
func TestTheReplacedControllerIsComposedFromItsOwnManifest(t *testing.T) {
now := catalogue.Manifest{Module: controllerModule, Version: "6", Resources: []map[string]any{
{"id": "settings", "type": "file", "path": "/etc/mesh-controller/settings", "content": "six\n"}}}
before := catalogue.Manifest{Module: controllerModule, Version: "5", Resources: []map[string]any{
{"id": "settings", "type": "file", "path": "/etc/mesh-controller/settings", "content": "five\n"}}}
in := compose.Input{Catalogue: map[string]catalogue.Manifest{controllerModule: now},
Nodes: []compose.Node{{Name: "anchor", Assigned: []string{controllerModule}, Stated: true}}}
fp := sdkdelivery.SourcePrefix2 + strings.Repeat("5", 64)
c, err := replacedComposition(in, replacedController{node: "anchor", fp: fp, manifest: before})
if err != nil {
t.Fatal(err)
}
if c.Module != controllerModule || c.FP != fp || c.Node != "anchor" || len(c.Resources) != 1 {
t.Fatalf("composed %s at %s on %s with %d resources", c.Module, c.FP, c.Node, len(c.Resources))
}
var f struct{ Content string }
_ = json.Unmarshal(c.Resources[0], &f)
if f.Content != "five\n" {
t.Errorf("the replaced controller's composition holds %q, its own manifest says five", f.Content)
}
if in.Catalogue[controllerModule].Version != "6" {
t.Errorf("composing the replaced controller changed the Input's catalogue")
}
}
// An earlier fingerprint is composed from its own build's manifest with today's inputs: the controller this one
// replaced (1945/AC6), and every kept composition of a module whose config changed (ADR 0545: on 2026-10-25 a want back
// to an earlier mesh-build-go on g14 applied the capacity of 0 its kept composition was made with). The composition
// holds what that manifest declares and today's config, the kept keys are those of the touched modules at a
// fingerprint other than the registered one, the build of a fingerprint is its newest that worked, and a write of
// mesh-config's resolved values names the node and module it touches.
func TestAnEarlierFingerprintIsComposedFromItsOwnManifest(t *testing.T) {
fp := func(c string) string { return sdkdelivery.SourcePrefix2 + strings.Repeat(c, 64) }
t.Run("from its own manifest, with today's inputs", func(t *testing.T) {
manifest := func(version string) catalogue.Manifest {
return catalogue.Manifest{Module: controllerModule, Version: version,
Settings: map[string]catalogue.SettingDeclaration{
"capacity": {Kind: catalogue.KindPreference, Default: float64(0), Why: "how many builds at once"}},
Resources: []map[string]any{{"id": "settings", "type": "file", "path": "/etc/mesh-controller/settings",
"content": version + " ${setting:capacity}\n"}}}
}
now, before := manifest("six"), manifest("five")
in := compose.Input{Catalogue: map[string]catalogue.Manifest{controllerModule: now},
Nodes: []compose.Node{{Name: "anchor", Assigned: []string{controllerModule}, Stated: true,
Settings: map[string][]catalogue.Layer{controllerModule: {{From: "mesh", Values: map[string]any{"capacity": float64(4)}}}}}}}
for name, compose := range map[string]func() (sdkdelivery.Composed, error){
"the replaced controller": func() (sdkdelivery.Composed, error) {
return replacedComposition(in, replacedController{node: "anchor", fp: fp("5"), manifest: before})
},
"a kept composition": func() (sdkdelivery.Composed, error) {
return keptComposition(in, "anchor", controllerModule, fp("5"), before)
},
} {
c, err := compose()
if err != nil {
t.Fatalf("%s: %v", name, err)
}
if c.Module != controllerModule || c.FP != fp("5") || c.Node != "anchor" || len(c.Resources) != 1 {
t.Fatalf("%s: composed %s at %s on %s with %d resources", name, c.Module, c.FP, c.Node, len(c.Resources))
}
var f struct{ Content string }
_ = json.Unmarshal(c.Resources[0], &f)
if f.Content != "five 4\n" {
t.Errorf("%s holds %q, want its own manifest's five with today's capacity 4", name, f.Content)
}
}
if in.Catalogue[controllerModule].Version != "six" {
t.Errorf("composing an earlier fingerprint changed the Input's catalogue")
}
})
// 1945/AC6 (unit half): the controller replaced is the newest earlier build of the controller that worked and names a
// fingerprint of the new form, not the one registered now.
func TestTheReplacedControllerIsItsNewestEarlierBuild(t *testing.T) {
now, before, older := sdkdelivery.SourcePrefix2+strings.Repeat("6", 64), sdkdelivery.SourcePrefix2+strings.Repeat("5", 64),
sdkdelivery.SourcePrefix2+strings.Repeat("4", 64)
m := []byte(`{"module":"mesh-controller","version":"5"}`)
builds := []inventory.Build{
{ID: "c6", SourceFingerprint: now, Manifest: m},
{ID: "c5-failed", SourceFingerprint: before, Failed: "no", Manifest: m},
{ID: "c5", SourceFingerprint: before, Manifest: m},
{ID: "c4", SourceFingerprint: older, Manifest: m},
}
if b, ok := replacedBuild(builds, now); !ok || b.ID != "c5" {
t.Errorf("the replaced build is %+v (%v), want c5", b, ok)
}
if _, ok := replacedBuild(builds[:1], now); ok {
t.Errorf("a controller with no earlier build replaced one")
}
t.Run("which build, which keys, which touch", func(t *testing.T) {
m := []byte(`{"module":"mesh-controller","version":"5"}`)
builds := []inventory.Build{
{ID: "c6", SourceFingerprint: fp("6"), Manifest: m},
{ID: "c5-failed", SourceFingerprint: fp("5"), Failed: "no", Manifest: m},
{ID: "c5", SourceFingerprint: fp("5"), Manifest: m},
{ID: "c4", SourceFingerprint: fp("4"), Manifest: m},
}
if b, ok := replacedBuild(builds, fp("6")); !ok || b.ID != "c5" {
t.Errorf("the replaced build is %+v (%v), want c5", b, ok)
}
if _, ok := replacedBuild(builds[:1], fp("6")); ok {
t.Errorf("a controller with no earlier build replaced one")
}
if b, ok := buildAt(builds, fp("5")); !ok || b.ID != "c5" {
t.Errorf("the build at 5 is %+v (%v), want c5, the one that worked", b, ok)
}
if _, ok := buildAt(builds, fp("3")); ok {
t.Errorf("a fingerprint with no build has one")
}
key := func(node, module, f string) string { k, _ := sdkdelivery.ComposedKey(node, module, f); return k }
keys := []string{key("g14", "mesh-build-go", fp("2")), key("g14", "mesh-build-go", fp("1")),
key("g14", "mesh-build-go", fp("3")), key("g14", "hello-web", fp("1")), key("ace", "mesh-build-go", fp("1"))}
registered := map[string]string{"mesh-build-go": fp("3"), "hello-web": fp("9")}
got := keptToFollow(keys, "g14", map[string]bool{"mesh-build-go": true}, registered)
if want := []keptKey{{"mesh-build-go", fp("1")}, {"mesh-build-go", fp("2")}}; !slices.Equal(got, want) {
t.Errorf("kept to follow %v, want %v", got, want)
}
if got := keptToFollow(keys, "g14", nil, registered); len(got) != 3 {
t.Errorf("every module's kept compositions: %v", got)
}
for k, want := range map[string][2]string{"g14.mesh-build-go.capacity": {"g14", "mesh-build-go"},
"g14.mesh-build-go.limits.cpu": {"g14", "mesh-build-go"}, "g14.mesh-build-go": {}, "": {}} {
node, module, ok := configTouched(k)
if [2]string{node, module} != want || ok != (want[0] != "") {
t.Errorf("%q touches %s %s (%v), want %v", k, node, module, ok, want)
}
}
// A value set before the consumer was made reaches the kept compositions: it reads each key once at creation.
if c := configFollowConfig(); c.DeliverPolicy != jetstream.DeliverLastPerSubjectPolicy ||
c.AckPolicy != jetstream.AckExplicitPolicy {
t.Errorf("the config consumer delivers %v, acknowledged %v; want each key's last value, acknowledged",
c.DeliverPolicy, c.AckPolicy)
}
})
}
// 1945/AC14 (unit half): the declarations' stream stays while one node's have names another node-engine than the one
+32 -10
View File
@@ -29,7 +29,13 @@ type composer struct {
type waiter struct {
nodes []string
grants bool
done chan error
done chan passResult
}
// passResult is a waiter's answer: the error of each of its nodes that failed, and of the grants when it asked for them.
type passResult struct {
failed map[string]error
grants error
}
func newComposer(compose func(context.Context, []string) map[string]error, grants func(context.Context) error) *composer {
@@ -39,7 +45,23 @@ func newComposer(compose func(context.Context, []string) map[string]error, grant
// compose asks for the nodes to be composed, and the grants written after when grants is true, and waits for the pass
// that does it, with the errors of its own nodes alone.
func (c *composer) compose(ctx context.Context, nodes []string, grants bool) error {
done := make(chan error, 1)
failed, grantsErr, err := c.composeEach(ctx, nodes, grants)
if err != nil {
return err
}
var errs []error
for _, n := range nodes {
if e := failed[n]; e != nil {
errs = append(errs, e)
}
}
return errors.Join(append(errs, grantsErr)...)
}
// composeEach is compose answering each node's error apart, and the grants' (ADR 0544 §5: a batch of take-ins asks one
// pass and settles each module by its own nodes); err is ctx's when it ends first.
func (c *composer) composeEach(ctx context.Context, nodes []string, grants bool) (map[string]error, error, error) {
done := make(chan passResult, 1)
c.mu.Lock()
for _, n := range nodes {
c.pending[n] = true
@@ -52,10 +74,10 @@ func (c *composer) compose(ctx context.Context, nodes []string, grants bool) err
default:
}
select {
case err := <-done:
return err
case r := <-done:
return r.failed, r.grants, nil
case <-ctx.Done():
return ctx.Err()
return nil, nil, ctx.Err()
}
}
@@ -90,16 +112,16 @@ func (c *composer) run(ctx context.Context) {
grantsErr = c.writeGrants(ctx)
}
for _, w := range waiters {
var errs []error
r := passResult{failed: map[string]error{}}
for _, n := range w.nodes {
if err := failed[n]; err != nil {
errs = append(errs, err)
r.failed[n] = err
}
}
if w.grants && grantsErr != nil {
errs = append(errs, grantsErr)
if w.grants {
r.grants = grantsErr
}
w.done <- errors.Join(errs...)
w.done <- r
}
}
}
+31 -8
View File
@@ -23,10 +23,13 @@ import (
// writes `<module>.<fingerprint hex>` then, naming the accounts it composed for the module, never removed. A module one
// of whose accounts has no password yet (the list leaves it out) has no grant until a later send carries it.
// grantsFrom is the grant of every module whose accounts all have a password in hashes, at the fingerprint current names
// for it; a module whose current build names no fingerprint of the new form has none. principals are the bus's users
// as broker.Users composes them.
func grantsFrom(principals []broker.Principal, hashes map[string]string, current map[string]string) map[string]sdkdelivery.ControllerGrant {
// grantsFrom is the grant of every module at the fingerprint current names for it, naming its accounts that have a
// sealed password in hashes; a module whose current build names no fingerprint of the new form has none. A module one
// of whose accounts has none is granted its sealed accounts alone, and when it has none, its grant is dropped (drop):
// an older grant naming a node whose account is gone would let the assigner want it (issue 1945, rule 6 per node), and
// an empty grant names every node. principals are the bus's users as broker.Users composes them.
func grantsFrom(principals []broker.Principal, hashes map[string]string, current map[string]string) (
map[string]sdkdelivery.ControllerGrant, []string) {
accounts, missing := map[string][]string{}, map[string]bool{}
for _, p := range principals {
if p.Module == "" || (p.Kind != broker.KindModule && p.Kind != broker.KindProcess) {
@@ -39,8 +42,9 @@ func grantsFrom(principals []broker.Principal, hashes map[string]string, current
accounts[p.Module] = append(accounts[p.Module], p.Username())
}
out := map[string]sdkdelivery.ControllerGrant{}
var drop []string
for module, fp := range current {
if missing[module] || !strings.HasPrefix(fp, sdkdelivery.SourcePrefix2) {
if !strings.HasPrefix(fp, sdkdelivery.SourcePrefix2) {
continue
}
key, err := sdkdelivery.ControllerGrantKey(module, fp)
@@ -48,10 +52,15 @@ func grantsFrom(principals []broker.Principal, hashes map[string]string, current
continue
}
names := accounts[module]
if missing[module] && len(names) == 0 {
drop = append(drop, key)
continue
}
sort.Strings(names)
out[key] = sdkdelivery.ControllerGrant{Accounts: names}
}
return out
sort.Strings(drop)
return out, drop
}
// currentFingerprints is, by module, the fingerprint of the build each registered module is registered at: its
@@ -248,13 +257,27 @@ func grantsIssued(ctx context.Context, inv *inventory.Inventory, js jetstream.Je
}
hashes := make(map[string]string, len(kept))
for name, u := range kept {
hashes[name] = u.PasswordHash
// A grant names an account whose credential is sealed to its node, never one only minted: the assigner wants a
// node only once the grant names its account (issue 1945, rule 6 per node).
if u.Sealed {
hashes[name] = u.PasswordHash
}
}
current, err := currentFingerprints(ctx, inv)
if err != nil {
return fmt.Errorf("the grants issued are not written: %w", err)
}
wrote, err := keepGrants(ctx, kv, grantsFrom(principals, hashes, current))
grants, drop := grantsFrom(principals, hashes, current)
wrote, err := keepGrants(ctx, kv, grants)
for _, key := range drop {
if _, gerr := kv.Get(ctx, key); gerr == nil {
if derr := kv.Delete(ctx, key); derr != nil {
err = errors.Join(err, fmt.Errorf("%s: %w", key, derr))
} else {
fmt.Printf("%sthe grant %s dropped: none of its accounts is sealed to its node\n", indent, key)
}
}
}
if wrote > 0 {
fmt.Printf("%s%d grant(s) issued written for the assigner\n", indent, wrote)
}
+17 -4
View File
@@ -23,12 +23,25 @@ func TestTheGrantsAListIssues(t *testing.T) {
y := broker.Principal{Kind: broker.KindModule, Node: "ace", Module: "mesh-y"}
node := broker.Principal{Kind: broker.KindNode, Node: "novox"}
hashes := map[string]string{x.Username(): "h1", xp.Username(): "h2", node.Username(): "h3"}
got := grantsFrom([]broker.Principal{x, xp, y, node}, hashes,
map[string]string{"mesh-x": grantFP('a'), "mesh-y": grantFP('b'), "web": grantFP('c'), "old": "src1:abc"})
// mesh-z is sealed on novox and not on ace (issue 1945: an older grant must not name ace): its grant names novox alone.
zn := broker.Principal{Kind: broker.KindModule, Node: "novox", Module: "mesh-z"}
za := broker.Principal{Kind: broker.KindModule, Node: "ace", Module: "mesh-z"}
hashes[zn.Username()] = "h4"
got, drop := grantsFrom([]broker.Principal{x, xp, y, node, zn, za}, hashes,
map[string]string{"mesh-x": grantFP('a'), "mesh-y": grantFP('b'), "web": grantFP('c'), "old": "src1:abc",
"mesh-z": grantFP('d')})
xKey, _ := sdkdelivery.ControllerGrantKey("mesh-x", grantFP('a'))
yKey, _ := sdkdelivery.ControllerGrantKey("mesh-y", grantFP('b'))
zKey, _ := sdkdelivery.ControllerGrantKey("mesh-z", grantFP('d'))
webKey, _ := sdkdelivery.ControllerGrantKey("web", grantFP('c'))
if len(got) != 2 {
t.Fatalf("grants = %v, want mesh-x's and web's", got)
if len(got) != 3 {
t.Fatalf("grants = %v, want mesh-x's, mesh-z's and web's", got)
}
if g := got[zKey]; len(g.Accounts) != 1 || g.Accounts[0] != zn.Username() {
t.Errorf("mesh-z's grant = %+v, want its sealed account alone", g)
}
if len(drop) != 1 || drop[0] != yKey {
t.Errorf("dropped %v, want mesh-y's grant, none of whose accounts is sealed", drop)
}
if g := got[xKey]; len(g.Accounts) != 2 || g.Accounts[0] != x.Username() && g.Accounts[1] != x.Username() {
t.Errorf("mesh-x's grant = %+v, want both its accounts", g)
+352
View File
@@ -0,0 +1,352 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"sort"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
sdkdelivery "git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
)
// **A kept composition follows its inputs** (novox/hq ADR 0545, issue 1945). The controller composes a module only at
// the fingerprint it is registered at, and keeps the compositions of up to five fingerprints (ADR 0539 §4) so that
// going back is one want. A kept one was never composed again, so it carried the config of the moment it was made: on
// 2026-10-25 a want back to an earlier mesh-build-go on g14 applied its capacity of 0, the value from before it was
// set. Now a write of mesh-config's resolved values for a module on a node, heard from that bucket, composes the node
// and then every kept fingerprint of that module on that node again, each from its own build's manifest with today's
// inputs, writing only those whose composition differs (writeComposed). The writes are read from a durable consumer,
// acknowledged once handled, so a change made while no controller ran is delivered to the next one, and a start
// composes again only what was written since the last write handled, never every kept composition.
// configTouched is the node and module a key of mesh-config_resolved (`<node>.<module>.<key>`) names.
func configTouched(key string) (node, module string, ok bool) {
parts := strings.SplitN(key, ".", 3)
if len(parts) != 3 || parts[0] == "" || parts[1] == "" {
return "", "", false
}
return parts[0], parts[1], true
}
// keptKey is one kept composition to compose again.
type keptKey struct{ module, fp string }
// keptToFollow is, of a node's composition keys, each of modules (every module when nil) at a fingerprint other than
// the one it is registered at, sorted: the kept compositions. The node's own composition and other nodes' are not.
func keptToFollow(keys []string, node string, modules map[string]bool, registered map[string]string) []keptKey {
var out []keptKey
for _, key := range keys {
n, module, fp, isNode, err := sdkdelivery.ParseComposedKey(key)
if err != nil || isNode || n != node || (modules != nil && !modules[module]) {
continue
}
if fp == registered[module] || !strings.HasPrefix(fp, sdkdelivery.SourcePrefix2) {
continue
}
out = append(out, keptKey{module, fp})
}
sort.Slice(out, func(i, j int) bool {
if out[i].module != out[j].module {
return out[i].module < out[j].module
}
return out[i].fp < out[j].fp
})
return out
}
// buildAt is, of a module's builds newest first, the newest that worked at fp and kept its manifest.
func buildAt(builds []inventory.Build, fp string) (inventory.Build, bool) {
for _, b := range builds {
if b.Worked() && b.SourceFingerprint == fp && len(b.Manifest) > 0 {
return b, true
}
}
return inventory.Build{}, false
}
// followKept composes again, on node, every kept composition of modules (every module when nil) with today's inputs,
// and writes those that differ. It answers the keys written; a fingerprint whose build's manifest is gone is said and
// left as it is.
func followKept(ctx context.Context, open *stores, node string, modules map[string]bool) ([]string, error) {
if _, err := theLease.epoch(ctx); err != nil {
return nil, err
}
bus, err := aBus()
if err != nil {
return nil, err
}
defer bus.Close()
js, err := jetstream.New(bus.Conn())
if err != nil {
return nil, err
}
inv := open.inventory
registered, err := currentFingerprints(ctx, inv)
if err != nil {
return nil, err
}
at, err := openComposedPlace(ctx, js, bus.Conn(), node)
if err != nil {
return nil, err
}
lister, err := at.bucket.ListKeys(ctx)
if err != nil {
if errors.Is(err, jetstream.ErrNoKeysFound) {
return nil, nil // empty-on-error: a node with no compositions keeps none to compose again
}
return nil, err
}
var keys []string
for k := range lister.Keys() {
keys = append(keys, k)
}
kept := keptToFollow(keys, node, modules, registered)
if len(kept) == 0 {
return nil, nil
}
sign, err := controllerSigner(ctx, open)
if err != nil {
return nil, err
}
upgrades, err := inv.Upgrades(ctx)
if err != nil {
return nil, err
}
at.sign = sign
at.policy = func(module string) (string, string, string) {
u, ok := upgrades[module]
if !ok {
return "", "", ""
}
return u.Policy(), u.From, u.Why
}
in, err := inputFor(ctx, open)
if err != nil {
return nil, err
}
var wrote []string
var errs []error
builds := map[string][]inventory.Build{}
for _, k := range kept {
if ctx.Err() != nil {
return wrote, errors.Join(append(errs, ctx.Err())...) // stopped: the rest is the next controller's
}
if _, read := builds[k.module]; !read {
if builds[k.module], err = inv.Builds(ctx, k.module, 20); err != nil {
errs = append(errs, fmt.Errorf("%s's builds: %w", k.module, err))
continue
}
}
b, found := buildAt(builds[k.module], k.fp)
if !found {
fmt.Printf("compositions: %s's kept %s on %s has no build whose manifest is kept, and is not composed again\n",
k.module, short(k.fp), node)
continue
}
var m catalogue.Manifest
if err := json.Unmarshal(b.Manifest, &m); err != nil {
errs = append(errs, fmt.Errorf("the manifest of %s's build %s: %w", k.module, b.ID, err))
continue
}
c, err := keptComposition(in.Input, node, k.module, k.fp, m)
if err != nil {
errs = append(errs, fmt.Errorf("%s at %s on %s: %w", k.module, short(k.fp), node, err))
continue
}
key, _ := sdkdelivery.ComposedKey(node, k.module, k.fp)
// One write on the node's keys at a time in this process, held only for the write: composing in memory holds
// nothing, so a sweep never keeps a take-in from the node (the review of ADR 0545).
unlock := lockComposing(node)
w, err := writeComposed(ctx, at, key, c)
unlock()
if err != nil {
errs = append(errs, err)
continue
}
if w {
wrote = append(wrote, key)
}
}
if len(wrote) > 0 {
fmt.Printf("compositions: %s: kept compositions composed again with today's inputs: %s\n", node, strings.Join(wrote, ", "))
}
return wrote, errors.Join(errs...)
}
// configFollowGather is how long the config consumer waits after a write for more before it composes: mesh-config writes
// a module's keys one by one.
var configFollowGather = time.Second
// configFollowConsumer is the controller's durable consumer of mesh-config's resolved values (ADR 0545 §1, §3).
const configFollowConsumer = broker.ConfigFollowConsumer
// configFollowConfig is that consumer: each resolved key's last value once when it is made, then each write,
// acknowledged after the write is handled, so what a stop cut is delivered to the next controller. The first start of
// the release that makes it composes again the kept compositions of every module with config, through the same gather,
// lock and acknowledgement: mesh-config never writes an unchanged value again, so a value set before this release (g14's
// mesh-build-go capacity of 2, against kept compositions carrying 0) would otherwise never reach them. Its deliver
// policy is set before its first delivery; an existing durable's cannot be changed.
func configFollowConfig() jetstream.ConsumerConfig {
return jetstream.ConsumerConfig{Durable: configFollowConsumer, FilterSubject: "$KV." + broker.ConfigResolvedBucket + ".>",
DeliverPolicy: jetstream.DeliverLastPerSubjectPolicy, AckPolicy: jetstream.AckExplicitPolicy, AckWait: takeInAckWait,
MaxAckPending: takeInPending, Description: "the controller's following of config into kept compositions (novox/hq ADR 0545)"}
}
// kindConfigNotFollowed is the condition of a config consumer that cannot be read: a change of config composes nothing
// until it is.
const kindConfigNotFollowed = "config-not-followed"
func configNotFollowedObservation(err error) conditions.Observation {
return conditions.Observation{Scope: conditions.ScopeCore, ID: "config-follow", Token: kindConfigNotFollowed,
Kind: kindConfigNotFollowed, Severity: conditions.Urgent, Resolver: conditions.ResolverOperator, Source: "config",
Summary: fmt.Sprintf("mesh-config's resolved values are not heard, so a change of config composes nothing: %v", err)}
}
// watchConfigFollow reads the writes of mesh-config's resolved values from the controller's durable consumer and, for
// each node and module they touch, composes the node and the module's kept compositions there again, until ctx ends
// (ADR 0545). A consumer that cannot be made or read raises config-not-followed and is tried again a minute later; it is
// cleared once it is read.
func watchConfigFollow(ctx context.Context, open *stores) {
goServing(ctx, func() {
for {
err := followConfigOnce(ctx, open)
if ctx.Err() != nil {
return
}
fmt.Printf("config: %v; tried again in a minute\n", err)
if keeper := conditionsFrom; keeper != nil {
_, _ = keeper.Observe(ctx, configNotFollowedObservation(err))
}
select {
case <-ctx.Done():
return
case <-time.After(time.Minute):
}
}
})
}
// followConfigOnce reads the consumer until it fails or ctx ends.
func followConfigOnce(ctx context.Context, open *stores) error {
bus, err := aBus()
if err != nil {
return err
}
defer bus.Close()
js, err := jetstream.New(bus.Conn())
if err != nil {
return err
}
c, err := js.CreateOrUpdateConsumer(ctx, "KV_"+broker.ConfigResolvedBucket, configFollowConfig())
if err != nil {
return fmt.Errorf("the consumer of %s: %w", broker.ConfigResolvedBucket, err)
}
it, err := c.Messages(jetstream.PullMaxMessages(startTakeIns))
if err != nil {
return fmt.Errorf("the consumer of %s: %w", broker.ConfigResolvedBucket, err)
}
defer it.Stop()
if keeper := conditionsFrom; keeper != nil {
_, _ = keeper.Clear(ctx, configNotFollowedObservation(nil).Key(), "mesh-config's resolved values are heard")
}
type next struct {
m jetstream.Msg
err error
}
got := make(chan next)
go func() {
for {
m, err := it.Next()
select {
case got <- next{m, err}:
case <-ctx.Done():
if m != nil {
_ = m.Nak()
}
return
}
if err != nil {
return
}
}
}()
var held []jetstream.Msg
touched := map[string]map[string]bool{}
var gather <-chan time.Time
for {
select {
case <-ctx.Done():
for _, m := range held {
_ = m.Nak() // delivered at once to the next controller
}
return nil
case n := <-got:
if n.err != nil {
for _, m := range held {
_ = m.Nak()
}
return fmt.Errorf("the consumer of %s stopped: %w", broker.ConfigResolvedBucket, n.err)
}
held = append(held, n.m)
key := strings.TrimPrefix(n.m.Subject(), "$KV."+broker.ConfigResolvedBucket+".")
if node, module, ok := configTouched(key); ok {
if touched[node] == nil {
touched[node] = map[string]bool{}
}
touched[node][module] = true
}
if gather == nil {
gather = time.After(configFollowGather)
}
case <-gather:
gather = nil
now, msgs := touched, held
touched, held = map[string]map[string]bool{}, nil
forgetResolved() // the next composition reads the values just written, not a read of the last 5 s
failed := false
for node, modules := range now {
if ctx.Err() != nil {
failed = true // stopped: the writes are refused below and delivered to the next controller
break
}
if err := composerFor(open).compose(ctx, []string{node}, false); err != nil {
fmt.Printf("config: %s is not composed after its config changed: %v\n", node, err)
failed = true
}
if _, err := followKept(ctx, open, node, modules); err != nil {
fmt.Printf("config: %s's kept compositions are not composed again: %v\n", node, err)
failed = true
}
}
for _, m := range msgs {
if failed && ctx.Err() != nil {
_ = m.Nak() // stopped: delivered at once to the next controller
} else if failed {
_ = m.NakWithDelay(doneKeyRetry) // tried again: the bus delivers the writes again
} else {
acking, cancel := context.WithTimeout(context.WithoutCancel(ctx), 10*time.Second)
if err := m.DoubleAck(acking); err != nil {
fmt.Printf("config: the bus did not take the acknowledgement of %s: %v\n", m.Subject(), err)
}
cancel()
}
}
}
}
}
// forgetResolved drops the last read of mesh-config's resolved values.
func forgetResolved() {
theResolved.mu.Lock()
theResolved.entries = nil
theResolved.mu.Unlock()
}
+11 -3
View File
@@ -39,6 +39,11 @@ import (
// errKeyWaits says a take-in from a key waits: for the builder, a reference, the store or the forge. Never a failure.
var errKeyWaits = errors.New("waits")
// errWaitsForAWrite is a take-in that waits for the builder or the indexer to write again (the key not done, its
// artifacts not referenced, no fingerprint of the new form): that write wakes it, so nothing looks again before it
// (ADR 0544 §4).
var errWaitsForAWrite = fmt.Errorf("%w for the builder's or the indexer's next write", errKeyWaits)
// errKeyFailed says the module's key failed: the builder's own words follow.
var errKeyFailed = errors.New("the builder's key failed")
@@ -298,12 +303,15 @@ func buildOfKey(ctx context.Context, keys keyReads, t keyTake, kc keyContext) (
waits := func(format string, a ...any) (inventory.Build, catalogue.Manifest, error) {
return inventory.Build{}, catalogue.Manifest{}, fmt.Errorf("%w: "+format, append([]any{errKeyWaits}, a...)...)
}
waitsForAWrite := func(format string, a ...any) (inventory.Build, catalogue.Manifest, error) {
return inventory.Build{}, catalogue.Manifest{}, fmt.Errorf("%w: "+format, append([]any{errWaitsForAWrite}, a...)...)
}
if t.Trunk.FP2 == "" && t.Hand == nil {
why := t.Trunk.None2
if why == "" {
why = "the indexer wrote no fingerprint of the new form"
}
return waits("%s at %.8s has no fingerprint to take a build by: %s", t.Module, t.Trunk.Commit, why)
return waitsForAWrite("%s at %.8s has no fingerprint to take a build by: %s", t.Module, t.Trunk.Commit, why)
}
k, err := keys.Build(ctx, t.Module, t.Trunk.FP2)
if t.Hand != nil {
@@ -319,9 +327,9 @@ func buildOfKey(ctx context.Context, keys keyReads, t keyTake, kc keyContext) (
case k.Failed():
return inventory.Build{}, catalogue.Manifest{}, fmt.Errorf("%w: %s", errKeyFailed, k.Says())
case !k.Done():
return waits("%s", k.Says())
return waitsForAWrite("%s", k.Says())
case len(k.Unreferenced()) > 0:
return waits("%s", k.Says())
return waitsForAWrite("%s", k.Says())
}
src, err := keys.Manifest(ctx, k.Build.Manifest)
if err != nil {
+109 -31
View File
@@ -9,8 +9,10 @@ import (
"net"
"os"
"path/filepath"
"slices"
"sort"
"strings"
"sync"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
@@ -965,24 +967,99 @@ func processPrincipalFor(m catalogue.Manifest, processID string) (broker.Process
return broker.ProcessPrincipal{}, false
}
// mintOnlyIfAbsent marks a context whose issue mints only accounts no row names yet (onlyIfAbsent).
// mintOnlyIfAbsent marks a context whose issue mints only accounts that are not issued (onlyIfAbsent).
type mintOnlyIfAbsent struct{}
// onlyIfAbsent is ctx for an issue that must never replace a minted account: the assign act's and the assignments'
// watch's, which may run at once for one account (novox/hq ADR 0543, the review of issue 1945).
// onlyIfAbsent is ctx for an issue that never replaces an issued account: the assign act's, the assignments' watch's
// and a registration's, which may run at once for one account (novox/hq ADR 0543, issue 1945).
func onlyIfAbsent(ctx context.Context) context.Context {
return context.WithValue(ctx, mintOnlyIfAbsent{}, true)
}
// mintBusUser mints u's password. By default it replaces whatever stands (a rotation: `module issue`); under
// onlyIfAbsent it mints only when no row names u, and won false says another caller minted it first and seals it,
// so this one seals nothing.
func mintBusUser(ctx context.Context, inv *inventory.Inventory, u inventory.BusUser) (string, bool, error) {
if only, _ := ctx.Value(mintOnlyIfAbsent{}).(bool); only {
return inv.MintBusPasswordIfAbsent(ctx, u)
// issuedNow collects the accounts an issue minted and sealed, for the line that says it (withIssued).
type issuedNow struct {
mu sync.Mutex
users []string
}
type issuedNowKey struct{}
// withIssued is ctx collecting the accounts its issues mint and seal.
func withIssued(ctx context.Context) (context.Context, *issuedNow) {
got := &issuedNow{}
return context.WithValue(ctx, issuedNowKey{}, got), got
}
func (g *issuedNow) list() []string {
if g == nil {
return nil
}
password, err := inv.MintBusPassword(ctx, u)
return password, err == nil, err
g.mu.Lock()
defer g.mu.Unlock()
return slices.Clone(g.users)
}
// accountIssuer is what issuing one account needs of the store.
type accountIssuer interface {
LockBusAccount(ctx context.Context, username string) (func(), error)
BusAccountIssued(ctx context.Context, username string) (minted, sealed bool, err error)
MintBusPassword(ctx context.Context, u inventory.BusUser) (string, error)
MintBusPasswordUnsealed(ctx context.Context, u inventory.BusUser) (string, error)
MarkBusAccountSealed(ctx context.Context, username string) error
}
// issueLockedFor is the longest an issue holds an account's lock: a seal cut by it leaves the account unsealed, issued
// again by the next issue.
const issueLockedFor = 2 * time.Minute
// issueAccount mints u's password and seals it to its node with seal. By default it replaces whatever stands (a
// rotation: `module issue`). Under onlyIfAbsent it issues only an account that is not issued, a row with no sealed
// credential counting as none (issue 1945, live on 2026-10-25 at 20:04: a row whose credential the node no longer held
// left ace's mesh-build-go with no credential), under the store's lock on the account, so two issuers never interleave
// their look, mint and seal: the hash the bus holds is always the password sealed. A seal that fails leaves the
// account unsealed, issued again by the next issue.
func issueAccount(ctx context.Context, accounts accountIssuer, u inventory.BusUser, seal func(password string) error) error {
only, _ := ctx.Value(mintOnlyIfAbsent{}).(bool)
if !only {
password, err := accounts.MintBusPassword(ctx, u)
if err != nil {
return err
}
if err := seal(password); err != nil {
return err
}
} else {
// A deadline on the locked section: it holds a connection of the store's pool across the vault and the bus.
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(ctx, issueLockedFor)
defer cancel()
release, err := accounts.LockBusAccount(ctx, u.Username)
if err != nil {
return err
}
defer release()
if minted, sealed, err := accounts.BusAccountIssued(ctx, u.Username); err != nil {
return err
} else if minted && sealed {
return nil
}
password, err := accounts.MintBusPasswordUnsealed(ctx, u)
if err != nil {
return err
}
if err := seal(password); err != nil {
return err
}
if err := accounts.MarkBusAccountSealed(ctx, u.Username); err != nil {
return fmt.Errorf("%s is sealed and not marked so, and is issued again by the next issue: %w", u.Username, err)
}
}
if got, _ := ctx.Value(issuedNowKey{}).(*issuedNow); got != nil {
got.mu.Lock()
got.users = append(got.users, u.Username)
got.mu.Unlock()
}
return nil
}
// issueOnTheNewBus gives an assigned module its credential on the bus being built.
@@ -1010,17 +1087,12 @@ func issueOnTheNewBus(ctx context.Context, inv *inventory.Inventory, m catalogue
}
if _, reads := m.OwnSecrets["broker"]; reads {
user := broker.Principal{Kind: broker.KindModule, Node: node, Module: m.Module}.Username()
password, won, err := mintBusUser(ctx, inv, inventory.BusUser{
Username: user, Kind: busKindOf(m.Module), Node: node, Module: m.Module,
})
if err != nil {
if err := issueAccount(ctx, inv, inventory.BusUser{Username: user, Kind: busKindOf(m.Module), Node: node,
Module: m.Module}, func(password string) error {
return issueWith(ctx, inv, m, node, busAddress, known, reachable, user, password)
}); err != nil {
return err
}
if won {
if err := issueWith(ctx, inv, m, node, busAddress, known, reachable, user, password); err != nil {
return err
}
}
}
// Every process with a bus declaration gets an account of its own (novox/hq ADR 0436 §1, §2), minted and
// delivered whether or not the module itself speaks on the bus — its own secret is broker-<process id>,
@@ -1043,12 +1115,15 @@ func issueOnTheNewBus(ctx context.Context, inv *inventory.Inventory, m catalogue
func issueProcess(ctx context.Context, inv *inventory.Inventory, m catalogue.Manifest, pr broker.ProcessPrincipal,
node, busAddress string, known broker.Broker, reachable string) error {
user := broker.Principal{Kind: broker.KindProcess, Node: node, Module: m.Module, Process: pr.Process}.Username()
password, won, err := mintBusUser(ctx, inv, inventory.BusUser{
Username: user, Kind: inventory.BusProcess, Node: node, Module: m.Module,
})
if err != nil || !won {
return err // lost: the caller that minted it seals it
}
return issueAccount(ctx, inv, inventory.BusUser{Username: user, Kind: inventory.BusProcess, Node: node, Module: m.Module},
func(password string) error {
return sealProcess(ctx, inv, m, pr, node, busAddress, known, reachable, user, password)
})
}
// sealProcess seals a process's minted account to its node, hands it to the vault and makes its work-queue consumers.
func sealProcess(ctx context.Context, inv *inventory.Inventory, m catalogue.Manifest, pr broker.ProcessPrincipal,
node, busAddress string, known broker.Broker, reachable, user, password string) error {
name := processSecretName(pr.Process)
held, err := heldProcessAccount(known, reachable, node, m.Module, user, password)
if err != nil {
@@ -1113,12 +1188,15 @@ func runsTheProvisionerClient(m catalogue.Manifest) bool {
func issueProvisionerClient(ctx context.Context, inv *inventory.Inventory, m catalogue.Manifest, node string,
known broker.Broker, reachable string) error {
user := broker.Principal{Kind: broker.KindProvisioner, Node: node, Module: m.Module}.Username()
password, won, err := mintBusUser(ctx, inv, inventory.BusUser{
Username: user, Kind: inventory.BusProvisioner, Node: node, Module: m.Module,
return issueAccount(ctx, inv, inventory.BusUser{Username: user, Kind: inventory.BusProvisioner, Node: node,
Module: m.Module}, func(password string) error {
return sealProvisionerClient(ctx, inv, m, node, known, reachable, user, password)
})
if err != nil || !won {
return err // lost: the caller that minted it seals it
}
}
// sealProvisionerClient seals the provisioner client's minted account to its machine and hands it to the vault.
func sealProvisionerClient(ctx context.Context, inv *inventory.Inventory, m catalogue.Manifest, node string,
known broker.Broker, reachable, user, password string) error {
held, err := json.Marshal(struct {
URL string `json:"url"`
Fingerprint string `json:"fingerprint,omitempty"`
+3
View File
@@ -208,6 +208,9 @@ func watchNewModules(ctx context.Context, open *stores) {
// composed, the replaced controller composed, and from now on each new fingerprint registered, composed and granted.
inBackground(ctx, func() { composeAtStart(ctx, open) })
watchDoneKeys(ctx, open)
// And a change of config composes the node and the touched modules' kept compositions again, heard from
// mesh-config's resolved values (ADR 0545, kept_compositions.go).
watchConfigFollow(ctx, open)
// And each assignment that comes to stand issues its node's bus accounts and the module's grant (ADR 0543): a module
// built long before is assigned to one more node with no build done again (issue 1945, assignment_accounts.go).
watchAssignmentAccounts(ctx, open)
+21
View File
@@ -70,6 +70,27 @@ var plainWordings = map[string]func(conditions.Observation) words{
"none can be given it yet. It is tried again every half minute; the details say what fails.", m),
Resolved: "the new build of " + m + " can be delivered"}
}),
// The take-in of done keys whose backlog stood still (take_in_consumer.go, novox/hq ADR 0544 §6).
kindTakeInStalled: worded(func(o conditions.Observation) words {
return words{Headline: "New builds are not being prepared for the machines",
Explanation: "The mesh prepares each new build for the machines as it is made, and has prepared none for " +
"five minutes while builds wait. Until it moves, no new build reaches a machine; the details say how many wait.",
Resolved: "new builds are being prepared again"}
}),
// A bucket of done keys whose take-in consumer could not be made (take_in_consumer.go, novox/hq ADR 0544 §6).
kindTakeInNotConsumed: worded(func(o conditions.Observation) words {
return words{Headline: "Some new builds are not being prepared",
Explanation: "The mesh could not start reading one of the two places new builds are announced in, so the " +
"builds announced there wait until the controller starts again. The details say which and why.",
Resolved: "the mesh reads where new builds are announced again"}
}),
// Config not followed into the compositions (kept_compositions.go, novox/hq ADR 0545).
kindConfigNotFollowed: worded(func(o conditions.Observation) words {
return words{Headline: "Changes of settings are not reaching the machines",
Explanation: "The mesh could not read the settings as they change, so a changed setting is not prepared for " +
"the machines until it can. It tries again every minute; the details say why.",
Resolved: "changes of settings reach the machines again"}
}),
// Machines and their node-engine.
"silent": worded(func(o conditions.Observation) words {
m := machineOr(o, "a machine")
+6 -3
View File
@@ -94,15 +94,18 @@ func issueMissingProcessAccounts(ctx context.Context, inv *inventory.Inventory,
if err != nil {
busAddress = ""
}
var ids []string
ctx, got := withIssued(ctx)
for _, pr := range missing {
if err := issueProcess(ctx, inv, m, pr, node, busAddress, known, reachable); err != nil {
return "", err
}
ids = append(ids, pr.Process)
}
issued := got.list()
if len(issued) == 0 {
return "", nil // another issuer sealed them meanwhile
}
return fmt.Sprintf("its process account(s) %s issued and sealed to %s, and arrive with the push",
strings.Join(ids, ", "), node), nil
strings.Join(issued, ", "), node), nil
}
// issueGainedProcessAccounts runs after a module's new definition is registered: on each machine assigned it, the
+4
View File
@@ -111,6 +111,9 @@ type meshStatus struct {
// why they could not be read.
Pending []pendingStatus `json:"pending,omitempty"`
PendingUnread string `json:"pendingUnread,omitempty"`
// TakeIn is the take-in's backlog of done keys: keys not delivered yet and keys being taken in (novox/hq ADR 0544
// §6). Absent where it cannot be read.
TakeIn *takeInBacklog `json:"take-in,omitempty"`
}
// pendingStatus is one pending assignment as status says it.
@@ -266,6 +269,7 @@ func statusAsJSON(asked answers) ([]byte, error) {
out.Failing = providerStandings(asked.conditions)
out.Overflowing = asked.overflowing
out.PendingUnread = asked.pendingUnread
out.TakeIn = asked.takeIn
for _, p := range asked.pending {
out.Pending = append(out.Pending, pendingStatus{Node: p.Node, Module: p.Module, State: p.State,
Build: p.Build, Repository: p.Repository, Path: p.Path, Since: p.Since, Settled: p.Settled, Note: p.Note})
+11
View File
@@ -151,6 +151,10 @@ func printStatus(asked answers) error {
}
printPending(asked.pending, asked.pendingUnread)
if b := asked.takeIn; b != nil && b.Total() > 0 {
fmt.Printf("the take-in of done keys has %d key(s) not taken in yet: %d waiting to be delivered, %d being taken in\n\n",
b.Total(), b.Pending, b.Taking)
}
if len(behind) > 0 {
var names []string
@@ -471,6 +475,13 @@ func theThreeQuestions(ctx context.Context, open *stores) (answers, error) {
out.pendingUnread = err.Error()
err = nil
}
// And the take-in's backlog of done keys, read from its consumers on the bus (novox/hq ADR 0544 §6): where there is
// a bus; a stall is a condition of its own, so this is said and judges nothing.
if _, onBus := broker.BusAddress(); onBus == nil {
if b, berr := readTakeInBacklog(ctx); berr == nil {
out.takeIn = &b
}
}
// And how many repairs were done by hand this week (novox/hq to-be 45 §7) — where there is a bus
// to read the log from; a process with none has no log to count.
if _, onBus := broker.BusAddress(); onBus == nil {
+452
View File
@@ -0,0 +1,452 @@
package main
import (
"context"
"errors"
"fmt"
"strings"
"sync"
"sync/atomic"
"time"
sdkdelivery "git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/conditions"
)
// The take-in's two durable pull consumers (novox/hq ADR 0544; research 072, milestones 2 and 4). On the indexer's
// modules bucket the consumer delivers each module's last trunk value when it is made, then each write: the start's look
// of a controller whose consumer is new, and nothing at a restart but what the controller before it left
// unacknowledged. On the builder's builds bucket it delivers each write from when it is made. A key is acknowledged once
// its module is taken in, has nothing new, or waits for a write that will be delivered; any other wait or failure is
// refused with a delay that grows with each refusal, and the bus delivers it again.
//
// **The bound on the store is the one take-in loop**, a batch of at most startTakeIns keys at a time (issues 1945 and
// 1277: a burst of take-ins once filled the store until it refused every connection). The consumers' own limit on
// unacknowledged keys is far above it, because a refused key stays unacknowledged until its delay ends: with a limit of
// a batch, a handful of keys refused for long (a manifest word this controller does not read yet) would stop a consumer
// for good, the build that reads the word among what it holds back (the review of ADR 0544).
// startTakeIns is the most a batch takes: the take-ins that may wait on the store at once.
const startTakeIns = 16
// takeInPending is how many keys each consumer may leave unacknowledged: delivered and being taken in, or refused and
// waiting for their delay. Far above a batch, so refused keys never stop the others.
const takeInPending = 1000
// takeInBuffered is how many delivered keys each consumer's reader holds ahead of the loop.
const takeInBuffered = startTakeIns / 2
// takeInAckWait is how long a delivered key may stay unacknowledged before the bus delivers it again: longer than a batch
// takes, its two composer passes among it.
const takeInAckWait = 10 * time.Minute
// takeInGather is how long a batch waits for more keys after its first, so a burst is one batch.
var takeInGather = 200 * time.Millisecond
// takeInStall is how long a backlog may stand without a key delivered before it is a condition (ADR 0544 §6).
var takeInStall = 5 * time.Minute
// takeInMsg is one delivered key, as the take-in uses it.
type takeInMsg struct {
delivered
ack func()
nak func(delay time.Duration)
}
// takeInKeyOf is the module a delivered key names, false for a delete or a purge, which takes nothing in.
func takeInKeyOf(bucket string, m jetstream.Msg) (delivered, bool) {
if op := m.Headers().Get("KV-Operation"); op == "DEL" || op == "PURGE" {
return delivered{}, false
}
key := strings.TrimPrefix(m.Subject(), "$KV."+bucket+".")
module, _, _ := strings.Cut(key, ".")
if module == "" {
return delivered{}, false
}
var deliveries uint64 = 1
if meta, err := m.Metadata(); err == nil {
deliveries = meta.NumDelivered
}
return delivered{module: module, again: deliveries > 1, deliveries: deliveries}, true
}
// takeInConsumer is one bucket's consumer of the take-in.
type takeInConsumer struct {
bucket string
consumer jetstream.Consumer
}
// takeInConfig is the take-in's consumer on bucket's stream.
func takeInConfig(bucket string) jetstream.ConsumerConfig {
deliver := jetstream.DeliverNewPolicy
if bucket == sdkdelivery.ModulesBucket {
deliver = jetstream.DeliverLastPerSubjectPolicy // the start's look: every module's trunk value once
}
return jetstream.ConsumerConfig{
Durable: broker.TakeInConsumer, FilterSubject: "$KV." + bucket + ".>", DeliverPolicy: deliver,
AckPolicy: jetstream.AckExplicitPolicy, AckWait: takeInAckWait, MaxAckPending: takeInPending,
Description: "the controller's take-in of done keys (novox/hq ADR 0544)",
}
}
// takeInConsumers makes or finds the take-in's consumer on each bucket's stream; one that cannot be made is said, and
// the other is still read.
func takeInConsumers(ctx context.Context, js jetstream.JetStream) []takeInConsumer {
var out []takeInConsumer
for _, bucket := range broker.TakeInBuckets {
c, err := js.CreateOrUpdateConsumer(ctx, "KV_"+bucket, takeInConfig(bucket))
o := takeInNotConsumedObservation(bucket, err)
if err != nil {
// Loud: a consumer whose deliver policy differs cannot be updated, and nothing of its bucket is taken in.
doneKeysSay("done keys: " + o.Summary)
if keeper := conditionsFrom; keeper != nil {
_, _ = keeper.Observe(ctx, o)
}
continue
}
// Made: a condition a controller before this one raised for it is cleared (a restart that works clears it).
if keeper := conditionsFrom; keeper != nil {
_, _ = keeper.Clear(ctx, o.Key(), "the take-in's consumer on "+bucket+" is made")
}
out = append(out, takeInConsumer{bucket: bucket, consumer: c})
}
return out
}
// watchDoneKeys takes in the done keys the two consumers deliver, batch after batch, until ctx ends, and watches their
// backlog. A consumer that cannot be made is said: its bucket is then not taken in until the next start.
func watchDoneKeys(ctx context.Context, open *stores) {
keys, closeKeys, err := openBuilderKeys()
if err != nil {
fmt.Printf("done keys: the builder's keys are not taken in: %v\n", err)
return
}
consumers := takeInConsumers(ctx, keys.JS)
if len(consumers) == 0 {
closeKeys()
return
}
in := make(chan takeInMsg)
// moving counts the keys handed to the loop and the keys it settled (acknowledged or refused, each as the bus took
// it): what tells a take-in that moves from one that holds work and does nothing (the stall watch, ADR 0544 §6).
moving := &takeInMoving{}
read := make([]jetstream.Consumer, 0, len(consumers))
// The connection closes only once the readers and the loop have ended, so the refusals they send at the stop reach
// the bus (the third review of ADR 0544).
var users sync.WaitGroup
for _, c := range consumers {
bucket := c.bucket
read = append(read, c.consumer)
it, err := c.consumer.Messages(jetstream.PullMaxMessages(takeInBuffered))
if err != nil {
fmt.Printf("done keys: the take-in's consumer on %s is not read: %v\n", bucket, err)
continue
}
goServing(ctx, func() {
<-ctx.Done()
it.Drain() // the keys it holds are handed out and refused at once below, not lost to their ack wait
})
users.Add(1)
goServing(ctx, func() {
defer users.Done()
for {
m, err := it.Next()
if err != nil {
if ctx.Err() == nil && !errors.Is(err, jetstream.ErrMsgIteratorClosed) {
fmt.Printf("done keys: the take-in's consumer on %s stopped: %v\n", bucket, err)
}
return
}
d, ok := takeInKeyOf(bucket, m)
if !ok {
_ = m.Ack()
continue
}
// Settled is the loop's attempt, whatever the bus answers: an acknowledgement it refuses (at a first start,
// until the bus node is composed) is delivered again and settled again, and never counts as work held
// (the third review of ADR 0544). Acknowledged synchronously and past the stop, so a refusal is said.
msg := takeInMsg{delivered: d, ack: func() {
acking, cancel := context.WithTimeout(context.WithoutCancel(ctx), 10*time.Second)
defer cancel()
if err := m.DoubleAck(acking); err != nil {
doneKeysSay(fmt.Sprintf("done keys: the bus did not take the acknowledgement of %s: %v", d.module, err))
}
moving.settled.Add(1)
},
nak: func(delay time.Duration) {
_ = m.NakWithDelay(delay)
moving.settled.Add(1)
}}
if ctx.Err() != nil {
_ = m.Nak()
continue // drained: each key it held, refused at once
}
select {
case in <- msg:
moving.handed.Add(1)
case <-ctx.Done():
_ = m.Nak() // delivered again at once to the controller after this one, not after the ack wait
continue
}
}
})
}
backlog := func(ctx context.Context) (takeInBacklog, error) { return backlogOf(ctx, read) }
steps := liveTakeInSteps(open, keys, keys.JS)
users.Add(1)
goServing(ctx, func() {
defer users.Done()
takeInLoop(ctx, in, steps, backlog)
})
goServing(ctx, func() { watchTakeInStall(ctx, backlog, moving) })
goServing(ctx, func() {
<-ctx.Done()
users.Wait()
closeKeys()
})
}
// takeInLoop gathers delivered keys into batches and takes each batch in, until ctx ends.
func takeInLoop(ctx context.Context, in <-chan takeInMsg, steps takeInSteps,
backlog func(context.Context) (takeInBacklog, error)) {
t := &takeInState{memo: map[string]string{}, raised: map[string]bool{}, saidWait: map[string]string{}}
for {
var batch []takeInMsg
select {
case <-ctx.Done():
return
case m := <-in:
batch = append(batch, m)
}
gather := time.NewTimer(takeInGather)
more:
for len(batch) < startTakeIns {
select {
case m := <-in:
batch = append(batch, m)
case <-gather.C:
break more
case <-ctx.Done():
gather.Stop()
for _, m := range batch {
m.nak(0)
}
return
}
}
gather.Stop()
said := t.take(ctx, steps, batch)
if b, err := backlog(ctx); err == nil {
said += fmt.Sprintf("; backlog %d", b.Total())
}
doneKeysSay("done keys: " + said)
}
}
// takeInState is what the take-in keeps between batches in this process: the trunk each module was last taken in at,
// the modules whose condition it raised, and the wait last said of each.
type takeInState struct {
memo map[string]string
raised map[string]bool
saidWait map[string]string
}
// take takes one batch in and acknowledges or refuses each key, raising and clearing the modules' conditions. It answers
// the batch in a line.
func (t *takeInState) take(ctx context.Context, steps takeInSteps, batch []takeInMsg) string {
keys := make([]delivered, len(batch))
for i, m := range batch {
keys[i] = m.delivered
}
settled, passes := steps.batch(ctx, keys, t.memo)
verdicts := make(map[string]verdict, len(settled))
var taken, nothing, waiting, failed int
for _, r := range settled {
verdicts[r.module] = r.verdict
o := doneKeyObservation(r.module)
switch {
case r.err != nil:
failed++
doneKeysSay(fmt.Sprintf("done keys: %s: %v", r.module, r.err))
if keeper := conditionsFrom; keeper != nil && !t.raised[r.module] && ctx.Err() == nil &&
!errors.Is(r.err, context.Canceled) {
o.Summary += r.err.Error()
if _, err := keeper.Observe(ctx, o); err == nil {
t.raised[r.module] = true
}
}
case r.waits != "":
waiting++
if t.saidWait[r.module] != r.waits {
doneKeysSay(fmt.Sprintf("done keys: %s %s", r.module, r.waits))
t.saidWait[r.module] = r.waits
}
case r.taken:
taken++
delete(t.saidWait, r.module)
if r.said != "" {
doneKeysSay("done keys: " + r.said)
}
// Cleared on every take-in, whoever raised it: a controller before a restart, whose key was delivered again.
if keeper := conditionsFrom; keeper != nil {
if _, err := keeper.Clear(ctx, o.Key(), "composed and granted"); err == nil {
delete(t.raised, r.module)
}
}
case r.nothingNew:
nothing++
}
}
for _, m := range batch {
if verdicts[m.module] == refused {
if ctx.Err() != nil {
m.nak(0) // cut by the stop: the next controller takes it at once
} else {
m.nak(refusalDelay(m.deliveries))
}
} else {
m.ack()
}
}
return fmt.Sprintf("%d key(s) of %d module(s): %d taken in, %d with nothing new, %d waiting, %d failed; %d composer pass(es)",
len(batch), len(settled), taken, nothing, waiting, failed, passes)
}
// takeInBacklog is the take-in's backlog on the bus: keys not delivered yet, and keys delivered and not acknowledged
// (being taken in, or refused and waiting for their delay).
type takeInBacklog struct {
Pending uint64 `json:"pending"`
Taking uint64 `json:"taking"`
}
// Total is every key the take-in has not acknowledged.
func (b takeInBacklog) Total() uint64 { return b.Pending + b.Taking }
// backlogOf reads the consumers' own counts from the bus.
func backlogOf(ctx context.Context, consumers []jetstream.Consumer) (takeInBacklog, error) {
var b takeInBacklog
for _, c := range consumers {
info, err := c.Info(ctx)
if err != nil {
return takeInBacklog{}, err
}
b.Pending += info.NumPending
b.Taking += uint64(info.NumAckPending)
}
return b, nil
}
// readTakeInBacklog is the take-in's backlog for a reader in any process (`status`): the consumers asked by name.
func readTakeInBacklog(ctx context.Context) (takeInBacklog, error) {
keys, closeKeys, err := openBuilderKeys()
if err != nil {
return takeInBacklog{}, err
}
defer closeKeys()
var consumers []jetstream.Consumer
for _, bucket := range broker.TakeInBuckets {
c, err := keys.JS.Consumer(ctx, "KV_"+bucket, broker.TakeInConsumer)
if err != nil {
return takeInBacklog{}, fmt.Errorf("the take-in's consumer on %s: %w", bucket, err)
}
consumers = append(consumers, c)
}
return backlogOf(ctx, consumers)
}
// kindTakeInStalled is the condition of a take-in that holds work and settles none (ADR 0544 §6); kindTakeInNotConsumed
// that of a bucket whose consumer could not be made.
const (
kindTakeInStalled = "take-in-stalled"
kindTakeInNotConsumed = "take-in-not-consumed"
)
func takeInStalledObservation(b takeInBacklog, since time.Time) conditions.Observation {
return conditions.Observation{Scope: conditions.ScopeCore, ID: "take-in", Token: kindTakeInStalled,
Kind: kindTakeInStalled, Severity: conditions.Urgent, Resolver: conditions.ResolverOperator, Source: "done keys",
Summary: fmt.Sprintf("the take-in of done keys has %d key(s) waiting and has settled none since %s",
b.Total(), since.UTC().Format(time.RFC3339))}
}
// takeInNotConsumedObservation is the condition of bucket's consumer not made, with its own id, so it is raised and
// cleared apart from a stall and from the other bucket's.
func takeInNotConsumedObservation(bucket string, err error) conditions.Observation {
summary := "the take-in's consumer on " + bucket + " is made"
if err != nil {
summary = fmt.Sprintf("the take-in's consumer on %s is not made, so its keys are not taken in until the next "+
"start: %v", bucket, err)
}
return conditions.Observation{Scope: conditions.ScopeCore, ID: "take-in-" + bucket, Token: kindTakeInNotConsumed,
Kind: kindTakeInNotConsumed, Severity: conditions.Urgent, Resolver: conditions.ResolverOperator,
Source: "done keys", Summary: summary}
}
// takeInMoving counts the keys handed to the take-in loop and the keys it settled.
type takeInMoving struct {
handed, settled atomic.Uint64
}
// stallWatch judges each look: stalled once there was work to do (keys not delivered yet, or keys handed to the loop
// and not settled) and nothing settled for takeInStall. A refused key waiting for its delay is no work of the loop's:
// with nothing else, a quiet mesh is not a stall; a loop that hangs on its batch is (the review of ADR 0544). Its first
// look always says it changed, so a condition a controller before this one raised is cleared once this one moves.
type stallWatch struct {
mu sync.Mutex
settled uint64
movedAt time.Time
stalled bool
known bool
}
// look takes a reading at now: the backlog, and how many keys were handed and settled; it answers whether the take-in
// stalls and whether that changed (always on the first look).
func (w *stallWatch) look(b takeInBacklog, handed, settled uint64, now time.Time) (stalled, changed bool) {
w.mu.Lock()
defer w.mu.Unlock()
work := b.Pending > 0 || handed > settled
if !work || settled != w.settled || w.movedAt.IsZero() {
w.movedAt = now
}
w.settled = settled
nowStalled := work && now.Sub(w.movedAt) >= takeInStall
changed = !w.known || nowStalled != w.stalled
w.stalled, w.known = nowStalled, true
return nowStalled, changed
}
// watchTakeInStall reads the backlog every minute and raises take-in-stalled while the take-in holds work and settles
// none, clearing it when it moves, and at its first look when it does not stall: a watchdog's timer, as research 072 §2
// keeps them until the health part leaves the controller.
func watchTakeInStall(ctx context.Context, backlog func(context.Context) (takeInBacklog, error), moving *takeInMoving) {
w := &stallWatch{}
tick := time.NewTicker(time.Minute)
defer tick.Stop()
for {
select {
case <-ctx.Done():
return
case <-tick.C:
}
b, err := backlog(ctx)
if err != nil {
continue
}
stalled, changed := w.look(b, moving.handed.Load(), moving.settled.Load(), time.Now())
keeper := conditionsFrom
if !changed || keeper == nil {
continue
}
w.mu.Lock()
since := w.movedAt
w.mu.Unlock()
o := takeInStalledObservation(b, since)
if stalled {
doneKeysSay("done keys: " + o.Summary)
_, _ = keeper.Observe(ctx, o)
} else {
_, _ = keeper.Clear(ctx, o.Key(), "the take-in moves")
}
}
}
+12
View File
@@ -182,3 +182,15 @@ func RaiseAfterDeclarations(r Raiser) error {
}
return AssertMeshConsumers(r)
}
// TakeInConsumer is the durable consumer the controller takes done keys in from, on the stream of each of TakeInBuckets
// (novox/hq ADR 0544 §1).
const TakeInConsumer = "mesh-controller-take-in"
// TakeInBuckets are the buckets the take-in consumes: the indexer's modules (each module's trunk value) and the builder's
// builds (each build's key).
var TakeInBuckets = []string{delivery.ModulesBucket, delivery.BuilderBuildsBucket}
// ConfigFollowConsumer is the durable consumer the controller follows mesh-config's resolved values with, on that
// bucket's stream (novox/hq ADR 0545).
const ConfigFollowConsumer = "mesh-controller-config-follow"
+11
View File
@@ -72,6 +72,17 @@ func TestAnAccountMayReadItsOwnMembershipAndNoOthers(t *testing.T) {
has(t, perms.Subscribe, "mesh.assignment.anchor.postgres")
has(t, perms.Publish, "$JS.API.DIRECT.GET.ASSIGNMENTS.mesh.assignment.anchor.postgres")
hasNot(t, perms.Subscribe, "mesh.assignment.>")
hasNot(t, perms.Publish, "mesh.assignment.*.*")
// The grants part alone publishes memberships (ADR 0547 §5), and defines no stream.
grants, err := PermissionsFor(Principal{Kind: KindModule, Node: "anchor", Module: GrantsModule, PasswordHash: "x"})
if err != nil {
t.Fatal(err)
}
has(t, grants.Publish, "mesh.assignment.*.*")
has(t, grants.Publish, "$JS.API.DIRECT.GET.ASSIGNMENTS.mesh.assignment.*.*")
hasNot(t, perms.Publish, "$JS.API.DIRECT.GET.ASSIGNMENTS.mesh.assignment.*.*")
hasNot(t, grants.Publish, "$JS.API.STREAM.CREATE.>")
hasNot(t, grants.Publish, "mesh.assignment.>")
}
// The runtime arriving on a machine changes nothing about what each module is issued (to-be 38 WP2):
+30
View File
@@ -432,6 +432,16 @@ var DegradableGrants = []DegradableGrant{
Reason: "carried only at genesis: the controller composes its own permissions into the bus module's composition, " +
"and writes a composition's digest only after the bus applied it (novox/hq ADR 0539 clauses 2 and 7)",
},
{
Subject: "$JS.ACK.KV_" + delivery.ModulesBucket + "." + TakeInConsumer + ".>",
Reason: "refused: an acknowledgement refused leaves the key delivered again after its wait, and the take-in's first " +
"batch with something new composes the bus's node, which grants it (novox/hq ADR 0544 §1)",
},
{
Subject: "$JS.ACK.KV_" + delivery.BuilderBuildsBucket + "." + TakeInConsumer + ".>",
Reason: "refused: an acknowledgement refused leaves the key delivered again after its wait, and the take-in's first " +
"batch with something new composes the bus's node, which grants it (novox/hq ADR 0544 §1)",
},
{
Subject: "$KV." + GrantsBucket + ".>",
Reason: "carried only at genesis: the controller composes its own permissions and pushes them to the bus at its " +
@@ -707,6 +717,11 @@ func PermissionsFor(p Principal) (Permissions, error) {
// ever, refused by the list it already has.
sub = append(sub, ControllerFollows...)
pub = append(pub, "$JS.ACK.EVENTS."+ControllerName+".>", "$JS.ACK.EVENTS."+MergesConsumerName+".>")
// And the take-in's two durable consumers on the indexer's and the builder's buckets (novox/hq ADR 0544 §1):
// acknowledged after each key is taken in, and no other consumer of those streams.
for _, bucket := range TakeInBuckets {
pub = append(pub, "$JS.ACK.KV_"+bucket+"."+TakeInConsumer+".>")
}
// **Where an enrolment's answer goes**, and `allow_responses` does not cover it. That
// permits one reply to the reply subject of a message the user received — and a message a
@@ -900,6 +915,12 @@ func PermissionsFor(p Principal) (Permissions, error) {
sub = append(sub, BusConnectSubject, BusDisconnectSubject)
pub = append(pub, BusConnzSubject)
}
// The grants part alone publishes the modules' memberships into the assignments stream (novox/hq ADR 0547 §5;
// the operator, 2026-10-25 22:31): one per subject, `mesh.assignment.<node>.<module>`, reading the last one of each
// first so it publishes only what changed, and no stream it defines.
if p.Module == GrantsModule {
pub = append(pub, "mesh.assignment.*.*", "$JS.API.DIRECT.GET."+AssignmentsStream+".mesh.assignment.*.*")
}
// The view alone reads the build seats' workers for the backlog of asks (issue 1887 A8a).
if p.Module == ViewModule {
pub = append(pub, BuildBacklogReads()...)
@@ -1258,6 +1279,11 @@ func PermissionsFor(p Principal) (Permissions, error) {
pub = append(pub, "mesh.seat."+v.Seat+"."+CallKind+"."+v.Verb+"."+CallerToken(p.Username()))
}
}
// And its consumer of mesh-config's resolved values, acknowledged once a change is composed (novox/hq ADR 0545):
// where mesh-config serves, as its verb, so genesis grants nothing of mesh-config.
if len(p.AsksTheConfig) > 0 {
pub = append(pub, "$JS.ACK.KV_"+ConfigResolvedBucket+"."+ConfigFollowConsumer+".>")
}
}
sort.Strings(pub)
@@ -1561,6 +1587,10 @@ func ComposeAccounts(principals []Principal) (string, error) {
// from them. No other principal is granted the subjects below, and nothing is a wildcard over them.
const WatchmanModule = "mesh-delivery-watchman"
// GrantsModule is the one module whose principal publishes the modules' memberships (novox/hq ADR 0547 §5): the grants
// part, which composes them from the manifests and placement. It defines no stream and is granted no other subject here.
const GrantsModule = "mesh-grants"
// The bus's own events, imported into MESH from the default system account `$SYS` under subjects of MESH's own
// (novox/hq ADR 0431 §1, issue 1147): the server says an account's connections opening and closing only to the system
// account, and the list of an account's connections only to a request made there. `$SYS` holds no user, so nobody
+1 -1
View File
@@ -29,7 +29,7 @@ accounts {
]
users = [
{ user: "controller", password: "$2a$11$cccccccccccccccccccccc", permissions: {
publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.DEAD_LETTER_NOTICES.controller.>", "$JS.ACK.EVENTS.controller-merges.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "$KV.mesh-controller_asked.>", "$KV.mesh-controller_assignments.>", "$KV.mesh-controller_builds.>", "$KV.mesh-controller_calls.>", "$KV.mesh-controller_composed.>", "$KV.mesh-controller_condition-history.>", "$KV.mesh-controller_conditions.>", "$KV.mesh-controller_declared-config.>", "$KV.mesh-controller_declared-packages.>", "$KV.mesh-controller_delivery-merges.>", "$KV.mesh-controller_grants.>", "$KV.mesh-controller_hand-acts.>", "$KV.mesh-controller_lease.>", "$KV.mesh-controller_predefined.>", "$KV.mesh-controller_rollouts.>", "$KV.mesh-controller_states-announced.>", "$SRV.INFO", "_INBOX.enrol.>", "mesh.again.>", "mesh.assignment.>", "mesh.events.dead.>", "mesh.mod.*.tool.>", "mesh.mod.mesh-controller.event.states", "mesh.node.>", "mesh.seat.mesh-controller.event.applied", "mesh.seat.mesh-controller.event.artifacts-gone", "mesh.seat.mesh-controller.event.assignment-changed", "mesh.seat.mesh-controller.event.bucket-deleted", "mesh.seat.mesh-controller.event.built-before", "mesh.seat.mesh-controller.event.checked", "mesh.seat.mesh-controller.event.condition-changed", "mesh.seat.mesh-controller.event.condition-cleared", "mesh.seat.mesh-controller.event.condition-raised", "mesh.seat.mesh-controller.event.declared-config-changed", "mesh.seat.mesh-controller.event.doctor-heartbeat", "mesh.seat.mesh-controller.event.healer-acted", "mesh.seat.mesh-controller.event.predefined-changed", "mesh.seat.mesh-controller.event.refused", "mesh.seat.mesh-controller.event.rolled-back", "mesh.seat.mesh-controller.event.secret-replaced", "mesh.seat.mesh-controller.tool.plans", "mesh.seat.mesh-controller.tool.push", "mesh.seat.mesh-controller.tool.upgrade", "mesh.seat.mesh-delivery.tool.close", "mesh.seat.mesh-delivery.tool.groups", "mesh.seat.mesh-delivery.tool.release", "mesh.seat.mesh-delivery.tool.stalled", "mesh.seat.mesh-delivery.tool.stop", "mesh.seat.mesh-delivery.tool.times", "mesh.seat.node-backup.tool.backed-up.*", "mesh.seat.node-backup.tool.now.*", "mesh.seat.node-intrusion-prevention.tool.banned.*", "mesh.seat.node-launcher.tool.secret.*", "mesh.seat.node-service-manager.tool.restart.*", "mesh.seat.operator-channel.tool.tell"] }
publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.DEAD_LETTER_NOTICES.controller.>", "$JS.ACK.EVENTS.controller-merges.>", "$JS.ACK.EVENTS.controller.>", "$JS.ACK.KV_mesh-delivery-builder_builds.mesh-controller-take-in.>", "$JS.ACK.KV_mesh-delivery-indexer_modules.mesh-controller-take-in.>", "$JS.API.>", "$KV.mesh-controller_asked.>", "$KV.mesh-controller_assignments.>", "$KV.mesh-controller_builds.>", "$KV.mesh-controller_calls.>", "$KV.mesh-controller_composed.>", "$KV.mesh-controller_condition-history.>", "$KV.mesh-controller_conditions.>", "$KV.mesh-controller_declared-config.>", "$KV.mesh-controller_declared-packages.>", "$KV.mesh-controller_delivery-merges.>", "$KV.mesh-controller_grants.>", "$KV.mesh-controller_hand-acts.>", "$KV.mesh-controller_lease.>", "$KV.mesh-controller_predefined.>", "$KV.mesh-controller_rollouts.>", "$KV.mesh-controller_states-announced.>", "$SRV.INFO", "_INBOX.enrol.>", "mesh.again.>", "mesh.assignment.>", "mesh.events.dead.>", "mesh.mod.*.tool.>", "mesh.mod.mesh-controller.event.states", "mesh.node.>", "mesh.seat.mesh-controller.event.applied", "mesh.seat.mesh-controller.event.artifacts-gone", "mesh.seat.mesh-controller.event.assignment-changed", "mesh.seat.mesh-controller.event.bucket-deleted", "mesh.seat.mesh-controller.event.built-before", "mesh.seat.mesh-controller.event.checked", "mesh.seat.mesh-controller.event.condition-changed", "mesh.seat.mesh-controller.event.condition-cleared", "mesh.seat.mesh-controller.event.condition-raised", "mesh.seat.mesh-controller.event.declared-config-changed", "mesh.seat.mesh-controller.event.doctor-heartbeat", "mesh.seat.mesh-controller.event.healer-acted", "mesh.seat.mesh-controller.event.predefined-changed", "mesh.seat.mesh-controller.event.refused", "mesh.seat.mesh-controller.event.rolled-back", "mesh.seat.mesh-controller.event.secret-replaced", "mesh.seat.mesh-controller.tool.plans", "mesh.seat.mesh-controller.tool.push", "mesh.seat.mesh-controller.tool.upgrade", "mesh.seat.mesh-delivery.tool.close", "mesh.seat.mesh-delivery.tool.groups", "mesh.seat.mesh-delivery.tool.release", "mesh.seat.mesh-delivery.tool.stalled", "mesh.seat.mesh-delivery.tool.stop", "mesh.seat.mesh-delivery.tool.times", "mesh.seat.node-backup.tool.backed-up.*", "mesh.seat.node-backup.tool.now.*", "mesh.seat.node-intrusion-prevention.tool.banned.*", "mesh.seat.node-launcher.tool.secret.*", "mesh.seat.node-service-manager.tool.restart.*", "mesh.seat.operator-channel.tool.tell"] }
subscribe: { allow: ["$JS.API.>", "$JS.EVENT.ADVISORY.CONSUMER.DELETED.>", "$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>", "$SRV.INFO", "$SRV.INFO.mesh-controller", "$SRV.INFO.mesh-controller.>", "$SRV.PING", "$SRV.PING.mesh-controller", "$SRV.PING.mesh-controller.>", "$SRV.STATS", "$SRV.STATS.mesh-controller", "$SRV.STATS.mesh-controller.>", "_DELIVER.controller", "_DELIVER.controller.>", "_INBOX.controller.>", "mesh.control.>", "mesh.mod.*.event.provisioner.failing", "mesh.mod.*.event.provisioner.recovered", "mesh.mod.*.event.provisioner.retirement", "mesh.mod.gitea.event.pull.merged", "mesh.mod.gitea.event.pull.updated", "mesh.mod.mesh-catalog.event.catching-up", "mesh.mod.mesh-catalog.event.upgraded", "mesh.seat.mesh-build-machine.event.built", "mesh.seat.mesh-controller.tool.>", "mesh.seat.node-build-agent.event.built", "mesh.seat.operator-channel.event.decided.mesh-controller"] }
allow_responses: { max: 1, ttl: "1m" }
} }
+109 -32
View File
@@ -7,6 +7,7 @@ import (
"errors"
"fmt"
"sync/atomic"
"time"
"github.com/jackc/pgx/v5"
"golang.org/x/crypto/bcrypt"
@@ -52,6 +53,8 @@ type BusUser struct {
// PasswordHash is what the composed file carries. The plaintext is returned once, by Mint, and
// then exists only where it was sealed.
PasswordHash string
// Sealed says its credential reached its node: an issue's mint is not sealed until then (ADR 0543 §2).
Sealed bool
}
// The kinds of bus user the mesh records. The same words the composer uses, so a row and a
@@ -100,52 +103,126 @@ func (i *Inventory) RecordBusPassword(ctx context.Context, u BusUser, password s
return i.writeBusUser(ctx, u, string(hash))
}
// writeBusUser is the row, whoever chose the password.
// writeBusUser is the row, whoever chose the password, taken as sealed.
func (i *Inventory) writeBusUser(ctx context.Context, u BusUser, hash string) error {
return i.writeBusUserSealed(ctx, u, hash, true)
}
// writeBusUserSealed is the row, sealed now or not yet.
func (i *Inventory) writeBusUserSealed(ctx context.Context, u BusUser, hash string, sealed bool) error {
if _, err := i.store.Pool().Exec(ctx,
`insert into bus_user (username, kind, node, module, password_hash)
values ($1, $2, $3, $4, $5)
`insert into bus_user (username, kind, node, module, password_hash, sealed_at)
values ($1, $2, $3, $4, $5, case when $6 then now() end)
on conflict (username) do update
set kind = excluded.kind, node = excluded.node, module = excluded.module,
password_hash = excluded.password_hash, minted_at = now()`,
u.Username, u.Kind, u.Node, u.Module, hash); err != nil {
password_hash = excluded.password_hash, minted_at = now(), sealed_at = excluded.sealed_at`,
u.Username, u.Kind, u.Node, u.Module, hash, sealed); err != nil {
return fmt.Errorf("cannot record the bus user %s: %w", u.Username, err)
}
return nil
}
// MintBusPasswordIfAbsent mints u's password only when no row names u yet, in one statement: of two callers that both
// found the account missing, exactly one wins, and the other is told it lost (won false, no password) and must seal
// nothing, so the hash the bus holds is always the password the node is sealed (novox/hq ADR 0543, the review of
// issue 1945: the assign act and the assignments' watch once each minted, the later hash beside the earlier seal).
func (i *Inventory) MintBusPasswordIfAbsent(ctx context.Context, u BusUser) (string, bool, error) {
if u.Username == "" || u.Kind == "" {
return "", false, errors.New("a bus user needs a username and a kind")
}
raw := make([]byte, 32)
if _, err := rand.Read(raw); err != nil {
return "", false, fmt.Errorf("cannot generate a bus password: %w", err)
}
password := base64.RawURLEncoding.EncodeToString(raw)
hash, err := hashBusPassword(password)
// LockBusAccount holds the store's lock on one account until the answered release is called, so two issuers of it
// (the assign act, the assignments' watch, a registration's process accounts) never interleave their look, mint and
// seal: the hash the bus holds is always the password sealed to the node (novox/hq ADR 0543 §2, issue 1945).
func (i *Inventory) LockBusAccount(ctx context.Context, username string) (func(), error) {
conn, err := i.store.Pool().Acquire(ctx)
if err != nil {
return "", false, fmt.Errorf("cannot hash a bus password: %w", err)
return nil, fmt.Errorf("the lock on %s cannot be taken: %w", username, err)
}
tag, err := i.store.Pool().Exec(ctx,
`insert into bus_user (username, kind, node, module, password_hash)
values ($1, $2, $3, $4, $5)
on conflict (username) do nothing`,
u.Username, u.Kind, u.Node, u.Module, string(hash))
if _, err := conn.Exec(ctx, `select pg_advisory_lock(hashtext($1))`, "bus_user/"+username); err != nil {
conn.Release()
return nil, fmt.Errorf("the lock on %s cannot be taken: %w", username, err)
}
return func() {
_, _ = conn.Exec(context.WithoutCancel(ctx), `select pg_advisory_unlock(hashtext($1))`, "bus_user/"+username)
conn.Release()
}, nil
}
// ForgetBusAccountsOf forgets the bus accounts of module on node, and the credentials sealed to the node for them (its
// `broker`, each process's `broker-<id>`, and with nodeOwned its provisioner client's): a module that no longer resolves
// on a node keeps no account there, so an assignment again mints and seals afresh, never trusting a row whose credential
// the node no longer holds (issue 1945, live on 2026-10-25 at 20:04: ace's mesh-build-go rows from 17:19 outlived its
// unassignment, nothing was issued again, and the module found no credential). The provisioner client's account is the
// node's (`provisioner.<node>`), forgotten only with nodeOwned. Each account's lock is held across its delete, so an issue
// in flight never writes a row back after it. It answers the accounts forgotten.
func (i *Inventory) ForgetBusAccountsOf(ctx context.Context, node, module string, nodeOwned bool) ([]string, error) {
if node == "" || module == "" {
return nil, errors.New("forgetting the bus accounts of no node or no module would forget others' accounts")
}
rows, err := i.store.Pool().Query(ctx,
`select username from bus_user where node = $1 and module = $2 and kind <> $3 and ($4 or kind <> $5)
order by username`, node, module, BusController, nodeOwned, BusProvisioner)
if err != nil {
return "", false, fmt.Errorf("cannot record the bus user %s: %w", u.Username, err)
return nil, err
}
if tag.RowsAffected() == 0 {
return "", false, nil
var names []string
for rows.Next() {
var n string
if err := rows.Scan(&n); err != nil {
rows.Close()
return nil, err
}
names = append(names, n)
}
return password, true, nil
rows.Close()
if err := rows.Err(); err != nil {
return nil, err
}
for _, n := range names { // in order, one lock at a time per issuer: no two hold each other's
release, err := i.LockBusAccount(ctx, n)
if err != nil {
return nil, err
}
defer release()
}
tx, err := i.store.Pool().Begin(ctx)
if err != nil {
return nil, err
}
defer func() { _ = tx.Rollback(context.WithoutCancel(ctx)) }()
if _, err := tx.Exec(ctx, `delete from bus_user where username = any($1)`, names); err != nil {
return nil, fmt.Errorf("the bus accounts of %s on %s are not forgotten: %w", module, node, err)
}
if _, err := tx.Exec(ctx,
`delete from module_secret s using node n
where s.node = n.id and n.name = $1 and s.module = $2
and (s.name = 'broker' or s.name like 'broker-%' or ($3 and s.name = 'provisioner-broker'))`,
node, module, nodeOwned); err != nil {
return nil, fmt.Errorf("the bus credentials of %s on %s are not forgotten: %w", module, node, err)
}
return names, tx.Commit(ctx)
}
// MintBusPasswordUnsealed mints u's password, replacing whatever stands, with no seal yet: the account is not issued
// until MarkBusAccountSealed says its credential reached the node (ADR 0543 §2).
func (i *Inventory) MintBusPasswordUnsealed(ctx context.Context, u BusUser) (string, error) {
return i.mintBusPassword(ctx, u, false)
}
// MarkBusAccountSealed says the credential last minted for username is sealed to its node.
func (i *Inventory) MarkBusAccountSealed(ctx context.Context, username string) error {
_, err := i.store.Pool().Exec(ctx, `update bus_user set sealed_at = now() where username = $1`, username)
return err
}
// BusAccountIssued says whether an account is minted, and whether its credential is sealed to its node.
func (i *Inventory) BusAccountIssued(ctx context.Context, username string) (minted, sealed bool, err error) {
var at *time.Time
err = i.store.Pool().QueryRow(ctx, `select sealed_at from bus_user where username = $1`, username).Scan(&at)
if errors.Is(err, pgx.ErrNoRows) {
return false, false, nil
}
return err == nil, err == nil && at != nil, err
}
func (i *Inventory) MintBusPassword(ctx context.Context, u BusUser) (string, error) {
return i.mintBusPassword(ctx, u, true)
}
// mintBusPassword mints, marked sealed when the caller seals it itself in the same act (every mint but an issue's).
func (i *Inventory) mintBusPassword(ctx context.Context, u BusUser, sealed bool) (string, error) {
if u.Username == "" || u.Kind == "" {
return "", errors.New("a bus user needs a username and a kind")
}
@@ -163,7 +240,7 @@ func (i *Inventory) MintBusPassword(ctx context.Context, u BusUser) (string, err
return "", fmt.Errorf("cannot hash a bus password: %w", err)
}
if err := i.writeBusUser(ctx, u, string(hash)); err != nil {
if err := i.writeBusUserSealed(ctx, u, string(hash), sealed); err != nil {
return "", err
}
return password, nil
@@ -176,7 +253,7 @@ func (i *Inventory) MintBusPassword(ctx context.Context, u BusUser) (string, err
// password, and the composer refuses it rather than writing a user anybody is.
func (i *Inventory) BusUsers(ctx context.Context) (map[string]BusUser, error) {
rows, err := i.store.Pool().Query(ctx,
`select username, kind, node, module, password_hash from bus_user order by username`)
`select username, kind, node, module, password_hash, sealed_at is not null from bus_user order by username`)
if err != nil {
return nil, err
}
@@ -184,7 +261,7 @@ func (i *Inventory) BusUsers(ctx context.Context) (map[string]BusUser, error) {
out := map[string]BusUser{}
for rows.Next() {
var u BusUser
if err := rows.Scan(&u.Username, &u.Kind, &u.Node, &u.Module, &u.PasswordHash); err != nil {
if err := rows.Scan(&u.Username, &u.Kind, &u.Node, &u.Module, &u.PasswordHash, &u.Sealed); err != nil {
return nil, err
}
out[u.Username] = u
+42 -34
View File
@@ -4,7 +4,9 @@ package inventory
import (
"context"
"slices"
"testing"
"time"
"golang.org/x/crypto/bcrypt"
)
@@ -63,48 +65,54 @@ func TestMintingAgainRotatesAndMintingIfAbsentHasOneWinner(t *testing.T) {
t.Fatal("the previous password still verifies, so a rotation revoked nothing")
}
// **Minting only if absent never rotates, and of several at once exactly one wins** (ADR 0543, the review of issue
// 1945): the assign act and the assignments' watch may both find an account missing; the winner's password is the
// one the store's hash verifies, so the one it seals is the one the bus accepts, and a loser seals nothing.
if _, won, err := inv.MintBusPasswordIfAbsent(ctx, aBusUser("shop")); err != nil || won {
t.Fatalf("a mint-if-absent over a minted account won %v (%v): it would replace a sealed password", won, err)
// **An issue's mint is not issued until its seal is marked, and forgotten with its assignment** (ADR 0543 §2, issue
// 1945, live on 2026-10-25 at 20:04): a row whose credential the node does not hold is not issued.
if minted, sealed, err := inv.BusAccountIssued(ctx, "one.shop"); err != nil || !minted || !sealed {
t.Fatalf("a mint that seals in its own act: minted %v sealed %v (%v)", minted, sealed, err)
}
const racers = 8
type minted struct {
password string
won bool
err error
if _, err := inv.MintBusPasswordUnsealed(ctx, aBusUser("racer")); err != nil {
t.Fatal(err)
}
results := make(chan minted, racers)
for range racers {
go func() {
p, won, err := inv.MintBusPasswordIfAbsent(ctx, aBusUser("racer"))
results <- minted{p, won, err}
}()
if minted, sealed, err := inv.BusAccountIssued(ctx, "one.racer"); err != nil || !minted || sealed {
t.Fatalf("an issue's mint before its seal: minted %v sealed %v (%v)", minted, sealed, err)
}
var winner string
wins := 0
for range racers {
r := <-results
if r.err != nil {
t.Fatal(r.err)
}
if r.won {
wins++
winner = r.password
} else if r.password != "" {
t.Fatal("a caller that lost was handed a password to seal")
}
if names, err := inv.BusMinted(ctx); err != nil || slices.Contains(names, "one.racer") {
t.Fatalf("an account not sealed counts as minted: %v (%v)", names, err)
}
if wins != 1 {
t.Fatalf("%d of %d concurrent mints won, want exactly one", wins, racers)
if err := inv.MarkBusAccountSealed(ctx, "one.racer"); err != nil {
t.Fatal(err)
}
stored, _, err := inv.BusUserHash(ctx, "one.racer")
if _, sealed, _ := inv.BusAccountIssued(ctx, "one.racer"); !sealed {
t.Fatal("an account marked sealed is not")
}
// The lock on an account holds a second issuer until the first lets go.
release, err := inv.LockBusAccount(ctx, "one.racer")
if err != nil {
t.Fatal(err)
}
if err := bcrypt.CompareHashAndPassword([]byte(stored), []byte(winner)); err != nil {
t.Fatal("the stored hash is not the winner's password: the node would be sealed a password the bus refuses")
waiting := make(chan struct{})
go func() {
again, err := inv.LockBusAccount(ctx, "one.racer")
if err == nil {
again()
}
close(waiting)
}()
select {
case <-waiting:
t.Fatal("a second issuer took the lock the first held")
case <-time.After(200 * time.Millisecond):
}
release()
<-waiting
if _, err := inv.ForgetBusAccountsOf(ctx, "one", "racer", true); err != nil {
t.Fatal(err)
}
if minted, _, _ := inv.BusAccountIssued(ctx, "one.racer"); minted {
t.Fatal("an unassigned module's account was kept")
}
if minted, _, _ := inv.BusAccountIssued(ctx, "one.shop"); !minted {
t.Fatal("forgetting one module's accounts forgot another's")
}
}
+15
View File
@@ -210,6 +210,21 @@ func (i *Inventory) hasModule(ctx context.Context, name string) (bool, error) {
// This is the whole of noticing. Nothing here builds anything — it writes down that the two
// halves differ, which is what makes *is this current?* answerable without building, and what
// makes a module that nobody rebuilt visible rather than silent.
// RecordBuiltFrom records that a registered module stands at commit, its manifest and fingerprint unchanged: a catalogue
// merge that left its sources alone (novox/hq ADR 0544 §3). The head moves with it, as a registration moves it; the
// module's builds and manifest stay as they are.
func (i *Inventory) RecordBuiltFrom(ctx context.Context, module, commit string) error {
tag, err := i.store.Pool().Exec(ctx,
`update module set built_from = $2, source_head = $2 where name = $1`, module, commit)
if err != nil {
return fmt.Errorf("cannot record %s at %s: %w", module, commit, err)
}
if tag.RowsAffected() == 0 {
return fmt.Errorf("%w: %s", ErrNoSuchModule, module)
}
return nil
}
func (i *Inventory) SourceMoved(ctx context.Context, module, head string) error {
tag, err := i.store.Pool().Exec(ctx,
`update module set source_head = $2, source_seen = now() where name = $1`, module, head)
@@ -0,0 +1,8 @@
-- A bus account says when the credential minted for it was sealed to its node (novox/hq ADR 0543 §2, issue 1945). The
-- assign act and the assignments' watch issue an account only when it is not issued: a row with no sealed credential
-- is not issued. On 2026-10-25 at 20:04 ace's mesh-build-go had a row minted at 17:19 whose credential the node no
-- longer held; minting nothing, the mesh sealed nothing, and the module crash-looped for its missing credential. Null
-- while an issue mints and seals; set once the credential is handed over. Every account standing before this migration
-- is taken as sealed: each works or worked, and its own issue sealed it.
alter table bus_user add column sealed_at timestamptz;
update bus_user set sealed_at = minted_at;
+4 -2
View File
@@ -226,8 +226,10 @@ func (i *Inventory) BusMinted(ctx context.Context) ([]string, error) {
return nil, err
}
out := make([]string, 0, len(kept))
for name := range kept {
out = append(out, name)
for name, u := range kept {
if u.Sealed { // an account whose credential is not sealed to its node is not issued (ADR 0543 §2)
out = append(out, name)
}
}
sort.Strings(out)
return out, nil