Compare commits

...
17 Commits
Author SHA1 Message Date
jschoubben c193d2b389 Pin mesh-sdk 8eafc0c, the L3 group's head on mesh-sdk main with #120 and #121 (issue 774)
mesh/delivery ready: it is merged on the Approve
mesh/check checking c193d2b38970 on 158348ede789 | novox/mesh-catalog#556 in group feat/774-vault-mints-l2
2026-10-28 21:56:06 +01:00
jschoubben 10447d8c1e mesh-config reads the placement beside the controller's assignments while both stand, as L2 has it and the vault does, and is listed as a reader of them (issue 774, ADR 0549 §5; re-review) 2026-10-28 21:54:36 +01:00
jschoubben 5be6576117 Follow L2 under L3: the vault and mesh-config read the seat mesh-placement's placement, and the check that only listed modules read the controller's assignments moves into noplacement, as L2 deleted the placement check (issue 774, ADR 0567, ADR 0549 §3, §6) 2026-10-28 21:54:36 +01:00
jschoubben 24fad2b92a mesh-vault: pin mesh-sdk 4e6e5cc, the L3 group's head rebased on L2 (issue 774) 2026-10-28 21:54:36 +01:00
jschoubben d3eb5cae80 mesh-vault: secret-ask answers the callers its manifest names, each named as the runtime proved it, within the same bounds; the controller's relay as before (issue 774, ADR 0563 §8) 2026-10-28 21:54:36 +01:00
jschoubben 2043200201 messenger: ask each machine's own engine whether it is root-free, the node seat node-engine's root-free, not the controller's (issue 774, ADR 0563 §6) 2026-10-28 21:54:36 +01:00
jschoubben ef618e8199 nats: the bus's certificate command moves here from mesh-controller with the same output, made once and checked, so the bus module makes its own certificate when a mesh is raised from nothing (issue 774, design 47 L3 s11) 2026-10-28 21:54:36 +01:00
jschoubben c00005c141 mesh-vault signs: its signing key and authority made once with their operator copies, the verb sign for a node's own certificate from bus state, the event signed, the state signers and the tools that read and reassert them (issue 774, ADR 0550 §2) 2026-10-28 21:54:36 +01:00
jschoubben bd5f170af0 The placement check's list of modules still reading the controller's assignments only shrinks against the trunk's copy (issue 774, ADR 0529 clause 2) 2026-10-28 21:54:36 +01:00
jschoubben 6bbb8329e4 mesh-config resolves for the modules the catalogue's placement puts on a node alone, and the placement check refuses any other module that reads the controller's assignments unless listed with why (issue 774, ADR 0549 §3, §6) 2026-10-28 21:54:36 +01:00
jschoubben e6b913cac7 mesh-vault: account_retire refuses while the controller holds the lease or before the wants are read whole, as the passes do (issue 774, review round 1) 2026-10-28 21:54:36 +01:00
jschoubben 411c1b8de8 mesh-vault: prove the vault whole on a real bus mints only from the lease write, and try a failed mint again at the pass; adopt the operator key before the watches open (issue 774, ADR 0561 §1-§4) 2026-10-28 21:54:36 +01:00
jschoubben 64e6688ec8 mesh-vault: tell where modules stand and which node holds the bus from the catalogue's placement first, so the vault does not depend on the controller's assignments for them (issue 774, ADR 0549 §3, §5) 2026-10-28 21:54:36 +01:00
jschoubben 2d57c9663a mesh-vault: serve issue-account, the hand path that mints one wanted account now, and record a rotation at most once per loaded list's revision; a covered account still never rotates in place (issue 774, ADR 0561 §4, §9) 2026-10-28 21:54:36 +01:00
jschoubben 5c1729523b mesh-vault: retire an account the grants part no longer wants only within ADR 0230's bounds, keeping its value, and re-enable it when wanted again; account_retire for a person past the bound (issue 774, ADR 0561 §2) 2026-10-28 21:54:36 +01:00
jschoubben 1628db6c80 mesh-vault: mint only while the controller's lease is handed away, read afresh before each mint, and say in vault_status what is held back and why (issue 774, ADR 0561 §3, §9) 2026-10-28 21:54:36 +01:00
jschoubben 0b9f19b612 mesh-vault: mint each account the grants part wants and the vault holds no value or adopted hash for, in the controller's shape, hashed before account-issued and secret-changed (issue 774, ADR 0561 §1, §2, §4) 2026-10-28 21:54:36 +01:00
40 changed files with 3298 additions and 133 deletions
+10
View File
@@ -0,0 +1,10 @@
# The modules of this catalogue that still read the controller's assignments (mesh-controller.assignments), each with
# why. The check of this directory (noplacement) refuses any other module that reads them, and an entry here whose module no longer
# does: the list only shrinks, and it is empty before the controller's lease is written to a holder that is no
# controller (novox/hq ADR 0549 §3, §6; issue 774).
gitea the provisioner's plan of a module (go/provider) rides only in an assignment's value; its home after the lease is issue 774's provisioner-plan subtask
keycloak the provisioner's plan of a module (go/provider), as gitea
mesh-config unioned with the placement while both stand, as the assigner and the vault do (ADR 0549 §5); its reading placement alone is a step before the lease write
mesh-vault the provisioner's plan, which only an assignment's value carries; its home is issue 774's provisioner-plan subtask
postgres the provisioner's plan of a module (go/provider), as gitea
route-adapter the provisioner's plan of a module (go/provider), as gitea
+4
View File
@@ -1,3 +1,7 @@
module git.novox.be/novox/mesh-catalog/checks/noplacement
go 1.22
require git.novox.be/novox/mesh-catalog/checks/trunk v0.0.0
replace git.novox.be/novox/mesh-catalog/checks/trunk => ../trunk
+73 -2
View File
@@ -8,7 +8,13 @@
// 2. a `particulars` file at the root (the mesh's node names come from the checker's run, §16);
//
// 3. a manifest naming a state, stream, event or bucket of the indexer's placement (`mesh-delivery-indexer.placement…`
// or `mesh-delivery-indexer_placement…`) in any string value, such as `reads`.
// or `mesh-delivery-indexer_placement…`) in any string value, such as `reads`;
//
// 4. a manifest, other than mesh-controller's, that reads the controller's assignments (`mesh-controller.assignments`):
// every reader takes placement from the seat mesh-placement (novox/hq ADR 0549 §3, ADR 0567), and the lease write is
// refused while one still reads them (ADR 0549 §6; issue 774). The modules that still read them for what the
// placement does not carry are named in checks/noplacement/assignments-readers, each with why; that list only
// shrinks: an entry whose module no longer reads them is refused too.
//
// go run . <catalogue root>
//
@@ -55,9 +61,74 @@ func Findings(root string) ([]string, error) {
out = append(out, fmt.Sprintf("%s: names %q, which the indexer no longer keeps; placement is the seat mesh-placement's (novox/hq ADR 0567 §12)", module, s))
}
}
readers, err := assignmentsReaders(root)
if err != nil {
return nil, err
}
out = append(out, readers...)
sort.Strings(out)
return out, nil
}
// AssignmentsRead is the controller's assignments as a manifest reads them, and Allowed the file, relative to the
// catalogue's root, that names the modules still reading them: `<module> <why>` per line, `#` a comment.
const (
AssignmentsRead = "mesh-controller.assignments"
Allowed = "checks/noplacement/assignments-readers"
)
// assignmentsReaders is every finding on the reads of the controller's assignments: a module other than the controller
// reading them that Allowed does not name, and a module Allowed names that no longer reads them (issue 774).
func assignmentsReaders(root string) ([]string, error) {
allowed := map[string]bool{}
raw, err := os.ReadFile(filepath.Join(root, Allowed))
if err != nil && !os.IsNotExist(err) {
return nil, err
}
for _, line := range strings.Split(string(raw), "\n") {
if f := strings.Fields(line); len(f) > 0 && !strings.HasPrefix(f[0], "#") {
allowed[f[0]] = true
}
}
manifests, err := filepath.Glob(filepath.Join(root, "modules", "*", "module.json"))
if err != nil {
return nil, err
}
var found []string
reads := map[string]bool{}
for _, path := range manifests {
raw, err := os.ReadFile(path)
if err != nil {
return nil, err
}
var m struct {
Module string `json:"module"`
Reads []string `json:"reads"`
}
if json.Unmarshal(raw, &m) != nil || m.Module == "mesh-controller" {
continue // a manifest that does not read is the manifest checks' to refuse
}
for _, r := range m.Reads {
if strings.TrimSpace(r) != AssignmentsRead {
continue
}
reads[m.Module] = true
if !allowed[m.Module] {
found = append(found, fmt.Sprintf("%s reads %s: read where modules run from mesh-delivery-assigner.placement, "+
"the seat mesh-placement's (novox/hq ADR 0549 §3, ADR 0567); the lease write is refused while a module reads "+
"the assignments (ADR 0549 §6)", m.Module, AssignmentsRead))
}
}
}
for module := range allowed {
if !reads[module] {
found = append(found, fmt.Sprintf("%s names %s, which no longer reads %s: take it off the list, which only shrinks",
Allowed, module, AssignmentsRead))
}
}
return found, nil
}
// named collects the strings of a decoded manifest that start with one of the Prefixes.
func named(v any, into []string) []string {
switch v := v.(type) {
@@ -95,5 +166,5 @@ func main() {
if len(found) > 0 {
os.Exit(1)
}
fmt.Println("no placement directory, and no manifest reads the indexer's placement")
fmt.Println("no placement directory, no manifest reads the indexer's placement, and only the modules listed read the controller's assignments")
}
+34 -1
View File
@@ -5,6 +5,8 @@ import (
"path/filepath"
"strings"
"testing"
"git.novox.be/novox/mesh-catalog/checks/trunk"
)
func TestFindingsRefuseWhatThePlacementPathLeft(t *testing.T) {
@@ -13,12 +15,18 @@ func TestFindingsRefuseWhatThePlacementPathLeft(t *testing.T) {
files map[string]string
want string // empty: clean
}{
{"clean", map[string]string{"modules/a/module.json": `{"module":"a","reads":["mesh-controller.assignments"]}`}, ""},
{"clean", map[string]string{"modules/a/module.json": `{"module":"a","reads":["mesh-delivery-assigner.placement"]}`}, ""},
{"a placement directory", map[string]string{"placement/ace.json": `{}`}, "placement: the repository holds no placement"},
{"a particulars file", map[string]string{"particulars": "ace\n"}, "particulars: the repository lists no node names"},
{"a read of the indexer's placement", map[string]string{"modules/a/module.json": `{"module":"a","reads":["mesh-delivery-indexer.placement"]}`}, `a: names "mesh-delivery-indexer.placement"`},
{"a read of its candidates", map[string]string{"modules/b/module.json": `{"module":"b","reads":["mesh-delivery-indexer.placement-candidates"]}`}, "placement-candidates"},
{"its event", map[string]string{"modules/c/module.json": `{"module":"c","on":[{"event":"mesh-delivery-indexer.placement-changed"}]}`}, "placement-changed"},
// Issue 774 (ADR 0549 §3, §6): a reader of the controller's assignments is refused unless the list names it, the
// controller's own manifest is not judged, and a name on the list whose module no longer reads them is refused.
{"an unlisted reader of the assignments", map[string]string{"modules/r/module.json": `{"module":"r","reads":["mesh-controller.assignments"]}`}, "r reads mesh-controller.assignments"},
{"a listed reader and the controller", map[string]string{"modules/l/module.json": `{"module":"l","reads":["mesh-controller.assignments"]}`,
"modules/mesh-controller/module.json": `{"module":"mesh-controller","reads":["mesh-controller.assignments"]}`, Allowed: "# why\nl its plan\n"}, ""},
{"a listed module that reads them no more", map[string]string{Allowed: "gone no longer reads\n"}, "names gone, which no longer reads"},
{"the seat's own state is allowed", map[string]string{"modules/d/module.json": `{"module":"d","reads":["mesh-placement.placement"]}`}, ""},
} {
t.Run(row.name, func(t *testing.T) {
@@ -52,4 +60,29 @@ func TestTheCatalogueHoldsNoPlacementOfTheRepositoryPath(t *testing.T) {
for _, f := range got {
t.Error(f)
}
// Issue 774 (ADR 0549 §3, §6; the comparison of novox/hq ADR 0529 clause 2): the list of modules still reading the
// controller's assignments gains none the trunk's copy at the merge base does not hold. Its first copy gains nothing.
path, ok := trunk.Copy(t, Allowed)
if !ok {
return
}
names := func(path string) map[string]bool {
raw, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
out := map[string]bool{}
for _, line := range strings.Split(string(raw), "\n") {
if f := strings.Fields(line); len(f) > 0 && !strings.HasPrefix(f[0], "#") {
out[f[0]] = true
}
}
return out
}
base := names(path)
for module := range names(filepath.Join("..", "..", Allowed)) {
if !base[module] {
t.Errorf("%s gains %s, which the trunk's copy does not hold: the list only shrinks", Allowed, module)
}
}
}
+11 -1
View File
@@ -53,6 +53,8 @@ type AccountHash struct {
Adopted bool `json:"adopted,omitempty"`
From string `json:"from,omitempty"`
Node string `json:"node,omitempty"`
// Rotating is a rotation under way over a second login (rotation.go, ADR 0561 §4): absent when none is.
Rotating *Rotating `json:"rotating,omitempty"`
}
// AccountStore is the vault's state of bus-account hashes: read and written by the vault's instances alone.
@@ -61,6 +63,8 @@ type AccountStore interface {
Put(a AccountHash) error
// Users is every user with a key.
Users() ([]string, error)
// Delete takes a user's key out: an account retired (retire.go).
Delete(user string) error
}
// busAccounts is AccountStore over the vault's own state through its runtime.
@@ -80,6 +84,8 @@ func (busAccounts) Get(user string) (*AccountHash, error) {
func (busAccounts) Users() ([]string, error) { return stdio.State(BusAccountsState).Keys() }
func (busAccounts) Delete(user string) error { return stdio.State(BusAccountsState).Delete(user) }
func (busAccounts) Put(a AccountHash) error {
_, err := stdio.State(BusAccountsState).Put(a.User, a)
return err
@@ -181,6 +187,10 @@ func (v *Vault) HashAccount(secret string, force bool) (Hashed, error) {
if err != nil {
return Hashed{}, err
}
if rec.Retired != nil {
v.hashes.paid(secret)
return Hashed{}, fmt.Errorf("%s is retired (%s): its hash is written again only when it is wanted again", secret, rec.Retired.Why)
}
user, password, err := accountOf(secret, value)
if err != nil {
return Hashed{}, err
@@ -259,7 +269,7 @@ func (v *Vault) HashHeld() (written []Hashed, failed map[string]string) {
return nil, failed
}
for _, r := range records {
if !Covered(r.Name) {
if !Covered(r.Name) || r.Retired != nil {
continue
}
h, err := v.HashAccount(r.Name, false)
+3 -26
View File
@@ -58,32 +58,6 @@ type adoptionKept struct {
failed string // why the last look found nothing to adopt, or ""
}
// busNode is the node the assignments name as holding the bus module: exactly one.
func (v *Vault) busNode() (string, error) {
if v.Assignments.Keys == nil {
return "", errors.New("the vault cannot read the assignments, so it cannot tell which node holds the bus")
}
keys, err := v.Assignments.Keys()
if err != nil {
return "", fmt.Errorf("the assignments' keys could not be read: %w", err)
}
var nodes []string
for _, k := range keys {
if node, module, ok := strings.Cut(k, "."); ok && module == BusModule {
nodes = append(nodes, node)
}
}
sort.Strings(nodes)
switch len(nodes) {
case 1:
return nodes[0], nil
case 0:
return "", fmt.Errorf("no node is assigned the bus module %s", BusModule)
}
return "", fmt.Errorf("%d nodes are assigned the bus module %s (%s): the vault adopts from one alone", len(nodes),
BusModule, strings.Join(nodes, ", "))
}
// AdoptNow is the tool account_adopt: the bus node's have read now and adopted, though it says the same as before.
func (v *Vault) AdoptNow(caller string) (Adoption, error) {
if v.Haves == nil {
@@ -143,6 +117,9 @@ func (v *Vault) adopt(node string, raw []byte, by string, force bool) (Adoption,
if err != nil {
return Adoption{}, fmt.Errorf("the have of %s on %s is not a have: %w", BusModule, node, err)
}
if have.UserList != nil {
v.LoadedListHeard(have.UserList.Revision)
}
if have.UserList == nil || have.UserList.Handed == nil {
return Adoption{}, fmt.Errorf("the have of %s on %s hands no hashes yet (its node-engine hands them once it "+
"heard the vault's key in its answer to node-key, and writes the list)", BusModule, node)
+37 -6
View File
@@ -177,6 +177,16 @@ func (v *Vault) takeCall(verb, id string) (*callKey, error) {
return c, nil
}
// openedBy is the caller that opened a secret-ask's call still open, as the ask's record names it; "" for none.
func (v *Vault) openedBy(id string) string {
v.callsMu.Lock()
defer v.callsMu.Unlock()
if c, ok := v.pending[id]; ok && c.verb == "secret-ask" && c.ask != nil {
return c.ask.Caller
}
return ""
}
// open opens what was sealed to the call's key, and forgets the key.
func (c *callKey) open(sealed string) ([]byte, error) {
defer func() { c.private = [32]byte{} }()
@@ -305,15 +315,33 @@ func (v *Vault) AcceptTake(caller, call, sealed string) (Changed, error) {
type AskOf struct {
Node, Module, Name, Desk string
// AskedBy is who asked the controller, as the bus named its caller; the controller's word, taken from the
// controller alone.
// controller alone. A caller asking directly is named by the runtime's proof, never by this (ADR 0563 §8).
AskedBy string
}
// askedByFor is who asked a secret-ask as the bus says it (novox/hq ADR 0563 §8): for a call relayed by the controller,
// the caller the bus named to the controller, which it takes from the controller alone, as before; for a caller the
// manifest names, that caller as the runtime proved it. Any other caller is refused: no caller names itself.
func (v *Vault) askedByFor(caller string, a AskOf) (string, error) {
switch {
case caller == ControllerCaller:
return askedByName(a.AskedBy), nil
case v.AskCallers.Allows(caller):
return askedByName(caller), nil
}
if caller == "" {
caller = stdio.Unproven
}
return "", fmt.Errorf("secret-ask answers the controller and the callers the vault's manifest names, and %s is "+
"neither; nobody was asked to type anything", caller)
}
// AskOpen records an ask for a module's own secret before any prompt opens, holding its bounds — one open ask
// per secret, three an hour (ADR 0277 §24, §25) — and answers the key the desk's prompt seals what is typed to,
// and who asked as the record says it, which the prompt names (§26).
func (v *Vault) AskOpen(caller string, a AskOf) (Opened, error) {
if err := fromTheController(caller, "secret-ask"); err != nil {
askedBy, err := v.askedByFor(caller, a)
if err != nil {
return Opened{}, err
}
secret, err := OwnSecret(a.Node, a.Module, a.Name)
@@ -323,7 +351,6 @@ func (v *Vault) AskOpen(caller string, a AskOf) (Opened, error) {
if !namePart.MatchString(a.Desk) {
return Opened{}, fmt.Errorf("nobody was asked to type anything: the desk %q is not a machine's name", a.Desk)
}
askedBy := askedByName(a.AskedBy)
v.Store.asksMu.Lock()
defer v.Store.asksMu.Unlock()
now := v.Now().UTC()
@@ -378,9 +405,13 @@ var askOutcomes = map[string]bool{"dismissed": true, "timed-out": true, "empty":
// AskEnd ends an ask: with what the desk sealed to its key, which the vault opens and keeps as typed, or with
// the outcome the desk answered instead. How it ended is recorded whatever happens.
func (v *Vault) AskEnd(caller, call, sealed, outcome string) (Changed, error) {
if err := fromTheController(caller, "secret-ask"); err != nil {
if _, err := v.askedByFor(caller, AskOf{}); err != nil {
return Changed{}, err
}
// An ask is ended by the caller that opened it, and by no other.
if opener := v.openedBy(call); opener != "" && opener != caller {
return Changed{}, fmt.Errorf("the ask %s was opened by %s, and %s may not end it", call, opener, caller)
}
if sealed == "" && !askOutcomes[outcome] {
return Changed{}, fmt.Errorf("an ask ends with what was sealed to its key, or with one of dismissed, " +
"timed-out, empty, refused or failed")
@@ -527,8 +558,8 @@ type Asked struct {
Unchanged bool `json:"unchanged,omitempty"`
}
// AskAtDesk is `secret-ask` (ADR 0349 clause 9, ADR 0277 §22–§27): asked by the controller alone, which names who
// asked it. The ask is recorded and its bounds held before any prompt opens; the vault asks the desk's prompt
// AskAtDesk is `secret-ask` (ADR 0349 clause 9, ADR 0277 §22–§27): asked by the controller, which names who asked it,
// or by a caller the vault's manifest names, named by the runtime's proof (ADR 0563 §8). The ask is recorded and its bounds held before any prompt opens; the vault asks the desk's prompt
// itself, in its own name, naming the asker as recorded and giving a key made for this one call; what is typed
// comes back sealed to that key and opens here. How the prompt ended is the answer and the record's.
func (v *Vault) AskAtDesk(caller string, a AskOf) (Asked, error) {
@@ -13,6 +13,7 @@ import (
"encoding/base64"
"encoding/json"
"errors"
stdio "git.novox.be/novox/mesh-sdk/go"
"os"
"path/filepath"
"strings"
@@ -173,6 +174,29 @@ func TestAnAskIsBoundedToOneOpenPerSecretAndThreeAnHour(t *testing.T) {
if _, err := w.AskOpen(ControllerCaller, telegramAsk("g14.claude-code")); err != nil {
t.Errorf("an ask left open past its time still held the secret: %v", err)
}
// 774 (ADR 0563 §8): a caller the manifest names asks directly, is named by its proven caller and never by the
// asked_by it gives, and is held to the same bounds; one it does not name is refused.
t.Run("a named caller asks itself, within the bounds", func(t *testing.T) {
x := newVault(t, nil)
x.AskCallers = stdio.Callers{"node-tools"}
if _, err := x.AskOpen("ace.claude-code", telegramAsk("x")); err == nil || !strings.Contains(err.Error(), "ace.claude-code") {
t.Errorf("a caller the manifest does not name opened an ask: %v", err)
}
opened, err := x.AskOpen("g14.node-tools", telegramAsk("person.operator"))
if err != nil || opened.AskedBy != "g14.node-tools" {
t.Fatalf("a named caller's ask: %+v, %v", opened, err)
}
if _, err := x.AskEnd(ControllerCaller, opened.Call, "", "dismissed"); err == nil {
t.Error("another caller ended a named caller's ask")
}
if _, err := x.AskOpen("ace.node-tools", telegramAsk("")); err == nil || !strings.Contains(err.Error(), "still open (asked by g14.node-tools)") {
t.Errorf("a second ask while a named caller's is open: %v", err)
}
recorded, _ := x.Store.Asks(10)
if len(recorded) == 0 || recorded[0].AskedBy != "g14.node-tools" || recorded[0].Caller != "g14.node-tools" {
t.Errorf("recorded as %+v", recorded)
}
})
}
// Only the controller reaches a desk's prompt, so only it opens and ends an ask; a value not sealed to the
@@ -48,6 +48,13 @@ func (f *fakeAccounts) Put(a AccountHash) error {
return nil
}
func (f *fakeAccounts) Delete(user string) error {
f.mu.Lock()
defer f.mu.Unlock()
delete(f.keys, user)
return nil
}
func (f *fakeAccounts) Users() ([]string, error) {
f.mu.Lock()
defer f.mu.Unlock()
@@ -0,0 +1,49 @@
package main
// The switch on the controller's lease (novox/hq ADR 0561 §3): minting is on only while the lease is handed to a holder
// that is no controller, and off for a controller's value, a key deleted or expired, or one that cannot be read. The
// write that hands it away turns it on and mints every wanted account the vault holds no value for; a lease handed back
// turns it off. No clock and no setting turns it. A key's age passing is heard by no watch, so the lease is read afresh
// before every mint (LeaseRead): a lease that expired since mints nothing.
import "git.novox.be/novox/mesh-sdk/go/buscompose"
// LeaseHeard is the watch of the controller's lease. When minting turns on or off, every wanted account is judged again:
// minted when on, held back with the new why when off.
func (v *Vault) LeaseHeard(raw []byte, found bool, readErr error) {
if !v.setLease(raw, found, readErr) {
return
}
on, why := v.mintingOn()
v.log("minting is %s: %s", map[bool]string{true: "ON", false: "off"}[on], why)
for _, user := range v.wantedUsers() {
if w, ok := v.wanted(user); ok {
if _, err := v.mintWanted(w, "the lease write"); err != nil {
v.log("%s is not minted: %v", user, err)
}
}
}
}
// setLease records what the lease says, and answers whether minting turned on or off by it.
func (v *Vault) setLease(raw []byte, found bool, readErr error) bool {
on, why := buscompose.LeaseHandedAway(raw, found, readErr)
m := &v.minting
m.mu.Lock()
defer m.mu.Unlock()
flipped := on != m.on || m.why == ""
if flipped {
m.since = v.Now().UTC()
}
m.on, m.why = on, why
return flipped
}
// leaseAfresh reads the lease again before a mint, where the vault can: a key that expired since is heard by no watch.
func (v *Vault) leaseAfresh() {
if v.LeaseRead == nil {
return
}
raw, found, err := v.LeaseRead()
v.setLease(raw, found, err)
}
+101 -2
View File
@@ -53,8 +53,9 @@ func build() *Env {
assigned = &WatchedAssignments{Fallback: readAssignment}
env.Vault = &Vault{Store: store, Own: own, Askers: Declared{Read: assigned.Read, Confirm: readAssignment, Store: store},
Assignments: Mirror{Keys: assignmentKeys, Read: readAssignment}, Emit: emit, Ask: stdio.Ask,
Now: time.Now, Log: say, Accounts: busAccounts{}, Haves: readHave,
HaveKeys: func() ([]string, error) { return stdio.State(HaveRead).Keys() }}
Now: time.Now, Log: say, Accounts: busAccounts{}, Haves: readHave, LeaseRead: readLease, WantRead: readWant,
HaveKeys: func() ([]string, error) { return stdio.State(HaveRead).Keys() }, Placement: placement, Signers: busSigners{},
AskCallers: askCallers(os.Getenv("MESH_VAULT_SECRET_ASK_CALLERS"))}
return env
}
@@ -120,6 +121,74 @@ func watchHandings(env *Env) bool {
return true
}
// watchWants mints from the grants part's wanted accounts as they change (mint.go, ADR 0561 §2): a watch, never a poll.
// It answers whether the watch began; one that did not is begun again at the next minute pass.
func watchWants(env *Env) bool {
if env.Vault == nil {
return true
}
err := stdio.State(WantedRead).Watch("", func(c stdio.StateChange) error {
env.Vault.WantHeard(c.Key, c.Value, c.Op == "delete" || c.Op == "purge" || len(c.Value) == 0)
return nil
})
if err != nil {
say("the grants part's wanted accounts are not watched, so nothing is minted; begun again at the next pass: %v", err)
return false
}
// The watch returns once every current key was handed: what has no key now is no longer wanted.
env.Vault.WantsRead()
return true
}
// placement reads the seat mesh-placement's placement through the runtime under the vault's own account (its manifest reads it).
var placement = PlacementReader{
Keys: func() ([]string, error) { return stdio.State(PlacementRead).Keys() },
Read: func(key string) ([]byte, error) {
e, err := stdio.State(PlacementRead).Get(key)
if err != nil || e == nil {
return nil, err
}
return e.Value, nil
},
}
// readWant is one wanted account of the grants part, read afresh (issue-account).
func readWant(user string) ([]byte, bool, error) {
e, err := stdio.State(WantedRead).Get(user)
if err != nil || e == nil {
return nil, false, err
}
return e.Value, true, nil
}
// readLease is the controller's lease key, read through the runtime under the vault's own account (its manifest reads it).
func readLease() ([]byte, bool, error) {
e, err := stdio.State(LeaseRead).Get(LeaseKey)
if err != nil || e == nil {
return nil, false, err
}
return e.Value, true, nil
}
// watchLease turns minting on and off as the controller's lease is written (lease.go, ADR 0561 §3): a watch, never a
// poll; and the lease is read afresh before every mint. It answers whether the watch began.
func watchLease(env *Env) bool {
if env.Vault == nil {
return true
}
env.Vault.LeaseHeard(readLease())
err := stdio.State(LeaseRead).Watch(LeaseKey, func(c stdio.StateChange) error {
deleted := c.Op == "delete" || c.Op == "purge" || len(c.Value) == 0
env.Vault.LeaseHeard(c.Value, !deleted, nil)
return nil
})
if err != nil {
say("the controller's lease is not watched, so minting stays as last read; begun again at the next pass: %v", err)
return false
}
return true
}
// adoptOperatorKey takes the operator key from the export the mesh writes to the vault's root (what
// `operator key set` recorded; the controller keeps only a public half there), and seals again any
// operator copy not yet sealed to the current key. Run every minute, so a failure partway is finished. It
@@ -173,7 +242,11 @@ func main() {
written, failed := env.Vault.HashHeld()
say("hashed %d bus accounts at start; %d could not be, tried again every minute: %v", len(written), len(failed), failed)
}
// The operator key first: nothing is kept without it, so a want heard before it would not be minted.
env.pass()
watched, watching := watchHandings(env), watchAssignments()
lease := watchLease(env)
wants := watchWants(env)
for {
if !watched {
watched = watchHandings(env)
@@ -181,9 +254,23 @@ func main() {
if !watching {
watching = watchAssignments()
}
if !lease {
lease = watchLease(env)
}
if !wants {
wants = watchWants(env)
}
env.pass()
if env.Vault != nil {
// The signing key and authority once the operator key is taken, and `signers` when it differs (ADR 0550 §2).
if err := env.Vault.SigningPass(); err != nil {
env.Vault.signingSaid(err)
} else {
env.Vault.signingSaid(nil)
}
env.Vault.HashOwed()
env.Vault.MintOwed()
env.Vault.RetirePass()
}
env.pairPass()
if env.Custody != nil {
@@ -197,3 +284,15 @@ func main() {
os.Exit(1)
}
}
// askCallers are the callers the vault's manifest names for secret-ask besides the controller (novox/hq ADR 0563 §8):
// MESH_VAULT_SECRET_ASK_CALLERS, comma-separated, each a module (on any machine), <machine>.<module> or person.<name>.
func askCallers(list string) stdio.Callers {
var out stdio.Callers
for _, c := range strings.Split(list, ",") {
if c = strings.TrimSpace(c); c != "" {
out = append(out, c)
}
}
return out
}
+376
View File
@@ -0,0 +1,376 @@
package main
// The vault mints every new bus account (novox/hq ADR 0561 §1 to §4, issue 774).
//
// The grants part says which accounts the mesh wants, in its state `accounts` that the vault alone reads (§2): one key
// per bus user, holding what the controller's credential carries beside the password. The vault watches it, and for a
// user it holds no value for and whose hash it did not adopt it makes a password, keeps the account whole (built by
// mesh-sdk's HeldCredential, byte for byte the controller's shape) as the module's own secret on that node
// (`own/<node>/<module>/<broker|broker-<process>|provisioner-broker>`), sealed, with the operator's copy; writes the
// bcrypt hash into `bus-accounts`; and says `account-issued`, then `secret-changed`. The node's engine asks `own` again
// on `secret-changed` and composes.
//
// **One issuer at every moment** (§3): minting is on only while the controller's lease is handed to a holder that is no
// controller (mesh-sdk LeaseHandedAway). Until then the controller mints, the vault keeps hashing what it is handed, and
// vault_status names every wanted account it holds back and why. **Nothing is minted anew** (§4): an account whose value
// the vault holds, or whose hash it adopted, is never minted; the node accounts and the controller's own are never wanted.
// **Never two values for one user** (§7): a user whose hash the vault already wrote from another of its secrets is held
// back, said, and never replaced but by a rotation.
import (
"crypto/rand"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"sort"
"strings"
"sync"
"time"
"git.novox.be/novox/mesh-sdk/go/buscompose"
)
// WantedRead is the grants part's state of wanted accounts, as the vault's manifest reads it.
const WantedRead = buscompose.GrantsAccounts
// LeaseRead is the controller's lease, as the vault's manifest reads it, and LeaseKey its one key.
const (
LeaseRead = buscompose.ControllerLease
LeaseKey = buscompose.ControllerLeaseKey
)
// AccountIssuedEvent is what the vault says when it minted an account (mesh-sdk buscompose.AccountIssued).
const AccountIssuedEvent = buscompose.AccountIssuedEvent
// minting is the switch and what the vault knows of the wanted accounts, kept in memory and read again from the bus at
// every start: the grants part's state and the lease are the record, never this.
type minting struct {
mu sync.Mutex
on bool
why string
since time.Time
// wants are the wanted accounts, by user, as last heard.
wants map[string]buscompose.WantedAccount
// heldBack are the wanted users the vault did not mint, with why (vault_status).
heldBack map[string]string
// owed are the wanted users whose mint failed, tried again at the pass (MintOwed).
owed map[string]bool
// minted are the accounts this vault minted since it started, by user (vault_status).
minted map[string]Minted
// loadedRevision is the highest revision of the user list the bus node said it loaded (rotation.go).
loadedRevision uint64
// retiring is what the passes keep of the accounts no longer wanted (retire.go).
retiring retiring
// mintMu keeps one mint at a time: a want heard, the switch turning on, and issue-account.
mintMu sync.Mutex
}
// Minted is what one mint did: the account and its secret, never its value or its hash.
type Minted struct {
User string `json:"user"`
Secret string `json:"secret"`
Generation int `json:"generation"`
At time.Time `json:"at"`
// Hashed says its hash is written; false leaves it owed, written at the next pass.
Hashed bool `json:"hashed"`
}
// MintState is what vault_status says of minting.
type MintState struct {
On bool `json:"on"`
Why string `json:"why"`
Since time.Time `json:"since,omitzero"`
Wanted int `json:"wanted"`
HeldBack map[string]string `json:"held_back"`
Minted []Minted `json:"minted"`
}
// MintState answers how minting stands.
func (v *Vault) MintState() MintState {
m := &v.minting
m.mu.Lock()
defer m.mu.Unlock()
out := MintState{On: m.on, Why: m.why, Since: m.since, Wanted: len(m.wants), HeldBack: map[string]string{}, Minted: []Minted{}}
if out.Why == "" {
out.Why = "the controller's lease has not been read yet, so nothing is minted"
}
for u, why := range m.heldBack {
out.HeldBack[u] = why
}
for _, x := range m.minted {
out.Minted = append(out.Minted, x)
}
sort.Slice(out.Minted, func(i, j int) bool { return out.Minted[i].User < out.Minted[j].User })
return out
}
// mintingOn says whether the vault mints now, and why.
func (v *Vault) mintingOn() (bool, string) {
v.minting.mu.Lock()
defer v.minting.mu.Unlock()
if v.minting.why == "" {
return false, "the controller's lease has not been read yet, so nothing is minted"
}
return v.minting.on, v.minting.why
}
func (v *Vault) holdBack(user, why string) {
m := &v.minting
m.mu.Lock()
defer m.mu.Unlock()
if m.heldBack == nil {
m.heldBack = map[string]string{}
}
m.heldBack[user] = why
}
func (v *Vault) noLongerHeld(user string) {
m := &v.minting
m.mu.Lock()
defer m.mu.Unlock()
delete(m.heldBack, user)
}
// AccountSecret is the vault's name of the own secret a wanted account is held in.
func AccountSecret(w buscompose.WantedAccount) (string, error) {
return OwnSecret(w.Node, w.Module, w.OwnSecret())
}
// WantHeard is the watch of the grants part's wanted accounts: a key put is a want, minted when minting is on; a key
// deleted is a want gone (retire.go).
func (v *Vault) WantHeard(user string, raw []byte, deleted bool) {
if deleted {
v.wantGone(user)
return
}
w, err := buscompose.ReadWantedAccount(user, raw)
if err != nil {
v.log("the wanted account %s is not read, so nothing is minted for it: %v", user, err)
v.holdBack(user, "its wanted key is not readable: "+err.Error())
return
}
v.minting.mu.Lock()
if v.minting.wants == nil {
v.minting.wants = map[string]buscompose.WantedAccount{}
}
v.minting.wants[user] = w
v.minting.mu.Unlock()
v.wantBack(user)
if _, err := v.mintWanted(w, "the grants part's want of "+user); err != nil {
v.log("%s is not minted: %v", user, err)
}
}
// errHeldBack is a want the vault does not mint now, with why: not an error of the vault.
type errHeldBack struct{ why string }
func (e errHeldBack) Error() string { return e.why }
// mintWanted mints w when minting is on, the vault holds no value for it and adopted no hash of it; otherwise it says
// why not and, where that is a hold, keeps it for vault_status.
func (v *Vault) mintWanted(w buscompose.WantedAccount, by string) (Minted, error) {
m, err := v.mintOnce(w, by)
var held errHeldBack
v.owe(w.User, err != nil && !errors.As(err, &held) && w.Valid() == nil)
return m, err
}
// mintOnce is one try of mintWanted.
func (v *Vault) mintOnce(w buscompose.WantedAccount, by string) (Minted, error) {
v.minting.mintMu.Lock()
defer v.minting.mintMu.Unlock()
if err := w.Valid(); err != nil {
v.holdBack(w.User, err.Error())
return Minted{}, err
}
secret, err := AccountSecret(w)
if err != nil {
v.holdBack(w.User, err.Error())
return Minted{}, err
}
if _, held, err := v.Store.Read(secret); err != nil {
return Minted{}, err
} else if held {
// Its value is held (the controller's, handed in, or one the vault minted): never minted anew (§4).
v.noLongerHeld(w.User)
return Minted{}, errHeldBack{secret + " is held already: an account the vault holds a value for is never minted anew (ADR 0561 §4)"}
}
if v.Accounts == nil {
v.holdBack(w.User, "the vault has no state of bus accounts to write the hash to")
return Minted{}, errors.New("the vault has no state of bus accounts to write the hash to")
}
was, err := v.Accounts.Get(w.User)
if err != nil {
v.holdBack(w.User, "its key in "+BusAccountsState+" cannot be read: "+err.Error())
return Minted{}, err
}
if was != nil && was.Adopted {
// The hash the bus loaded is kept (§4): not minted, and not held back either.
v.noLongerHeld(w.User)
return Minted{}, errHeldBack{w.User + "'s hash was adopted from the list the bus loaded: it keeps it, and is never minted (ADR 0561 §4)"}
}
if was != nil && was.Of != "" && was.Of != secret {
why := fmt.Sprintf("%s holds the hash of %s for %s, not of %s: a user has one value, so nothing is minted until "+
"a person sees why (ADR 0561 §7)", BusAccountsState, was.Of, w.User, secret)
v.holdBack(w.User, why)
return Minted{}, errHeldBack{why}
}
v.leaseAfresh()
if on, why := v.mintingOn(); !on {
v.holdBack(w.User, "held back: "+why)
return Minted{}, errHeldBack{why}
}
raw := make([]byte, 30)
if _, err := rand.Read(raw); err != nil {
return Minted{}, err
}
value, err := buscompose.HeldCredential(w, base64.RawURLEncoding.EncodeToString(raw))
if err != nil {
v.holdBack(w.User, err.Error())
return Minted{}, err
}
c, err := v.keepLocked(secret, string(value), "the vault", "", OriginMade, keepOptions{neverReplace: true, quiet: true})
if err != nil {
v.holdBack(w.User, "its value could not be kept: "+err.Error())
return Minted{}, err
}
out := Minted{User: w.User, Secret: secret, Generation: c.Generation, At: c.At}
// The hash first, then account-issued, then secret-changed: the grants part composes the user before the node asks.
if _, err := v.HashAccount(secret, false); err != nil {
v.hashes.owe(secret, err.Error())
v.log("%s is minted and kept, and ITS HASH IS NOT WRITTEN, written at the next pass: %v", w.User, err)
} else {
out.Hashed = true
if v.Emit != nil {
e := buscompose.AccountIssued{User: w.User, Kind: w.Kind, Node: w.Node, Module: w.Module, Process: w.Process,
Generation: c.Generation, At: c.At}
body := map[string]any{}
b, _ := json.Marshal(e)
_ = json.Unmarshal(b, &body)
if err := v.Emit(AccountIssuedEvent, body); err != nil {
v.log("%s is minted and hashed, but %s could not be said: %v", w.User, AccountIssuedEvent, err)
}
}
}
v.announce(c)
v.minting.mu.Lock()
if v.minting.minted == nil {
v.minting.minted = map[string]Minted{}
}
v.minting.minted[w.User] = out
v.minting.mu.Unlock()
v.noLongerHeld(w.User)
v.log("minted the bus account %s as %s (generation %d), for %s", w.User, secret, c.Generation, by)
return out, nil
}
// MintOwed tries again, at the vault's minute pass, every wanted account whose mint failed (its value could not be kept,
// its key read, its password made): a want heard is never lost to one refusal. A want held back is not owed: the switch
// or the want changing judges it again.
func (v *Vault) MintOwed() {
v.minting.mu.Lock()
var owed []string
for u := range v.minting.owed {
owed = append(owed, u)
}
v.minting.mu.Unlock()
sort.Strings(owed)
for _, u := range owed {
if w, ok := v.wanted(u); ok {
if _, err := v.mintWanted(w, "the pass, after a mint that failed"); err != nil {
v.log("%s is still not minted: %v", u, err)
}
} else {
v.owe(u, false)
}
}
}
// owe marks user's mint as failed (tried again at the pass) or not.
func (v *Vault) owe(user string, owed bool) {
v.minting.mu.Lock()
defer v.minting.mu.Unlock()
if !owed {
delete(v.minting.owed, user)
return
}
if v.minting.owed == nil {
v.minting.owed = map[string]bool{}
}
v.minting.owed[user] = true
}
// heldBackOrMinted is a mint's answer as a tool gives it.
func heldBackOrMinted(m Minted, err error) (any, error) {
var held errHeldBack
if errors.As(err, &held) {
return map[string]any{"minted": false, "why": held.why}, nil
}
if err != nil {
return nil, err
}
return map[string]any{"minted": true, "account": m}, nil
}
// IssueAccount is the seat's verb issue-account (ADR 0561 §9): one wanted user minted now, its key read afresh from the
// grants part's state. The recovery when a want was not heard: it mints nothing a want would not, and is refused while
// minting is off, for a node's account and the controller's (never minted here, §4), and for a user nobody wants.
func (v *Vault) IssueAccount(caller, user string) (any, error) {
user = strings.TrimSpace(user)
if user == "" {
return nil, errors.New("issue-account names a bus user (`user`), e.g. ace.mesh-build-go")
}
if nodeOrController(user) {
return nil, fmt.Errorf("%s is never minted by the vault: a node's account and the controller's keep the hash they "+
"have (ADR 0561 §4)", user)
}
v.leaseAfresh()
if on, why := v.mintingOn(); !on {
return nil, fmt.Errorf("not minted: %s", why)
}
raw, found, err := []byte(nil), false, error(nil)
if v.WantRead != nil {
raw, found, err = v.WantRead(user)
} else if w, ok := v.wanted(user); ok {
raw, err = json.Marshal(w)
found = err == nil
}
if err != nil {
return nil, fmt.Errorf("the wanted account %s could not be read: %w", user, err)
}
if !found {
return nil, fmt.Errorf("the grants part holds no wanted account %s: the vault mints only what placement composes "+
"(grants_accounts lists them)", user)
}
w, err := buscompose.ReadWantedAccount(user, raw)
if err != nil {
return nil, err
}
v.log("issue-account %s, asked by %s", user, someoneOr("", caller))
return heldBackOrMinted(v.mintWanted(w, "issue-account by "+someoneOr("", caller)))
}
// wantedUsers is every wanted user heard, sorted.
func (v *Vault) wantedUsers() []string {
v.minting.mu.Lock()
defer v.minting.mu.Unlock()
out := make([]string, 0, len(v.minting.wants))
for u := range v.minting.wants {
out = append(out, u)
}
sort.Strings(out)
return out
}
func (v *Vault) wanted(user string) (buscompose.WantedAccount, bool) {
v.minting.mu.Lock()
defer v.minting.mu.Unlock()
w, ok := v.minting.wants[user]
return w, ok
}
// nodeOrController says user is a node's account or the controller's: never minted here (ADR 0561 §4, §9).
func nodeOrController(user string) bool {
return user == ControllerCaller || strings.HasPrefix(user, "node.")
}
@@ -0,0 +1,422 @@
//go:build integration
package main
// The vault whole against a real bus (novox/hq ADR 0561 §1 to §4, issue 774's acceptance 12): the built binary, played
// over its real stdin and stdout as the node's tool runtime plays it (tools/call with the caller in _meta; mesh/state.*
// and mesh/publish answered from a real nats-server's key-value buckets and subjects), its store in a directory of its
// own. A want written to the grants part's state is minted only once the controller's lease is handed away: the hash is
// in bus-accounts, own answers the value sealed to the node, the value is the controller's shape and its hash's
// password, account-issued is said; an adopted user is never minted, and a lease handed back mints nothing.
import (
"bufio"
"context"
"crypto/rand"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"os/exec"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"golang.org/x/crypto/bcrypt"
"golang.org/x/crypto/nacl/box"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/seal"
"git.novox.be/novox/mesh-sdk/go/testbus"
)
// wire is one line between the runtime and the bundle (the SDK's stdio message).
type wire struct {
ID json.RawMessage `json:"id,omitempty"`
Method string `json:"method,omitempty"`
Params json.RawMessage `json:"params,omitempty"`
Result json.RawMessage `json:"result,omitempty"`
Error *struct {
Code int `json:"code"`
Message string `json:"message"`
} `json:"error,omitempty"`
}
// player is the node's tool runtime as far as the vault needs it, over a real bus.
type player struct {
t *testing.T
ctx context.Context
nc *nats.Conn
js jetstream.JetStream
mu sync.Mutex
in io.Writer
next int
pending map[string]chan wire
said []string
}
// bucketOf is the bucket a state names: the vault's own by its local name, another's as `<owner>.<name>`.
func bucketOf(state string) string {
if owner, name, ok := strings.Cut(state, "."); ok {
return owner + "_" + name
}
return "mesh-vault_" + state
}
func (p *player) send(m map[string]any) {
raw, _ := json.Marshal(m)
p.mu.Lock()
defer p.mu.Unlock()
_, _ = p.in.Write(append(raw, '\n'))
}
func (p *player) answer(id json.RawMessage, result any, err error) {
if err != nil {
p.send(map[string]any{"id": id, "error": map[string]any{"code": -32000, "message": err.Error()}})
return
}
p.send(map[string]any{"id": id, "result": result})
}
func (p *player) kv(state string) (jetstream.KeyValue, error) {
return p.js.KeyValue(p.ctx, bucketOf(state))
}
// serve answers what the bundle asks, and hands the answers to what the test asked it.
func (p *player) serve(out io.Reader) {
scanner := bufio.NewScanner(out)
scanner.Buffer(make([]byte, 1<<20), 1<<24)
for scanner.Scan() {
var m wire
if json.Unmarshal(scanner.Bytes(), &m) != nil {
continue
}
if m.Method == "" {
var key string
_ = json.Unmarshal(m.ID, &key)
p.mu.Lock()
ch := p.pending[key]
delete(p.pending, key)
p.mu.Unlock()
if ch != nil {
ch <- m
}
continue
}
go p.handle(m)
}
}
func (p *player) handle(m wire) {
var a struct {
State string `json:"state"`
Key string `json:"key"`
Value json.RawMessage `json:"value"`
Body json.RawMessage `json:"body"`
}
_ = json.Unmarshal(m.Params, &a)
switch m.Method {
case "mesh/state.get":
kv, err := p.kv(a.State)
if err != nil {
p.answer(m.ID, nil, err)
return
}
e, err := kv.Get(p.ctx, a.Key)
if errors.Is(err, jetstream.ErrKeyNotFound) {
p.answer(m.ID, nil, nil)
return
}
if err != nil {
p.answer(m.ID, nil, err)
return
}
p.answer(m.ID, map[string]any{"key": e.Key(), "value": json.RawMessage(e.Value()), "revision": e.Revision()}, nil)
case "mesh/state.put":
kv, err := p.kv(a.State)
if err == nil {
var rev uint64
rev, err = kv.Put(p.ctx, a.Key, a.Value)
if err == nil {
p.answer(m.ID, map[string]any{"revision": rev}, nil)
return
}
}
p.answer(m.ID, nil, err)
case "mesh/state.keys":
kv, err := p.kv(a.State)
if err != nil {
p.answer(m.ID, nil, err)
return
}
keys, err := kv.Keys(p.ctx)
if errors.Is(err, jetstream.ErrNoKeysFound) {
keys, err = []string{}, nil
}
p.answer(m.ID, keys, err)
case "mesh/state.delete":
kv, err := p.kv(a.State)
if err == nil {
err = kv.Delete(p.ctx, a.Key)
}
p.answer(m.ID, map[string]any{}, err)
case "mesh/state.watch":
kv, err := p.kv(a.State)
if err != nil {
p.answer(m.ID, nil, err)
return
}
key := strings.ReplaceAll(a.Key, "**", ">")
if key == "" {
key = ">"
}
w, err := kv.Watch(p.ctx, key)
if err != nil {
p.answer(m.ID, nil, err)
return
}
var watch string
_ = json.Unmarshal(m.ID, &watch)
current := true
for e := range w.Updates() {
if e == nil {
current = false
p.answer(m.ID, map[string]any{}, nil)
continue
}
op := "put"
if e.Operation() != jetstream.KeyValuePut {
op = "delete"
}
change := map[string]any{"state": a.State, "key": e.Key(), "op": op, "revision": e.Revision(), "current": current}
if op == "put" {
change["value"] = json.RawMessage(e.Value())
}
p.mu.Lock()
p.next++
id := fmt.Sprintf("w-%d", p.next)
p.mu.Unlock()
p.send(map[string]any{"id": id, "method": "mesh/state", "params": map[string]any{"watch": watch, "change": change}})
}
case "mesh/publish":
var e struct {
Key string `json:"key"`
Body json.RawMessage `json:"body"`
}
_ = json.Unmarshal(m.Params, &e)
p.mu.Lock()
p.said = append(p.said, e.Key+" "+string(e.Body))
p.mu.Unlock()
p.answer(m.ID, map[string]any{}, p.nc.Publish("mesh.mod.mesh-vault.event."+e.Key, e.Body))
default:
// mesh/ask and the rest: nothing answers on this bus, as a call with no responder.
p.answer(m.ID, nil, errors.New("nats: no responders available for request"))
}
}
// call is a tools/call of the vault as caller, answered as the tool's JSON.
func (p *player) call(name, caller string, args map[string]any) (json.RawMessage, error) {
p.mu.Lock()
p.next++
id := fmt.Sprintf("c-%d", p.next)
ch := make(chan wire, 1)
p.pending[id] = ch
p.mu.Unlock()
p.send(map[string]any{"id": id, "method": "tools/call", "params": map[string]any{"name": name, "arguments": args,
"_meta": map[string]any{"mesh/caller": caller}}})
select {
case m := <-ch:
var r struct {
Content []struct {
Text string `json:"text"`
} `json:"content"`
IsError bool `json:"isError"`
}
if m.Error != nil {
return nil, errors.New(m.Error.Message)
}
if err := json.Unmarshal(m.Result, &r); err != nil || len(r.Content) != 1 {
return nil, fmt.Errorf("%s answered %s", name, m.Result)
}
if r.IsError {
return nil, errors.New(r.Content[0].Text)
}
return json.RawMessage(r.Content[0].Text), nil
case <-time.After(30 * time.Second):
return nil, fmt.Errorf("%s did not answer in 30s", name)
}
}
func (p *player) put(t *testing.T, state, key string, v any) {
t.Helper()
kv, err := p.kv(state)
if err != nil {
t.Fatal(err)
}
raw, _ := json.Marshal(v)
if _, err := kv.Put(p.ctx, key, raw); err != nil {
t.Fatal(err)
}
}
func (p *player) hashOf(t *testing.T, user string) *AccountHash {
t.Helper()
kv, err := p.kv(BusAccountsState)
if err != nil {
t.Fatal(err)
}
e, err := kv.Get(p.ctx, user)
if errors.Is(err, jetstream.ErrKeyNotFound) {
return nil
}
if err != nil {
t.Fatal(err)
}
var a AccountHash
if err := json.Unmarshal(e.Value(), &a); err != nil {
t.Fatal(err)
}
return &a
}
func eventually(t *testing.T, what string, ok func() bool) {
t.Helper()
for end := time.Now().Add(30 * time.Second); time.Now().Before(end); time.Sleep(50 * time.Millisecond) {
if ok() {
return
}
}
t.Fatalf("never: %s", what)
}
// 774/AC12.
func TestTheVaultWholeMintsOnTheBusOnlyFromTheLeaseWrite(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
t.Cleanup(cancel)
dir := t.TempDir()
bin := filepath.Join(dir, "mesh-vault")
build := exec.Command("go", "build", "-o", bin, ".")
build.Stderr = os.Stderr
if err := build.Run(); err != nil {
t.Fatalf("the vault does not build: %v", err)
}
nc, err := nats.Connect(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(nc.Close)
js, err := jetstream.New(nc)
if err != nil {
t.Fatal(err)
}
for _, b := range []string{"mesh-vault_bus-accounts", "mesh-grants_accounts", "mesh-controller_lease",
"mesh-controller_assignments", "node-engine_have", "mesh-delivery-assigner_placement"} {
if _, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: b, History: 2}); err != nil {
t.Fatal(err)
}
}
// The operator key, as the mesh exports it to the vault's root.
opPub, _, err := box.GenerateKey(rand.Reader)
if err != nil {
t.Fatal(err)
}
root := filepath.Join(dir, "root")
_ = os.MkdirAll(root, 0o700)
export, _ := json.Marshal(map[string]string{"operator-key": base64.StdEncoding.EncodeToString(opPub[:])})
if err := os.WriteFile(filepath.Join(root, "export.json"), export, 0o600); err != nil {
t.Fatal(err)
}
cmd := exec.CommandContext(ctx, bin)
cmd.Env = append(os.Environ(), "MESH_VAULT_STORE="+filepath.Join(dir, "store"), "MESH_VAULT_STATE="+filepath.Join(dir, "state"),
"MESH_VAULT_ROOT="+root, "MESH_VAULT_LEDGER="+filepath.Join(dir, "ledger"), "MESH_RECEIVES="+filepath.Join(dir, "mesh.json"),
"MESH_NODE=novox", "MESH_MODULE=mesh-vault")
stdin, _ := cmd.StdinPipe()
stdout, _ := cmd.StdoutPipe()
cmd.Stderr = os.Stderr
if err := cmd.Start(); err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = stdin.Close(); cancel(); _ = cmd.Wait() })
p := &player{t: t, ctx: ctx, nc: nc, js: js, in: stdin, pending: map[string]chan wire{}}
go p.serve(stdout)
status := func() MintState {
raw, err := p.call("vault_status", "person.jochen", nil)
if err != nil {
return MintState{}
}
var s struct {
Minting MintState `json:"minting"`
}
_ = json.Unmarshal(raw, &s)
return s.Minting
}
// The controller holds the lease: a want is held back, nothing is minted.
p.put(t, "mesh-controller.lease", LeaseKey, map[string]any{"instance": "novox/1/2026-10-27", "epoch": 57})
x, y, z := aWant("mod-x"), aWant("mod-y"), aWant("mod-z")
p.put(t, "mesh-vault.bus-accounts", y.User, AccountHash{User: y.User, Hash: "$2a$04$adoptedadoptedadoptedadoptedadoptedadoptedadopte", Adopted: true})
p.put(t, "mesh-grants.accounts", x.User, x)
p.put(t, "mesh-grants.accounts", y.User, y)
eventually(t, "the want held back under the controller's lease", func() bool {
return strings.Contains(status().HeldBack[x.User], "controller holds the lease")
})
if p.hashOf(t, x.User) != nil {
t.Fatal("minted under the controller's lease")
}
// The write that hands the lease away mints it.
p.put(t, "mesh-controller.lease", LeaseKey, map[string]any{"handed-to": "the operator's hand-over"})
eventually(t, "the account minted at the lease write", func() bool { return p.hashOf(t, x.User) != nil })
nodePub, nodePriv := aSealingKey(t)
if _, err := p.call(Seat+".node-key", "node.ace", map[string]any{"key": nodePub}); err != nil {
t.Fatal(err)
}
raw, err := p.call(Seat+".own", "node.ace", map[string]any{})
if err != nil {
t.Fatal(err)
}
var own OwnAnswer
if err := json.Unmarshal(raw, &own); err != nil {
t.Fatal(err)
}
if len(own.Own) != 1 || own.Own[0].Module != "mod-x" || own.Own[0].Name != "broker" || own.Own[0].Origin != OriginMade {
t.Fatalf("own answers %+v", own.Own)
}
value, err := seal.Open(own.Own[0].Sealed, nodePriv)
if err != nil {
t.Fatal(err)
}
var held struct{ Password string }
_ = json.Unmarshal([]byte(value), &held)
shape, _ := buscompose.HeldCredential(x, held.Password)
if value != string(shape) {
t.Errorf("the minted value %q is not the controller's shape %q", value, shape)
}
if h := p.hashOf(t, x.User); bcrypt.CompareHashAndPassword([]byte(h.Hash), []byte(held.Password)) != nil || h.Adopted || h.Generation != 1 {
t.Errorf("bus-accounts %s holds %+v, not the minted password's hash", x.User, h)
}
if h := p.hashOf(t, y.User); h == nil || !h.Adopted {
t.Errorf("the adopted %s was replaced: %+v", y.User, h)
}
p.mu.Lock()
said := strings.Join(p.said, "\n")
p.mu.Unlock()
if !strings.Contains(said, AccountIssuedEvent+` {"`) || !strings.Contains(said, `"user":"ace.mod-x"`) || strings.Contains(said, held.Password) {
t.Errorf("the vault said: %s", said)
}
// Handed back to a controller: a new want mints nothing.
p.put(t, "mesh-controller.lease", LeaseKey, map[string]any{"instance": "novox/2/2026-10-28", "epoch": 58})
p.put(t, "mesh-grants.accounts", z.User, z)
eventually(t, "the new want held back once the lease is the controller's again", func() bool {
return strings.Contains(status().HeldBack[z.User], "controller holds the lease")
})
if p.hashOf(t, z.User) != nil {
t.Error("minted after the lease was handed back")
}
}
@@ -0,0 +1,244 @@
package main
import (
"encoding/json"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/seal"
)
// aWant is a wanted account of a module on ace, as the grants part writes it.
func aWant(module string) buscompose.WantedAccount {
return buscompose.WantedAccount{User: "ace." + module, Kind: buscompose.KindModule, Node: "ace", Module: module,
URL: "nats://novox.mesh:4222", Fingerprint: "ab:cd", Claims: []buscompose.ClaimedSeat{{Seat: "s", Scope: "node"}}}
}
func wantRaw(t *testing.T, w buscompose.WantedAccount) []byte {
t.Helper()
raw, err := json.Marshal(w)
if err != nil {
t.Fatal(err)
}
return raw
}
// mintingVault is a vault with a state of bus accounts, ace's sealing key stated, ace's modules assigned, and minting
// turned on or off as on says.
func mintingVault(t *testing.T, on bool) (testVault, *fakeAccounts, string) {
t.Helper()
plans := assignments{"ace.mod-x": `{"node":"ace","module":"mod-x"}`, "ace.mod-y": `{"node":"ace","module":"mod-y"}`}
v := newVault(t, nil)
v.Askers = Declared{Read: plans.read, Store: v.Store}
accounts := &fakeAccounts{}
v.Accounts = accounts
pub, priv := aSealingKey(t)
if err := v.recordNodeKey("ace", pub, "node.ace"); err != nil {
t.Fatal(err)
}
if on {
v.LeaseHeard([]byte(`{"handed-to":"the operator's hand-over"}`), true, nil)
} else {
v.LeaseHeard([]byte(`{"instance":"novox/1/2026-10-27","epoch":57}`), true, nil)
}
return v, accounts, priv
}
// 774/AC7 (novox/hq ADR 0561 §1, §2, §4): a want with no value, minting on, is minted once: kept whole in the controller's
// shape as the module's broker, its hash in bus-accounts, said account-issued before secret-changed, and answered by own
// sealed to the node; a value held or a hash adopted is never minted.
func TestTheVaultMintsAWantedAccountOnceAndNothingItHolds(t *testing.T) {
v, accounts, nodePriv := mintingVault(t, true)
w := aWant("mod-x")
before := len(v.bus.events)
v.WantHeard(w.User, wantRaw(t, w), false)
rec, held, err := v.Store.Read("own/ace/mod-x/broker")
if err != nil || !held || rec.Generation != 1 || rec.Origin != OriginMade {
t.Fatalf("the minted account is kept as %+v (held %v, %v)", rec, held, err)
}
value, err := seal.Open(rec.AtRest, v.Own.PrivateKey)
if err != nil {
t.Fatal(err)
}
var got struct{ Password string }
_ = json.Unmarshal([]byte(value), &got)
shape, _ := buscompose.HeldCredential(w, got.Password)
if len(got.Password) != 40 || value != string(shape) {
t.Errorf("the minted value is %q, not the controller's shape %q", value, shape)
}
if !accounts.hashOf(w.User, got.Password, 1) {
t.Errorf("%s's hash is not the minted password's: %+v", w.User, accounts.keys[w.User])
}
var order []string
for _, e := range v.bus.events[before:] {
order = append(order, e["event"].(string))
}
if strings.Join(order, ",") != AccountHashedEvent+","+AccountIssuedEvent+","+ChangedEvent {
t.Errorf("said %v, want the hash, account-issued, then secret-changed", order)
}
for _, raw := range v.bus.raw[before:] {
if strings.Contains(raw, got.Password) || strings.Contains(raw, "$2a$") {
t.Errorf("a value or a hash was said: %s", raw)
}
}
own, err := v.OwnOfNode("node.ace")
if err != nil || len(own.Own) != 1 {
t.Fatalf("own answered %+v, %v", own, err)
}
if opened, err := seal.Open(own.Own[0].Sealed, nodePriv); err != nil || opened != value {
t.Errorf("own's sealed value opens to %q, %v", opened, err)
}
// Heard again, or for a value held, or for a hash adopted: nothing is minted, nothing is held back.
v.WantHeard(w.User, wantRaw(t, w), false)
if rec, _, _ := v.Store.Read("own/ace/mod-x/broker"); rec.Generation != 1 {
t.Errorf("a want heard again minted generation %d", rec.Generation)
}
accounts.keys["ace.mod-y"] = AccountHash{User: "ace.mod-y", Hash: "$2a$04$adopted", Adopted: true}
y := aWant("mod-y")
v.WantHeard(y.User, wantRaw(t, y), false)
if _, held, _ := v.Store.Read("own/ace/mod-y/broker"); held || accounts.keys["ace.mod-y"].Hash != "$2a$04$adopted" {
t.Errorf("an adopted user was minted")
}
if st := v.MintState(); len(st.HeldBack) != 0 || len(st.Minted) != 1 {
t.Errorf("vault_status says %+v: an adopted user is not held back, one account minted", st)
}
}
// 774/AC8 (novox/hq ADR 0561 §3): while the controller holds the lease nothing is minted and vault_status says why and
// names each wanted account held back (never an adopted one); the write that hands the lease away mints them; a lease
// handed back, expired or unreadable turns minting off again, read afresh before every mint.
func TestNothingIsMintedUntilTheLeaseIsHandedAway(t *testing.T) {
v, accounts, _ := mintingVault(t, false)
accounts.keys = map[string]AccountHash{"ace.mod-y": {User: "ace.mod-y", Hash: "$2a$04$adopted", Adopted: true}}
for _, m := range []string{"mod-x", "mod-y"} {
w := aWant(m)
v.WantHeard(w.User, wantRaw(t, w), false)
}
st := v.MintState()
if st.On || !strings.Contains(st.Why, "controller holds the lease") || len(st.HeldBack) != 1 ||
!strings.Contains(st.HeldBack["ace.mod-x"], "controller holds the lease") {
t.Fatalf("under the controller's lease vault_status says %+v", st)
}
if _, held, _ := v.Store.Read("own/ace/mod-x/broker"); held {
t.Fatal("minted while the controller holds the lease")
}
// Handed away: the wanted account is minted at that write, the adopted one is not.
lease := []byte(`{"handed-to":"the operator's hand-over"}`)
v.LeaseRead = func() ([]byte, bool, error) { return lease, true, nil }
v.LeaseHeard(lease, true, nil)
if _, held, _ := v.Store.Read("own/ace/mod-x/broker"); !held {
t.Fatalf("not minted at the lease write: %+v", v.MintState())
}
if _, held, _ := v.Store.Read("own/ace/mod-y/broker"); held {
t.Error("an adopted account was minted at the lease write")
}
if st := v.MintState(); !st.On || len(st.HeldBack) != 0 {
t.Errorf("after the lease write vault_status says %+v", st)
}
// The key expired (no watch hears a key's age pass): read afresh before the next mint, nothing is minted.
v.LeaseRead = func() ([]byte, bool, error) { return nil, false, nil }
z := aWant("mod-z")
v.WantHeard(z.User, wantRaw(t, z), false)
if _, held, _ := v.Store.Read("own/ace/mod-z/broker"); held {
t.Error("minted after the lease's key expired")
}
if st := v.MintState(); st.On || !strings.Contains(st.HeldBack["ace.mod-z"], "no lease is written") {
t.Errorf("after the key expired vault_status says %+v", st)
}
}
// 774/AC10 issue-account (novox/hq ADR 0561 §9): the verb mints one wanted user now, read afresh; it is refused while
// minting is off, for a node's account and the controller's, and for a user nobody wants; a user held already is said so.
func TestIssueAccountMintsOneWantedUserNowAndRefusesTheRest(t *testing.T) {
v, _, _ := mintingVault(t, false)
w := aWant("mod-x")
raw := wantRaw(t, w)
v.WantRead = func(user string) ([]byte, bool, error) {
if user == w.User {
return raw, true, nil
}
return nil, false, nil
}
for user, says := range map[string]string{
"ace.mod-x": "controller holds the lease",
"node.ace": "never minted",
"controller": "never minted",
} {
if _, err := v.IssueAccount("person.jochen", user); err == nil || !strings.Contains(err.Error(), says) {
t.Errorf("%s: %v, want a refusal saying %q", user, err, says)
}
}
v.LeaseHeard([]byte(`{"handed-to":"the operator's hand-over"}`), true, nil)
if _, err := v.IssueAccount("person.jochen", "ace.nobody"); err == nil || !strings.Contains(err.Error(), "no wanted account") {
t.Errorf("a user nobody wants: %v", err)
}
got, err := v.IssueAccount("person.jochen", w.User)
if err != nil || got.(map[string]any)["minted"] != true {
t.Fatalf("issue-account: %v, %v", got, err)
}
if again, err := v.IssueAccount("person.jochen", w.User); err != nil || again.(map[string]any)["minted"] != false ||
!strings.Contains(again.(map[string]any)["why"].(string), "held already") {
t.Errorf("issue-account again: %v, %v", again, err)
}
}
// 774/AC10 the rotation's rule (novox/hq ADR 0561 §4): at most one rotation begins per revision of the loaded user list,
// over the second login `<user>--b`, never of a node's account, the controller's or an adopted hash, and `rotate` of a
// covered account stays refused, so nothing rotates in place.
func TestOneRotationBeginsPerRevisionOfTheLoadedList(t *testing.T) {
v, accounts, _ := mintingVault(t, true)
for _, m := range []string{"mod-x", "mod-y"} {
w := aWant(m)
v.WantHeard(w.User, wantRaw(t, w), false)
}
if _, err := v.BeginRotation("ace.mod-x"); err == nil || !strings.Contains(err.Error(), "not known yet") {
t.Errorf("began with no loaded revision known: %v", err)
}
v.LoadedListHeard(570)
r, err := v.BeginRotation("ace.mod-x")
if err != nil || r.Login != "ace.mod-x--b" || r.Revision != 570 || r.Generation != 2 || accounts.keys["ace.mod-x"].Rotating == nil {
t.Fatalf("the first rotation: %+v, %v", r, err)
}
if _, err := v.BeginRotation("ace.mod-y"); err == nil || !strings.Contains(err.Error(), "one rotation begins per revision") {
t.Errorf("a second rotation in one revision: %v", err)
}
v.LoadedListHeard(571)
if _, err := v.BeginRotation("ace.mod-y"); err != nil {
t.Errorf("the next revision's rotation: %v", err)
}
if _, err := v.BeginRotation("ace.mod-x"); err == nil || !strings.Contains(err.Error(), "rotating already") {
t.Errorf("an account rotating began again: %v", err)
}
accounts.keys["node.ace"] = AccountHash{User: "node.ace", Hash: "$2a$04$n", Adopted: true}
for _, u := range []string{"node.ace", "controller"} {
if _, err := v.BeginRotation(u); err == nil {
t.Errorf("%s began a rotation", u)
}
}
if _, err := v.RotateFor(ControllerCaller, "own/ace/mod-x/broker", Narrow{}, "the test", "a rotation in place"); err == nil {
t.Error("a covered account rotated in place")
}
}
// A mint that failed (here: no operator key yet, so nothing can be kept) is tried again at the pass, never lost.
func TestAFailedMintIsTriedAgainAtThePass(t *testing.T) {
v, accounts, _ := mintingVault(t, true)
key, _ := v.Store.OperatorKey()
if err := v.Store.setOperatorKey("", "the test", v.Now()); err != nil {
t.Fatal(err)
}
w := aWant("mod-x")
v.WantHeard(w.User, wantRaw(t, w), false)
if _, ok := accounts.keys[w.User]; ok {
t.Fatal("minted with no operator key")
}
if err := v.Store.setOperatorKey(key, "the test", v.Now()); err != nil {
t.Fatal(err)
}
v.MintOwed()
if _, ok := accounts.keys[w.User]; !ok {
t.Errorf("not minted at the pass after the failure: %+v", v.MintState())
}
}
+18 -9
View File
@@ -84,22 +84,31 @@ func (v *Vault) credentialRefusal(node string, c CredentialAsked, plan *planRead
if c.Consumer != node {
return fmt.Sprintf("%s asks as a consumer on %s, and is not that machine", node, c.Consumer)
}
// The consuming module stands on the machine: read, never assumed (an assignment that cannot be read refuses).
// The consuming module stands on the machine: read from the placement and the assignments, never assumed (one
// that cannot be read refuses). The assignments' mirror can be behind, so one it does not hold is confirmed directly.
placed, err := v.placedOn(node, c.ConsumerModule)
if err != nil {
return fmt.Sprintf(notYetPrefix+"where %s stands could not be read: %v", c.ConsumerModule, err)
}
if placed {
return ""
}
raw, why, err := plan.read(c.ConsumerModule)
if why != "" {
if why != "" && v.Placement.Read == nil {
return why
}
if err != nil {
return fmt.Sprintf(notYetPrefix+"the assignment %s.%s could not be read: %v", node, c.ConsumerModule, err)
}
if raw == nil {
// Confirmed directly before refusing: the mirror may not hold a module assigned a moment ago.
if raw, err = plan.confirmed(c.ConsumerModule); err != nil {
if why == "" {
if err != nil {
return fmt.Sprintf(notYetPrefix+"the assignment %s.%s could not be read: %v", node, c.ConsumerModule, err)
}
if raw == nil {
if raw, err = plan.confirmed(c.ConsumerModule); err != nil {
return fmt.Sprintf(notYetPrefix+"the assignment %s.%s could not be read: %v", node, c.ConsumerModule, err)
}
}
}
if raw == nil {
return fmt.Sprintf("%s is not on %s: neither assigned there nor required by a module there", c.ConsumerModule, node)
return fmt.Sprintf("%s is not on %s: neither placed nor assigned there, nor required by a module there", c.ConsumerModule, node)
}
return ""
case "provider":
+134
View File
@@ -0,0 +1,134 @@
package main
// Where modules stand, from the placement (novox/hq ADR 0549 §3, §5; ADR 0561; issue 774), the seat mesh-placement's
// state held by the assigner (ADR 0567). The controller's assignments mirror goes with its lease, so the vault reads which
// module stands on which node from the seat's placement first, and the assignments beside it while both stand: a module either names stands. The provisioner's plan,
// which only an assignment's value carries, is read from the assignments still: its home after the lease is a subtask of
// its own (issue 774).
import (
"errors"
"fmt"
"slices"
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// PlacementRead is the seat mesh-placement's placement, as the vault's manifest reads it (mesh-sdk's contract).
const PlacementRead = delivery.PlacementRead
// PlacementReader reads the seat's placement: every key, and one key's value (nil when there is none).
type PlacementReader struct {
Keys func() ([]string, error)
Read func(key string) ([]byte, error)
}
// placements is every node's placement, by node; none when the vault reads no placement.
func (v *Vault) placements() (map[string]delivery.Placement, error) {
out := map[string]delivery.Placement{}
if v.Placement.Keys == nil || v.Placement.Read == nil {
return out, nil
}
keys, err := v.Placement.Keys()
if err != nil {
return nil, fmt.Errorf("the placement's keys could not be read: %w", err)
}
for _, k := range keys {
node, err := delivery.ParsePlacementKey(k)
if err != nil {
continue
}
raw, err := v.Placement.Read(k)
if err != nil {
return nil, fmt.Errorf("the placement of %s could not be read: %w", node, err)
}
if raw == nil {
continue
}
p, err := delivery.DecodePlacement(raw)
if err != nil || p.Node != node {
continue
}
out[node] = p
}
return out, nil
}
// placedOn says module is placed on node by the catalogue.
func (v *Vault) placedOn(node, module string) (bool, error) {
if v.Placement.Read == nil {
return false, nil
}
key, err := delivery.PlacementKey(node)
if err != nil {
return false, err
}
raw, err := v.Placement.Read(key)
if err != nil || raw == nil {
return false, err
}
p, err := delivery.DecodePlacement(raw)
if err != nil {
return false, fmt.Errorf("the placement of %s is not readable: %w", node, err)
}
return slices.Contains(p.Assigned(), module), nil
}
// standsOn says module stands on node: placed there, or assigned there while the assignments stand. A read that fails is
// an error, never "not there".
func (v *Vault) standsOn(node, module string) (bool, error) {
placed, err := v.placedOn(node, module)
if err != nil || placed {
return placed, err
}
d, ok := v.Askers.(Declared)
if !ok || d.Read == nil {
if v.Placement.Read == nil {
return false, errors.New("the vault reads neither the placement nor the assignments")
}
return false, nil
}
raw, err := standing(d.Read)(node + "." + module)
return raw != nil, err
}
// busNode is the node running the bus's module: from the placement, and from the assignments while no placement names
// one. The vault adopts from one node alone.
func (v *Vault) busNode() (string, error) {
placed, err := v.placements()
if err != nil {
return "", err
}
var nodes []string
for node, p := range placed {
if slices.Contains(p.Assigned(), BusModule) {
nodes = append(nodes, node)
}
}
from := "placed"
if len(nodes) == 0 && v.Assignments.Keys != nil {
from = "assigned"
keys, err := v.Assignments.Keys()
if err != nil {
return "", fmt.Errorf("the assignments' keys could not be read: %w", err)
}
for _, k := range keys {
if node, module, ok := strings.Cut(k, "."); ok && module == BusModule {
nodes = append(nodes, node)
}
}
} else if len(nodes) == 0 && len(placed) == 0 {
return "", errors.New("the vault reads neither the placement nor the assignments, so it cannot tell which node holds the bus")
}
sort.Strings(nodes)
switch len(nodes) {
case 1:
return nodes[0], nil
case 0:
return "", fmt.Errorf("no node is %s the bus module %s", from, BusModule)
}
return "", fmt.Errorf("%d nodes are %s the bus module %s (%s): the vault adopts from one alone", len(nodes), from,
BusModule, strings.Join(nodes, ", "))
}
@@ -0,0 +1,66 @@
package main
import (
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// placementsOf is the seat mesh-placement's placement of each node, as the vault reads it.
func placementsOf(t *testing.T, nodes map[string][]string) PlacementReader {
t.Helper()
values := map[string][]byte{}
for node, modules := range nodes {
p := delivery.Placement{V: delivery.Version, Node: node}
for _, m := range modules {
p.Modules = append(p.Modules, delivery.PlacedModule{Module: m})
}
raw, err := delivery.EncodePlacement(p)
if err != nil {
t.Fatal(err)
}
key, _ := delivery.PlacementKey(node)
values[key] = raw
}
return PlacementReader{
Keys: func() ([]string, error) {
var out []string
for k := range values {
out = append(out, k)
}
return out, nil
},
Read: func(key string) ([]byte, error) { return values[key], nil },
}
}
// 774/AC11 (novox/hq ADR 0561, ADR 0549 §3, §5): with no assignments to read, the vault tells from the seat's
// placement which node holds the bus and whether a consumer's module stands on its node; while both stand, a module
// either names stands.
func TestTheVaultReadsWhereModulesStandFromPlacement(t *testing.T) {
v := newVault(t, nil)
v.Placement = placementsOf(t, map[string][]string{"novox": {"nats", "web"}, "ace": {"shop"}})
if node, err := v.busNode(); err != nil || node != "novox" {
t.Errorf("the bus node from placement: %q, %v", node, err)
}
asked := func(node, module string) CredentialAsked {
return CredentialAsked{Role: "consumer", Provision: "postgres", Consumer: node, ConsumerModule: module, Provider: "novox"}
}
if why := v.credentialRefusal("ace", asked("ace", "shop"), v.plansOf("ace")); why != "" {
t.Errorf("shop stands on ace by placement, refused: %s", why)
}
if why := v.credentialRefusal("ace", asked("ace", "web"), v.plansOf("ace")); !strings.Contains(why, "is not on ace") {
t.Errorf("web is not placed on ace, and is answered %q", why)
}
// While both stand, an assignment still names a module placement does not yet.
v.Askers = Declared{Read: assignments{"ace.web": `{"node":"ace","module":"web"}`}.read, Store: v.Store}
if why := v.credentialRefusal("ace", asked("ace", "web"), v.plansOf("ace")); why != "" {
t.Errorf("web assigned on ace, refused: %s", why)
}
// Two nodes placed with the bus: the vault adopts from neither.
v.Placement = placementsOf(t, map[string][]string{"novox": {"nats"}, "ace": {"nats"}})
if _, err := v.busNode(); err == nil || !strings.Contains(err.Error(), "2 nodes") {
t.Errorf("two bus nodes: %v", err)
}
}
+264
View File
@@ -0,0 +1,264 @@
package main
// A wanted account no longer wanted is retired within ADR 0230's bounds (novox/hq ADR 0561 §2, issue 774).
//
// The grants part deletes a wanted account's key when no input names its user any more. The vault then retires the
// account: it takes its hash out of `bus-accounts`, so the grants part's next list drops the user, and marks the value
// retired in its store, keeping it (a value is deleted only by a person, ADR 0230). As ADR 0230 §15 and §18 have it, the
// same set of accounts gone must stand for five passes and ten minutes first, and a set of more than three, or more than
// half of the accounts held, waits for a person, who retires them with `account_retire`. Wanted again, an account is
// re-enabled at once: the mark taken off and its hash written again. Nothing is retired while the controller holds the
// lease (it forgets the accounts it mints), or before the wanted accounts are read whole.
import (
"errors"
"fmt"
"sort"
"strings"
"time"
)
// retiring is what the passes keep of the accounts no longer wanted, under minting.mu.
type retiring struct {
wantsRead bool
key string
passes int
first time.Time
waiting []string
why string
retired []string
}
// RetireState is what vault_status says of retirement.
type RetireState struct {
Gone []string `json:"gone"`
Passes int `json:"passes"`
Since time.Time `json:"since,omitzero"`
Waiting []string `json:"waiting_for_a_person"`
Why string `json:"why"`
Retired []string `json:"retired"`
}
// RetireState answers how retirement stands.
func (v *Vault) RetireState() RetireState {
m := &v.minting
m.mu.Lock()
defer m.mu.Unlock()
r := m.retiring
out := RetireState{Passes: r.passes, Since: r.first, Why: r.why, Waiting: append([]string{}, r.waiting...),
Retired: append([]string{}, r.retired...), Gone: []string{}}
if r.key != "" {
out.Gone = strings.Split(r.key, " ")
}
return out
}
// WantsRead says the wanted accounts were read whole once (the watch handed every current key): only then is an account
// with no key known to be no longer wanted.
func (v *Vault) WantsRead() {
v.minting.mu.Lock()
defer v.minting.mu.Unlock()
v.minting.retiring.wantsRead = true
}
// wantGone is a wanted account's key deleted: forgotten as wanted, its retirement left to the passes.
func (v *Vault) wantGone(user string) {
v.minting.mu.Lock()
defer v.minting.mu.Unlock()
delete(v.minting.wants, user)
delete(v.minting.heldBack, user)
}
// wantBack is a wanted account's key written: a retired account of it is re-enabled at once.
func (v *Vault) wantBack(user string) {
w, ok := v.wanted(user)
if !ok {
return
}
secret, err := AccountSecret(w)
if err != nil {
return
}
rec, held, err := v.Store.Read(secret)
if err != nil || !held || rec.Retired == nil {
return
}
rec.Retired = nil
if err := v.Store.Write(rec); err != nil {
v.log("%s is wanted again and NOT RE-ENABLED: its mark could not be taken off: %v", user, err)
return
}
if _, err := v.HashAccount(secret, true); err != nil {
v.hashes.owe(secret, err.Error())
v.log("%s is wanted again and its hash is not written yet, written at the next pass: %v", user, err)
}
v.minting.mu.Lock()
r := &v.minting.retiring
for i, u := range r.retired {
if u == user {
r.retired = append(r.retired[:i], r.retired[i+1:]...)
break
}
}
v.minting.mu.Unlock()
v.log("%s is wanted again: re-enabled, its hash written again", user)
}
// goneAccounts are the users whose hash the vault wrote from a value it holds and that no wanted key names, sorted, and
// how many such hashes it holds in all.
func (v *Vault) goneAccounts() ([]string, int, error) {
users, err := v.Accounts.Users()
if err != nil {
return nil, 0, err
}
var gone []string
held := 0
for _, u := range users {
a, err := v.Accounts.Get(u)
if err != nil {
return nil, 0, err
}
if a == nil || a.Adopted || a.Of == "" {
continue
}
held++
if _, wanted := v.wanted(u); !wanted {
gone = append(gone, u)
}
}
sort.Strings(gone)
return gone, held, nil
}
// RetirePass is one pass of retirement, at the vault's minute pass.
func (v *Vault) RetirePass() {
if v.Accounts == nil {
return
}
v.leaseAfresh()
on, leaseWhy := v.mintingOn()
m := &v.minting
m.mu.Lock()
read := m.retiring.wantsRead
m.mu.Unlock()
reset := func(why string) {
m.mu.Lock()
defer m.mu.Unlock()
m.retiring.key, m.retiring.passes, m.retiring.first, m.retiring.waiting, m.retiring.why = "", 0, time.Time{}, nil, why
}
switch {
case !read:
reset("the wanted accounts are not read whole yet, so none is known to be gone")
return
case !on:
reset("nothing is retired: " + leaseWhy)
return
}
gone, held, err := v.goneAccounts()
if err != nil {
reset("the accounts held could not be read: " + err.Error())
return
}
if len(gone) == 0 {
reset("")
return
}
now := v.Now().UTC()
key := strings.Join(gone, " ")
m.mu.Lock()
r := &m.retiring
if r.key != key {
r.key, r.passes, r.first, r.waiting = key, 0, now, nil
}
r.passes++
passes, since := r.passes, r.first
m.mu.Unlock()
setWhy := func(why string, waiting []string) {
m.mu.Lock()
defer m.mu.Unlock()
m.retiring.why, m.retiring.waiting = why, waiting
}
if passes < StablePasses || now.Sub(since) < StableFor {
setWhy(fmt.Sprintf("%d no longer wanted (%s): retired once the same set stands for %d passes and %s (ADR 0230)",
len(gone), key, StablePasses, StableFor), nil)
return
}
if len(gone) > RetireAtOnce || float64(len(gone)) > RetireFraction*float64(held) {
why := fmt.Sprintf("%d accounts are no longer wanted, more than %d or more than half of the %d held at once: they "+
"wait for a person (account_retire), ADR 0230 §18", len(gone), RetireAtOnce, held)
setWhy(why, gone)
v.log("%s: %s", why, key)
return
}
for _, u := range gone {
if err := v.retireOne(u, "no longer wanted by the grants part", "the vault's pass"); err != nil {
v.log("%s is not retired: %v", u, err)
}
}
reset("")
}
// retireOne marks the value of user's account retired and takes its hash out of bus-accounts.
func (v *Vault) retireOne(user, why, by string) error {
a, err := v.Accounts.Get(user)
if err != nil {
return err
}
if a == nil || a.Adopted || a.Of == "" {
return fmt.Errorf("the vault holds no account of %s it made or was handed: an adopted hash is never retired here", user)
}
rec, held, err := v.Store.Read(a.Of)
if err != nil {
return err
}
if held {
rec.Retired = &RetiredMark{At: v.Now().UTC().Format(time.RFC3339), Why: why + ", by " + by}
if err := v.Store.Write(rec); err != nil {
return fmt.Errorf("its value could not be marked retired: %w", err)
}
}
if err := v.Accounts.Delete(user); err != nil {
return fmt.Errorf("its hash could not be taken out of %s: %w", BusAccountsState, err)
}
v.minting.mu.Lock()
v.minting.retiring.retired = append(v.minting.retiring.retired, user)
v.minting.mu.Unlock()
v.log("retired the bus account %s (%s): its hash is out of %s and its value kept, marked; %s", user, a.Of,
BusAccountsState, why)
return nil
}
// RetireAccounts is the tool account_retire: a person retires accounts no longer wanted, past ADR 0230's bound, with why.
func (v *Vault) RetireAccounts(caller string, users []string, why string) (any, error) {
if strings.TrimSpace(why) == "" {
return nil, errors.New("account_retire says why (`why`)")
}
if len(users) == 0 {
return nil, errors.New("account_retire names the accounts (`users`), as vault_status lists them waiting")
}
// As the passes: never while the controller holds the lease (it forgets the accounts it mints), nor before the wanted
// accounts are read whole (an account not read yet is not known to be gone).
v.leaseAfresh()
if on, why := v.mintingOn(); !on {
return nil, fmt.Errorf("nothing was retired: %s", why)
}
v.minting.mu.Lock()
read := v.minting.retiring.wantsRead
v.minting.mu.Unlock()
if !read {
return nil, errors.New("nothing was retired: the wanted accounts are not read whole yet, so none is known to be gone")
}
for _, u := range users {
if _, wanted := v.wanted(u); wanted {
return nil, fmt.Errorf("%s is still wanted by the grants part: only an account no longer wanted is retired; nothing was retired", u)
}
}
var done []string
for _, u := range users {
if err := v.retireOne(u, strings.TrimSpace(why), someoneOr("", caller)); err != nil {
return map[string]any{"retired": done}, fmt.Errorf("%s: %w", u, err)
}
done = append(done, u)
}
return map[string]any{"retired": done}, nil
}
@@ -0,0 +1,89 @@
package main
import (
"strings"
"testing"
"time"
)
// 774/AC9 (novox/hq ADR 0561 §2, ADR 0230 §15, §18): a wanted account whose key is deleted is retired (its hash taken
// out of bus-accounts, its value kept, marked) only once the same set has stood for five passes and ten minutes, and never
// more than three, or more than half of those held, at once: past that it waits for a person (account_retire). Wanted
// again, it is re-enabled at once; and a retired account is not hashed again at a start.
func TestAWantGoneRetiresItsAccountOnlyWithinTheBounds(t *testing.T) {
v, accounts, _ := mintingVault(t, true)
clock := time.Date(2026, 10, 28, 9, 0, 0, 0, time.UTC)
v.Now = func() time.Time { return clock }
modules := []string{"m1", "m2", "m3", "m4", "m5", "m6", "m7", "m8"}
for _, m := range modules {
w := aWant(m)
v.WantHeard(w.User, wantRaw(t, w), false)
}
v.WantsRead()
if len(accounts.keys) != 8 {
t.Fatalf("minted %d", len(accounts.keys))
}
pass := func(n int, every time.Duration) {
for i := 0; i < n; i++ {
clock = clock.Add(every)
v.RetirePass()
}
}
v.WantHeard("ace.m1", nil, true)
pass(5, time.Second)
if _, ok := accounts.keys["ace.m1"]; !ok {
t.Fatal("retired after five passes in five seconds: ADR 0230 waits ten minutes too")
}
pass(1, 10*time.Minute)
if _, ok := accounts.keys["ace.m1"]; ok {
t.Fatalf("not retired after five passes and ten minutes: %+v", v.RetireState())
}
if rec, held, _ := v.Store.Read("own/ace/m1/broker"); !held || rec.Retired == nil {
t.Fatalf("the retired account's value is %+v (held %v): kept, and marked", rec, held)
}
if written, _ := v.HashHeld(); len(written) != 0 {
t.Errorf("a start hashed a retired account again: %+v", written)
}
// Wanted again: re-enabled at once.
w := aWant("m1")
v.WantHeard(w.User, wantRaw(t, w), false)
if rec, _, _ := v.Store.Read("own/ace/m1/broker"); rec.Retired != nil || accounts.keys["ace.m1"].Hash == "" {
t.Fatalf("wanted again, ace.m1 is %+v with hash %v", rec.Retired, accounts.keys["ace.m1"])
}
// Four gone at once is over the bound: it waits for a person, who retires them by hand.
for _, m := range modules[4:] {
v.WantHeard("ace."+m, nil, true)
}
pass(6, 3*time.Minute)
st := v.RetireState()
if len(accounts.keys) != 8 || len(st.Waiting) != 4 || !strings.Contains(st.Why, "more than 3") {
t.Fatalf("over the bound: %d keys, %+v", len(accounts.keys), st)
}
if _, err := v.RetireAccounts("person.jochen", []string{"ace.m5"}, ""); err == nil {
t.Error("retired by hand with no why")
}
if _, err := v.RetireAccounts("person.jochen", []string{"ace.m2"}, "test"); err == nil {
t.Error("retired by hand an account still wanted")
}
// Never while the controller holds the lease, nor before the wants are read whole.
held := v.LeaseRead
v.LeaseRead = func() ([]byte, bool, error) { return []byte(`{"instance":"novox/1","epoch":57}`), true, nil }
if _, err := v.RetireAccounts("person.jochen", []string{"ace.m5"}, "the modules left"); err == nil || !strings.Contains(err.Error(), "controller holds the lease") {
t.Errorf("retired by hand under the controller's lease: %v", err)
}
v.LeaseRead = held
v.LeaseHeard([]byte(`{"handed-to":"the operator's hand-over"}`), true, nil)
v.minting.mu.Lock()
v.minting.retiring.wantsRead = false
v.minting.mu.Unlock()
if _, err := v.RetireAccounts("person.jochen", []string{"ace.m5"}, "the modules left"); err == nil || !strings.Contains(err.Error(), "not read whole") {
t.Errorf("retired by hand before the wants were read: %v", err)
}
v.WantsRead()
if _, err := v.RetireAccounts("person.jochen", []string{"ace.m5", "ace.m6"}, "the modules left"); err != nil {
t.Fatal(err)
}
if _, ok := accounts.keys["ace.m5"]; ok || len(accounts.keys) != 6 {
t.Errorf("after a person's retirement %d keys remain", len(accounts.keys))
}
}
@@ -0,0 +1,94 @@
package main
// The record of a bus account's rotation, and its one rule (novox/hq ADR 0561 §4; issue 774, the lead's decision of
// 2026-10-27). A module's, a process's or a provisioner client's account rotates over two credentials (ADR 0114 §10):
// a second login, `<user>--b`, carrying the first login's permissions verbatim, which the bus composer composes only while
// the account's key in `bus-accounts` names it in `rotating`. The vault starts at most one rotation per revision of the
// user list the bus loaded. What composes and delivers the second login is a subtask of its own, before the lease write;
// until it is built nothing begins a rotation, and `rotate` of a covered account stays refused, so nothing rotates in
// place.
import (
"fmt"
"strings"
"time"
)
// SecondLoginSuffix makes an account's second login: reserved, so no module or process may be named to end in it.
const SecondLoginSuffix = "--b"
// Rotating is a rotation under way, as the account's key in bus-accounts says it: the second login, the generation of
// the value it carries, since when, and the revision of the loaded user list it began at.
type Rotating struct {
Login string `json:"login"`
Generation int `json:"generation"`
Since time.Time `json:"since"`
Revision uint64 `json:"revision"`
}
// LoadedListHeard keeps the revision of the user list the bus node says it loaded (its have of the bus module).
func (v *Vault) LoadedListHeard(revision uint64) {
v.minting.mu.Lock()
defer v.minting.mu.Unlock()
if revision > v.minting.loadedRevision {
v.minting.loadedRevision = revision
}
}
// BeginRotation records a rotation of user's account at the loaded list's revision, under the one rule: none for a
// node's account or the controller's, none while minting is off, none of an account the vault did not make or was not
// handed, none twice for one account, and none when another began at this revision.
func (v *Vault) BeginRotation(user string) (Rotating, error) {
if nodeOrController(user) {
return Rotating{}, fmt.Errorf("%s does not rotate here: a node's account and the controller's keep the hash they have (ADR 0561 §4)", user)
}
if on, why := v.mintingOn(); !on {
return Rotating{}, fmt.Errorf("no rotation begins: %s", why)
}
v.minting.mu.Lock()
revision := v.minting.loadedRevision
v.minting.mu.Unlock()
if revision == 0 {
return Rotating{}, fmt.Errorf("no rotation begins: the revision of the user list the bus loaded is not known yet")
}
users, err := v.Accounts.Users()
if err != nil {
return Rotating{}, err
}
var mine *AccountHash
began := ""
for _, u := range users {
a, err := v.Accounts.Get(u)
if err != nil {
return Rotating{}, err
}
if a == nil {
continue
}
if a.Rotating != nil && a.Rotating.Revision == revision {
began = u
}
if u == user {
mine = a
}
}
switch {
case mine == nil || mine.Adopted || mine.Of == "":
return Rotating{}, fmt.Errorf("the vault holds no account of %s it made or was handed, so it rotates none", user)
case mine.Rotating != nil:
return Rotating{}, fmt.Errorf("%s is rotating already, since %s (second login %s)", user, mine.Rotating.Since.Format(time.RFC3339), mine.Rotating.Login)
case strings.HasSuffix(user, SecondLoginSuffix):
return Rotating{}, fmt.Errorf("%s is a second login, which never rotates of its own", user)
case began != "":
return Rotating{}, fmt.Errorf("no rotation of %s begins: %s's began at the loaded list's revision %d, and one "+
"rotation begins per revision (ADR 0561 §4)", user, began, revision)
}
r := Rotating{Login: user + SecondLoginSuffix, Generation: mine.Generation + 1, Since: v.Now().UTC(), Revision: revision}
next := *mine
next.Rotating = &r
if err := v.Accounts.Put(next); err != nil {
return Rotating{}, err
}
v.log("a rotation of %s begins at the loaded list's revision %d: second login %s", user, revision, r.Login)
return r, nil
}
@@ -0,0 +1,463 @@
package main
// The vault signs (novox/hq ADR 0550 §2; issue 774, design 47's L3 item d). It makes an ed25519 signing key and an x509
// authority once, keeps each as its own secret with its operator copy (ADR 0113 §22), publishes their public halves in its
// state `signers`, and signs with them on its verb `sign`: a node's certificate for that node's node-engine alone, built
// from bus state (the name from the node's placement, the serving key from its record), never from bytes the caller chose.
// Each signature is said as `signed` and kept in the record of hand-outs, never a key. The controller's key is never
// copied here.
import (
"crypto/ed25519"
"crypto/rand"
"crypto/x509"
"encoding/base64"
"encoding/json"
"encoding/pem"
"errors"
"fmt"
"strings"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/overlay"
)
// The vault's own signing secrets: each an ed25519 seed in base64 (never PEM, which the vault takes as a private key
// someone sent it and refuses).
const (
SigningKeySecret = "signing/key"
AuthorityKeySecret = "signing/authority-key"
// signedOutcome marks a signature in the record of hand-outs.
signedOutcome = "signed"
)
// signingSecret says name is one of the vault's signing secrets, which no asker is ever handed.
func signingSecret(name string) bool { return name == SigningKeySecret || name == AuthorityKeySecret }
// SignersStore is the vault's state `signers` (delivery.SignersBucket): a key's value, or nil; and a write.
type SignersStore interface {
Get(key string) ([]byte, error)
Put(key string, value []byte) error
}
// busSigners is SignersStore over the vault's own state through its runtime.
type busSigners struct{}
func (busSigners) Get(key string) ([]byte, error) {
e, err := stdio.State(delivery.SignersState).Get(key)
if err != nil || e == nil {
return nil, err
}
return e.Value, nil
}
func (busSigners) Put(key string, value []byte) error {
_, err := stdio.State(delivery.SignersState).Put(key, json.RawMessage(value))
return err
}
// signingKeys are the vault's two keys, opened, and the authority's certificate.
type signingKeys struct {
key ed25519.PrivateKey
keyMade time.Time
authority ed25519.PrivateKey
authorityMade time.Time
certificate string
parsed *x509.Certificate
}
// EnsureSigning makes the signing key and the authority once, when the store holds neither, and answers whether it made
// them. **Never made again over what is held** (as OwnKey refuses a new key beside secrets): one held without the other,
// or none held while `signers` already publishes a key, is a lost key, refused loudly with nothing made, because every
// node trusts what was published and a new key would be one nobody trusts.
func (v *Vault) EnsureSigning() (bool, error) {
_, keyHeld, err := v.Store.Read(SigningKeySecret)
if err != nil {
return false, err
}
_, authHeld, err := v.Store.Read(AuthorityKeySecret)
if err != nil {
return false, err
}
switch {
case keyHeld && authHeld:
return false, nil
case keyHeld != authHeld:
missing := AuthorityKeySecret
if authHeld {
missing = SigningKeySecret
}
return false, fmt.Errorf("THE VAULT'S %s IS MISSING BESIDE THE OTHER SIGNING SECRET: nothing is made, since a new one "+
"would be trusted by no node; put it back from its operator copy (novox/hq ADR 0113 §21, ADR 0550 §2)", missing)
}
if v.Signers != nil {
if raw, err := v.Signers.Get(delivery.SignersKeyKey); err != nil {
return false, fmt.Errorf("the vault's state %s could not be read, so no signing key is made: %w", delivery.SignersRead, err)
} else if raw != nil {
return false, fmt.Errorf("THE VAULT'S SIGNING KEYS ARE MISSING WHILE %s PUBLISHES ONE: nothing is made, since every "+
"node trusts the published key; put them back from their operator copies (novox/hq ADR 0113 §21)", delivery.SignersRead)
}
}
for _, name := range []string{SigningKeySecret, AuthorityKeySecret} {
_, priv, err := ed25519.GenerateKey(rand.Reader)
if err != nil {
return false, err
}
if _, err := v.keepWith(name, base64.StdEncoding.EncodeToString(priv.Seed()), "the vault", "", OriginMade,
keepOptions{neverReplace: true, quiet: true}); err != nil {
return false, fmt.Errorf("the vault's %s is not made: %w", name, err)
}
}
v.log("made the vault's signing key and authority, each kept with its operator copy (ADR 0550 §2)")
return true, nil
}
// openSeed is a signing secret opened: its private key and when it was made.
func (v *Vault) openSeed(name string) (ed25519.PrivateKey, time.Time, error) {
r, held, err := v.Store.Read(name)
if err != nil {
return nil, time.Time{}, err
}
if !held {
return nil, time.Time{}, fmt.Errorf("the vault holds no %s yet: it is made once the operator key is taken", name)
}
value, err := openAtRest(r, v)
if err != nil {
return nil, time.Time{}, fmt.Errorf("%s does not open with the vault's key: %w", name, err)
}
seed, err := base64.StdEncoding.DecodeString(value)
if err != nil || len(seed) != ed25519.SeedSize {
return nil, time.Time{}, fmt.Errorf("%s is not an ed25519 seed", name)
}
return ed25519.NewKeyFromSeed(seed), r.MadeAt, nil
}
// keys opens both signing secrets and derives the authority's certificate from its key and when it was made, so it is the
// same certificate at every start.
func (v *Vault) keys() (signingKeys, error) {
key, keyMade, err := v.openSeed(SigningKeySecret)
if err != nil {
return signingKeys{}, err
}
auth, authMade, err := v.openSeed(AuthorityKeySecret)
if err != nil {
return signingKeys{}, err
}
pub := auth.Public().(ed25519.PublicKey)
tmpl := delivery.AuthorityTemplate(pub, authMade)
der, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, pub, auth)
if err != nil {
return signingKeys{}, fmt.Errorf("the vault's authority certificate could not be made: %w", err)
}
parsed, err := x509.ParseCertificate(der)
if err != nil {
return signingKeys{}, err
}
return signingKeys{key: key, keyMade: keyMade, authority: auth, authorityMade: authMade, parsed: parsed,
certificate: string(pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der}))}, nil
}
// SignersValues are the two keys of `signers` as the vault holds them now.
func (v *Vault) SignersValues() (delivery.SignerKey, delivery.SignerAuthority, error) {
k, err := v.keys()
if err != nil {
return delivery.SignerKey{}, delivery.SignerAuthority{}, err
}
fp, err := delivery.AuthorityFingerprint(k.certificate)
if err != nil {
return delivery.SignerKey{}, delivery.SignerAuthority{}, err
}
return delivery.NewSignerKey(k.key.Public().(ed25519.PublicKey), k.keyMade),
delivery.SignerAuthority{Certificate: k.certificate, Fingerprint: fp, Made: k.authorityMade.UTC()}, nil
}
// PublishSigners writes `signers` from the keys the vault holds, each key only when its value differs (or always with
// force, signers_reassert), and answers which keys it wrote.
func (v *Vault) PublishSigners(force bool) ([]string, error) {
if v.Signers == nil {
return nil, errors.New("the vault has no state " + delivery.SignersRead + " to write")
}
key, auth, err := v.SignersValues()
if err != nil {
return nil, err
}
var written []string
for name, value := range map[string]any{delivery.SignersKeyKey: key, delivery.SignersAuthorityKey: auth} {
raw, err := json.Marshal(value)
if err != nil {
return written, err
}
if !force {
held, err := v.Signers.Get(name)
if err != nil {
return written, fmt.Errorf("%s %s could not be read: %w", delivery.SignersRead, name, err)
}
if same(held, raw) {
continue
}
}
if err := v.Signers.Put(name, raw); err != nil {
return written, fmt.Errorf("%s %s could not be written: %w", delivery.SignersRead, name, err)
}
written = append(written, name)
}
if len(written) > 0 {
v.log("wrote %s %s (ADR 0550 §2)", delivery.SignersRead, strings.Join(written, " and "))
}
return written, nil
}
// same says two JSON values are the same value.
func same(a, b []byte) bool {
if a == nil || b == nil {
return false
}
var x, y any
if json.Unmarshal(a, &x) != nil || json.Unmarshal(b, &y) != nil {
return false
}
ra, _ := json.Marshal(x)
rb, _ := json.Marshal(y)
return string(ra) == string(rb)
}
// SigningPass makes the keys once and publishes `signers` when it differs: each minute pass, after the operator key.
func (v *Vault) SigningPass() error {
if _, err := v.EnsureSigning(); err != nil {
return err
}
if v.Signers == nil {
return nil
}
_, err := v.PublishSigners(false)
return err
}
// SignArgs are `sign`'s arguments: the kind, and the kind's fields.
type SignArgs struct {
Kind string
Node string
Reply string
}
// Sign answers caller's `sign`. The caller is the one the runtime proved; who may ask which kind is the vault's rule,
// never an argument: a node certificate for a machine's node-engine alone, about its own machine; an enrolment for no
// caller until adoption's record names one (ADR 0561 §5); the authority for the vault itself, never asked.
func (v *Vault) Sign(caller string, a SignArgs) (delivery.SignAnswer, error) {
if caller == "" || caller == stdio.Unproven || !busUser.MatchString(caller) {
return delivery.SignAnswer{}, errors.New("the vault signs only for a caller the runtime proved")
}
switch a.Kind {
case delivery.KindNodeCertificate:
return v.signNodeCertificate(caller, a.Node)
case delivery.KindEnrolment:
return delivery.SignAnswer{}, fmt.Errorf("%s may not ask for an enrolment's signature: no caller is named for it until "+
"adoption's record names the seat that adopts (novox/hq ADR 0561 §5)", caller)
case delivery.KindAuthority:
return delivery.SignAnswer{}, fmt.Errorf("the authority is the vault's own, signed for itself; %s may not ask it", caller)
}
return delivery.SignAnswer{}, fmt.Errorf("%q is not a kind the vault signs (%s)", a.Kind, strings.Join(delivery.SignKinds, ", "))
}
// signNodeCertificate signs the certificate of the calling machine's internal name for the serving key its record states.
func (v *Vault) signNodeCertificate(caller, asked string) (delivery.SignAnswer, error) {
node, ok := nodeEngineOf(caller)
if !ok {
return delivery.SignAnswer{}, fmt.Errorf("a node certificate is signed for a machine's node-engine alone, about its own "+
"machine; %s is not a node-engine", caller)
}
if asked != "" && asked != node {
return delivery.SignAnswer{}, fmt.Errorf("%s asks for the certificate of %s, and is %s's node-engine: a machine asks for its own alone", caller, asked, node)
}
name, err := v.internalName(node)
if err != nil {
return delivery.SignAnswer{}, err
}
serving, err := v.servingKey(node)
if err != nil {
return delivery.SignAnswer{}, err
}
k, err := v.keys()
if err != nil {
return delivery.SignAnswer{}, err
}
der, err := x509.CreateCertificate(rand.Reader, delivery.NodeCertificateTemplate(name, serving, k.authorityMade),
k.parsed, serving, k.authority)
if err != nil {
return delivery.SignAnswer{}, fmt.Errorf("the certificate of %s could not be signed: %w", name, err)
}
fp, _ := delivery.AuthorityFingerprint(k.certificate)
answer := delivery.SignAnswer{Kind: delivery.KindNodeCertificate, Node: node, Authority: k.certificate,
Certificate: string(pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der})), Signer: fp, Digest: delivery.DigestOf(der)}
if err := v.saySigned(caller, AuthorityKeySecret, answer); err != nil {
return delivery.SignAnswer{}, err
}
return answer, nil
}
// internalName is node's name inside the mesh: only a node with an address on the private network in its placement has
// one (a certificate for a name nothing resolves is one nothing can check).
func (v *Vault) internalName(node string) (string, error) {
if v.Placement.Read == nil {
return "", errors.New("the vault reads no placement, so it cannot tell a node's internal name")
}
key, err := delivery.PlacementKey(node)
if err != nil {
return "", err
}
raw, err := v.Placement.Read(key)
if err != nil {
return "", fmt.Errorf(notYetPrefix+"the placement of %s could not be read: %w", node, err)
}
if raw == nil {
return "", fmt.Errorf("%s has no placement of the seat mesh-placement, so it has no name inside the mesh", node)
}
p, err := delivery.DecodePlacement(raw)
if err != nil {
return "", fmt.Errorf("the placement of %s is not readable: %w", node, err)
}
if p.Decided.Address == "" {
return "", fmt.Errorf("%s is not on the private network (its placement names no address), so it has no name inside the mesh to be certified for", node)
}
return overlay.InternalName(node), nil
}
// servingKey is the serving key node's record states, `<node>.node` of the have bucket, which that node alone writes.
func (v *Vault) servingKey(node string) (ed25519.PublicKey, error) {
if v.Haves == nil {
return nil, errors.New("the vault cannot read the have bucket (" + HaveRead + "), so it cannot read a node's record")
}
key, err := delivery.HaveNodeKey(node)
if err != nil {
return nil, err
}
raw, err := v.Haves(key)
if err != nil {
return nil, fmt.Errorf(notYetPrefix+"%s %s could not be read: %w", HaveRead, key, err)
}
if raw == nil {
return nil, fmt.Errorf("%s has written no record (%s %s), so it states no serving key", node, HaveRead, key)
}
h, err := delivery.DecodeHaveNode(raw)
if err != nil {
return nil, fmt.Errorf("the record of %s is not readable: %w", node, err)
}
if h.Record == nil || h.Record.ServingKey == "" {
return nil, fmt.Errorf("%s's record states no serving key, so there is no key to certify", node)
}
pub, err := base64.StdEncoding.DecodeString(h.Record.ServingKey)
if err != nil || len(pub) != ed25519.PublicKeySize {
return nil, fmt.Errorf("%s's serving key is not an ed25519 public key in base64", node)
}
return ed25519.PublicKey(pub), nil
}
// saySigned keeps the signature in the record of hand-outs and says `signed`: who, which kind, which node, the digest and
// the signer's fingerprint. Never a key. A signature whose record cannot be written is not answered.
func (v *Vault) saySigned(caller, with string, a delivery.SignAnswer) error {
at := v.Now().UTC()
if err := v.recordHanding(Handing{To: caller, Name: with, Outcome: signedOutcome,
Why: a.Kind + " " + a.Node + " " + a.Digest}); err != nil {
return fmt.Errorf("the signature is not answered: its record could not be written: %w", err)
}
if v.Emit == nil {
return nil
}
raw, _ := json.Marshal(delivery.Signed{Caller: caller, Kind: a.Kind, Node: a.Node, Digest: a.Digest, Signer: a.Signer, At: at})
var body map[string]any
_ = json.Unmarshal(raw, &body)
if err := v.Emit(delivery.SignedEvent, body); err != nil {
v.log("signed %s for %s, but %s could not be said: %v", a.Kind, caller, delivery.SignedEvent, err)
}
return nil
}
// SignedRow is one signature as vault_signed lists it.
type SignedRow struct {
At time.Time `json:"at"`
Caller string `json:"caller"`
With string `json:"with"`
Kind string `json:"kind"`
Node string `json:"node,omitempty"`
Digest string `json:"digest"`
}
// Signatures are the last signatures the vault made, newest first, at most limit.
func (v *Vault) Signatures(limit int) ([]SignedRow, error) {
all, err := v.Store.Handed(10000)
if err != nil {
return nil, err
}
out := []SignedRow{}
for _, h := range all {
if h.Outcome != signedOutcome || len(out) >= limit {
continue
}
f := strings.Fields(h.Why)
row := SignedRow{At: h.At, Caller: h.To, With: h.Name}
if len(f) == 3 {
row.Kind, row.Node, row.Digest = f[0], f[1], f[2]
}
out = append(out, row)
}
return out, nil
}
// SigningState is what vault_status and vault_signers say of the signing secrets: whether each is held, its operator
// copy, and the fingerprints held against those published.
func (v *Vault) SigningState() map[string]any {
out := map[string]any{}
for _, name := range []string{SigningKeySecret, AuthorityKeySecret} {
r, held, err := v.Store.Read(name)
row := map[string]any{"held": held}
if err != nil {
row["error"] = err.Error()
}
if held {
row["generation"], row["made"], row["operator_copy"] = r.Generation, r.MadeAt, r.Operator.Sealed != ""
}
out[name] = row
}
key, auth, err := v.SignersValues()
if err != nil {
out["error"] = err.Error()
return out
}
out["key_fingerprint"], out["authority_fingerprint"] = key.Fingerprint, auth.Fingerprint
if v.Signers == nil {
return out
}
published := map[string]any{}
match := true
for name, want := range map[string]string{delivery.SignersKeyKey: key.Fingerprint, delivery.SignersAuthorityKey: auth.Fingerprint} {
raw, err := v.Signers.Get(name)
var got struct {
Fingerprint string `json:"fingerprint"`
}
if err != nil || raw == nil || json.Unmarshal(raw, &got) != nil {
published[name], match = nil, false
continue
}
published[name] = got.Fingerprint
match = match && got.Fingerprint == want
}
out["published"], out["published_matches"] = published, match
return out
}
// signingSaid says why the signing pass did not finish, once, and again when it changes.
func (v *Vault) signingSaid(err error) {
line := ""
if err != nil {
line = err.Error()
}
v.mu.Lock()
changed := line != v.signingWhy
v.signingWhy = line
v.mu.Unlock()
if changed && line != "" {
v.log("the vault signs nothing yet: %s", line)
}
}
@@ -0,0 +1,196 @@
package main
import (
"crypto/ed25519"
"crypto/rand"
"encoding/base64"
"encoding/json"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/overlay"
"git.novox.be/novox/mesh-sdk/go/seal"
)
// signersState is the vault's state `signers` in memory, counting writes.
type signersState struct {
keys map[string][]byte
writes int
}
func (s *signersState) Get(key string) ([]byte, error) { return s.keys[key], nil }
func (s *signersState) Put(key string, value []byte) error {
s.keys[key] = append([]byte(nil), value...)
s.writes++
return nil
}
// L3 s2 to s5 (novox/hq ADR 0550 §2): the vault makes its signing key and authority once, with their operator copies and
// no other holder; publishes their public halves only when they differ; signs a node's certificate for that node's
// node-engine alone, from its placement's name and its record's serving key; and no answer, event or record carries a
// key.
func TestTheVaultSigns(t *testing.T) {
v := newVault(t, nil)
state := &signersState{keys: map[string][]byte{}}
v.Signers = state
servingPub, _, _ := ed25519.GenerateKey(rand.Reader)
placements := map[string]delivery.Placement{
"ace": {Node: "ace", Decided: delivery.NodeDecided{Address: "10.77.0.3"}},
"g14": {Node: "g14", Decided: delivery.NodeDecided{Address: "10.77.0.4"}},
"plain": {Node: "plain"},
}
v.Placement = PlacementReader{Read: func(key string) ([]byte, error) {
node, err := delivery.ParsePlacementKey(key)
if err != nil {
return nil, err
}
p, ok := placements[node]
if !ok {
return nil, nil
}
return delivery.EncodePlacement(p)
}}
records := map[string]string{"ace": base64.StdEncoding.EncodeToString(servingPub), "plain": base64.StdEncoding.EncodeToString(servingPub)}
v.Haves = func(key string) ([]byte, error) {
node, _, _ := strings.Cut(key, ".")
serving, ok := records[node]
if !ok {
return nil, nil
}
return delivery.EncodeHaveNode(delivery.HaveNode{Record: &delivery.NodeRecord{Node: node, ServingKey: serving}})
}
// s2: made once, kept with their operator copies; a second pass makes nothing new and writes nothing again (s5).
if err := v.SigningPass(); err != nil {
t.Fatal(err)
}
var seeds []string
for _, name := range []string{SigningKeySecret, AuthorityKeySecret} {
r, held, err := v.Store.Read(name)
if err != nil || !held || r.Generation != 1 {
t.Fatalf("%s: held %v at generation %d, %v", name, held, r.Generation, err)
}
value, _ := seal.Open(r.AtRest, v.Own.PrivateKey)
if got := openOperator(t, v.operatorPublic, v.operatorPrivate, r.Operator.Sealed); got != value {
t.Errorf("%s: the operator copy does not open to the value held", name)
}
seeds = append(seeds, value)
}
key1, auth1, err := v.SignersValues()
if err != nil {
t.Fatal(err)
}
writes := state.writes
if writes != 2 {
t.Errorf("signers written %d times at the first pass, want its two keys", writes)
}
if err := v.SigningPass(); err != nil {
t.Fatal(err)
}
key2, auth2, _ := v.SignersValues()
if key2 != key1 || auth2 != auth1 || state.writes != writes {
t.Errorf("a second pass changed the keys or wrote signers again (%d writes)", state.writes)
}
if _, err := delivery.DecodeSignerKey(state.keys[delivery.SignersKeyKey]); err != nil {
t.Errorf("signers' key: %v", err)
}
authority, err := delivery.DecodeSignerAuthority(state.keys[delivery.SignersAuthorityKey])
if err != nil {
t.Fatalf("signers' authority: %v", err)
}
// No other holder: `get` hands neither, even to an asker a rule names for it.
asker, _ := seal.GenerateKeyPair()
v.Askers = AllowList{"node.ace": {SigningKeySecret, AuthorityKeySecret}}
for _, name := range []string{SigningKeySecret, AuthorityKeySecret} {
if _, err := v.Get("node.ace", name, asker.PublicKey); err == nil {
t.Errorf("get handed %s", name)
}
}
v.Askers = nil
if forced, err := v.PublishSigners(true); err != nil || len(forced) != 2 {
t.Errorf("signers_reassert wrote %v, %v", forced, err)
}
// s3, s4: who may ask which kind, and the certificate built from bus state.
for _, row := range []struct {
name, caller, kind, node, refused string
}{
{"a module is not a node-engine", "ace.mesh-grants", delivery.KindNodeCertificate, "", "not a node-engine"},
{"another node's certificate", "node.g14", delivery.KindNodeCertificate, "ace", "its own alone"},
{"a node not on the private network", "node.plain", delivery.KindNodeCertificate, "", "not on the private network"},
{"a node that states no serving key", "node.g14", delivery.KindNodeCertificate, "", "no record"},
{"an enrolment has no caller yet", "node.ace", delivery.KindEnrolment, "", "ADR 0561 §5"},
{"the authority is the vault's own", "node.ace", delivery.KindAuthority, "", "the vault's own"},
{"an unproven caller", "", delivery.KindNodeCertificate, "", "proved"},
{"the node's own certificate", "node.ace", delivery.KindNodeCertificate, "ace", ""},
} {
v.bus.raw = nil
got, err := v.Sign(row.caller, SignArgs{Kind: row.kind, Node: row.node})
if row.refused != "" {
if err == nil || !strings.Contains(err.Error(), row.refused) {
t.Errorf("%s: %v, want refused saying %q", row.name, err, row.refused)
}
if len(v.bus.raw) != 0 {
t.Errorf("%s: refused, and said %v", row.name, v.bus.raw)
}
continue
}
if err != nil {
t.Fatalf("%s: %v", row.name, err)
}
if err := delivery.CheckNodeCertificate(got.Certificate, []string{authority.Certificate}, overlay.InternalName("ace"),
servingPub, authority.Made.Add(1)); err != nil {
t.Errorf("%s: the certificate does not check against signers' authority: %v", row.name, err)
}
if got.Signer != authority.Fingerprint || got.Digest == "" {
t.Errorf("%s: signer %s, digest %q", row.name, got.Signer, got.Digest)
}
// Said once as signed, kept in the record, and no key in any of it.
answer, _ := json.Marshal(got)
said := strings.Join(v.bus.raw, "\n")
if !strings.Contains(said, `"event":"signed"`) || !strings.Contains(said, got.Digest) {
t.Errorf("%s: signed not said: %s", row.name, said)
}
rows, _ := v.Signatures(10)
if len(rows) != 1 || rows[0].Caller != "node.ace" || rows[0].Kind != delivery.KindNodeCertificate || rows[0].Digest != got.Digest {
t.Errorf("%s: vault_signed lists %+v", row.name, rows)
}
status, _ := json.Marshal(v.SigningState())
for _, seed := range seeds {
raw, _ := base64.StdEncoding.DecodeString(seed)
priv := ed25519.NewKeyFromSeed(raw)
for _, text := range []string{string(answer), said, string(status), string(state.keys[delivery.SignersKeyKey]),
string(state.keys[delivery.SignersAuthorityKey])} {
if strings.Contains(text, seed) || strings.Contains(text, base64.StdEncoding.EncodeToString(priv)) {
t.Errorf("%s: a signing key's private half is in what the vault says: %s", row.name, text)
}
}
}
}
// A serving key changed in the record is the one certified next, never the one before.
other, _, _ := ed25519.GenerateKey(rand.Reader)
records["ace"] = base64.StdEncoding.EncodeToString(other)
got, err := v.Sign("node.ace", SignArgs{Kind: delivery.KindNodeCertificate})
if err != nil || delivery.CheckNodeCertificate(got.Certificate, []string{authority.Certificate}, overlay.InternalName("ace"),
servingPub, authority.Made.Add(1)) == nil {
t.Errorf("a certificate for a serving key the record no longer states: %v", err)
}
// s2: one secret lost beside the other, or both lost while signers publishes a key, makes nothing.
lost := newVault(t, nil)
lost.Signers = &signersState{keys: map[string][]byte{delivery.SignersKeyKey: state.keys[delivery.SignersKeyKey]}}
if made, err := lost.EnsureSigning(); made || err == nil || !strings.Contains(err.Error(), "MISSING") {
t.Errorf("keys made while signers publishes one: %v, %v", made, err)
}
half := newVault(t, nil)
if _, err := half.Put(SigningKeySecret, seeds[0], "the test"); err != nil {
t.Fatal(err)
}
if made, err := half.EnsureSigning(); made || err == nil || !strings.Contains(err.Error(), "MISSING") {
t.Errorf("an authority made beside a held signing key: %v, %v", made, err)
}
if _, held, _ := half.Store.Read(AuthorityKeySecret); held {
t.Error("the authority was made all the same")
}
}
@@ -52,6 +52,8 @@ type Record struct {
// (move-in with replace_running, owntake.go): kept sealed as it was, so the replacement can be read and undone. Any later
// change of the secret drops it.
Previous *PreviousValue `json:"previous,omitempty"`
// Retired marks a bus account no longer wanted (retire.go, ADR 0561 §2): its value kept, its hash taken out.
Retired *RetiredMark `json:"retired,omitempty"`
}
// PreviousValue is a replaced value as the store kept it: its generation, both its seals, and who changed it when.
+94 -4
View File
@@ -122,7 +122,8 @@ func definitions(env *Env) []toolDef {
}},
{Name: Seat + ".secret-ask",
Description: "Ask the operator for a module's own secret at a desk (ADR 0349 clause 9, ADR 0277 §22–§27), " +
"asked by the controller alone: node, module, name, desk and asked_by. The ask is recorded before any " +
"asked by the controller (naming asked_by) or by a caller the vault's manifest names, itself named by the " +
"bus (ADR 0563 §8): node, module, name and desk. The ask is recorded before any " +
"prompt opens — one open ask per secret and three an hour — and the vault asks the desk's prompt itself, " +
"in its own name, with a key made for the call; what is typed opens in the vault alone. Answers how the " +
"prompt ended (given, dismissed, timed-out, empty, refused or failed) and who asked as recorded; never " +
@@ -130,7 +131,7 @@ func definitions(env *Env) []toolDef {
Input: map[string]any{"type": "object", "properties": map[string]any{
"node": str("the machine the secret is for"), "module": str("the module"),
"name": str("the secret's name in the module"), "desk": str("the machine whose prompt asks"),
"asked_by": str("who asked the controller, as the bus named its caller")}},
"asked_by": str("from the controller alone: who asked it, as the bus named its caller")}},
run: func(caller string, a map[string]any) (any, error) {
v, err := env.ready()
if err != nil {
@@ -303,6 +304,20 @@ func definitions(env *Env) []toolDef {
// clause 9, adopt.go): a public half, never the private one.
return map[string]any{"recorded": true, "vault_key": v.Own.PublicKey}, nil
}},
{Name: Seat + ".issue-account",
Description: "Mint one bus account the grants part wants, now (novox/hq ADR 0561 §9): the recovery when the vault did not " +
"hear a want. Reads the wanted key afresh from mesh-grants.accounts and mints it as a want would: only while " +
"the controller's lease is handed away, never for a node's account or the controller's, never a user nobody " +
"wants, and never anew for one whose value the vault holds or whose hash it adopted (it says so). Answers " +
"whether it minted, the secret and generation, never the value or the hash.",
Input: map[string]any{"user": str("the bus user, e.g. ace.mesh-build-go")},
run: func(caller string, a map[string]any) (any, error) {
v, err := env.ready()
if err != nil {
return nil, err
}
return v.IssueAccount(caller, arg(a, "user"))
}},
{Name: Seat + ".own",
Description: "The own secrets of this machine's modules, each sealed to the machine's sealing key (novox/hq ADR 0547 §2, " +
"ADR 0550 §5, issue 1126): from the node-engine of the machine alone, about its own machine (the bus proves which), " +
@@ -476,6 +491,28 @@ func definitions(env *Env) []toolDef {
}
return env.Vault.HashUser(arg(a, "user"), caller)
}},
{Name: "account_retire",
Description: "Retire bus accounts the grants part no longer wants, by a person's act with why (novox/hq ADR 0561 §2, " +
"ADR 0230 §18): the vault takes each one's hash out of its state `bus-accounts`, so the grants part's next user " +
"list drops the user, and keeps its value marked retired. For the set vault_status lists as waiting for a " +
"person (more than three, or more than half, gone at once); refused for an account still wanted, an adopted " +
"hash, and with no why. Wanted again, an account is re-enabled at once.",
Input: map[string]any{"users": map[string]any{"type": "array", "items": str("a bus user")},
"why": str("why they are retired")},
run: func(caller string, a map[string]any) (any, error) {
if env.Vault == nil {
return nil, errors.New("the vault is not ready")
}
var users []string
if list, ok := a["users"].([]any); ok {
for _, u := range list {
if s, ok := u.(string); ok {
users = append(users, strings.TrimSpace(s))
}
}
}
return env.Vault.RetireAccounts(caller, users, arg(a, "why"))
}},
{Name: "secret_vault_export",
Description: "Every secret the vault itself holds, as its copy sealed to the operator key, in the shape of " +
"`secret export`: saved to a file, `nox secret recover vault mesh-vault <name> --key <operator key file> " +
@@ -537,6 +574,57 @@ func definitions(env *Env) []toolDef {
}
return map[string]any{"deleted": consumer, "freed_bytes": -1}, nil
}},
{Name: Seat + ".sign",
Description: "Sign with the vault's key or authority (novox/hq ADR 0550 §2). Kind node-certificate: the certificate " +
"of the calling machine's internal name for the serving key its record states, built from bus state alone and " +
"asked by that machine's node-engine only (the bus proves which); kind enrolment: no caller is named until " +
"adoption's record (ADR 0561 §5). Answers the certificate, the authority, the signer's fingerprint and the " +
"digest, and says signed. Never a key.",
Input: map[string]any{"kind": str("node-certificate or enrolment"),
"node": str("the machine, for a node certificate: the caller's own (left out, the caller's)"),
"reply": str("the enrolment reply's bytes in base64, for an enrolment")},
run: func(caller string, a map[string]any) (any, error) {
v, err := env.ready()
if err != nil {
return nil, err
}
return v.Sign(caller, SignArgs{Kind: arg(a, "kind"), Node: arg(a, "node"), Reply: arg(a, "reply")})
}},
{Name: "vault_signers",
Description: "The vault's signing key and authority (novox/hq ADR 0550 §2): whether each is held with its operator " +
"copy, the fingerprints the vault holds, what its state signers publishes, and whether they match. Never a key.",
run: func(string, map[string]any) (any, error) {
v, err := env.ready()
if err != nil {
return nil, err
}
return v.SigningState(), nil
}},
{Name: "vault_signed",
Description: "The last signatures the vault made, newest first: when, the caller the bus proved, the kind, the node " +
"and the digest of what was signed. Never a key.",
Input: map[string]any{"limit": map[string]any{"type": "number", "description": "how many (default 50)"}},
run: func(_ string, a map[string]any) (any, error) {
v, err := env.ready()
if err != nil {
return nil, err
}
return v.Signatures(limitOf(a))
}},
{Name: "signers_reassert",
Description: "Write the vault's state signers again from the keys it holds, whether or not it differs (novox/hq " +
"ADR 0550 §2): the by-hand path for a state lost or doubted. Makes no key.",
run: func(string, map[string]any) (any, error) {
v, err := env.ready()
if err != nil {
return nil, err
}
written, err := v.PublishSigners(true)
if err != nil {
return nil, err
}
return map[string]any{"written": written}, nil
}},
{Name: "vault_health",
Description: "The vault's runtime health, as the node-engine asks it (novox/hq ADR 0366): unhealthy once " +
"re-sealing the operator copies has failed on three passes in a row, saying what failed and why; " +
@@ -544,7 +632,9 @@ func definitions(env *Env) []toolDef {
run: func(string, map[string]any) (any, error) { return env.health(), nil }},
{Name: "vault_status",
Description: "What needs a person's eye in the vault: operator copies still sealed to an earlier key, the operator " +
"key the mesh's export holds if the vault refused it, and how often the record of hand-outs was rolled.",
"key the mesh's export holds if the vault refused it, how often the record of hand-outs was rolled, and " +
"minting (novox/hq ADR 0561 §3, §9): whether the vault mints bus accounts now and why (only while the " +
"controller's lease is handed away), every wanted account it holds back with why, and those it minted.",
run: func(string, map[string]any) (any, error) {
if env.Vault == nil {
return map[string]any{"ready": false, "error": fmt.Sprint(env.Broken)}, nil
@@ -556,7 +646,7 @@ func definitions(env *Env) []toolDef {
key, _ := env.Vault.Store.OperatorKey()
return map[string]any{"ready": true, "operator_key": key, "copies_on_an_earlier_key": left,
"operator_key_refused": env.keyRefused(), "handed_rolled": env.Vault.Store.Rolled(),
"ok": left == 0 && env.keyRefused() == "" && key != ""}, nil
"minting": env.Vault.MintState(), "retiring": env.Vault.RetireState(), "signing": env.Vault.SigningState(), "ok": left == 0 && env.keyRefused() == "" && key != ""}, nil
}},
}
}
+30 -5
View File
@@ -40,8 +40,12 @@ type Vault struct {
// account (ADR 0259 §8's runs-as, issue 723): the desks' secret prompt and each issuing provider's tool, which
// the bus lets the vault's account alone call in its own name. Nil asks nobody.
Ask func(key string, body any) (json.RawMessage, error)
Now func() time.Time
Log func(format string, args ...any)
// AskCallers are the callers besides the controller that may ask secret-ask themselves, as the vault's manifest names
// them (MESH_VAULT_SECRET_ASK_CALLERS; novox/hq ADR 0563 §8): each is named by the runtime's proof. Empty: the
// controller alone.
AskCallers stdio.Callers
Now func() time.Time
Log func(format string, args ...any)
// Accounts is the vault's state of bus-account hashes (accounts.go, ADR 0547 clause 9): nil writes none.
Accounts AccountStore
// HashCost is the bcrypt cost of those hashes: zero is the controller's, bcrypt's default.
@@ -58,6 +62,19 @@ type Vault struct {
hashes hashing
// adoption is the last adoption (adopt.go), under mu.
adoption adoptionKept
// WantRead reads one wanted account afresh (issue-account): nil reads the wants as the watch handed them.
WantRead func(user string) ([]byte, bool, error)
// Placement reads the seat's placement (placed.go): where modules stand once the assignments are gone.
Placement PlacementReader
// LeaseRead reads the controller's lease afresh before a mint (lease.go): nil reads it only as the watch hands it.
LeaseRead func() ([]byte, bool, error)
// minting is the switch on the lease and what the vault knows of the wanted bus accounts (mint.go).
minting minting
// Signers is the vault's state `signers`, where it publishes its signing halves (signing.go, ADR 0550 §2): nil writes
// none.
Signers SignersStore
// signingWhy is why the last signing pass did not finish, under mu (signing.go).
signingWhy string
// callsMu guards pending: the keys of the calls open now (intake.go), never written anywhere.
callsMu sync.Mutex
pending map[string]*callKey
@@ -106,8 +123,10 @@ func (v *Vault) keep(name, value, by, via, origin string) (Changed, error) {
// was first made, and where it was moved from.
type keepOptions struct {
neverReplace bool
madeAt time.Time
movedFrom *MovedFrom
// quiet keeps secret-changed for the caller to say once the rest is done (a minted account: mint.go).
quiet bool
madeAt time.Time
movedFrom *MovedFrom
// replaceOnly, keepPrevious and refuse are what move-in with replace_running keeps (owntake.go): only a value held is replaced,
// keeping its origin and the value it replaces, and refuse, under the store's lock, says why a held value is not.
replaceOnly bool
@@ -208,7 +227,9 @@ func (v *Vault) keepLocked(name, value, by, via, origin string, opt keepOptions)
return Changed{}, err
}
c := Changed{Name: name, Generation: rec.Generation, At: now, By: by}
v.announce(c)
if !opt.quiet {
v.announce(c)
}
return c, nil
}
@@ -283,6 +304,10 @@ func (v *Vault) Get(caller, name, publicKey string) (Answer, error) {
if err := validName(name); err != nil {
return Answer{}, err
}
// The vault's signing secrets are handed to nobody: it signs with them itself (ADR 0550 §2).
if signingSecret(name) {
return Answer{}, fmt.Errorf("%s is the vault's own signing secret and is handed to nobody: ask the verb sign", name)
}
// A value issued for one holder is that holder's alone, whatever else would allow it (issue 749).
if why := notTheHolder(caller, name); why != "" {
v.log("refused %s to %s: %s", name, caller, why)
+28 -10
View File
@@ -131,9 +131,11 @@ func filesUnder(t *testing.T, dir string) map[string]string {
// The manifest declares one bucket, and no secret is in it: the vault's only place for a secret is its own store on its
// own disk (ADR 0349 §7, ADR 0201 §13). The one bucket is bus-accounts, the bcrypt hashes of the bus accounts it holds
// (ADR 0547 clause 9), opened in accounts.go alone. It reads two buckets of others', and only reads them: the
// controller's assignments, for what each node's plan names (askers.go, design 57 V4), and the node-engines' haves, where
// the bus node hands the loaded list's hashes sealed to the vault (adopt.go, ADR 0547 clause 9), in main.go alone.
// (ADR 0547 clause 9), opened in accounts.go alone. It reads five buckets of others', and only reads them: the
// controller's assignments, for what each node's plan names (askers.go, design 57 V4), the node-engines' haves, where
// the bus node hands the loaded list's hashes sealed to the vault (adopt.go, ADR 0547 clause 9), the grants part's wanted
// accounts and the controller's lease (mint.go, lease.go, ADR 0561 §2, §3), and the seat's placement (placed.go), in
// main.go alone.
func TestNoBucketHoldsASecret(t *testing.T) {
raw, err := os.ReadFile("../../module.json")
if err != nil {
@@ -143,12 +145,13 @@ func TestNoBucketHoldsASecret(t *testing.T) {
if err := json.Unmarshal(raw, &manifest); err != nil {
t.Fatal(err)
}
if s, _ := json.Marshal(manifest["state"]); string(s) != `[{"history":2,"name":"bus-accounts"}]` {
t.Errorf("the manifest declares buckets %s: a secret is in no bucket, sealed or not (ADR 0201 §13), and the one "+
"bucket is the accounts' hashes (ADR 0547 clause 9)", s)
if s, _ := json.Marshal(manifest["state"]); string(s) != `[{"history":2,"name":"bus-accounts"},"signers"]` {
t.Errorf("the manifest declares buckets %s: a secret is in no bucket, sealed or not (ADR 0201 §13), and the "+
"buckets are the accounts' hashes (ADR 0547 clause 9) and the signing keys' public halves (ADR 0550 §2)", s)
}
if r, _ := json.Marshal(manifest["reads"]); string(r) != `["mesh-controller.assignments","node-engine.have"]` {
t.Errorf("the manifest reads %s: the vault reads the controller's assignments and the haves alone", r)
if r, _ := json.Marshal(manifest["reads"]); string(r) != `["mesh-controller.assignments","node-engine.have","mesh-grants.accounts","mesh-controller.lease","mesh-delivery-assigner.placement"]` {
t.Errorf("the manifest reads %s: the vault reads the controller's assignments, the haves and the grants part's "+
"wanted accounts alone", r)
}
sources, _ := filepath.Glob("*.go")
bucket := regexp.MustCompile(`stdio\.State\(|nats-io|jetstream|KeyValue`)
@@ -166,14 +169,29 @@ func TestNoBucketHoldsASecret(t *testing.T) {
code = []byte(strings.Replace(string(code), "stdio.State(BusAccountsState).Get(", "", 1))
code = []byte(strings.Replace(string(code), "stdio.State(BusAccountsState).Put(", "", 1))
code = []byte(strings.Replace(string(code), "stdio.State(BusAccountsState).Keys(", "", 1))
code = []byte(strings.Replace(string(code), "stdio.State(BusAccountsState).Delete(", "", 1))
}
// The haves, in main.go alone: one read, one watch and their keys (own_drift), never a write; and one watch of
// the assignments, which the vault's rules read from memory once it caught up (issue 1126).
// The signing keys' public halves, in signing.go alone: one read and one write of its own bucket (ADR 0550 §2).
if f == "signing.go" {
code = []byte(strings.Replace(string(code), "stdio.State(delivery.SignersState).Get(", "", 1))
code = []byte(strings.Replace(string(code), "stdio.State(delivery.SignersState).Put(", "", 1))
}
// The haves, in main.go alone: one read, one watch and their keys (own_drift), never a write; one watch of the
// assignments, which the vault's rules read from memory once it caught up (issue 1126); and one watch of the wanted
// accounts, the grants part's, which hold no secret (ADR 0561 §2).
if f == "main.go" {
code = []byte(strings.Replace(string(code), "stdio.State(AssignmentsRead).Watch(", "", 1))
code = []byte(strings.Replace(string(code), "stdio.State(HaveRead).Get(", "", 1))
code = []byte(strings.Replace(string(code), "stdio.State(HaveRead).Watch(", "", 1))
code = []byte(strings.Replace(string(code), "stdio.State(HaveRead).Keys(", "", 1))
code = []byte(strings.Replace(string(code), "stdio.State(WantedRead).Watch(", "", 1))
code = []byte(strings.Replace(string(code), "stdio.State(WantedRead).Get(", "", 1))
// And the controller's lease, one read and one watch (ADR 0561 §3).
code = []byte(strings.Replace(string(code), "stdio.State(LeaseRead).Get(", "", 1))
code = []byte(strings.Replace(string(code), "stdio.State(LeaseRead).Watch(", "", 1))
// And the seat's placement, where modules stand (placed.go): its keys and one key, read.
code = []byte(strings.Replace(string(code), "stdio.State(PlacementRead).Keys(", "", 1))
code = []byte(strings.Replace(string(code), "stdio.State(PlacementRead).Get(", "", 1))
}
if m := bucket.Find(code); m != nil {
t.Errorf("%s opens a bucket (%s): the vault keeps every secret in its own store only", f, m)
+1 -1
View File
@@ -3,7 +3,7 @@ module mesh-vault
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261028174713-38dbe1d3a388
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261028205404-8eafc0c5ce3c
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
golang.org/x/crypto v0.57.0
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261028174713-38dbe1d3a388 h1:ycu/sNieTdLpD3bzhZOCXdzcoqXUOxPRmqWV019REpo=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261028174713-38dbe1d3a388/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261028205404-8eafc0c5ce3c h1:hcqOCd5q3M0ZiEzB6MhwxT/AOKkm9gaskKqPcpkAAg8=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261028205404-8eafc0c5ce3c/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
+19 -6
View File
@@ -16,17 +16,23 @@
"rotated",
"secret-changed",
"account-hashed",
"own-replaced"
"own-replaced",
"account-issued",
"signed"
],
"reads": [
"mesh-controller.assignments",
"node-engine.have"
"node-engine.have",
"mesh-grants.accounts",
"mesh-controller.lease",
"mesh-delivery-assigner.placement"
],
"state": [
{
"name": "bus-accounts",
"history": 2
}
},
"signers"
],
"receives": {
"secret": "${dir:grants}/mesh.json"
@@ -144,7 +150,8 @@
"MESH_VAULT_LEDGER": "${dir:ledger}",
"MESH_VAULT_ROOT": "${dir:root}",
"MESH_VAULT_STORE": "${dir:store}",
"MESH_VAULT_STATE": "${dir:state}"
"MESH_VAULT_STATE": "${dir:state}",
"MESH_VAULT_SECRET_ASK_CALLERS": "node-tools"
}
}
]
@@ -157,6 +164,7 @@
"secret_own",
"account_hash",
"account_adopt",
"account_retire",
"secret_asks",
"secret_vault_export",
"provisioner_retirement",
@@ -166,7 +174,10 @@
"vault_status",
"vault_health",
"own_drift",
"own_take_switch"
"own_take_switch",
"vault_signers",
"vault_signed",
"signers_reassert"
],
"claims": [
{
@@ -181,7 +192,9 @@
"move-in",
"node-key",
"move-asks",
"own"
"own",
"issue-account",
"sign"
]
}
]
+1 -1
View File
@@ -133,7 +133,7 @@ func pruneGrants(grants map[string]Grant, kind string, now time.Time, keep int)
// stands, closes the ask with a warrant for its asker (asks.go); Details are shown where they were asked.
// Anything else is refused, and said there. The same answer is acted on once.
func (h *Holder) Chosen(kind string, c asks.Choice) (string, error) {
// The controller's root-free word is asked before the work is taken (the confirmation review of 2026-10-09):
// The machines' engines' root-free word is asked before the work is taken (the confirmation review of 2026-10-09):
// up to RootFreeWithin, which no ask, answer or link should wait behind. Asked fresh here, for this answer,
// and judged with in it.
root := h.rootWordFor(kind)
@@ -600,6 +600,16 @@ func TestAnAskIsRefusedInWordsToItsAsker(t *testing.T) {
if _, kept := store.asks["mesh-delivery.d-1"]; kept {
t.Error("a refused ask was recorded")
}
// A node's engine is an asker under node-engine-<node> (novox/hq ADR 0563 §3, changing ADR 0259 §2): its ask is
// recorded under that name, with its own bound of open asks; a name that is no asker's is dropped.
submit(t, h, "node-engine-ace", heldAsk(c, "hand-over-1"))
submit(t, h, "node.ace", heldAsk(c, "hand-over-2"))
if rec, kept := store.asks["node-engine-ace.hand-over-1"]; !kept || rec.Asker != "node-engine-ace" {
t.Errorf("a node's engine's ask was not recorded under its name: %+v", rec)
}
if _, kept := store.asks["node.ace.hand-over-2"]; kept {
t.Error("an ask under a name that is no asker's was recorded")
}
// No channel can carry anything: held with the reason (ADR 0397), never queued in silence.
h2, _, _, c2, store2, w2 := newRouter(t)
h2.Bench.Stand("telegram", Standing{Ready: false, Why: "no bot token", At: c2.now()})
+51 -37
View File
@@ -242,8 +242,9 @@ func (h *Holder) carriers(level asks.Level) []string {
return out
}
// RootVerdict is the controller's word on one machine (its verb root-free, novox/hq ADR 0259 §8): free only
// when it measured, recently, that no agent can become root there without a person.
// RootVerdict is a machine's engine's word on that machine (the node seat node-engine's root-free, novox/hq ADR 0259
// §8 as ADR 0563 §6 changes it): free only when it measured, recently, that no agent can become root there without a
// person.
type RootVerdict struct {
Machine string `json:"machine"`
Free bool `json:"free"`
@@ -251,51 +252,64 @@ type RootVerdict struct {
Judged string `json:"judged,omitempty"`
}
// RootFreeVerb is the controller's verb the router asks, as this module, which machines are root-free.
const RootFreeVerb = "seat:" + ControllerSeat + ".root-free"
// EngineSeat is the node seat each machine's engine holds, and RootFreeVerb its verb the router asks of a machine, as
// this module: `seat:node-engine.root-free@<machine>`, which that machine's engine alone answers (ADR 0563 §6).
const (
EngineSeat = "node-engine"
RootFreeVerb = "seat:" + EngineSeat + ".root-free"
)
// RootFreeWithin is how long the router waits for the controller's word; RootFreeKept how long a word is
// used to render a message's buttons. An answer is always judged on a word asked then.
// RootFreeWithin is how long the router waits for the machines' words; RootFreeKept how long a word is used to render a
// message's buttons. An answer is always judged on a word asked then.
const (
RootFreeWithin = 10 * time.Second
RootFreeKept = 30 * time.Second
)
// askRootFree asks the controller about machines, failing on no answer in time.
// askRootFree asks each machine's own engine about that machine, at once, failing on no answer in time. A machine's
// word is the verdict its engine gives on it; a verdict an engine gives on another machine is not taken.
func (h *Holder) askRootFree(machines []string) (map[string]RootVerdict, error) {
if h.RootFree != nil {
return h.RootFree(machines)
}
if h.Ask == nil {
return nil, errors.New("this router cannot ask the controller")
return nil, errors.New("this router cannot ask a machine's engine")
}
type answer struct {
raw json.RawMessage
err error
machine string
raw json.RawMessage
err error
}
done := make(chan answer, 1)
go func() {
raw, err := h.Ask(RootFreeVerb, map[string]any{"machines": machines})
done <- answer{raw, err}
}()
var a answer
select {
case a = <-done:
case <-time.After(RootFreeWithin):
return nil, fmt.Errorf("the controller did not answer within %s", RootFreeWithin)
}
if a.err != nil {
return nil, a.err
}
var body struct {
Machines []RootVerdict `json:"machines"`
}
if err := json.Unmarshal(a.raw, &body); err != nil {
return nil, fmt.Errorf("the controller's answer could not be read: %v", err)
done := make(chan answer, len(machines))
for _, m := range machines {
go func(m string) {
raw, err := h.Ask(RootFreeVerb+"@"+m, map[string]any{})
done <- answer{m, raw, err}
}(m)
}
out := map[string]RootVerdict{}
for _, v := range body.Machines {
out[v.Machine] = v
deadline := time.After(RootFreeWithin)
for range machines {
var a answer
select {
case a = <-done:
case <-deadline:
return nil, fmt.Errorf("a machine's engine did not answer within %s", RootFreeWithin)
}
if a.err != nil {
return nil, fmt.Errorf("%s's engine: %w", a.machine, a.err)
}
var body struct {
Machines []RootVerdict `json:"machines"`
}
if err := json.Unmarshal(a.raw, &body); err != nil {
return nil, fmt.Errorf("%s's engine's answer could not be read: %v", a.machine, err)
}
for _, v := range body.Machines {
if v.Machine == a.machine {
out[v.Machine] = v
}
}
}
return out, nil
}
@@ -329,7 +343,7 @@ func (h *Holder) rootFree(k KindInfo, cached bool) string {
case cached:
// Read inside the work, which never waits on the controller: no recent word is no pass. warmRootFree
// asks outside it.
return "the controller's word on whether an agent can become root there is not recent"
return "the machines' engines' word on whether an agent can become root there is not recent"
default:
got, err := h.askRootFree(machines)
if err != nil {
@@ -351,7 +365,7 @@ func (h *Holder) rootFree(k KindInfo, cached bool) string {
v, said := verdicts[m]
switch {
case !said:
return "the controller said nothing of whether an agent can become root on " + m
return "its engine said nothing of whether an agent can become root on " + m
case !v.Free:
why := v.Why
if why == "" {
@@ -364,7 +378,7 @@ func (h *Holder) rootFree(k KindInfo, cached bool) string {
return ""
}
// warmRootFree asks the controller's root-free word for every kind that proves its sender, outside the work,
// warmRootFree asks the machines' engines' root-free word for every kind that proves its sender, outside the work,
// so what is rendered and offered inside it reads a recent word.
// force asks even a recent word again: taking an ask, whose refusal its asker keeps until something changes.
func (h *Holder) warmRootFree(force bool) {
@@ -390,7 +404,7 @@ func (h *Holder) warmRootFree(force bool) {
}
}
// rootWordFor asks the controller's root-free word for a kind fresh, before the work is taken, and answers
// rootWordFor asks the machines' engines' root-free word for a kind fresh, before the work is taken, and answers
// what judging an answer of that kind then reads: the word for that very kind's placement, or why there is none.
// A kind that does not prove its sender is asked nothing: its answers never approve.
func (h *Holder) rootWordFor(kind string) func(KindInfo) string {
@@ -453,7 +467,7 @@ func (h *Holder) TakeAsk(verb, asker string, body json.RawMessage) error {
if verb == ConditionVerb {
// Before the work lock: Condition takes it, and decides as for the controller's conditions.
if !asks.UsableID(strings.ReplaceAll(asker, "-", "_")) {
h.Logf("[messenger] a condition from %q was taken and dropped: not a module's name", asker)
h.Logf("[messenger] a condition from %q was taken and dropped: not an asker's name (a module's, or a node's engine's)", asker)
return nil
}
h.takeCondition(asker, body)
@@ -466,7 +480,7 @@ func (h *Holder) TakeAsk(verb, asker string, body json.RawMessage) error {
h.init()
h.mu.Unlock()
if !asks.UsableID(strings.ReplaceAll(asker, "-", "_")) {
h.Logf("[messenger] an ask from %q was taken and dropped: not a module's name", asker)
h.Logf("[messenger] an ask from %q was taken and dropped: not an asker's name (a module's, or a node's engine's)", asker)
return nil
}
if verb == "cancel" {
+2 -10
View File
@@ -256,16 +256,8 @@ func (r *busRuntime) runtime(id json.RawMessage, method string, params json.RawM
case "mesh/seat.publish":
return map[string]any{}, r.nc.Publish(p.Subject, p.Body)
case "mesh/ask":
if p.Key == RootFreeVerb {
var a struct {
Machines []string `json:"machines"`
}
_ = json.Unmarshal(p.Body, &a)
var out []RootVerdict
for _, m := range a.Machines {
out = append(out, RootVerdict{Machine: m, Free: true})
}
return map[string]any{"machines": out}, nil
if m, ok := strings.CutPrefix(p.Key, RootFreeVerb+"@"); ok {
return map[string]any{"machines": []RootVerdict{{Machine: m, Free: true}}}, nil
}
return nil, fmt.Errorf("the test runtime answers nothing for %s", p.Key)
}
+2 -1
View File
@@ -315,7 +315,8 @@ type Holder struct {
Emit func(event string, body any) error
// Ask calls a seat verb through the mesh, as this module: the answers the operator chooses.
Ask Asker
// RootFree asks the controller's verb root-free about machines; nil asks it through Ask (askRootFree).
// RootFree asks each machine's engine's root-free (the node seat node-engine's, ADR 0563 §6); nil asks it through Ask
// (askRootFree).
RootFree func(machines []string) (map[string]RootVerdict, error)
// Zone is the operator's time zone, which every time in a message is said in; nil is this machine's.
Zone func() *time.Location
@@ -87,7 +87,7 @@ func TestItDeclaresTheSeatsOfAnAskAndHoldsTheOperatorChannel(t *testing.T) { //
// It performs nothing: the verbs it calls read the open conditions, whether root is free, and which issues
// track which conditions.
if !reflect.DeepEqual(m.Emits, []string{"refused"}) || !reflect.DeepEqual(m.Invokes, []string{"seat:" + ControllerSeat + ".conditions",
"seat:" + ControllerSeat + ".root-free", TrackingVerb}) {
"seat:" + EngineSeat + ".root-free", TrackingVerb}) {
t.Fatalf("emits %v invokes %v", m.Emits, m.Invokes)
}
var state []string
+19 -7
View File
@@ -8,6 +8,7 @@ import (
"os"
"path/filepath"
"regexp"
"sort"
"strings"
"sync"
"testing"
@@ -66,26 +67,37 @@ func TestApprovalNeedsTheControllersWordThatRootIsFree(t *testing.T) {
}
}
// H3: the word is asked of the controller's verb, as this module, and read from its answer.
func TestTheRootWordIsAskedOfTheControllersVerb(t *testing.T) {
// H3, as ADR 0563 §6 changes it: the word on each machine is asked of that machine's own engine, the node seat
// node-engine's root-free there, as this module, and read from its answer; a machine's word is that machine's alone.
func TestTheRootWordIsAskedOfEachMachinesEngine(t *testing.T) {
h, _, _, _, _, _ := newRouter(t)
h.RootFree = nil
var asked []string
h.Ask = func(key string, body any) (json.RawMessage, error) {
asked = append(asked, key)
raw, _ := json.Marshal(body)
if !strings.Contains(string(raw), `"machines":["anchor"]`) {
t.Errorf("asked about %s", raw)
machine := strings.TrimPrefix(key, "seat:node-engine.root-free@")
if machine == "elsewhere" {
machine = "anchor" // an engine that answers for another machine is not believed
}
return json.RawMessage(`{"machines":[{"machine":"anchor","free":true,"why":"judged","judged":"2026-10-09T10:00:00Z"}]}`), nil
return json.RawMessage(`{"machines":[{"machine":"` + machine + `","free":true,"why":"judged","judged":"2026-10-09T10:00:00Z"}]}`), nil
}
info, _ := h.Bench.Kind("telegram")
if why := h.rootFree(info, false); why != "" {
t.Fatalf("free not read: %s", why)
}
if len(asked) != 1 || asked[0] != "seat:mesh-controller.root-free" {
if len(asked) != 1 || asked[0] != "seat:node-engine.root-free@anchor" {
t.Errorf("asked %v", asked)
}
asked = nil
h.Node = "elsewhere"
if why := h.rootFree(info, false); !strings.Contains(why, "said nothing of whether an agent can become root on elsewhere") {
t.Errorf("one machine's word was taken for another's: %q", why)
}
sort.Strings(asked)
if len(asked) != 2 || asked[0] != "seat:node-engine.root-free@anchor" || asked[1] != "seat:node-engine.root-free@elsewhere" {
t.Errorf("the router's machine and the channel's were asked as %v", asked)
}
h.Node = "anchor"
h.Ask = func(string, any) (json.RawMessage, error) { return json.RawMessage(`not json`), nil }
if why := h.rootFree(info, false); !strings.Contains(why, "could not be read") {
t.Errorf("an unreadable answer was a pass: %q", why)
+1 -1
View File
@@ -90,7 +90,7 @@
],
"invokes": [
"seat:mesh-controller.conditions",
"seat:mesh-controller.root-free",
"seat:node-engine.root-free",
"seat:issue-tracker.tracking"
],
"state": [
@@ -0,0 +1,174 @@
package main
import (
"crypto/rand"
"crypto/rsa"
"crypto/x509"
"crypto/x509/pkix"
"encoding/pem"
"errors"
"fmt"
"math/big"
"net"
"os"
"path/filepath"
"time"
)
// The bus's own certificate, made by the mesh rather than borrowed from an image. Moved here from mesh-controller's
// `broker certificate` with the same output (design 47's L3 s11): the command belongs to the bus module, and the
// controller's binary is retired after its lease is handed away (ADR 0547 §11). Nothing changes at the lease write: the
// certificate in /var/lib/mesh-broker-tls stays, and every pin stays valid.
//
// **The foundation asked a third-party image for a tool it never said must be there** (novox/hq
// 04-ISSUES/146). The bootstrap made this certificate by running `openssl` inside the broker's
// image, which worked while the broker was one that happened to carry it and stopped the day the
// bus changed: the new one has a shell and no openssl, so the step exited 127 and no mesh could be
// raised. Substituting another image the bundle names does not help — none of them carry it
// either.
//
// So the program that needs a certificate makes one. It is the mesh's own binary, already on the
// machine at this point in the bootstrap (the schema step ran it), and it needs nothing from the
// image it writes into but a mounted directory.
//
// **Self-signed, and that is the design** — a host pins this server's exact certificate and
// authenticates with a password (novox/hq ADR 0004). There is no authority above it to ask, and at
// this moment in a bootstrap there is no mesh to ask one of.
//
// Idempotent, because the step is applied again on every reconcile and a second certificate would
// be one the hosts that pinned the first no longer believe.
// busCertificateNames is what the bus is reached by: the container name on a mesh network, and the
// loopback address the machine's own foundation dials.
var busCertificateNames = []string{"mesh-broker"}
const busCertificateLife = 10 * 365 * 24 * time.Hour
// busCertificate makes the bus's certificate in a directory, or says whether one is there.
//
// mesh-nats-snapshot certificate --into /tls make it if it is not there
// mesh-nats-snapshot certificate --check --into /tls exit non-zero unless a usable pair is
func busCertificate(args []string) error {
into, check := "", false
for i := 0; i < len(args); i++ {
switch args[i] {
case "--check":
check = true
case "--into":
if i+1 >= len(args) {
return errors.New("--into needs a directory")
}
into = args[i+1]
i++
default:
return fmt.Errorf("certificate [--check] --into <directory>: %q", args[i])
}
}
if into == "" {
return errors.New("certificate [--check] --into <directory>")
}
crt, key := filepath.Join(into, "tls.crt"), filepath.Join(into, "tls.key")
if usable, err := busCertificateUsable(crt, key); err != nil {
return err
} else if usable {
fmt.Printf("the bus already has a certificate at %s, and it was left alone\n", crt)
return nil
}
if check {
// Said as a failure, because that is what the caller asked: a bootstrap's verify runs
// this and a false answer is what makes the step run.
return fmt.Errorf("no usable certificate and key at %s", into)
}
return writeBusCertificate(crt, key)
}
// busCertificateUsable says whether a certificate and its key are both there and parse.
//
// Both, and parsed rather than stat'ed: a half-written pair is the state a bootstrap interrupted
// between the two files leaves behind, and a step that treated it as done would hand the server a
// certificate with no key and report success.
func busCertificateUsable(crt, key string) (bool, error) {
certPEM, err := os.ReadFile(crt)
if errors.Is(err, os.ErrNotExist) {
return false, nil
}
if err != nil {
return false, err
}
keyPEM, err := os.ReadFile(key)
if errors.Is(err, os.ErrNotExist) {
return false, nil
}
if err != nil {
return false, err
}
if _, err := tlsPairParses(certPEM, keyPEM); err != nil {
return false, nil
}
return true, nil
}
func tlsPairParses(certPEM, keyPEM []byte) (*x509.Certificate, error) {
block, _ := pem.Decode(certPEM)
if block == nil || block.Type != "CERTIFICATE" {
return nil, errors.New("not a certificate")
}
certificate, err := x509.ParseCertificate(block.Bytes)
if err != nil {
return nil, err
}
keyBlock, _ := pem.Decode(keyPEM)
if keyBlock == nil {
return nil, errors.New("not a key")
}
if _, err := x509.ParsePKCS8PrivateKey(keyBlock.Bytes); err != nil {
if _, err := x509.ParsePKCS1PrivateKey(keyBlock.Bytes); err != nil {
return nil, err
}
}
return certificate, nil
}
func writeBusCertificate(crt, key string) error {
private, err := rsa.GenerateKey(rand.Reader, 2048)
if err != nil {
return err
}
serial, err := rand.Int(rand.Reader, new(big.Int).Lsh(big.NewInt(1), 128))
if err != nil {
return err
}
template := &x509.Certificate{
SerialNumber: serial,
Subject: pkix.Name{CommonName: busCertificateNames[0]},
DNSNames: busCertificateNames,
IPAddresses: []net.IP{net.ParseIP("127.0.0.1")},
NotBefore: time.Now().Add(-time.Hour),
NotAfter: time.Now().Add(busCertificateLife),
KeyUsage: x509.KeyUsageDigitalSignature | x509.KeyUsageKeyEncipherment,
ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
BasicConstraintsValid: true,
}
der, err := x509.CreateCertificate(rand.Reader, template, template, &private.PublicKey, private)
if err != nil {
return err
}
pkcs8, err := x509.MarshalPKCS8PrivateKey(private)
if err != nil {
return err
}
// **The key first, and only then the certificate**, so the pair a reader finds is never a
// certificate whose key has not been written yet — the one order in which an interruption
// leaves something that looks finished (novox/hq 04-ISSUES/014, a key present and unusable).
if err := os.WriteFile(key, pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: pkcs8}), 0o600); err != nil {
return err
}
if err := os.WriteFile(crt, pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der}), 0o644); err != nil {
return err
}
fmt.Printf("made the bus a certificate for %v, valid until %s\n %s\n %s\n",
busCertificateNames, template.NotAfter.Format(time.RFC3339), crt, key)
return nil
}
@@ -0,0 +1,138 @@
package main
import (
"crypto/tls"
"crypto/x509"
"os"
"path/filepath"
"strings"
"testing"
)
// L3 s11 (design 47): the bus's certificate command, moved from mesh-controller with the same output, made once, and
// its check false without a usable pair. Its cases, as the controller's tests had them, are rows of one test.
func TestTheBusCertificate(t *testing.T) {
for _, row := range []struct {
name string
run func(*testing.T)
}{
{"it loads as a server loads it", loadsAsAServersWould},
{"it is made once", isMadeOnce},
{"the check is false until there is a pair", checkIsFalseUntilAPair},
{"a certificate without its key is not usable", withoutItsKeyIsNotUsable},
{"where to write is required", whereToWriteIsRequired},
} {
t.Run(row.name, row.run)
}
}
// novox/hq 04-ISSUES/146. The bootstrap could not make the bus a certificate: it asked an image for
// `openssl` and the image it asks has none. What replaces it is this command, so what is checked is
// what the bootstrap needs from it — a pair a TLS server can actually load, made once and only once.
func loadsAsAServersWould(t *testing.T) {
into := t.TempDir()
if err := busCertificate([]string{"--into", into}); err != nil {
t.Fatalf("the bus could not be given a certificate: %v", err)
}
// The check a cheaper test would not make. The key was present and valid and the server could
// not start, once, because nothing loaded the pair the way a server loads it
// (novox/hq 04-ISSUES/014).
pair, err := tls.LoadX509KeyPair(filepath.Join(into, "tls.crt"), filepath.Join(into, "tls.key"))
if err != nil {
t.Fatalf("a TLS server cannot load what was written: %v", err)
}
leaf := pair.Leaf
if leaf == nil {
if leaf, err = x509.ParseCertificate(pair.Certificate[0]); err != nil {
t.Fatal(err)
}
}
if err := leaf.VerifyHostname("mesh-broker"); err != nil {
t.Errorf("the certificate is not for the name the bus is reached by: %v", err)
}
if len(leaf.IPAddresses) == 0 || leaf.IPAddresses[0].String() != "127.0.0.1" {
t.Errorf("the certificate does not cover the loopback address the foundation dials: %v", leaf.IPAddresses)
}
// The key is not readable by anything else on the machine; the certificate is public and is.
key, err := os.Stat(filepath.Join(into, "tls.key"))
if err != nil {
t.Fatal(err)
}
if key.Mode().Perm() != 0o600 {
t.Errorf("the key is %v", key.Mode().Perm())
}
crt, err := os.Stat(filepath.Join(into, "tls.crt"))
if err != nil {
t.Fatal(err)
}
if crt.Mode().Perm() != 0o644 {
t.Errorf("the certificate is %v, which the server runs as another user cannot read", crt.Mode().Perm())
}
}
// **Made once.** The step is applied again on every reconcile, and a second certificate is one the
// hosts that pinned the first no longer believe (novox/hq ADR 0004).
func isMadeOnce(t *testing.T) {
into := t.TempDir()
if err := busCertificate([]string{"--into", into}); err != nil {
t.Fatal(err)
}
first, err := os.ReadFile(filepath.Join(into, "tls.crt"))
if err != nil {
t.Fatal(err)
}
if err := busCertificate([]string{"--into", into}); err != nil {
t.Fatal(err)
}
again, err := os.ReadFile(filepath.Join(into, "tls.crt"))
if err != nil {
t.Fatal(err)
}
if string(first) != string(again) {
t.Fatal("running it twice replaced the certificate every host had pinned")
}
}
// The verify half: false before, true after, which is what makes the bootstrap run the step at all.
func checkIsFalseUntilAPair(t *testing.T) {
into := t.TempDir()
if err := busCertificate([]string{"--check", "--into", into}); err == nil {
t.Fatal("an empty directory reported a usable certificate")
}
if err := busCertificate([]string{"--into", into}); err != nil {
t.Fatal(err)
}
if err := busCertificate([]string{"--check", "--into", into}); err != nil {
t.Fatalf("the certificate it just made does not satisfy its own check: %v", err)
}
}
// A half-written pair is not a pair. An interrupted bootstrap leaves exactly this, and a step that
// called it done would hand the server a certificate with no key and report success.
func withoutItsKeyIsNotUsable(t *testing.T) {
into := t.TempDir()
if err := busCertificate([]string{"--into", into}); err != nil {
t.Fatal(err)
}
if err := os.Remove(filepath.Join(into, "tls.key")); err != nil {
t.Fatal(err)
}
if err := busCertificate([]string{"--check", "--into", into}); err == nil {
t.Fatal("a certificate with no key passed the check")
}
if err := busCertificate([]string{"--into", into}); err != nil {
t.Fatal(err)
}
if _, err := tls.LoadX509KeyPair(filepath.Join(into, "tls.crt"), filepath.Join(into, "tls.key")); err != nil {
t.Fatalf("it did not replace the unusable pair: %v", err)
}
}
func whereToWriteIsRequired(t *testing.T) {
if err := busCertificate(nil); err == nil || !strings.Contains(err.Error(), "--into") {
t.Fatalf("it did not ask where to write: %v", err)
}
}
@@ -17,6 +17,7 @@
// mesh-nats-snapshot verify [--from FILE]
// mesh-nats-snapshot restore --into DIR [--account MESH] [--from FILE]
// mesh-nats-snapshot restore --server URL --credential FILE [--from FILE]
// mesh-nats-snapshot certificate [--check] --into DIR
//
// **Read-only on the live bus.** A snapshot neither pauses nor reconfigures a stream; the server
// keeps accepting writes while it streams the copy out. The credential it uses is the bus's own
@@ -47,6 +48,8 @@ func main() {
err = runVerify(os.Args[2:])
case "restore":
err = runRestore(os.Args[2:])
case "certificate":
err = busCertificate(os.Args[2:])
case "-h", "--help", "help":
usage()
return
@@ -67,6 +70,7 @@ func usage() {
verify [--from bus.tar] every archive against the manifest
restore --into DIR [--account MESH] [--from bus.tar] into a new store directory, by a server of its own
restore --server URL --credential FILE [--from bus.tar] into a running server that holds none of them
certificate [--check] --into DIR the bus's self-signed certificate, made once
The credential is the JSON the mesh seals to the bus's own module: url, fingerprint, user, password.
`)