Compare commits
17
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
05c0396f6b | ||
|
|
c36677b9e5 | ||
|
|
b82aa1691c | ||
|
|
3fe19c9c0e | ||
|
|
424fc5a281 | ||
|
|
56a9a1b082 | ||
|
|
df94be0e60 | ||
|
|
8f5e8bcb6e | ||
|
|
ca3c3e6dfd | ||
|
|
2051d8b80b | ||
|
|
3e84581927 | ||
|
|
4d9cb0827f | ||
|
|
ac90b35b49 | ||
|
|
f2d765320a | ||
|
|
a92c08fd5b | ||
|
|
991ffcfda0 | ||
|
|
2927cb2ceb |
+18
-8
@@ -45,28 +45,38 @@ base64-encoded. The Go bundle `cmd/forge` serves the commit, and the reads an ag
|
||||
| `gitea_search_code` | r | the lines of a ref's text files matching a literal string or a regular expression, optionally under a path |
|
||||
|
||||
**Where a commit may go** is the rule `gitea_api` already holds for a file written (novox/hq ADR 0266):
|
||||
never the repository's default branch, never a branch a protection rule names (a glob rule covers by the same
|
||||
matcher as `guard.ts`). The rules are read before anything is sent; rules that cannot be read refuse the
|
||||
never the repository's default branch, never a branch a protection rule names. A rule covers a branch as the
|
||||
forge itself decides it (Gitea 1.27, novox/hq issue 539): a plain name covers that branch in any case; a glob
|
||||
(`*`, `**`, `?`, `[abc]`, `[!abc]`, `{a,b}`, `\` escaping) covers case-sensitively with `/` as separator; a glob
|
||||
the forge cannot compile covers its own text. `cmd/forge/rules.go` and `guard.ts` hold this one rule. The rules are read before anything is sent; rules that cannot be read refuse the
|
||||
commit. Nothing about merging changes: a merge still needs an independent review at the head and both
|
||||
required checks green, through `gitea_merge_pull_request`.
|
||||
|
||||
**How the commit is made.** `files` become one call to the forge's multi-file contents endpoint: a path the
|
||||
ref holds is updated, one it does not is created, and an update or delete carries the blob read just
|
||||
before, so a file someone changed meanwhile is refused by the forge rather than overwritten. A `patch` goes
|
||||
to the forge's diff endpoint, which applies it; for a small change to a large file this is far shorter than
|
||||
the file. Either way the commit is the forge's own, made as the forge's admin account, so its sha is not the
|
||||
before, so a file someone changed meanwhile is refused by the forge rather than overwritten. The blobs are read at
|
||||
the commit the branch (or, for a new branch, its base) was read at, never at its name. A `patch` goes to the
|
||||
forge's diff endpoint, which applies it; for a small change to a large file this is far shorter than the
|
||||
file. Neither endpoint takes a commit to build on, only a branch (Gitea 1.27), so the parent answered is the
|
||||
one the forge says it made the commit on; when the branch moved between the read and the commit, the answer
|
||||
says so. One commit carries at most 200 files, 4 MiB per file and 16 MiB together, or a patch of 4 MiB. Either way the commit is the forge's own, made as the forge's admin account, so its sha is not the
|
||||
local commit's: the merge check judges the head on the forge. After the write the branch is read back, and a
|
||||
branch not at the commit the forge named is a failure, never a success.
|
||||
|
||||
**Code search** has no endpoint on the forge, so the ref's archive (`/archive/{ref}.tar.gz`, at most 64 MiB)
|
||||
is read once and searched in memory; a binary file is passed over, a file over 4 MiB is named under
|
||||
`skipped`, and at most `max` matches (default 100) are answered, with `truncated` when more were found.
|
||||
`skipped`, and at most `max` matches (default 100) are answered, with `truncated` when more were found. A
|
||||
search that reaches the call's deadline stops there and answers what it found, with `stopped_at_deadline`.
|
||||
|
||||
It reaches the forge as the npm-registry bundle does (below), and one tool call takes at most 50 seconds.
|
||||
It reaches the forge through the same HTTP client as the npm-registry bundle (`internal/forgehttp`), and one
|
||||
tool call takes at most 50 seconds.
|
||||
`go test ./cmd/forge` runs it against a fake forge: the files of one commit created, updated and deleted on
|
||||
a new branch from its base; an existing branch committed on; the default branch, a protected branch and
|
||||
unreadable rules refused before anything is sent; a patch sent to the forge; a branch read back at another
|
||||
commit failing; every malformed ask refused; and each read summarised.
|
||||
commit failing; a branch or base that moved under the commit said, with the parent the forge used; a file
|
||||
changed under the commit refused; a commit over its size limits refused; protection globs matched as the
|
||||
forge matches them; a code search stopped at its deadline; every malformed ask refused; and each read
|
||||
summarised.
|
||||
|
||||
## The package registry's retention (novox/hq ADR 0251 §4, to-be 51)
|
||||
|
||||
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"unicode/utf8"
|
||||
)
|
||||
@@ -52,8 +53,14 @@ type CommittedFile struct {
|
||||
Operation string `json:"operation"`
|
||||
}
|
||||
|
||||
// maxFiles is the most files one commit carries.
|
||||
const maxFiles = 200
|
||||
// What one commit carries at most (novox/hq issue 539): files, one file's content, one patch, and every
|
||||
// file's content together, in bytes as committed (content_base64 decoded). Larger work is several commits.
|
||||
const (
|
||||
maxFiles = 200
|
||||
maxFileBytes = 4 << 20
|
||||
maxPatchBytes = 4 << 20
|
||||
maxCommitBytes = 16 << 20
|
||||
)
|
||||
|
||||
// readCommitAsk reads the tool's arguments.
|
||||
func readCommitAsk(args map[string]any) (CommitAsk, error) {
|
||||
@@ -109,8 +116,11 @@ func readCommitAsk(args map[string]any) (CommitAsk, error) {
|
||||
return a, refuse("give files or patch, not both: one commit carries one of them. Nothing was sent")
|
||||
case len(a.Files) > maxFiles:
|
||||
return a, refuse("%d files is more than the %d one commit carries. Nothing was sent", len(a.Files), maxFiles)
|
||||
case len(a.Patch) > maxPatchBytes:
|
||||
return a, refuse("the patch is %s, more than the %s one commit carries: split it into several commits. Nothing was sent", mib(len(a.Patch)), mib(maxPatchBytes))
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
total := 0
|
||||
for i, fc := range a.Files {
|
||||
if strings.HasPrefix(fc.Path, "/") {
|
||||
return a, refuse("files[%d].path %q is relative to the repository's root, without a leading slash. Nothing was sent", i, fc.Path)
|
||||
@@ -134,15 +144,35 @@ func readCommitAsk(args map[string]any) (CommitAsk, error) {
|
||||
if fc.Content != nil && !utf8.ValidString(*fc.Content) {
|
||||
return a, refuse("files[%d].content (%s) is not UTF-8 text: give it as content_base64. Nothing was sent", i, fc.Path)
|
||||
}
|
||||
if fc.ContentBase64 != nil {
|
||||
if _, err := base64.StdEncoding.DecodeString(*fc.ContentBase64); err != nil {
|
||||
size := 0
|
||||
switch {
|
||||
case fc.Content != nil:
|
||||
size = len(*fc.Content)
|
||||
case fc.ContentBase64 != nil:
|
||||
if base64.StdEncoding.DecodedLen(len(*fc.ContentBase64)) > maxFileBytes+3 {
|
||||
size = base64.StdEncoding.DecodedLen(len(*fc.ContentBase64))
|
||||
break
|
||||
}
|
||||
raw, err := base64.StdEncoding.DecodeString(*fc.ContentBase64)
|
||||
if err != nil {
|
||||
return a, refuse("files[%d].content_base64 (%s) is not base64: %v. Nothing was sent", i, fc.Path, err)
|
||||
}
|
||||
size = len(raw)
|
||||
}
|
||||
if size > maxFileBytes {
|
||||
return a, refuse("files[%d] (%s) is %s, more than the %s one file of a commit may be. Nothing was sent", i, fc.Path, mib(size), mib(maxFileBytes))
|
||||
}
|
||||
total += size
|
||||
}
|
||||
if total > maxCommitBytes {
|
||||
return a, refuse("the files are %s together, more than the %s one commit carries: split them into several commits. Nothing was sent", mib(total), mib(maxCommitBytes))
|
||||
}
|
||||
return a, nil
|
||||
}
|
||||
|
||||
// mib says a size in bytes in MiB, to a tenth.
|
||||
func mib(n int) string { return fmt.Sprintf("%.1f MiB", float64(n)/(1<<20)) }
|
||||
|
||||
// targetRefused says why a commit may not go on a branch: the default branch, or one a protection rule names.
|
||||
func targetRefused(target, defaultBranch string, rules []Protection) error {
|
||||
if target == defaultBranch {
|
||||
@@ -172,7 +202,7 @@ func commit(ctx context.Context, f *Forge, a CommitAsk) (*Committed, error) {
|
||||
return nil, err
|
||||
}
|
||||
var r Repo
|
||||
if err := f.getJSON(ctx, repo, &r); err != nil {
|
||||
if err := f.GetJSON(ctx, repo, &r); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if r.DefaultBranch == "" {
|
||||
@@ -180,7 +210,7 @@ func commit(ctx context.Context, f *Forge, a CommitAsk) (*Committed, error) {
|
||||
}
|
||||
// Read the rules, or refuse: a branch the rules could not be read for is never written (fails closed).
|
||||
var rules []Protection
|
||||
if err := f.getJSON(ctx, repo+"/branch_protections", &rules); err != nil {
|
||||
if err := f.GetJSON(ctx, repo+"/branch_protections", &rules); err != nil {
|
||||
return nil, fmt.Errorf("the protection rules of %s/%s could not be read, so the commit was not judged and nothing was sent: %w", a.Owner, a.Repo, err)
|
||||
}
|
||||
if err := targetRefused(a.Branch, r.DefaultBranch, rules); err != nil {
|
||||
@@ -192,9 +222,13 @@ func commit(ctx context.Context, f *Forge, a CommitAsk) (*Committed, error) {
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
ref := a.Branch // where the files are read and the commit goes
|
||||
// The files are judged at the commit just read, never at a branch name that could move between two
|
||||
// reads (novox/hq issue 539): a file the branch changes meanwhile is refused by the forge, whose update
|
||||
// and delete carry the blob read here.
|
||||
var ref string
|
||||
if existing != nil {
|
||||
out.Parent = existing.Commit.ID
|
||||
ref = existing.Commit.ID
|
||||
if a.Base != "" {
|
||||
out.Said = append(out.Said, fmt.Sprintf("%s exists, at %s: base %s was not used", a.Branch, short(existing.Commit.ID), a.Base))
|
||||
}
|
||||
@@ -204,7 +238,6 @@ func commit(ctx context.Context, f *Forge, a CommitAsk) (*Committed, error) {
|
||||
if out.Base == "" {
|
||||
out.Base = r.DefaultBranch
|
||||
}
|
||||
ref = out.Base
|
||||
b, err := f.branch(ctx, repo, out.Base)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
@@ -213,8 +246,6 @@ func commit(ctx context.Context, f *Forge, a CommitAsk) (*Committed, error) {
|
||||
return nil, refuse("%s does not exist, and its base %s is not a branch of %s/%s. Nothing was sent", a.Branch, out.Base, a.Owner, a.Repo)
|
||||
}
|
||||
out.Parent = b.Commit.ID
|
||||
// The files are judged at the commit just read; a file the base changes meanwhile is refused by the
|
||||
// forge, whose update and delete carry the blob read here.
|
||||
ref = b.Commit.ID
|
||||
}
|
||||
|
||||
@@ -227,14 +258,14 @@ func commit(ctx context.Context, f *Forge, a CommitAsk) (*Committed, error) {
|
||||
on["branch"] = a.Branch
|
||||
}
|
||||
|
||||
// The forge takes no commit to build on, only a branch (Gitea 1.27: neither /contents nor /diffpatch
|
||||
// carries a last commit), so the parent is read from what it made, never assumed (novox/hq issue 539).
|
||||
var made struct {
|
||||
Commit struct {
|
||||
SHA string `json:"sha"`
|
||||
} `json:"commit"`
|
||||
Commit madeCommit `json:"commit"`
|
||||
}
|
||||
if a.Patch != "" {
|
||||
on["content"] = a.Patch
|
||||
if err := f.sendJSON(ctx, http.MethodPost, repo+"/diffpatch", on, &made); err != nil {
|
||||
if err := f.SendJSON(ctx, http.MethodPost, repo+"/diffpatch", on, &made); err != nil {
|
||||
return nil, fmt.Errorf("the forge did not take the patch: %w", err)
|
||||
}
|
||||
} else {
|
||||
@@ -266,7 +297,7 @@ func commit(ctx context.Context, f *Forge, a CommitAsk) (*Committed, error) {
|
||||
out.Files = append(out.Files, CommittedFile{Path: fc.Path, Operation: op["operation"].(string)})
|
||||
}
|
||||
on["files"] = ops
|
||||
if err := f.sendJSON(ctx, http.MethodPost, repo+"/contents", on, &made); err != nil {
|
||||
if err := f.SendJSON(ctx, http.MethodPost, repo+"/contents", on, &made); err != nil {
|
||||
return nil, fmt.Errorf("the forge did not take the commit: %w", err)
|
||||
}
|
||||
}
|
||||
@@ -274,6 +305,25 @@ func commit(ctx context.Context, f *Forge, a CommitAsk) (*Committed, error) {
|
||||
return nil, fmt.Errorf("the forge answered the commit on %s without saying the commit it made: read the branch (gitea_list_branches) before trying again", a.Branch)
|
||||
}
|
||||
out.Commit = made.Commit.SHA
|
||||
read := out.Parent
|
||||
parent, err := f.parentOf(ctx, repo, made.Commit)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("the forge made %s on %s, but which commit it went on could not be read: %w", out.Commit, a.Branch, err)
|
||||
}
|
||||
out.Parent = parent
|
||||
if parent != read {
|
||||
moved := "branch " + a.Branch
|
||||
if out.Created {
|
||||
moved = "base " + out.Base
|
||||
}
|
||||
how := "each file it changes was checked against " + short(read) + " by its blob, so none of them changed in between"
|
||||
if a.Patch != "" {
|
||||
how = "the forge applied the patch there, and it refuses a patch whose context changed"
|
||||
}
|
||||
out.Said = append(out.Said, fmt.Sprintf("the %s moved from %s to %s between the read and the commit, so the "+
|
||||
"commit went on %s: %s. What else changed in between: gitea_compare %s...%s",
|
||||
moved, short(read), short(parent), short(parent), how, short(read), short(parent)))
|
||||
}
|
||||
|
||||
// Evidence over assertion: the branch is read back, and it must be at the commit the forge said.
|
||||
after, err := f.branch(ctx, repo, a.Branch)
|
||||
@@ -294,6 +344,30 @@ func commit(ctx context.Context, f *Forge, a CommitAsk) (*Committed, error) {
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// madeCommit is the commit the forge answers a write with.
|
||||
type madeCommit struct {
|
||||
SHA string `json:"sha"`
|
||||
Parents []struct {
|
||||
SHA string `json:"sha"`
|
||||
} `json:"parents"`
|
||||
}
|
||||
|
||||
// parentOf is the commit the forge made its commit on: from its answer, or read from the commit when the
|
||||
// answer does not say it.
|
||||
func (f *Forge) parentOf(ctx context.Context, repo string, made madeCommit) (string, error) {
|
||||
if len(made.Parents) == 0 {
|
||||
var read madeCommit
|
||||
if err := f.GetJSON(ctx, repo+"/git/commits/"+url.PathEscape(made.SHA)+"?stat=false&verification=false&files=false", &read); err != nil {
|
||||
return "", err
|
||||
}
|
||||
made = read
|
||||
}
|
||||
if len(made.Parents) != 1 || made.Parents[0].SHA == "" {
|
||||
return "", fmt.Errorf("the forge says %s has %d parents, not one", made.SHA, len(made.Parents))
|
||||
}
|
||||
return made.Parents[0].SHA, nil
|
||||
}
|
||||
|
||||
// blobAt is the blob sha of a file at a ref, "" when there is no file there.
|
||||
func (f *Forge) blobAt(ctx context.Context, repo, path, ref string) (string, error) {
|
||||
seg, err := segments("file path", path)
|
||||
@@ -305,7 +379,7 @@ func (f *Forge) blobAt(ctx context.Context, repo, path, ref string) (string, err
|
||||
SHA string `json:"sha"`
|
||||
}
|
||||
var list []any
|
||||
body, err := f.do(ctx, http.MethodGet, repo+"/contents/"+seg+"?ref="+queryEscape(ref), nil, maxBody)
|
||||
body, err := f.Do(ctx, http.MethodGet, repo+"/contents/"+seg+"?ref="+queryEscape(ref), nil, maxBody)
|
||||
if err != nil {
|
||||
if errors.Is(err, errNotFound) {
|
||||
return "", nil
|
||||
|
||||
@@ -1,156 +1,43 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea/internal/forgehttp"
|
||||
)
|
||||
|
||||
// The forge as this bundle reaches it: its own HTTP interface on the machine, as the forge's admin account,
|
||||
// which the vault delivered into a file this module owns (novox/hq ADR 0086), by basic authentication — the
|
||||
// same way the npm-registry bundle beside it does. The password is read when a call is made and never logged,
|
||||
// answered or put in an error.
|
||||
// The forge as this bundle reaches it: its own HTTP interface on the machine, as the forge's admin account
|
||||
// (internal/forgehttp, shared with the npm-registry bundle beside it).
|
||||
|
||||
// Forge is the forge's HTTP interface.
|
||||
type Forge struct {
|
||||
URL string // e.g. http://127.0.0.1:3000
|
||||
User string
|
||||
PasswordFile string
|
||||
HTTP *http.Client
|
||||
forgehttp.Client
|
||||
}
|
||||
|
||||
// ForgeFromEnv is the forge as the manifest hands it to this bundle.
|
||||
func ForgeFromEnv() (*Forge, error) {
|
||||
f := &Forge{
|
||||
URL: strings.TrimRight(os.Getenv("MESH_GITEA_URL"), "/"),
|
||||
User: os.Getenv("MESH_GITEA_ADMIN_USER"),
|
||||
PasswordFile: os.Getenv("MESH_GITEA_ADMIN_PASSWORD_FILE"),
|
||||
}
|
||||
var missing []string
|
||||
for _, kv := range [][2]string{{"MESH_GITEA_URL", f.URL}, {"MESH_GITEA_ADMIN_USER", f.User},
|
||||
{"MESH_GITEA_ADMIN_PASSWORD_FILE", f.PasswordFile}} {
|
||||
if kv[1] == "" {
|
||||
missing = append(missing, kv[0])
|
||||
}
|
||||
}
|
||||
c, missing := forgehttp.FromEnv()
|
||||
if len(missing) > 0 {
|
||||
return nil, fmt.Errorf("the forge cannot be reached: %s not set by the manifest", strings.Join(missing, ", "))
|
||||
}
|
||||
return f, nil
|
||||
c.Timeout = deadline
|
||||
return &Forge{Client: c}, nil
|
||||
}
|
||||
|
||||
// errNotFound is the forge answering 404.
|
||||
var errNotFound = errors.New("not found")
|
||||
|
||||
// statusError is an answer the forge gave that is not a success, said without the request's credential.
|
||||
type statusError struct {
|
||||
method, path string
|
||||
status int
|
||||
said string
|
||||
}
|
||||
|
||||
func (e *statusError) Error() string {
|
||||
return fmt.Sprintf("the forge answered %d to %s %s: %s", e.status, e.method, e.path, e.said)
|
||||
}
|
||||
|
||||
func (f *Forge) client() *http.Client {
|
||||
if f.HTTP != nil {
|
||||
return f.HTTP
|
||||
}
|
||||
return &http.Client{Timeout: 50 * time.Second}
|
||||
}
|
||||
|
||||
func (f *Forge) password() (string, error) {
|
||||
raw, err := os.ReadFile(f.PasswordFile)
|
||||
if err != nil {
|
||||
// The path is the manifest's, not a secret; the content never appears.
|
||||
return "", fmt.Errorf("the forge's admin password cannot be read from its file: %w", err)
|
||||
}
|
||||
return strings.TrimSpace(string(raw)), nil
|
||||
}
|
||||
var errNotFound = forgehttp.ErrNotFound
|
||||
|
||||
// maxBody is the largest JSON answer read from the forge; maxArchive the largest archive code search reads.
|
||||
const (
|
||||
maxBody = 32 << 20
|
||||
maxBody = forgehttp.MaxBody
|
||||
maxArchive = 64 << 20
|
||||
)
|
||||
|
||||
// do sends one request and answers the body, read up to limit; 404 is errNotFound. A body the limit cut is
|
||||
// refused, never read as if it were whole.
|
||||
func (f *Forge) do(ctx context.Context, method, path string, payload any, limit int64) ([]byte, error) {
|
||||
password, err := f.password()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var body io.Reader
|
||||
if payload != nil {
|
||||
raw, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
body = bytes.NewReader(raw)
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, method, f.URL+path, body)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
req.SetBasicAuth(f.User, password)
|
||||
req.Header.Set("Accept", "application/json")
|
||||
if payload != nil {
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
}
|
||||
res, err := f.client().Do(req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("the forge did not answer %s %s: %w", method, path, err)
|
||||
}
|
||||
defer res.Body.Close()
|
||||
got, err := io.ReadAll(io.LimitReader(res.Body, limit+1))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if int64(len(got)) > limit {
|
||||
return nil, fmt.Errorf("the forge's answer to %s %s is larger than %d MiB and was not read whole", method, path, limit>>20)
|
||||
}
|
||||
switch {
|
||||
case res.StatusCode == http.StatusNotFound:
|
||||
return nil, fmt.Errorf("%s %s: %w", method, path, errNotFound)
|
||||
case res.StatusCode >= 300:
|
||||
said := strings.TrimSpace(string(got))
|
||||
if len(said) > 300 {
|
||||
said = said[:300] + "…"
|
||||
}
|
||||
return nil, &statusError{method, path, res.StatusCode, said}
|
||||
}
|
||||
return got, nil
|
||||
}
|
||||
|
||||
func (f *Forge) getJSON(ctx context.Context, path string, into any) error {
|
||||
return f.sendJSON(ctx, http.MethodGet, path, nil, into)
|
||||
}
|
||||
|
||||
func (f *Forge) sendJSON(ctx context.Context, method, path string, payload, into any) error {
|
||||
body, err := f.do(ctx, method, path, payload, maxBody)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if into == nil {
|
||||
return nil
|
||||
}
|
||||
if err := json.Unmarshal(body, into); err != nil {
|
||||
return fmt.Errorf("the forge's answer to %s %s is not readable: %w", method, path, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Refused is a refusal of policy or of an argument, in the words the caller sees.
|
||||
type Refused struct{ msg string }
|
||||
|
||||
@@ -204,36 +91,6 @@ func (p Protection) name() string {
|
||||
return p.BranchName
|
||||
}
|
||||
|
||||
// ruleCovers is whether a protection rule's name — a branch, or a glob of them (`*`, `**`, `?`) — covers a
|
||||
// branch: the TypeScript bundle's ruleCovers (guard.ts), which gitea_api's file writes are judged by.
|
||||
func ruleCovers(rule, branch string) bool {
|
||||
if rule == branch {
|
||||
return true
|
||||
}
|
||||
if !strings.ContainsAny(rule, "*?[") {
|
||||
return false
|
||||
}
|
||||
var re strings.Builder
|
||||
re.WriteString("^")
|
||||
for i := 0; i < len(rule); i++ {
|
||||
c := rule[i]
|
||||
switch {
|
||||
case c == '*' && i+1 < len(rule) && rule[i+1] == '*':
|
||||
re.WriteString(".*")
|
||||
i++
|
||||
case c == '*':
|
||||
re.WriteString("[^/]*")
|
||||
case c == '?':
|
||||
re.WriteString("[^/]")
|
||||
default:
|
||||
re.WriteString(regexp.QuoteMeta(string(c)))
|
||||
}
|
||||
}
|
||||
re.WriteString("$")
|
||||
ok, err := regexp.MatchString(re.String(), branch)
|
||||
return err == nil && ok
|
||||
}
|
||||
|
||||
// Branch is one branch with the commit it points at.
|
||||
type Branch struct {
|
||||
Name string `json:"name"`
|
||||
@@ -249,7 +106,7 @@ func (f *Forge) branch(ctx context.Context, repo, name string) (*Branch, error)
|
||||
return nil, err
|
||||
}
|
||||
var b Branch
|
||||
if err := f.getJSON(ctx, repo+"/branches/"+seg, &b); err != nil {
|
||||
if err := f.GetJSON(ctx, repo+"/branches/"+seg, &b); err != nil {
|
||||
if errors.Is(err, errNotFound) {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
@@ -13,9 +13,12 @@ import (
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"gitea/internal/forgehttp"
|
||||
)
|
||||
|
||||
// fakeForge is the forge's HTTP interface as far as these tools use it.
|
||||
@@ -37,6 +40,16 @@ type fakeForge struct {
|
||||
comments map[string][]map[string]any
|
||||
commits []map[string]any
|
||||
dir []map[string]any
|
||||
// moves: a branch moved by someone else right after it is first read — to a commit, with its tree.
|
||||
moves map[string]move
|
||||
noParents bool // the forge's answer to a write does not say the parents
|
||||
parentsBroken bool // and GET git/commits fails
|
||||
parents map[string]string // commit -> parent, as GET git/commits says it
|
||||
}
|
||||
|
||||
type move struct {
|
||||
to string
|
||||
tree map[string]string
|
||||
}
|
||||
|
||||
func newFake(t *testing.T) *fakeForge {
|
||||
@@ -62,7 +75,7 @@ func (f *fakeForge) serve() *Forge {
|
||||
}
|
||||
srv := httptest.NewServer(http.HandlerFunc(f.handle))
|
||||
f.t.Cleanup(srv.Close)
|
||||
return &Forge{URL: srv.URL, User: "mesh-admin", PasswordFile: pw}
|
||||
return &Forge{Client: forgehttp.Client{URL: srv.URL, User: "mesh-admin", PasswordFile: pw}}
|
||||
}
|
||||
|
||||
func (f *fakeForge) handle(w http.ResponseWriter, r *http.Request) {
|
||||
@@ -92,6 +105,11 @@ func (f *fakeForge) handle(w http.ResponseWriter, r *http.Request) {
|
||||
return
|
||||
}
|
||||
reply(map[string]any{"name": name, "commit": map[string]any{"id": c}})
|
||||
if m, ok := f.moves[name]; ok {
|
||||
delete(f.moves, name)
|
||||
f.trees[m.to] = m.tree
|
||||
f.branches[name] = m.to
|
||||
}
|
||||
case r.Method == "GET" && (path == repo+"/contents" || path == repo+"/contents/docs"):
|
||||
reply(f.dir)
|
||||
case r.Method == "GET" && strings.HasPrefix(path, repo+"/contents/"):
|
||||
@@ -148,19 +166,28 @@ func (f *fakeForge) handle(w http.ResponseWriter, r *http.Request) {
|
||||
if !f.staleRead {
|
||||
f.branches[target] = made
|
||||
}
|
||||
reply(map[string]any{"commit": map[string]any{"sha": made}})
|
||||
reply(map[string]any{"commit": f.made(made, from)})
|
||||
case r.Method == "POST" && path == repo+"/diffpatch":
|
||||
var body map[string]any
|
||||
_ = json.NewDecoder(r.Body).Decode(&body)
|
||||
f.patches = append(f.patches, body)
|
||||
f.next++
|
||||
made := fmt.Sprintf("p%d", f.next)
|
||||
from := f.branches[body["branch"].(string)]
|
||||
target := body["branch"].(string)
|
||||
if nb, ok := body["new_branch"].(string); ok && nb != "" {
|
||||
target = nb
|
||||
}
|
||||
f.branches[target] = made
|
||||
reply(map[string]any{"commit": map[string]any{"sha": made}})
|
||||
reply(map[string]any{"commit": f.made(made, from)})
|
||||
case r.Method == "GET" && strings.HasPrefix(path, repo+"/git/commits/"):
|
||||
sha := strings.TrimPrefix(path, repo+"/git/commits/")
|
||||
p, ok := f.parents[sha]
|
||||
if !ok || f.parentsBroken {
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
reply(map[string]any{"sha": sha, "parents": []any{map[string]any{"sha": p}}})
|
||||
case r.Method == "GET" && path == repo+"/commits":
|
||||
f.writes = append(f.writes, map[string]any{"query": r.URL.RawQuery})
|
||||
reply(f.commits)
|
||||
@@ -180,6 +207,18 @@ func (f *fakeForge) handle(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
}
|
||||
|
||||
// made is the forge's answer about a commit it made on a parent, with the parents unless noParents.
|
||||
func (f *fakeForge) made(sha, parent string) map[string]any {
|
||||
if f.parents == nil {
|
||||
f.parents = map[string]string{}
|
||||
}
|
||||
f.parents[sha] = parent
|
||||
if f.noParents {
|
||||
return map[string]any{"sha": sha}
|
||||
}
|
||||
return map[string]any{"sha": sha, "parents": []any{map[string]any{"sha": parent}}}
|
||||
}
|
||||
|
||||
func filesArg(files ...map[string]any) []any {
|
||||
out := make([]any, len(files))
|
||||
for i, f := range files {
|
||||
@@ -359,17 +398,27 @@ func TestACommitAskThatCannotBeReadIsRefused(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRuleCoversIsTheGuardsRule(t *testing.T) {
|
||||
for _, c := range []struct {
|
||||
rule, branch string
|
||||
want bool
|
||||
}{
|
||||
{"main", "main", true}, {"main", "mainx", false}, {"release/*", "release/1.0", true},
|
||||
{"release/*", "release/a/b", false}, {"release/**", "release/a/b", true}, {"v?", "v1", true},
|
||||
{"a.b", "axb", false},
|
||||
} {
|
||||
if got := ruleCovers(c.rule, c.branch); got != c.want {
|
||||
t.Errorf("%s covers %s: %v", c.rule, c.branch, got)
|
||||
// The forge's own answers (Gitea 1.27's Match), one table for this bundle and guard.ts alike (novox/hq issue 539).
|
||||
// `forge` is the forge's answer, which this port must give; `guard` is where guard.ts deliberately covers more.
|
||||
func TestRuleCoversAnswersAsTheForgeDoes(t *testing.T) {
|
||||
raw, err := os.ReadFile("../../test/rule-cases.json")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var cases []struct {
|
||||
Rule string `json:"rule"`
|
||||
Branch string `json:"branch"`
|
||||
Forge bool `json:"forge"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &cases); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(cases) < 100 {
|
||||
t.Fatalf("only %d cases", len(cases))
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := ruleCovers(c.Rule, c.Branch); got != c.Forge {
|
||||
t.Errorf("%q covers %q: %v, the forge says %v", c.Rule, c.Branch, got, c.Forge)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -581,3 +630,115 @@ func TestToolsAreTheOnesTheReadmeNamesAndTheManifestBuilds(t *testing.T) {
|
||||
t.Error("the manifest builds no cmd/forge bundle")
|
||||
}
|
||||
}
|
||||
|
||||
// novox/hq issue 539: the forge takes a branch, never a commit to build on, so the answer's parent is the one
|
||||
// the forge made the commit on, and a branch that moves between the read and the commit is said.
|
||||
func TestAnExistingBranchMovedUnderTheCommitIsSaidAndItsParentIsTheOneTheForgeUsed(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.moves = map[string]move{"fix/1-old": {to: "c2", tree: map[string]string{"README.md": "hello\n", "docs/a.md": "a on fix\n", "other.md": "x\n"}}}
|
||||
got, err := commit(context.Background(), f.serve(), ask(t, map[string]any{"branch": "fix/1-old",
|
||||
"files": filesArg(map[string]any{"path": "docs/a.md", "content": "a fixed\n"})}))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got.Parent != "c2" || f.branches["fix/1-old"] != got.Commit {
|
||||
t.Fatalf("parent %s, want c2 (the commit the forge built on): %+v", got.Parent, got)
|
||||
}
|
||||
said := strings.Join(got.Said, " ")
|
||||
if !strings.Contains(said, "moved from c1 to c2") || !strings.Contains(said, "after c2") {
|
||||
t.Fatalf("the move was not said: %v", got.Said)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnExistingBranchsFilesAreJudgedAtTheCommitReadNotAtWhereTheBranchMovedTo(t *testing.T) {
|
||||
f := newFake(t)
|
||||
// Someone else changes docs/a.md on the branch between the read and the commit: that change is not
|
||||
// overwritten by a blob read at the branch's new head.
|
||||
f.moves = map[string]move{"fix/1-old": {to: "c2", tree: map[string]string{"README.md": "hello\n", "docs/a.md": "theirs\n"}}}
|
||||
_, err := commit(context.Background(), f.serve(), ask(t, map[string]any{"branch": "fix/1-old",
|
||||
"files": filesArg(map[string]any{"path": "docs/a.md", "content": "mine\n"})}))
|
||||
if err == nil || !strings.Contains(err.Error(), "sha does not match") {
|
||||
t.Fatalf("a file changed under the commit was overwritten: %v", err)
|
||||
}
|
||||
if f.trees["c2"]["docs/a.md"] != "theirs\n" || f.branches["fix/1-old"] != "c2" {
|
||||
t.Fatalf("the branch was changed: %v", f.branches)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAPatchOnABaseThatMovedNamesTheCommitTheForgeAppliedItTo(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.moves = map[string]move{"main": {to: "c9", tree: map[string]string{"README.md": "hello\n"}}}
|
||||
patch := "diff --git a/README.md b/README.md\n--- a/README.md\n+++ b/README.md\n@@ -1 +1 @@\n-hello\n+hi\n"
|
||||
got, err := commit(context.Background(), f.serve(), ask(t, map[string]any{"branch": "fix/6", "patch": patch}))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got.Parent != "c9" || !strings.Contains(strings.Join(got.Said, " "), "base main moved from c0 to c9") {
|
||||
t.Fatalf("%+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAParentTheAnswerDoesNotSayIsReadFromTheCommit(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.noParents = true
|
||||
got, err := commit(context.Background(), f.serve(), ask(t, map[string]any{"branch": "fix/7",
|
||||
"files": filesArg(map[string]any{"path": "x", "content": "x"})}))
|
||||
if err != nil || got.Parent != "c0" {
|
||||
t.Fatalf("%v %+v", err, got)
|
||||
}
|
||||
// A parent that cannot be read is a failure, said with the commit that was made.
|
||||
f = newFake(t)
|
||||
f.noParents, f.parentsBroken = true, true
|
||||
_, err = commit(context.Background(), f.serve(), ask(t, map[string]any{"branch": "fix/8",
|
||||
"files": filesArg(map[string]any{"path": "x", "content": "x"})}))
|
||||
if err == nil || !strings.Contains(err.Error(), "the forge made n1 on fix/8, but which commit it went on could not be read") {
|
||||
t.Fatalf("%v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestACommitLargerThanItsLimitsIsRefusedBeforeAnythingIsSent(t *testing.T) {
|
||||
big := strings.Repeat("x", maxFileBytes+1)
|
||||
quarter := strings.Repeat("y", maxFileBytes)
|
||||
many := filesArg()
|
||||
for i := 0; i < maxCommitBytes/maxFileBytes+1; i++ {
|
||||
many = append(many, map[string]any{"path": fmt.Sprintf("f%d", i), "content": quarter})
|
||||
}
|
||||
for name, args := range map[string]map[string]any{
|
||||
"one file": {"files": filesArg(map[string]any{"path": "x", "content": big})},
|
||||
"one file b64": {"files": filesArg(map[string]any{"path": "x", "content_base64": base64.StdEncoding.EncodeToString([]byte(big))})},
|
||||
"the patch": {"patch": "diff --git a/x b/x\n" + big},
|
||||
"all files": {"files": many},
|
||||
} {
|
||||
full := map[string]any{"owner": "acme", "repo": "app", "message": "m", "branch": "b"}
|
||||
for k, v := range args {
|
||||
full[k] = v
|
||||
}
|
||||
_, err := readCommitAsk(full)
|
||||
if err == nil || !strings.Contains(err.Error(), "MiB") || !strings.Contains(err.Error(), "Nothing was sent") {
|
||||
t.Errorf("%s: %v", name, err)
|
||||
}
|
||||
}
|
||||
// At the limit is taken.
|
||||
if _, err := readCommitAsk(map[string]any{"owner": "acme", "repo": "app", "message": "m", "branch": "b",
|
||||
"files": filesArg(map[string]any{"path": "x", "content_base64": base64.StdEncoding.EncodeToString([]byte(quarter))})}); err != nil {
|
||||
t.Errorf("a file at the limit was refused: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCodeSearchStopsAtItsDeadlineAndSaysSo(t *testing.T) {
|
||||
gz := archive(t, map[string]string{"a.md": "forge\n", "b.md": "forge\n"})
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
out := &Found{Matches: []Match{}}
|
||||
if err := searchArchive(ctx, gz, "", regexp.MustCompile("forge"), 100, out); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !out.Stopped || out.Files != 0 {
|
||||
t.Fatalf("a search past its deadline went on: %+v", out)
|
||||
}
|
||||
// Within its deadline, the same archive is searched whole.
|
||||
out = &Found{Matches: []Match{}}
|
||||
if err := searchArchive(context.Background(), gz, "", regexp.MustCompile("forge"), 100, out); err != nil || out.Stopped || len(out.Matches) != 2 {
|
||||
t.Fatalf("%v %+v", err, out)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -53,7 +53,9 @@ func Tools(forge func() (*Forge, error)) []stdio.Tool {
|
||||
"independent review approved and whose required checks passed (gitea_merge_pull_request). Carries " +
|
||||
"either files — each {path, content (text) | content_base64 | delete: true}, created or updated as " +
|
||||
"the branch holds them — or patch, a unified diff as git diff writes it, which the forge applies. " +
|
||||
"Answers the commit it made and its parent, read back from the branch. Replaces git commit and git " +
|
||||
"At most 200 files, 4 MiB each and 16 MiB together, or a 4 MiB patch. Answers the commit it made and " +
|
||||
"the parent the forge made it on, read back from the branch, and says when the branch moved under " +
|
||||
"it. Replaces git commit and git " +
|
||||
"push of a feature branch. (a)",
|
||||
Input: map[string]any{
|
||||
"type": "object",
|
||||
|
||||
@@ -72,7 +72,7 @@ func history(ctx context.Context, f *Forge, owner, name, ref, path string, limit
|
||||
q.Set("verification", "false")
|
||||
q.Set("files", "false")
|
||||
var got []forgeCommit
|
||||
if err := f.getJSON(ctx, repo+"/commits?"+q.Encode(), &got); err != nil {
|
||||
if err := f.GetJSON(ctx, repo+"/commits?"+q.Encode(), &got); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
h := &History{Repository: owner + "/" + name, Ref: ref, Path: path, Page: page, Commits: []CommitLine{}, More: len(got) >= limit}
|
||||
@@ -121,7 +121,7 @@ func compare(ctx context.Context, f *Forge, owner, name, base, head string) (*Co
|
||||
Total int `json:"total_commits"`
|
||||
Commits []forgeCommit `json:"commits"`
|
||||
}
|
||||
if err := f.getJSON(ctx, repo+"/compare/"+b+"..."+h, &got); err != nil {
|
||||
if err := f.GetJSON(ctx, repo+"/compare/"+b+"..."+h, &got); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
c := &Comparison{Repository: owner + "/" + name, Base: base, Head: head, Total: got.Total, Commits: []CommitLine{}, Files: []FileLine{}}
|
||||
@@ -195,7 +195,7 @@ func reviews(ctx context.Context, f *Forge, owner, name string, number int) (*Re
|
||||
SHA string `json:"sha"`
|
||||
} `json:"head"`
|
||||
}
|
||||
if err := f.getJSON(ctx, pr, &pull); err != nil {
|
||||
if err := f.GetJSON(ctx, pr, &pull); err != nil {
|
||||
if errors.Is(err, errNotFound) {
|
||||
return nil, refuse("%s/%s has no pull request %d", owner, name, number)
|
||||
}
|
||||
@@ -218,7 +218,7 @@ func reviews(ctx context.Context, f *Forge, owner, name string, number int) (*Re
|
||||
var all []forgeReview
|
||||
for p := 1; ; p++ {
|
||||
var batch []forgeReview
|
||||
if err := f.getJSON(ctx, fmt.Sprintf("%s/reviews?limit=%d&page=%d", pr, page, p), &batch); err != nil {
|
||||
if err := f.GetJSON(ctx, fmt.Sprintf("%s/reviews?limit=%d&page=%d", pr, page, p), &batch); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
all = append(all, batch...)
|
||||
@@ -240,7 +240,7 @@ func reviews(ctx context.Context, f *Forge, owner, name string, number int) (*Re
|
||||
CommitID string `json:"commit_id"`
|
||||
Resolver *forgeUser `json:"resolver"`
|
||||
}
|
||||
if err := f.getJSON(ctx, fmt.Sprintf("%s/reviews/%d/comments", pr, r.ID), &cs); err != nil {
|
||||
if err := f.GetJSON(ctx, fmt.Sprintf("%s/reviews/%d/comments", pr, r.ID), &cs); err != nil {
|
||||
return nil, fmt.Errorf("review %d's line comments could not be read: %w", r.ID, err)
|
||||
}
|
||||
for _, c := range cs {
|
||||
@@ -292,7 +292,7 @@ func listDir(ctx context.Context, f *Forge, owner, name, path, ref string) (*Lis
|
||||
if ref != "" {
|
||||
at += "?ref=" + queryEscape(ref)
|
||||
}
|
||||
body, err := f.do(ctx, "GET", at, nil, maxBody)
|
||||
body, err := f.Do(ctx, "GET", at, nil, maxBody)
|
||||
if err != nil {
|
||||
if errors.Is(err, errNotFound) {
|
||||
return nil, refuse("%s/%s has no %q at %s", owner, name, path, orDefault(ref))
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"regexp"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// ruleCovers is whether a protection rule's name covers a branch, as the forge itself decides it (Gitea 1.27,
|
||||
// models/git/protected_branch.go Match and modules/glob, novox/hq issue 539): a name without a glob character
|
||||
// covers the branch of that name in any case; any other name is a glob with `/` as its separator — `*` any
|
||||
// run within a segment, `**` any run across them, `?` one character of a segment, `[abc]`, `[a-z]` and `[!abc]`
|
||||
// one character, `{a,b}` either, `\` the next character itself — matched case-sensitively; a glob the forge
|
||||
// cannot compile covers its own text exactly. The TypeScript bundle's ruleCovers (guard.ts) is the same rule.
|
||||
func ruleCovers(rule, branch string) bool {
|
||||
if !strings.ContainsAny(rule, `*?\[]{}`) {
|
||||
return strings.EqualFold(rule, branch)
|
||||
}
|
||||
re, err := globRegexp(rule)
|
||||
if err != nil {
|
||||
return rule == branch
|
||||
}
|
||||
return re.MatchString(branch)
|
||||
}
|
||||
|
||||
// globRegexp is the forge's glob as a regular expression, compiled as modules/glob compiles it with `/` as
|
||||
// the separator.
|
||||
func globRegexp(glob string) (*regexp.Regexp, error) {
|
||||
g := []rune(glob)
|
||||
var b strings.Builder
|
||||
pos := 0
|
||||
var compile func(sub bool) error
|
||||
chars := func() error {
|
||||
b.WriteByte('[')
|
||||
if pos < len(g) && g[pos] == '!' {
|
||||
pos++
|
||||
b.WriteByte('^')
|
||||
}
|
||||
for pos < len(g) {
|
||||
c := g[pos]
|
||||
pos++
|
||||
switch {
|
||||
case c == ']':
|
||||
b.WriteByte(']')
|
||||
return nil
|
||||
case c == '\\':
|
||||
if pos >= len(g) {
|
||||
return errors.New("unterminated character class escape")
|
||||
}
|
||||
b.WriteByte('\\')
|
||||
b.WriteRune(g[pos])
|
||||
pos++
|
||||
default:
|
||||
b.WriteRune(c)
|
||||
}
|
||||
}
|
||||
return errors.New("unterminated character class")
|
||||
}
|
||||
compile = func(sub bool) error {
|
||||
for pos < len(g) {
|
||||
c := g[pos]
|
||||
pos++
|
||||
if sub && c == '}' {
|
||||
b.WriteByte(')')
|
||||
return nil
|
||||
}
|
||||
switch c {
|
||||
case '*':
|
||||
if pos < len(g) && g[pos] == '*' {
|
||||
pos++
|
||||
b.WriteString(".*")
|
||||
} else {
|
||||
b.WriteString("[^/]*")
|
||||
}
|
||||
case '?':
|
||||
b.WriteString("[^/]")
|
||||
case '+':
|
||||
b.WriteString(`\+`)
|
||||
case '[':
|
||||
if err := chars(); err != nil {
|
||||
return err
|
||||
}
|
||||
case '{':
|
||||
b.WriteByte('(')
|
||||
if err := compile(true); err != nil {
|
||||
return err
|
||||
}
|
||||
case ',':
|
||||
if sub {
|
||||
b.WriteByte('|')
|
||||
} else {
|
||||
b.WriteByte(',')
|
||||
}
|
||||
case '\\':
|
||||
if pos >= len(g) {
|
||||
return errors.New("no character to escape")
|
||||
}
|
||||
b.WriteByte('\\')
|
||||
b.WriteRune(g[pos])
|
||||
pos++
|
||||
case '.', '^', '$', '(', ')', '|':
|
||||
b.WriteByte('\\')
|
||||
b.WriteRune(c)
|
||||
default:
|
||||
b.WriteRune(c)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
b.WriteByte('^')
|
||||
if err := compile(false); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
b.WriteByte('$')
|
||||
return regexp.Compile(b.String())
|
||||
}
|
||||
@@ -32,6 +32,7 @@ type Found struct {
|
||||
Files int `json:"files_searched"`
|
||||
Skipped []string `json:"skipped,omitempty"`
|
||||
Truncated bool `json:"truncated"`
|
||||
Stopped bool `json:"stopped_at_deadline,omitempty"`
|
||||
Said []string `json:"said"`
|
||||
}
|
||||
|
||||
@@ -69,7 +70,7 @@ func searchCode(ctx context.Context, f *Forge, a SearchAsk) (*Found, error) {
|
||||
ref := a.Ref
|
||||
if ref == "" {
|
||||
var r Repo
|
||||
if err := f.getJSON(ctx, repo, &r); err != nil {
|
||||
if err := f.GetJSON(ctx, repo, &r); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if r.Empty || r.DefaultBranch == "" {
|
||||
@@ -87,7 +88,7 @@ func searchCode(ctx context.Context, f *Forge, a SearchAsk) (*Found, error) {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
body, err := f.do(ctx, "GET", repo+"/archive/"+seg+".tar.gz", nil, maxArchive)
|
||||
body, err := f.Do(ctx, "GET", repo+"/archive/"+seg+".tar.gz", nil, maxArchive)
|
||||
if err != nil {
|
||||
if errors.Is(err, errNotFound) {
|
||||
return nil, refuse("%s/%s has no ref %s", a.Owner, a.Repo, ref)
|
||||
@@ -95,11 +96,15 @@ func searchCode(ctx context.Context, f *Forge, a SearchAsk) (*Found, error) {
|
||||
return nil, err
|
||||
}
|
||||
out := &Found{Repository: a.Owner + "/" + a.Repo, Ref: ref, Query: a.Query, Matches: []Match{}}
|
||||
if err := searchArchive(body, prefix, re, a.Max, out); err != nil {
|
||||
if err := searchArchive(ctx, body, prefix, re, a.Max, out); err != nil {
|
||||
return nil, fmt.Errorf("the archive of %s at %s could not be read: %w", out.Repository, ref, err)
|
||||
}
|
||||
said := fmt.Sprintf("%d matches in %d files searched", len(out.Matches), out.Files)
|
||||
if out.Truncated {
|
||||
switch {
|
||||
case out.Stopped:
|
||||
said = fmt.Sprintf("the search stopped at its deadline of %s after %d files, with %d matches: the rest of the "+
|
||||
"archive was not searched. Narrow with path", deadline, out.Files, len(out.Matches))
|
||||
case out.Truncated:
|
||||
said = fmt.Sprintf("the first %d matches, and more were found: narrow with path or the query", len(out.Matches))
|
||||
}
|
||||
out.Said = append(out.Said, said)
|
||||
@@ -110,18 +115,32 @@ func searchCode(ctx context.Context, f *Forge, a SearchAsk) (*Found, error) {
|
||||
}
|
||||
|
||||
// searchArchive reads a gzipped tar the forge made — every path under one top directory — and collects the
|
||||
// lines that match, from text files under the prefix.
|
||||
func searchArchive(gz []byte, prefix string, re *regexp.Regexp, max int, out *Found) error {
|
||||
// lines that match, from text files under the prefix. It stops at the call's deadline (novox/hq issue 539):
|
||||
// what it found by then is kept, and out.Stopped says the rest was not searched.
|
||||
func searchArchive(ctx context.Context, gz []byte, prefix string, re *regexp.Regexp, max int, out *Found) error {
|
||||
zr, err := gzip.NewReader(bytes.NewReader(gz))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
tr := tar.NewReader(zr)
|
||||
tr := tar.NewReader(ctxReader{ctx, zr})
|
||||
stopped := func(err error) bool {
|
||||
if ctx.Err() != nil && (err == nil || errors.Is(err, ctx.Err())) {
|
||||
out.Stopped = true
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
for {
|
||||
if stopped(nil) {
|
||||
return nil
|
||||
}
|
||||
h, err := tr.Next()
|
||||
if err == io.EOF {
|
||||
return nil
|
||||
}
|
||||
if stopped(err) {
|
||||
return nil
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -140,6 +159,9 @@ func searchArchive(gz []byte, prefix string, re *regexp.Regexp, max int, out *Fo
|
||||
continue
|
||||
}
|
||||
content, err := io.ReadAll(io.LimitReader(tr, maxFileSearched+1))
|
||||
if stopped(err) {
|
||||
return nil
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -169,3 +191,16 @@ func searchArchive(gz []byte, prefix string, re *regexp.Regexp, max int, out *Fo
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ctxReader is a reader that ends when its context does: decompressing is the long part of a search.
|
||||
type ctxReader struct {
|
||||
ctx context.Context
|
||||
r io.Reader
|
||||
}
|
||||
|
||||
func (c ctxReader) Read(p []byte) (int, error) {
|
||||
if err := c.ctx.Err(); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return c.r.Read(p)
|
||||
}
|
||||
|
||||
@@ -2,45 +2,33 @@ package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea/internal/forgehttp"
|
||||
)
|
||||
|
||||
// The forge as this bundle reaches it: its own HTTP interface on the machine, as the forge's admin
|
||||
// account, which the vault delivered into a file this module owns (novox/hq ADR 0086). Basic
|
||||
// authentication, so this bundle mints nothing and keeps nothing: the TypeScript bundle's token is its
|
||||
// own. The password is read when a call is made and never logged, answered or put in an error.
|
||||
// The forge as this bundle reaches it: its own HTTP interface on the machine, as the forge's admin account
|
||||
// (internal/forgehttp, shared with the forge bundle beside it). Basic authentication, so this bundle mints
|
||||
// nothing and keeps nothing: the TypeScript bundle's token is its own.
|
||||
|
||||
// Forge is the forge's HTTP interface.
|
||||
type Forge struct {
|
||||
URL string // e.g. http://127.0.0.1:3000
|
||||
User string
|
||||
PasswordFile string
|
||||
Owner string // the package registry's owner, as the seat serves it
|
||||
HTTP *http.Client
|
||||
forgehttp.Client
|
||||
Owner string // the package registry's owner, as the seat serves it
|
||||
}
|
||||
|
||||
// ForgeFromEnv is the forge as the manifest hands it to this bundle.
|
||||
func ForgeFromEnv() (*Forge, error) {
|
||||
f := &Forge{
|
||||
URL: strings.TrimRight(os.Getenv("MESH_GITEA_URL"), "/"),
|
||||
User: os.Getenv("MESH_GITEA_ADMIN_USER"),
|
||||
PasswordFile: os.Getenv("MESH_GITEA_ADMIN_PASSWORD_FILE"),
|
||||
Owner: os.Getenv("MESH_NPM_OWNER"),
|
||||
}
|
||||
var missing []string
|
||||
for name, v := range map[string]string{"MESH_GITEA_URL": f.URL, "MESH_GITEA_ADMIN_USER": f.User,
|
||||
"MESH_GITEA_ADMIN_PASSWORD_FILE": f.PasswordFile, "MESH_NPM_OWNER": f.Owner} {
|
||||
if v == "" {
|
||||
missing = append(missing, name)
|
||||
}
|
||||
c, missing := forgehttp.FromEnv()
|
||||
f := &Forge{Client: c, Owner: os.Getenv("MESH_NPM_OWNER")}
|
||||
if f.Owner == "" {
|
||||
missing = append(missing, "MESH_NPM_OWNER")
|
||||
}
|
||||
if len(missing) > 0 {
|
||||
return nil, fmt.Errorf("the forge cannot be reached: %s not set by the manifest", strings.Join(missing, ", "))
|
||||
@@ -49,86 +37,14 @@ func ForgeFromEnv() (*Forge, error) {
|
||||
}
|
||||
|
||||
// errNotFound is the forge answering 404.
|
||||
var errNotFound = errors.New("not found")
|
||||
|
||||
// statusError is an answer the forge gave that is not a success, said without the request's credential.
|
||||
type statusError struct {
|
||||
method, path string
|
||||
status int
|
||||
said string
|
||||
}
|
||||
|
||||
func (e *statusError) Error() string {
|
||||
return fmt.Sprintf("the forge answered %d to %s %s: %s", e.status, e.method, e.path, e.said)
|
||||
}
|
||||
|
||||
func (f *Forge) client() *http.Client {
|
||||
if f.HTTP != nil {
|
||||
return f.HTTP
|
||||
}
|
||||
return &http.Client{Timeout: 20 * time.Second}
|
||||
}
|
||||
|
||||
func (f *Forge) password() (string, error) {
|
||||
raw, err := os.ReadFile(f.PasswordFile)
|
||||
if err != nil {
|
||||
// The path is the manifest's, not a secret; the content never appears.
|
||||
return "", fmt.Errorf("the forge's admin password cannot be read from its file: %w", err)
|
||||
}
|
||||
return strings.TrimSpace(string(raw)), nil
|
||||
}
|
||||
var errNotFound = forgehttp.ErrNotFound
|
||||
|
||||
// maxBody is the largest answer read from the forge.
|
||||
const maxBody = 32 << 20
|
||||
const maxBody = forgehttp.MaxBody
|
||||
|
||||
// do sends one request and answers the body; 404 is errNotFound.
|
||||
// do sends one request without a body and answers the body; 404 is errNotFound.
|
||||
func (f *Forge) do(ctx context.Context, method, path string) ([]byte, error) {
|
||||
password, err := f.password()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, method, f.URL+path, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
req.SetBasicAuth(f.User, password)
|
||||
req.Header.Set("Accept", "application/json")
|
||||
res, err := f.client().Do(req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("the forge did not answer %s %s: %w", method, path, err)
|
||||
}
|
||||
defer res.Body.Close()
|
||||
// One byte past the cap is read, so a body the cap cut is told from one that fits it exactly: a
|
||||
// cut body is refused, never parsed as if it were whole.
|
||||
body, err := io.ReadAll(io.LimitReader(res.Body, maxBody+1))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(body) > maxBody {
|
||||
return nil, fmt.Errorf("the forge's answer to %s %s is larger than %d MiB and was not read whole", method, path, maxBody>>20)
|
||||
}
|
||||
switch {
|
||||
case res.StatusCode == http.StatusNotFound:
|
||||
return nil, fmt.Errorf("%s %s: %w", method, path, errNotFound)
|
||||
case res.StatusCode >= 300:
|
||||
said := strings.TrimSpace(string(body))
|
||||
if len(said) > 300 {
|
||||
said = said[:300] + "…"
|
||||
}
|
||||
return nil, &statusError{method, path, res.StatusCode, said}
|
||||
}
|
||||
return body, nil
|
||||
}
|
||||
|
||||
func (f *Forge) getJSON(ctx context.Context, path string, into any) error {
|
||||
body, err := f.do(ctx, http.MethodGet, path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := json.Unmarshal(body, into); err != nil {
|
||||
return fmt.Errorf("the forge's answer to %s is not readable: %w", path, err)
|
||||
}
|
||||
return nil
|
||||
return f.Do(ctx, method, path, nil, maxBody)
|
||||
}
|
||||
|
||||
// PackageVersion is one version of one package, as the forge lists it.
|
||||
@@ -146,7 +62,7 @@ func (f *Forge) Packages(ctx context.Context) ([]PackageVersion, error) {
|
||||
for page := 1; page <= 1000; page++ {
|
||||
var batch []PackageVersion
|
||||
path := fmt.Sprintf("/api/v1/packages/%s?type=npm&page=%d&limit=%d", url.PathEscape(f.Owner), page, pageSize)
|
||||
if err := f.getJSON(ctx, path, &batch); err != nil {
|
||||
if err := f.GetJSON(ctx, path, &batch); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
all = append(all, batch...)
|
||||
@@ -166,7 +82,7 @@ func (f *Forge) Size(ctx context.Context, name, version string) (int64, error) {
|
||||
var files []struct {
|
||||
Size int64 `json:"size"`
|
||||
}
|
||||
if err := f.getJSON(ctx, f.versionPath(name, version)+"/files", &files); err != nil {
|
||||
if err := f.GetJSON(ctx, f.versionPath(name, version)+"/files", &files); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
var total int64
|
||||
@@ -182,7 +98,7 @@ func (f *Forge) DistTags(ctx context.Context, name string) (map[string]string, e
|
||||
DistTags map[string]string `json:"dist-tags"`
|
||||
}
|
||||
path := fmt.Sprintf("/api/packages/%s/npm/%s", url.PathEscape(f.Owner), url.PathEscape(name))
|
||||
if err := f.getJSON(ctx, path, &meta); err != nil {
|
||||
if err := f.GetJSON(ctx, path, &meta); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return meta.DistTags, nil
|
||||
@@ -208,7 +124,7 @@ func (f *Forge) Repositories(ctx context.Context) ([]Repository, error) {
|
||||
var found struct {
|
||||
Data []Repository `json:"data"`
|
||||
}
|
||||
if err := f.getJSON(ctx, fmt.Sprintf("/api/v1/repos/search?page=%d&limit=%d", page, pageSize), &found); err != nil {
|
||||
if err := f.GetJSON(ctx, fmt.Sprintf("/api/v1/repos/search?page=%d&limit=%d", page, pageSize), &found); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
all = append(all, found.Data...)
|
||||
@@ -239,7 +155,7 @@ func (f *Forge) Files(ctx context.Context, repo Repository) ([]string, error) {
|
||||
}
|
||||
path := fmt.Sprintf("/api/v1/repos/%s/git/trees/%s?recursive=true&page=%d&per_page=1000",
|
||||
repoPath(repo.FullName), url.PathEscape(repo.DefaultBranch), page)
|
||||
if err := f.getJSON(ctx, path, &tree); err != nil {
|
||||
if err := f.GetJSON(ctx, path, &tree); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, e := range tree.Tree {
|
||||
|
||||
@@ -15,6 +15,8 @@ import (
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea/internal/forgehttp"
|
||||
)
|
||||
|
||||
// fakeForge answers the forge's interface from maps, and records every delete.
|
||||
@@ -38,7 +40,7 @@ func (f *fakeForge) serve() *Forge {
|
||||
if err := os.WriteFile(file, []byte(f.password+"\n"), 0o600); err != nil {
|
||||
f.t.Fatal(err)
|
||||
}
|
||||
return &Forge{URL: srv.URL, User: "admin", PasswordFile: file, Owner: "acme"}
|
||||
return &Forge{Client: forgehttp.Client{URL: srv.URL, User: "admin", PasswordFile: file}, Owner: "acme"}
|
||||
}
|
||||
|
||||
var (
|
||||
@@ -501,7 +503,7 @@ func TestAnAnswerTheCapCutIsRefused(t *testing.T) {
|
||||
if err := os.WriteFile(file, []byte("x\n"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
f := &Forge{URL: srv.URL, User: "admin", PasswordFile: file, Owner: "acme"}
|
||||
f := &Forge{Client: forgehttp.Client{URL: srv.URL, User: "admin", PasswordFile: file}, Owner: "acme"}
|
||||
if _, err := f.do(context.Background(), http.MethodGet, "/big"); err == nil || !strings.Contains(err.Error(), "not read whole") {
|
||||
t.Fatalf("a cut answer: %v", err)
|
||||
}
|
||||
|
||||
+98
-13
@@ -164,21 +164,106 @@ export function judgeApiCall(method: string, rawPath: string, body: unknown): Ap
|
||||
};
|
||||
}
|
||||
|
||||
/** Whether a protection rule's name — a branch, or a glob of them (`*`, `**`) — covers a branch. */
|
||||
/**
|
||||
* Whether a protection rule's name covers a branch, as the forge itself decides it (Gitea 1.27,
|
||||
* models/git/protected_branch.go Match and modules/glob; novox/hq issue 539): a name without a glob character
|
||||
* covers the branch of that name in any case; any other name is a glob with `/` as its separator — `*` any run
|
||||
* within a segment, `**` any run across them, `?` one character of a segment, `[abc]`, `[a-z]` and `[!abc]` one
|
||||
* character, `{a,b}` either, `\` the next character itself — matched case-sensitively; a glob the forge cannot
|
||||
* compile covers its own text exactly. The forge bundle's ruleCovers (cmd/forge/rules.go) is the same rule, and
|
||||
* both are held to one table of answers (test/rule-cases.json).
|
||||
*
|
||||
* The forge hands the compiled glob to Go's regexp, which reads some things JavaScript's reads otherwise: a
|
||||
* backslash before a letter or digit (`\d`, `\w`, `\pL`, `\b`, `\A` are classes or anchors there), a class
|
||||
* that opens with `]` (`[]a]` holds `]` and `a` there, an empty class here) or with `^` (a negation there),
|
||||
* and a POSIX class (`[[:digit:]]`).
|
||||
* A rule using any of them is taken to cover every branch: refusing a write the forge would take is safe,
|
||||
* taking one it guards is not.
|
||||
*/
|
||||
export function ruleCovers(rule: string, branch: string): boolean {
|
||||
if (rule === branch) return true;
|
||||
if (!/[*?[]/.test(rule)) return false;
|
||||
let re = "";
|
||||
for (let i = 0; i < rule.length; i++) {
|
||||
const c = rule[i];
|
||||
if (c === "*" && rule[i + 1] === "*") {
|
||||
re += ".*";
|
||||
i++;
|
||||
} else if (c === "*") re += "[^/]*";
|
||||
else if (c === "?") re += "[^/]";
|
||||
else re += c.replace(/[.+^${}()|[\]\\]/g, "\\$&");
|
||||
if (!/[*?\\[\]{}]/.test(rule)) return rule.toLowerCase() === branch.toLowerCase();
|
||||
if (goReadsOtherwise(rule)) return true;
|
||||
const re = globRegExp(rule);
|
||||
return re ? re.test(branch) : rule === branch;
|
||||
}
|
||||
|
||||
/** Whether a glob holds what Go's regexp reads otherwise than JavaScript's (see ruleCovers): fail closed. */
|
||||
function goReadsOtherwise(rule: string): boolean {
|
||||
return /\\[\p{L}\p{N}]/u.test(rule) || /\[!?[\]^]/.test(rule) || /\[[^\]]*\[:/.test(rule);
|
||||
}
|
||||
|
||||
/** The forge's glob as a regular expression, compiled as its modules/glob compiles it with `/` as separator. */
|
||||
function globRegExp(glob: string): RegExp | null {
|
||||
const g = Array.from(glob);
|
||||
let out = "";
|
||||
let pos = 0;
|
||||
// A character as itself: escaped when it is syntax to a regular expression, and only then — the `u` flag
|
||||
// refuses any other escape, and `-` may be escaped only inside a class.
|
||||
const lit = (c: string) => c.replace(/[.*+?^${}()|[\]\\/]/g, "\\$&");
|
||||
const litInClass = (c: string) => c.replace(/[\]\\[^-]/g, "\\$&");
|
||||
const chars = (): boolean => {
|
||||
out += "[";
|
||||
if (g[pos] === "!") {
|
||||
pos++;
|
||||
out += "^";
|
||||
}
|
||||
while (pos < g.length) {
|
||||
const c = g[pos++];
|
||||
if (c === "]") {
|
||||
out += "]";
|
||||
return true;
|
||||
}
|
||||
if (c === "\\") {
|
||||
if (pos >= g.length) return false;
|
||||
out += litInClass(g[pos++]);
|
||||
} else out += c === "[" || c === "^" ? "\\" + c : c;
|
||||
}
|
||||
return false;
|
||||
};
|
||||
const compile = (sub: boolean): boolean => {
|
||||
while (pos < g.length) {
|
||||
const c = g[pos++];
|
||||
if (sub && c === "}") {
|
||||
out += ")";
|
||||
return true;
|
||||
}
|
||||
switch (c) {
|
||||
case "*":
|
||||
if (g[pos] === "*") {
|
||||
pos++;
|
||||
out += ".*";
|
||||
} else out += "[^/]*";
|
||||
break;
|
||||
case "?":
|
||||
out += "[^/]";
|
||||
break;
|
||||
case "[":
|
||||
if (!chars()) return false;
|
||||
break;
|
||||
case "{":
|
||||
out += "(";
|
||||
if (!compile(true)) return false;
|
||||
break;
|
||||
case ",":
|
||||
out += sub ? "|" : ",";
|
||||
break;
|
||||
case "\\":
|
||||
if (pos >= g.length) return false;
|
||||
out += lit(g[pos++]);
|
||||
break;
|
||||
default:
|
||||
out += lit(c);
|
||||
}
|
||||
}
|
||||
// An alternative left open is a glob the forge cannot compile.
|
||||
return !sub;
|
||||
};
|
||||
if (!compile(false)) return null;
|
||||
try {
|
||||
return new RegExp(`^${out}$`, "u");
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
return new RegExp(`^${re}$`).test(branch);
|
||||
}
|
||||
|
||||
/** Refuses a file written to the default branch or to a branch a protection rule names. */
|
||||
|
||||
@@ -0,0 +1,151 @@
|
||||
// Package forgehttp is the forge's HTTP interface as this module's Go bundles reach it (cmd/forge,
|
||||
// cmd/npm-registry): the forge on its own machine, as its admin account, whose password the vault delivered
|
||||
// into a file this module owns (novox/hq ADR 0086), by basic authentication. The password is read when a call
|
||||
// is made and never logged, answered or put in an error. One copy for both bundles (novox/hq issue 539).
|
||||
package forgehttp
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// MaxBody is the largest JSON answer read from the forge.
|
||||
const MaxBody = 32 << 20
|
||||
|
||||
// Client is the forge's HTTP interface.
|
||||
type Client struct {
|
||||
URL string // e.g. http://127.0.0.1:3000
|
||||
User string
|
||||
PasswordFile string
|
||||
HTTP *http.Client // when nil, a client with Timeout
|
||||
Timeout time.Duration // one request's limit when HTTP is nil; 20 s when zero
|
||||
}
|
||||
|
||||
// FromEnv is the forge as the manifest hands it to a bundle, and the names of the variables it left unset.
|
||||
func FromEnv() (Client, []string) {
|
||||
c := Client{
|
||||
URL: strings.TrimRight(os.Getenv("MESH_GITEA_URL"), "/"),
|
||||
User: os.Getenv("MESH_GITEA_ADMIN_USER"),
|
||||
PasswordFile: os.Getenv("MESH_GITEA_ADMIN_PASSWORD_FILE"),
|
||||
}
|
||||
var missing []string
|
||||
for _, kv := range [][2]string{{"MESH_GITEA_URL", c.URL}, {"MESH_GITEA_ADMIN_USER", c.User},
|
||||
{"MESH_GITEA_ADMIN_PASSWORD_FILE", c.PasswordFile}} {
|
||||
if kv[1] == "" {
|
||||
missing = append(missing, kv[0])
|
||||
}
|
||||
}
|
||||
return c, missing
|
||||
}
|
||||
|
||||
// ErrNotFound is the forge answering 404.
|
||||
var ErrNotFound = errors.New("not found")
|
||||
|
||||
// StatusError is an answer the forge gave that is not a success, said without the request's credential.
|
||||
type StatusError struct {
|
||||
Method, Path string
|
||||
Status int
|
||||
Said string
|
||||
}
|
||||
|
||||
func (e *StatusError) Error() string {
|
||||
return fmt.Sprintf("the forge answered %d to %s %s: %s", e.Status, e.Method, e.Path, e.Said)
|
||||
}
|
||||
|
||||
func (c *Client) client() *http.Client {
|
||||
if c.HTTP != nil {
|
||||
return c.HTTP
|
||||
}
|
||||
timeout := c.Timeout
|
||||
if timeout <= 0 {
|
||||
timeout = 20 * time.Second
|
||||
}
|
||||
return &http.Client{Timeout: timeout}
|
||||
}
|
||||
|
||||
func (c *Client) password() (string, error) {
|
||||
raw, err := os.ReadFile(c.PasswordFile)
|
||||
if err != nil {
|
||||
// The path is the manifest's, not a secret; the content never appears.
|
||||
return "", fmt.Errorf("the forge's admin password cannot be read from its file: %w", err)
|
||||
}
|
||||
return strings.TrimSpace(string(raw)), nil
|
||||
}
|
||||
|
||||
// Do sends one request, with payload as its JSON body when not nil, and answers the body, read up to limit;
|
||||
// 404 is ErrNotFound. One byte past the limit is read, so a body the limit cut is told from one that fits it
|
||||
// exactly: a cut body is refused, never read as if it were whole.
|
||||
func (c *Client) Do(ctx context.Context, method, path string, payload any, limit int64) ([]byte, error) {
|
||||
password, err := c.password()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var body io.Reader
|
||||
if payload != nil {
|
||||
raw, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
body = bytes.NewReader(raw)
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, method, c.URL+path, body)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
req.SetBasicAuth(c.User, password)
|
||||
req.Header.Set("Accept", "application/json")
|
||||
if payload != nil {
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
}
|
||||
res, err := c.client().Do(req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("the forge did not answer %s %s: %w", method, path, err)
|
||||
}
|
||||
defer res.Body.Close()
|
||||
got, err := io.ReadAll(io.LimitReader(res.Body, limit+1))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if int64(len(got)) > limit {
|
||||
return nil, fmt.Errorf("the forge's answer to %s %s is larger than %d MiB and was not read whole", method, path, limit>>20)
|
||||
}
|
||||
switch {
|
||||
case res.StatusCode == http.StatusNotFound:
|
||||
return nil, fmt.Errorf("%s %s: %w", method, path, ErrNotFound)
|
||||
case res.StatusCode >= 300:
|
||||
said := strings.TrimSpace(string(got))
|
||||
if len(said) > 300 {
|
||||
said = said[:300] + "…"
|
||||
}
|
||||
return nil, &StatusError{method, path, res.StatusCode, said}
|
||||
}
|
||||
return got, nil
|
||||
}
|
||||
|
||||
// GetJSON reads one answer into into.
|
||||
func (c *Client) GetJSON(ctx context.Context, path string, into any) error {
|
||||
return c.SendJSON(ctx, http.MethodGet, path, nil, into)
|
||||
}
|
||||
|
||||
// SendJSON sends one request and reads its answer into into, when into is not nil.
|
||||
func (c *Client) SendJSON(ctx context.Context, method, path string, payload, into any) error {
|
||||
body, err := c.Do(ctx, method, path, payload, MaxBody)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if into == nil {
|
||||
return nil
|
||||
}
|
||||
if err := json.Unmarshal(body, into); err != nil {
|
||||
return fmt.Errorf("the forge's answer to %s %s is not readable: %w", method, path, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -7,6 +7,7 @@
|
||||
import { test, after, beforeEach } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { createServer, type IncomingMessage, type ServerResponse } from "node:http";
|
||||
import { readFileSync } from "node:fs";
|
||||
|
||||
import { GiteaClient } from "../dist/client.js";
|
||||
import { ConfiguredToken } from "../dist/token.js";
|
||||
@@ -146,6 +147,25 @@ test("gitea_api refuses a file written to the default branch or one a rule guard
|
||||
assert.deepEqual(writes(), ["PUT /repos/novox/mesh-catalog/contents/modules/sudo/module.json"]);
|
||||
});
|
||||
|
||||
test("a protection rule covers a branch as the forge decides it: classes, alternatives, a plain name in any case (issue 539)", async () => {
|
||||
// One table with the forge bundle (cmd/forge/rules.go): `forge` is the forge's answer; `guard` is where this
|
||||
// guard deliberately covers more — a rule Go's regexp reads otherwise than JavaScript's — and never less.
|
||||
const cases = JSON.parse(readFileSync(new URL("./rule-cases.json", import.meta.url), "utf8")) as
|
||||
Array<{ rule: string; branch: string; forge: boolean; guard?: boolean }>;
|
||||
assert.ok(cases.length >= 100);
|
||||
for (const c of cases) {
|
||||
assert.ok(c.guard === undefined || (c.guard && !c.forge), `${c.rule}: the guard may only cover more than the forge`);
|
||||
assert.equal(ruleCovers(c.rule, c.branch), c.guard ?? c.forge, `${c.rule} covers ${c.branch}`);
|
||||
}
|
||||
const put = (branch: string) => tool("gitea_api").run({ path: "/repos/novox/mesh-catalog/contents/README.md",
|
||||
method: "PUT", body: { content: "e30=", sha: "x", branch } });
|
||||
state.rules.push({ rule_name: "release-[ab]" }, { rule_name: "{stable,lts}/*" });
|
||||
await assert.rejects(put("release-a"), /protection rule release-\[ab\]/);
|
||||
await assert.rejects(put("lts/2"), /protection rule \{stable,lts\}\/\*/);
|
||||
await assert.rejects(put("MAIN"), /protection rule main/);
|
||||
assert.deepEqual(writes(), []);
|
||||
});
|
||||
|
||||
test("gitea_api still writes what an author writes", async () => {
|
||||
await tool("gitea_api").run({ path: "/repos/novox/mesh-catalog/issues/3/comments", method: "POST", body: { body: "hi" } });
|
||||
await tool("gitea_api").run({ path: "/repos/novox/mesh-catalog/pulls/7/reviews", method: "POST", body: { event: "COMMENT" } });
|
||||
|
||||
@@ -0,0 +1,285 @@
|
||||
[
|
||||
{"rule":"main","branch":"main","forge":true,"why":"a plain name covers its branch"},
|
||||
{"rule":"main","branch":"mainx","forge":false},
|
||||
{"rule":"main","branch":"MAIN","forge":true,"why":"a plain name covers in any case (EqualFold)"},
|
||||
{"rule":"Main","branch":"main","forge":true},
|
||||
{"rule":"release/*","branch":"release/1.0","forge":true,"why":"* stays within a segment"},
|
||||
{"rule":"release/*","branch":"release/a/b","forge":false},
|
||||
{"rule":"release/**","branch":"release/a/b","forge":true,"why":"** crosses segments"},
|
||||
{"rule":"v?","branch":"v1","forge":true},
|
||||
{"rule":"v?","branch":"v/","forge":false,"why":"? is not the separator"},
|
||||
{"rule":"a.b","branch":"axb","forge":false,"why":". is itself"},
|
||||
{"rule":"Feat/*","branch":"feat/x","forge":false,"why":"a glob is matched with case"},
|
||||
{"rule":"release-[ab]","branch":"release-a","forge":true,"why":"a class"},
|
||||
{"rule":"release-[ab]","branch":"release-c","forge":false},
|
||||
{"rule":"release-[!ab]","branch":"release-c","forge":true,"why":"a negated class"},
|
||||
{"rule":"release-[!ab]","branch":"release-a","forge":false},
|
||||
{"rule":"v[0-9].x","branch":"v7.x","forge":true,"why":"a range"},
|
||||
{"rule":"v[0-9].x","branch":"vA.x","forge":false},
|
||||
{"rule":"{main,master}","branch":"master","forge":true,"why":"alternatives"},
|
||||
{"rule":"{main,master}","branch":"mainline","forge":false},
|
||||
{"rule":"rel/{a,b}/*","branch":"rel/b/1","forge":true},
|
||||
{"rule":"rel/{a,b}/*","branch":"rel/c/1","forge":false},
|
||||
{"rule":"a\\*","branch":"a*","forge":true,"why":"an escape"},
|
||||
{"rule":"a\\*","branch":"ab","forge":false},
|
||||
{"rule":"a+b","branch":"a+b","forge":true,"why":"+ is itself"},
|
||||
{"rule":"a+b","branch":"aab","forge":false},
|
||||
{"rule":"{open","branch":"{open","forge":true,"why":"a glob the forge cannot compile covers its own text"},
|
||||
{"rule":"[open","branch":"[open","forge":true},
|
||||
{"rule":"[open","branch":"o","forge":false},
|
||||
{"rule":"release-\\d","branch":"release-1","forge":true,"why":"Go reads \\d as a digit"},
|
||||
{"rule":"release-\\d","branch":"release-d","forge":false,"guard":true},
|
||||
{"rule":"v\\w*","branch":"v2","forge":true},
|
||||
{"rule":"main\\s","branch":"main","forge":false,"guard":true},
|
||||
{"rule":"[]a]","branch":"a","forge":true,"why":"a class opening with ] holds ]"},
|
||||
{"rule":"[]a]","branch":"]","forge":true},
|
||||
{"rule":"[]a]","branch":"b","forge":false,"guard":true},
|
||||
{"rule":"rel-[\\d]","branch":"rel-1","forge":true},
|
||||
{"rule":"v[\\pL]","branch":"va","forge":true,"why":"\\p is a Unicode class"},
|
||||
{"rule":"v\\pL","branch":"va","forge":true},
|
||||
{"rule":"[[:alpha:]]","branch":"a","forge":true,"why":"a POSIX class"},
|
||||
{"rule":"rel-[[:digit:]]","branch":"rel-1","forge":true},
|
||||
{"rule":"rel-[[:digit:]]","branch":"rel-x","forge":false,"guard":true},
|
||||
{"rule":"[^+]*","branch":"-[1","forge":true,"why":"a class opening with ^ is negated in Go"},
|
||||
{"rule":"d[^](","branch":"d[^](","forge":true},
|
||||
{"rule":"]?","branch":":,1},","forge":false},
|
||||
{"rule":"Ab,","branch":"Ab,","forge":true},
|
||||
{"rule":"\\:[}-|","branch":",{","forge":false},
|
||||
{"rule":"^,*.d::","branch":"/x","forge":false},
|
||||
{"rule":"","branch":"A*:","forge":false},
|
||||
{"rule":"!|A{","branch":":.{]","forge":false},
|
||||
{"rule":"","branch":"]éb","forge":false},
|
||||
{"rule":"/{)w","branch":"/{)w","forge":true},
|
||||
{"rule":"*^w]","branch":"}!.","forge":false},
|
||||
{"rule":"*),^?*d","branch":"*),^?*d","forge":true},
|
||||
{"rule":"A)^}!","branch":"","forge":false},
|
||||
{"rule":"}^d?","branch":"1\\.","forge":false},
|
||||
{"rule":"","branch":"","forge":true},
|
||||
{"rule":"$?[","branch":"$?[","forge":true},
|
||||
{"rule":"","branch":"","forge":true},
|
||||
{"rule":"+","branch":"+","forge":true},
|
||||
{"rule":"","branch":"},[","forge":false},
|
||||
{"rule":"xA[d:a)","branch":"]1.!bx","forge":false},
|
||||
{"rule":"^/}(","branch":"","forge":false},
|
||||
{"rule":"-|x.x1?","branch":"-|x.x1?","forge":true},
|
||||
{"rule":"x+)A]-A","branch":"x+)A]-A","forge":true},
|
||||
{"rule":"x*a","branch":"x*a","forge":true},
|
||||
{"rule":"$xb","branch":"}é","forge":false},
|
||||
{"rule":"wAx?/a,","branch":"","forge":false},
|
||||
{"rule":"[,","branch":"[,","forge":true},
|
||||
{"rule":"}!","branch":"}!","forge":true},
|
||||
{"rule":"*:|\\w","branch":"","forge":false,"guard":true},
|
||||
{"rule":"","branch":"","forge":true},
|
||||
{"rule":"é","branch":"é","forge":true},
|
||||
{"rule":"..(","branch":"{\\{+","forge":false},
|
||||
{"rule":"*|{","branch":"1.}-","forge":false},
|
||||
{"rule":"!+,","branch":"}++-*","forge":false},
|
||||
{"rule":"x\\({","branch":"*","forge":false},
|
||||
{"rule":"","branch":"","forge":true},
|
||||
{"rule":"","branch":"","forge":true},
|
||||
{"rule":"?-{+-","branch":"?-{+-","forge":true},
|
||||
{"rule":"/-{é|$","branch":"b!.[","forge":false},
|
||||
{"rule":"a-+dxA","branch":"1AA,xA","forge":false},
|
||||
{"rule":"]|(","branch":"]|(","forge":true},
|
||||
{"rule":"}","branch":"{!x{d","forge":false},
|
||||
{"rule":"/{*](","branch":"!]","forge":false},
|
||||
{"rule":"($+{-}|","branch":"/","forge":false},
|
||||
{"rule":"(-b.-1}","branch":"*A*+,1","forge":false},
|
||||
{"rule":"1[d|)wa","branch":"+}[!}}","forge":false},
|
||||
{"rule":"w","branch":":.","forge":false},
|
||||
{"rule":"?}\\*","branch":"","forge":false},
|
||||
{"rule":"?$*","branch":"?$*","forge":true},
|
||||
{"rule":"","branch":"!/{béa","forge":false},
|
||||
{"rule":"","branch":"A1","forge":false},
|
||||
{"rule":"","branch":"","forge":true},
|
||||
{"rule":"!a$1/b{","branch":"!a$1/b{","forge":true},
|
||||
{"rule":":dww/|d","branch":":dww/|d","forge":true},
|
||||
{"rule":"!|*}","branch":"xéxé{b","forge":false},
|
||||
{"rule":"1é||","branch":"a+.","forge":false},
|
||||
{"rule":"/+],","branch":"!-\\[{é","forge":false},
|
||||
{"rule":"a}(","branch":"a}(","forge":true},
|
||||
{"rule":"|A?.","branch":"}.","forge":false},
|
||||
{"rule":"d$a^d","branch":"*{","forge":false},
|
||||
{"rule":"-ba$+A","branch":"-ba$+A","forge":true},
|
||||
{"rule":"-.|,?a!","branch":"-.|,?a!","forge":true},
|
||||
{"rule":"A","branch":"A","forge":true},
|
||||
{"rule":"w","branch":"w","forge":true},
|
||||
{"rule":".!1!+a","branch":".!1!+a","forge":true},
|
||||
{"rule":"+$w:}","branch":"+$w:}","forge":true},
|
||||
{"rule":"","branch":"","forge":true},
|
||||
{"rule":")]","branch":")]","forge":true},
|
||||
{"rule":"a})^,}1","branch":"a})^,}1","forge":true},
|
||||
{"rule":":[:!:","branch":":[:!:","forge":true},
|
||||
{"rule":"w?w:]1!","branch":"\\.]-","forge":false},
|
||||
{"rule":"*$/","branch":"*$/","forge":true},
|
||||
{"rule":"é](","branch":"é](","forge":true},
|
||||
{"rule":"x$a/","branch":"x$a/","forge":true},
|
||||
{"rule":"*)b+[^","branch":"*)b+[^","forge":true},
|
||||
{"rule":"/]/?","branch":"/]/?","forge":true},
|
||||
{"rule":"{x,A","branch":"!Ab","forge":false},
|
||||
{"rule":"","branch":"","forge":true},
|
||||
{"rule":"-,","branch":"xé}A[.","forge":false},
|
||||
{"rule":"w)Ad","branch":"w)Ad","forge":true},
|
||||
{"rule":":??","branch":":??","forge":true},
|
||||
{"rule":"x/xé|","branch":"x/xé|","forge":true},
|
||||
{"rule":"","branch":"1!","forge":false},
|
||||
{"rule":"!![é[","branch":"x","forge":false},
|
||||
{"rule":"wb$","branch":"wb$","forge":true},
|
||||
{"rule":")","branch":")","forge":true},
|
||||
{"rule":"Aw:)(1","branch":"{b\\,","forge":false},
|
||||
{"rule":"","branch":"","forge":true},
|
||||
{"rule":"1*}?d","branch":"1*}?d","forge":true},
|
||||
{"rule":"","branch":"","forge":true},
|
||||
{"rule":"?-b.","branch":"?-b.","forge":true},
|
||||
{"rule":"","branch":"","forge":true},
|
||||
{"rule":"d","branch":",aé1","forge":false},
|
||||
{"rule":"+","branch":"+{","forge":false},
|
||||
{"rule":"]d","branch":"]}:/bA","forge":false},
|
||||
{"rule":"*?+:|","branch":"!é.","forge":false},
|
||||
{"rule":"}$d","branch":"}$d","forge":true},
|
||||
{"rule":"/","branch":"/","forge":true},
|
||||
{"rule":"a$+b)b","branch":"d}","forge":false},
|
||||
{"rule":"?$[/A","branch":"","forge":false},
|
||||
{"rule":"é){","branch":"é){","forge":true},
|
||||
{"rule":"$(d","branch":"{é}","forge":false},
|
||||
{"rule":"b(","branch":"b(","forge":true},
|
||||
{"rule":"^^+]","branch":"^^+]","forge":true},
|
||||
{"rule":",)","branch":",)","forge":true},
|
||||
{"rule":"d-.A-","branch":"d-.A-","forge":true},
|
||||
{"rule":"$","branch":"\\/","forge":false},
|
||||
{"rule":"}+[\\|\\$","branch":"x[!b","forge":false},
|
||||
{"rule":"(d1aé!é","branch":"A[-","forge":false},
|
||||
{"rule":"x","branch":"1","forge":false},
|
||||
{"rule":"x","branch":"dx","forge":false},
|
||||
{"rule":"]|)]","branch":"]|)]","forge":true},
|
||||
{"rule":"","branch":"","forge":true},
|
||||
{"rule":"[.*?","branch":"[.*?","forge":true},
|
||||
{"rule":"[?","branch":"[?","forge":true},
|
||||
{"rule":"","branch":"","forge":true},
|
||||
{"rule":"),/","branch":"a1,,1","forge":false},
|
||||
{"rule":"}+","branch":"}+","forge":true},
|
||||
{"rule":"$d\\,d","branch":"$d\\,d","forge":false},
|
||||
{"rule":"d1","branch":"d1","forge":true},
|
||||
{"rule":"-!^.:","branch":"-!^.:","forge":true},
|
||||
{"rule":"x","branch":"+].A11","forge":false},
|
||||
{"rule":"","branch":"-]","forge":false},
|
||||
{"rule":"}]}.","branch":"","forge":false},
|
||||
{"rule":"^{wx","branch":"]1a!","forge":false},
|
||||
{"rule":"^","branch":"*]x,","forge":false},
|
||||
{"rule":"(:x*x","branch":"","forge":false},
|
||||
{"rule":"|[/*a/","branch":"}","forge":false},
|
||||
{"rule":"","branch":"","forge":true},
|
||||
{"rule":"","branch":"","forge":true},
|
||||
{"rule":"1\\^","branch":"!é!x{","forge":false},
|
||||
{"rule":"(.b,AA+","branch":"(.b,AA+","forge":true},
|
||||
{"rule":"xx","branch":"","forge":false},
|
||||
{"rule":"d)^):","branch":"/}!","forge":false},
|
||||
{"rule":"|","branch":",é*]*[","forge":false},
|
||||
{"rule":"d*d:*1","branch":"\\b,","forge":false},
|
||||
{"rule":":]","branch":"[","forge":false},
|
||||
{"rule":"([|!\\^é","branch":"([|!\\^é","forge":true},
|
||||
{"rule":"$[w)x-*","branch":"","forge":false},
|
||||
{"rule":")","branch":")","forge":true},
|
||||
{"rule":".d^)?","branch":"x\\é","forge":false},
|
||||
{"rule":"1|{w","branch":"1|{w","forge":true},
|
||||
{"rule":"é$,(]$$","branch":"é$,(]$$","forge":true},
|
||||
{"rule":"/","branch":"/","forge":true},
|
||||
{"rule":"","branch":"","forge":true},
|
||||
{"rule":"+","branch":"","forge":false},
|
||||
{"rule":"]","branch":"","forge":false},
|
||||
{"rule":"+:a:","branch":"+:a:","forge":true},
|
||||
{"rule":"/*","branch":"d.}","forge":false},
|
||||
{"rule":"|/ba1-","branch":"|/ba1-","forge":true},
|
||||
{"rule":"","branch":"+","forge":false},
|
||||
{"rule":"A","branch":"A","forge":true},
|
||||
{"rule":",-!]","branch":",-!]","forge":true},
|
||||
{"rule":"$*d!)","branch":"","forge":false},
|
||||
{"rule":")é+x.d","branch":")é+x.d","forge":true},
|
||||
{"rule":"éa$\\[1","branch":"1/,","forge":false},
|
||||
{"rule":"}?1|]d","branch":"}?1|]d","forge":true},
|
||||
{"rule":"[","branch":"[","forge":true},
|
||||
{"rule":"-1.*+","branch":"-1.*+","forge":true},
|
||||
{"rule":"Aw|{","branch":"Aw|{","forge":true},
|
||||
{"rule":"adAAAa","branch":"A","forge":false},
|
||||
{"rule":"","branch":"","forge":true},
|
||||
{"rule":"d/^{:","branch":"d/^{:","forge":true},
|
||||
{"rule":"é:(,^","branch":"é:(,^","forge":true},
|
||||
{"rule":"","branch":"","forge":true},
|
||||
{"rule":"","branch":"","forge":true},
|
||||
{"rule":"!ébé({+","branch":"!ébé({+","forge":true},
|
||||
{"rule":"x","branch":"x","forge":true},
|
||||
{"rule":"|","branch":",:*","forge":false},
|
||||
{"rule":"","branch":"","forge":true},
|
||||
{"rule":"x|","branch":"x|","forge":true},
|
||||
{"rule":"a","branch":"aaxéb,","forge":false},
|
||||
{"rule":"/w-),1é","branch":"/w-),1é","forge":true},
|
||||
{"rule":"|*)^,","branch":"|*)^,","forge":true},
|
||||
{"rule":"){é.!b.","branch":"a-}:","forge":false},
|
||||
{"rule":"(\\","branch":"(\\","forge":true},
|
||||
{"rule":".w(\\","branch":"+","forge":false},
|
||||
{"rule":"/d[","branch":"/d[","forge":true},
|
||||
{"rule":"(}$x!}","branch":"a,bb","forge":false},
|
||||
{"rule":"w{","branch":"w{","forge":true},
|
||||
{"rule":"[+a\\A(+","branch":"[+a\\A(+","forge":true},
|
||||
{"rule":"(1^{)bb","branch":"(1^{)bb","forge":true},
|
||||
{"rule":"^1","branch":"é-}","forge":false},
|
||||
{"rule":"*,+}1):","branch":"*,+}1):","forge":true},
|
||||
{"rule":"[A[(.x,","branch":"]db!*","forge":false},
|
||||
{"rule":"","branch":"Ad}","forge":false},
|
||||
{"rule":".","branch":".","forge":true},
|
||||
{"rule":"w!é{x","branch":"é.{","forge":false},
|
||||
{"rule":"$\\^","branch":"bx]+","forge":false},
|
||||
{"rule":"1)","branch":"+*d","forge":false},
|
||||
{"rule":"x)}/])","branch":"]","forge":false},
|
||||
{"rule":",a}.1}","branch":",a}.1}","forge":true},
|
||||
{"rule":"-a{(","branch":"-a{(","forge":true},
|
||||
{"rule":"1","branch":"1","forge":true},
|
||||
{"rule":"","branch":"","forge":true},
|
||||
{"rule":"}[é$","branch":"]d-!","forge":false},
|
||||
{"rule":"xé{é+?","branch":"xé{é+?","forge":true},
|
||||
{"rule":"/A1!","branch":"xAA\\::","forge":false},
|
||||
{"rule":"1][b-.(","branch":"{/-b![","forge":false},
|
||||
{"rule":"(d}(","branch":"(d}(","forge":true},
|
||||
{"rule":"{*}","branch":"!ébd","forge":true},
|
||||
{"rule":"()^^)","branch":"()^^)","forge":true},
|
||||
{"rule":"1(1A{$","branch":"*,[","forge":false},
|
||||
{"rule":"A:!^\\","branch":"A:!^\\","forge":true},
|
||||
{"rule":"-]:b]","branch":"-]:b]","forge":true},
|
||||
{"rule":",","branch":",","forge":true},
|
||||
{"rule":"[-:]","branch":"[-:]","forge":false},
|
||||
{"rule":"","branch":"é:b1:d","forge":false},
|
||||
{"rule":"1\\!]?d","branch":"","forge":false},
|
||||
{"rule":"","branch":"bb!]+","forge":false},
|
||||
{"rule":"/da","branch":"b]d[","forge":false},
|
||||
{"rule":",]","branch":",]","forge":true},
|
||||
{"rule":"[","branch":"[","forge":true},
|
||||
{"rule":"a]{A-","branch":"-+","forge":false},
|
||||
{"rule":"A?A^1","branch":"A?A^1","forge":true},
|
||||
{"rule":"\\-$","branch":"","forge":false},
|
||||
{"rule":"+","branch":"*[}[","forge":false},
|
||||
{"rule":",b?{dw","branch":",b?{dw","forge":true},
|
||||
{"rule":"d[$b1","branch":"/","forge":false},
|
||||
{"rule":"})|xAA","branch":"é,!","forge":false},
|
||||
{"rule":"]--","branch":"]--","forge":true},
|
||||
{"rule":"],/b","branch":"],/b","forge":true},
|
||||
{"rule":"/","branch":"/","forge":true},
|
||||
{"rule":"*aé","branch":"*aé","forge":true},
|
||||
{"rule":"{","branch":"{","forge":true},
|
||||
{"rule":":aé","branch":"","forge":false},
|
||||
{"rule":".\\}","branch":".\\}","forge":false},
|
||||
{"rule":"wA]*","branch":"\\]!d{+","forge":false},
|
||||
{"rule":"a/1x$/","branch":"[1!:/A","forge":false},
|
||||
{"rule":"[.wb}d.","branch":"-!*..a","forge":false},
|
||||
{"rule":".[{()a","branch":"+}","forge":false},
|
||||
{"rule":"]/!:,","branch":"{Aéa","forge":false},
|
||||
{"rule":"é{ba[-","branch":"],,1d/","forge":false},
|
||||
{"rule":"(w*","branch":"1}.!","forge":false},
|
||||
{"rule":"{?","branch":"1","forge":false},
|
||||
{"rule":")","branch":",+!","forge":false},
|
||||
{"rule":":","branch":"{/+A][","forge":false},
|
||||
{"rule":".1{)","branch":"a\\bA","forge":false},
|
||||
{"rule":"w-1+x1-","branch":"*]1}bb","forge":false},
|
||||
{"rule":"xb$$","branch":"éx:]-:","forge":false},
|
||||
{"rule":"++$!?*$","branch":"bA","forge":false},
|
||||
{"rule":"]{","branch":"[xd*x.","forge":false}
|
||||
]
|
||||
@@ -132,8 +132,16 @@ as text. `get` serves each, and any other field a record of hq wrote under `fiel
|
||||
- `partial`: both ends of a change agree;
|
||||
- `rests-on-live`: no accepted record extends a superseded one.
|
||||
|
||||
A failure is kept as a finding and makes the module unhealthy, so the mesh raises a condition naming the rule
|
||||
and the records. The acceptance stands.
|
||||
A failure is kept as a finding and told to the operator once, as a warning naming the rule and the records
|
||||
(`operator-channel.notify`, keyed `knowledge.finding.<rule>.<records>`, sent again at the next step when it
|
||||
could not be sent, and cleared once the rule holds for them again). It is a fault of the records, which their
|
||||
authors repair, not of the module: the module's health names it and stays healthy, so it never holds the
|
||||
module's own rollouts (the operator's decision of 2026-10-12 on hq issue 504). The acceptance stands.
|
||||
|
||||
`cited` is also judged ahead, in the review: a decision that names no home and that no accepted (or
|
||||
superseded) record links is not judged and not accepted. Its review, by the knowledge base itself, asks changes
|
||||
naming the rule (hq issue 504). It is not refused at propose, because no record can link a number only
|
||||
reserved: a record citing a new one is proposed after it and accepted, and the new one is proposed again.
|
||||
|
||||
## The import
|
||||
|
||||
@@ -181,14 +189,14 @@ the consolidated schema for a fresh database is `schema.sql`. A test holds the t
|
||||
|
||||
- **the bundle**: the verbs, `knowledge_health`, the operator's answers heard, and once a minute (and at once when
|
||||
a record is proposed) the router's record read, reservations never proposed withdrawn, the reviews of proposals,
|
||||
and the drafting worker. It never changes the schema, and
|
||||
and the drafting worker; and it tells the whole-set findings. It never changes the schema, and
|
||||
refuses to serve while the schema is behind.
|
||||
- **`prepare`** (run once): brings the schema to this version.
|
||||
- **`keep`** (judged by its unit): judges the whole record set every five minutes and keeps the findings.
|
||||
- **`watch`** (judged by `knowledge_health`, run as the machine's account, the one the runtime runs the bundle
|
||||
as, so the tool can read what it writes): every minute, reads whether the database serves and what the
|
||||
findings say, and leaves its verdict in `${dir:state}/health.json`. That is the file the health tool answers
|
||||
from. A verdict older than three minutes is unhealthy.
|
||||
from. A verdict older than three minutes is unhealthy; a whole-set finding is named in it but leaves it healthy.
|
||||
|
||||
## Settings
|
||||
|
||||
|
||||
@@ -514,6 +514,73 @@ func (k *KB) judgeSet(ctx context.Context) []Finding {
|
||||
return findings
|
||||
}
|
||||
|
||||
// findingPrefix keys the notice of one whole-set finding: the prefix, the rule, and the records it names.
|
||||
const findingPrefix = "knowledge.finding."
|
||||
|
||||
func findingKey(f Finding) string {
|
||||
names := make([]string, len(f.Records))
|
||||
for i, r := range f.Records {
|
||||
names[i] = strings.ReplaceAll(strings.ToLower(r), " ", "-")
|
||||
}
|
||||
return findingPrefix + f.Rule + "." + strings.Join(names, ".")
|
||||
}
|
||||
|
||||
// TellFindings, called by the background work and never under the write lock, tells the operator, once, each finding of the rules that judge the whole record set, as a warning
|
||||
// naming the rule and the records, and clears the notice once the rule holds for them again. A finding is a fault
|
||||
// of the records, which their authors repair; it is not the module failing, so it leaves the module's health as it
|
||||
// is and never holds the module's own rollouts (the operator's decision of 2026-10-12 on novox/hq issue 504, as
|
||||
// issue 508 decided for a proposal never reviewed). A notice that cannot be sent is not kept as told, and is sent
|
||||
// again at the next step.
|
||||
func (k *KB) TellFindings() {
|
||||
if k.Notify == nil || k.Clear == nil {
|
||||
return
|
||||
}
|
||||
ctx := context.Background()
|
||||
findings, err := k.Store.Findings(ctx)
|
||||
if err != nil {
|
||||
k.logf("[mesh-knowledge] the whole-set findings cannot be read to tell them: %v", err)
|
||||
return
|
||||
}
|
||||
now := map[string]bool{}
|
||||
for _, f := range findings {
|
||||
key := findingKey(f)
|
||||
now[key] = true
|
||||
first, err := k.Store.Told(ctx, key)
|
||||
if err != nil || !first {
|
||||
continue
|
||||
}
|
||||
records := strings.Join(f.Records, ", ")
|
||||
summary := fmt.Sprintf("The rule %s, which judges the whole record set, fails: %s.", f.Rule, f.Says)
|
||||
explanation := "The knowledge base works; its records break a rule. The acceptance stands. The author of " + records +
|
||||
" repairs it: a decision cited by nothing names its home in a progressive insight (propose with homes and insight), " +
|
||||
"or a record that cites it is accepted. get and links show the record."
|
||||
resolved := fmt.Sprintf("The rule %s holds again for %s", f.Rule, records)
|
||||
if err := k.Notify(key, summary, "A rule of the record set fails: "+records, explanation, resolved); err != nil {
|
||||
k.logf("[mesh-knowledge] the operator could not be told that %s fails for %s: %v; told again at the next step", f.Rule, records, err)
|
||||
if uerr := k.Store.Untold(ctx, key); uerr != nil {
|
||||
k.logf("[mesh-knowledge] and that it was not told could not be kept: %v", uerr)
|
||||
}
|
||||
}
|
||||
}
|
||||
told, err := k.Store.ToldOf(ctx, findingPrefix)
|
||||
if err != nil {
|
||||
k.logf("[mesh-knowledge] the notices of whole-set findings cannot be read: %v", err)
|
||||
return
|
||||
}
|
||||
for _, key := range told {
|
||||
if now[key] {
|
||||
continue
|
||||
}
|
||||
if err := k.Clear(key, ""); err != nil {
|
||||
k.logf("[mesh-knowledge] the notice %s could not be cleared: %v; it is cleared again at the next step", key, err)
|
||||
continue
|
||||
}
|
||||
if err := k.Store.Untold(ctx, key); err != nil {
|
||||
k.logf("[mesh-knowledge] the notice %s was cleared, and that could not be kept: %v", key, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ---- reservations that were never proposed -------------------------------------------------------------------
|
||||
|
||||
// Withdraw withdraws every number reserved longer ago than the settings allow and never proposed: it stays spent,
|
||||
|
||||
@@ -40,6 +40,9 @@ func TestMain(m *testing.M) {
|
||||
serve(k, os.Getenv("MESH_KNOWLEDGE_STATUS"))
|
||||
os.Exit(0)
|
||||
}
|
||||
// Every test runs in a zone that is not UTC, so a time said locally is proven to be, wherever the tests run. It
|
||||
// is set once, before any goroutine starts: a test that set it while a bus test's server read it raced.
|
||||
time.Local = testZone
|
||||
code := m.Run()
|
||||
stopPostgres()
|
||||
os.Exit(code)
|
||||
@@ -436,9 +439,13 @@ func valid(n int) ProposeAsk {
|
||||
{Name: "Consequences", Text: "Merges land sooner."},
|
||||
{Name: "How it is checked", Text: "The controller's merge-window tests."},
|
||||
},
|
||||
Links: []LinkAsk{{Kind: LinkExtends, To: "0276"}, {Kind: LinkChanges, To: "0276 §1", From: 1}}}
|
||||
Links: []LinkAsk{{Kind: LinkExtends, To: "0276"}, {Kind: LinkChanges, To: "0276 §1", From: 1}},
|
||||
Homes: []string{testHome}}
|
||||
}
|
||||
|
||||
// testHome is the hq design valid names as its home: a decision is accepted only cited (novox/hq issue 504).
|
||||
const testHome = "03-DESIGN/01-to-be/55-the-knowledge-base.md"
|
||||
|
||||
// imported is a world whose knowledge base holds the fixture's records.
|
||||
func imported(t *testing.T, s Store) *world {
|
||||
t.Helper()
|
||||
|
||||
@@ -164,6 +164,8 @@ func background(k *KB) {
|
||||
logf("[mesh-knowledge] withdrew %d reservation(s) never proposed", n)
|
||||
}
|
||||
review()
|
||||
// The keeper judges the record set in a process of its own, which tells nobody: its findings are told here.
|
||||
k.TellFindings()
|
||||
for {
|
||||
said, err := k.DraftStep()
|
||||
if err != nil {
|
||||
@@ -552,10 +554,11 @@ type HealthAnswer struct {
|
||||
At time.Time `json:"at,omitempty"`
|
||||
}
|
||||
|
||||
// Health is whether the knowledge base works: unhealthy when its database cannot serve, or when a rule that judges
|
||||
// the whole record set fails — the mesh raises a condition naming the rule and the records. A proposal that ran out
|
||||
// of review tries waits on its author, not on the module: it is told to its author and the operator, and named
|
||||
// here, but leaves the module healthy, so it never holds the module's own rollouts (novox/hq issue 508).
|
||||
// Health is whether the knowledge base works: unhealthy when its database cannot serve. A rule that judges the whole
|
||||
// record set and fails is a fault of the records, which their authors repair, not of the module: it is told to the
|
||||
// operator as a warning naming the rule and the records (TellFindings), and named here, but leaves the module
|
||||
// healthy (the operator's decision of 2026-10-12 on novox/hq issue 504). So does a proposal that ran out of review
|
||||
// tries (novox/hq issue 508). Neither holds the module's own rollouts.
|
||||
func (k *KB) Health(ready Healther) HealthAnswer {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 8*time.Second)
|
||||
defer cancel()
|
||||
@@ -586,17 +589,14 @@ func (k *KB) Health(ready Healther) HealthAnswer {
|
||||
waiting = fmt.Sprintf("%d proposed record(s) could not be reviewed in %d tries and wait for their authors to propose them again: %s",
|
||||
n, MostReviewTries, strings.Join(unreviewed, "; "))
|
||||
}
|
||||
why := "the database serves and every rule that judges the whole record set holds"
|
||||
if len(says) > 0 {
|
||||
if len(says) > 8 {
|
||||
says = append(says[:8], fmt.Sprintf("and %d more", len(says)-8))
|
||||
}
|
||||
why := fmt.Sprintf("%d rule(s) that judge the whole record set fail: %s", len(findings), strings.Join(says, "; "))
|
||||
if waiting != "" {
|
||||
why += ". " + waiting
|
||||
}
|
||||
return HealthAnswer{State: "unhealthy", Why: why}
|
||||
why = fmt.Sprintf("the database serves; %d rule(s) that judge the whole record set fail, told to the operator as a warning "+
|
||||
"for the records' authors to repair: %s", len(findings), strings.Join(says, "; "))
|
||||
}
|
||||
why := "the database serves and every rule that judges the whole record set holds"
|
||||
if waiting != "" {
|
||||
why += "; " + waiting
|
||||
}
|
||||
@@ -605,10 +605,11 @@ func (k *KB) Health(ready Healther) HealthAnswer {
|
||||
|
||||
func healthTool(status string, now func() time.Time) stdio.Tool {
|
||||
return stdio.Tool{Name: HealthTool,
|
||||
Description: "How the knowledge base is, as its watch last saw it: healthy when its database serves this version and every " +
|
||||
"rule that judges the whole record set holds; unhealthy otherwise, naming the rule and the records — a decision accepted " +
|
||||
"that nothing cites, a supersession or a change said on one end only, a record extending a superseded one. A " +
|
||||
"proposed record that ran out of review tries is named but leaves it healthy: it waits on its author (novox/hq issue 508).",
|
||||
Description: "How the knowledge base is, as its watch last saw it: healthy when its database serves this version, unhealthy " +
|
||||
"otherwise. A rule that judges the whole record set and fails — a decision accepted that nothing cites, a supersession " +
|
||||
"or a change said on one end only, a record extending a superseded one — is named but leaves it healthy: it is told " +
|
||||
"to the operator as a warning for the records' authors (novox/hq issue 504). So is a proposed record that ran out of " +
|
||||
"review tries: it waits on its author (novox/hq issue 508).",
|
||||
Run: func(map[string]any) (any, error) { return ReadStatus(status, now()), nil }}
|
||||
}
|
||||
|
||||
@@ -918,7 +919,8 @@ func Tools(k *KB, caller func(map[string]any) string) []stdio.Tool {
|
||||
"link to nothing, a clause without a bold lead, a superseded record cited alone, an unknown topic, a retired word. A " +
|
||||
"proposal is revised by proposing it again; an accepted record changes only by a progressive insight (insight: what " +
|
||||
"stood before). The knowledge base then reviews each version itself, in a judgement that sees only the record, the " +
|
||||
"records it links and the rules: on approve it is accepted (a record that needs the operator waits for the operator's " +
|
||||
"records it links and the rules. A decision that names no home and that no accepted record links is not judged: " +
|
||||
"its review asks changes for the whole-set rule cited, which its acceptance would break. On approve it is accepted (a record that needs the operator waits for the operator's " +
|
||||
"Approve), on changes it stays proposed with the findings for its author to revise, and when no judgement can be had " +
|
||||
"it stays proposed and says so. Nothing proposed is in force.",
|
||||
Input: schema([]string{"number", "title"}, map[string]any{
|
||||
@@ -933,7 +935,7 @@ func Tools(k *KB, caller func(map[string]any) string) []stdio.Tool {
|
||||
"links": map[string]any{"type": "array", "description": "its links: kind, to (\"0276\", \"0276 §4\", \"research 045\", \"issue 365\", a design's path), from (the clause it goes out from)",
|
||||
"items": map[string]any{"type": "object", "properties": map[string]any{"kind": map[string]any{"type": "string", "enum": LinkKinds},
|
||||
"to": text("the target"), "from": integer("the clause of this record it goes out from")}, "required": []string{"kind", "to"}}},
|
||||
"homes": texts("hq documents that cite it, such as a design's path"),
|
||||
"homes": texts("hq documents that cite it, such as a design's path; a decision needs one unless an accepted record links it"),
|
||||
"needs-operator": flag("it holds only once the operator approves it"),
|
||||
"status": map[string]any{"type": "string", "enum": ResearchStatuses, "description": "a research effort's status"},
|
||||
"insight": text("for an accepted record: a progressive insight, saying what stood before"),
|
||||
|
||||
@@ -408,7 +408,10 @@ func (k *KB) reviewOne(key Key) (string, error) {
|
||||
k.mu.Lock()
|
||||
e, err := k.Store.Get(ctx, key)
|
||||
var linked []*Record
|
||||
var all []*Entry
|
||||
var allErr error
|
||||
if err == nil && e != nil {
|
||||
all, allErr = k.Store.All(ctx)
|
||||
seen := map[Key]bool{key: true}
|
||||
for _, l := range e.Working().Links {
|
||||
to, ok := l.To.Record()
|
||||
@@ -430,9 +433,20 @@ func (k *KB) reviewOne(key Key) (string, error) {
|
||||
var verdict string
|
||||
var findings []string
|
||||
by := ReviewBy
|
||||
if len(question) > MostReviewQuestion {
|
||||
uncited := ""
|
||||
if allErr == nil {
|
||||
uncited = UncitedOnAcceptance(w, all)
|
||||
}
|
||||
switch {
|
||||
case allErr != nil:
|
||||
err = fmt.Errorf("the record set cannot be read to judge whether a record cites it: %v", allErr)
|
||||
case uncited != "":
|
||||
// The whole-set rule cited, judged ahead: changes, asked by the knowledge base itself without a judgement,
|
||||
// so its acceptance never turns the module's health red for it (novox/hq issue 504).
|
||||
verdict, findings = VerdictChanges, []string{"the rule cited, which judges the whole record set: " + uncited}
|
||||
case len(question) > MostReviewQuestion:
|
||||
err = fmt.Errorf("the record is %d bytes as a question, and a judgement takes at most %d: shorten it", len(question), MostReviewQuestion)
|
||||
} else {
|
||||
default:
|
||||
var answer json.RawMessage
|
||||
var judge string
|
||||
answer, judge, err = k.Judge(question, reviewSchema, k.judgeModel())
|
||||
|
||||
@@ -155,13 +155,16 @@ func TestProposedAgainAfterTheLastTryClearsTheNotice(t *testing.T) {
|
||||
})
|
||||
}
|
||||
|
||||
// inZone sets the local zone to one that is not UTC for one test, so a time said locally is proven to be, wherever
|
||||
// the test runs.
|
||||
// testZone is the local zone every test runs in, set once by TestMain: not UTC.
|
||||
var testZone = time.FixedZone("CEST", 2*60*60)
|
||||
|
||||
// inZone says the test needs the local zone not to be UTC; TestMain set it before any goroutine started. Setting it
|
||||
// here, under a running bus server's goroutines, was a data race the race detector failed.
|
||||
func inZone(t *testing.T) {
|
||||
t.Helper()
|
||||
was := time.Local
|
||||
time.Local = time.FixedZone("CEST", 2*60*60)
|
||||
t.Cleanup(func() { time.Local = was })
|
||||
if time.Local != testZone {
|
||||
t.Fatal("the local zone is not the tests' own: TestMain sets it")
|
||||
}
|
||||
}
|
||||
|
||||
// novox/hq issue 497: a clear the operator's channel refused keeps its notice as told, and is sent again at the next
|
||||
|
||||
@@ -546,6 +546,36 @@ func JudgeSet(records []*Record) []Finding {
|
||||
return out
|
||||
}
|
||||
|
||||
// UncitedOnAcceptance is the whole-set rule `cited` judged ahead, on a decision's working version as it would
|
||||
// stand once accepted, over the records as JudgeSet reads them: "" when it names a home or a record accepted (or
|
||||
// superseded) links it, else what the rule would say. hq held this rule at the merge of the record's own pull
|
||||
// request, which carried the design citing it; the knowledge base holds it in the review, before the judgement, so
|
||||
// an acceptance never leaves a decision cited by nothing (novox/hq issue 504). It is not refused at propose: no
|
||||
// record can link a number only reserved, so a record that will cite a new one is proposed after it, accepted, and
|
||||
// the new one proposed again. A citing record changed by a correction after the acceptance is what the rule, run
|
||||
// after every acceptance, still says. An imported record was held to hq's checks, and its correction is not judged
|
||||
// here, as JudgeSet does not judge it.
|
||||
func UncitedOnAcceptance(r *Record, all []*Entry) string {
|
||||
if r.Kind != KindDecision || len(r.Homes) > 0 || (r.Pending != nil && r.AcceptedAt.IsZero()) {
|
||||
return ""
|
||||
}
|
||||
k := r.Key()
|
||||
for _, e := range all {
|
||||
from := e.Current
|
||||
if from.Key() == k || !(from.Accepted() || from.Status == StatusSuperseded) {
|
||||
continue
|
||||
}
|
||||
for _, l := range from.Links {
|
||||
if to, ok := l.To.Record(); ok && to == k {
|
||||
return ""
|
||||
}
|
||||
}
|
||||
}
|
||||
return fmt.Sprintf("%s would be accepted with nothing citing it: it names no home and no accepted record links it. Name "+
|
||||
"its home (homes: the hq design or playbook that cites it), or have the record that links it accepted first; then "+
|
||||
"propose this one again", k)
|
||||
}
|
||||
|
||||
// RuleFrom is the day each dated rule began to hold a record, as hq's checks hold it: a record dated earlier was
|
||||
// never held to it, and a dated correction of it (a progressive insight, which keeps the record's date) is not.
|
||||
// words.py held the retired words from 2026-10-07 and the daemons' lines from 2026-10-09; cycle.py the deferrals
|
||||
|
||||
@@ -1,38 +1,119 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// to-be 55 §10: "A whole-set rule that fails is loud — the module's test: an acceptance that leaves a decision
|
||||
// cited by nothing raises the condition." The condition is the mesh's, raised when the module's health answers
|
||||
// unhealthy; the test reads the health answer the node-engine reads.
|
||||
func TestAnAcceptanceThatLeavesADecisionCitedByNothingMakesTheModuleUnhealthy(t *testing.T) {
|
||||
// A decision nothing would cite is not accepted: its review asks changes itself, without a judgement, naming the
|
||||
// rule cited, and the module stays healthy (novox/hq issue 504). Before, the review approved it, the acceptance
|
||||
// broke the rule, and the module turned unhealthy for a record its author could only repair after the fact.
|
||||
func TestADecisionNothingWouldCiteIsNotAccepted(t *testing.T) {
|
||||
eachStore(t, func(t *testing.T, s Store) {
|
||||
w := imported(t, s)
|
||||
if h := w.k.Health(nil); !h.Healthy {
|
||||
t.Fatalf("unhealthy after the import of real records: %s", h.Why)
|
||||
}
|
||||
n := w.reserved(t, "author@g14")
|
||||
w.accepted(t, "author@g14", valid(n))
|
||||
h := w.k.Health(nil)
|
||||
if h.Healthy || h.State != "unhealthy" || !strings.Contains(h.Why, "the rule cited") || !strings.Contains(h.Why, itoaKey(n)) {
|
||||
t.Fatalf("an accepted decision nothing cites left the module %+v", h)
|
||||
p := valid(n)
|
||||
p.Homes = nil
|
||||
if _, err := w.k.Propose("author@g14", p); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// The acceptance stands: the condition says it, and undoes nothing.
|
||||
judged := len(w.judge.reviews)
|
||||
w.reviewAll(t)
|
||||
e := mustGet(t, s, Key{KindDecision, n})
|
||||
if e.Current.Status != StatusProposed {
|
||||
t.Fatalf("a decision nothing cites was %s by its review", e.Current.Status)
|
||||
}
|
||||
rv := e.Current.Reviews
|
||||
if len(rv) == 0 || rv[len(rv)-1].Verdict != VerdictChanges || !strings.Contains(rv[len(rv)-1].Findings, "the rule cited") ||
|
||||
rv[len(rv)-1].By != ReviewBy {
|
||||
t.Fatalf("its review asked no changes for the rule cited: %+v", rv)
|
||||
}
|
||||
if len(w.judge.reviews) != judged {
|
||||
t.Fatal("a judgement was spent on a record the rule cited refuses")
|
||||
}
|
||||
if h := w.k.Health(nil); !h.Healthy {
|
||||
t.Fatalf("unhealthy for a record not accepted: %s", h.Why)
|
||||
}
|
||||
// Named with its home, it is accepted, and the module stays healthy.
|
||||
p.Homes = []string{testHome}
|
||||
w.accepted(t, "author@g14", p)
|
||||
if h := w.k.Health(nil); !h.Healthy {
|
||||
t.Fatalf("unhealthy after a decision with its home was accepted: %s", h.Why)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// A new decision is cited by a record proposed after it: no record can link a number only reserved. Once the
|
||||
// citing record is accepted, the new one proposed again is accepted. A correction of the citing record that drops
|
||||
// the link still leaves the decision cited by nothing, and that is loud: told to the operator once, as a warning
|
||||
// naming the rule and the record, while the module stays healthy and its rollouts are not held (the operator's
|
||||
// decision of 2026-10-12 on novox/hq issue 504; to-be 55 §10). A record that cites it clears the notice.
|
||||
func TestARecordAcceptedCitingANewDecisionLetsItBeAcceptedAndDroppingTheLinkIsLoud(t *testing.T) {
|
||||
eachStore(t, func(t *testing.T, s Store) {
|
||||
w := imported(t, s)
|
||||
n := w.reserved(t, "author@g14")
|
||||
a := valid(n)
|
||||
a.Homes = nil
|
||||
if _, err := w.k.Propose("author@g14", a); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
w.reviewAll(t)
|
||||
m := w.reserved(t, "author@g14")
|
||||
b := valid(m)
|
||||
b.Links = append(b.Links, LinkAsk{Kind: LinkCites, To: itoa4(n)})
|
||||
w.accepted(t, "author@g14", b)
|
||||
w.accepted(t, "author@g14", a) // its revision: an accepted record links it now
|
||||
if h := w.k.Health(nil); !h.Healthy {
|
||||
t.Fatalf("unhealthy with both accepted, one citing the other: %s", h.Why)
|
||||
}
|
||||
w.k.TellFindings()
|
||||
if len(w.told) != 0 {
|
||||
t.Fatalf("told with every rule holding: %v", w.told)
|
||||
}
|
||||
// A correction of the citing record drops its link: the rule, run after the acceptance, says so.
|
||||
b.Links = b.Links[:len(b.Links)-1]
|
||||
b.Insight = "it cited " + itoaKey(n)
|
||||
w.accepted(t, "author@g14", b)
|
||||
h := w.k.Health(nil)
|
||||
if !h.Healthy || !strings.Contains(h.Why, "the rule cited") || !strings.Contains(h.Why, itoaKey(n)) {
|
||||
t.Fatalf("a decision left cited by nothing: the health should stay healthy and name it, not %+v", h)
|
||||
}
|
||||
w.k.TellFindings()
|
||||
w.k.TellFindings()
|
||||
if len(w.told) != 1 || !strings.Contains(w.told[0], "rule cited") || !strings.Contains(w.told[0], itoaKey(n)) ||
|
||||
!strings.HasPrefix(w.toldKeys[0], findingPrefix) {
|
||||
t.Fatalf("the operator is not told once, naming the rule and the record: %v %v", w.told, w.toldKeys)
|
||||
}
|
||||
// A restart tells nothing again: what was told is kept in the store.
|
||||
told := w.toldKeys[0]
|
||||
restarted := newWorld(t, s, fixture())
|
||||
restarted.clock = w.clock
|
||||
restarted.k.TellFindings()
|
||||
if len(restarted.told) != 0 {
|
||||
t.Fatalf("told again after a restart: %v", restarted.told)
|
||||
}
|
||||
w.told = nil
|
||||
if mustGet(t, s, Key{KindDecision, n}).Current.Status != StatusAccepted {
|
||||
t.Fatal("the acceptance was undone")
|
||||
}
|
||||
// A record that cites it heals it.
|
||||
m := w.reserved(t, "author@g14")
|
||||
p := valid(m)
|
||||
p.Links = append(p.Links, LinkAsk{Kind: LinkCites, To: itoa4(n)})
|
||||
w.accepted(t, "author@g14", p)
|
||||
h = w.k.Health(nil)
|
||||
if strings.Contains(h.Why, itoaKey(n)+" is accepted and nothing cites it") {
|
||||
k := w.reserved(t, "author@g14")
|
||||
c := valid(k)
|
||||
c.Links = append(c.Links, LinkAsk{Kind: LinkCites, To: itoa4(n)})
|
||||
w.accepted(t, "author@g14", c)
|
||||
if h := w.k.Health(nil); strings.Contains(h.Why, itoaKey(n)+" is accepted and nothing cites it") {
|
||||
t.Fatalf("still said after a record cited it: %s", h.Why)
|
||||
}
|
||||
w.k.TellFindings()
|
||||
if len(w.cleared) != 1 || w.cleared[0][0] != told {
|
||||
t.Fatalf("the notice is not cleared once the rule holds again: %v", w.cleared)
|
||||
}
|
||||
w.k.TellFindings()
|
||||
if len(w.cleared) != 1 || len(w.told) != 0 {
|
||||
t.Fatalf("after the clear: cleared %v, told %v", w.cleared, w.told)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
@@ -83,3 +164,29 @@ func TestEachWholeSetRuleFails(t *testing.T) {
|
||||
t.Errorf("a consistent set fails: %+v", f)
|
||||
}
|
||||
}
|
||||
|
||||
// A finding's notice the operator's channel refused is not kept as told: it is sent again at the next step.
|
||||
func TestAFindingsNoticeThatFailsIsSentAgain(t *testing.T) {
|
||||
eachStore(t, func(t *testing.T, s Store) {
|
||||
w := newWorld(t, s, fixture())
|
||||
f := Finding{Rule: "cited", Records: []string{"ADR 0300"}, Says: "ADR 0300 is accepted and nothing cites it"}
|
||||
if err := s.SetFindings(context.Background(), []Finding{f}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
tries := 0
|
||||
w.k.Notify = func(key, summary, headline, explanation, resolved string) error {
|
||||
tries++
|
||||
if tries == 1 {
|
||||
return errors.New("nothing answers operator-channel.notify")
|
||||
}
|
||||
w.told = append(w.told, summary)
|
||||
return nil
|
||||
}
|
||||
w.k.TellFindings()
|
||||
w.k.TellFindings()
|
||||
w.k.TellFindings()
|
||||
if tries != 2 || len(w.told) != 1 {
|
||||
t.Fatalf("a refused notice: %d tries, told %v", tries, w.told)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
# mesh-reads
|
||||
|
||||
A node's files and accounts, read through the mesh (novox/hq ADR 0310, extending ADR 0268 §5; hq to-be 41
|
||||
WP8; issue 418). It holds the node seat `node-reads` on every node, the control-node included, so reading a
|
||||
file, searching one, listing a directory, the owner and mode of each component of a path and looking up an
|
||||
account or a group never go through the login shell's `execute`, and an approval is asked for an act, never
|
||||
for a read.
|
||||
|
||||
## Who it reads as
|
||||
|
||||
The account `mesh-reads`, which this module makes on every node (`runs-as`, ADR 0259 §8): no login shell, a
|
||||
home of its own under `/var/lib`, and no group beside its own. The tool runner serves it in a runtime of its
|
||||
own, on its own bus credential, never the machine's runtime, which runs as the operator's account. So the
|
||||
node's file modes bound what it sees: a file only the operator, root or a module's own account may read is
|
||||
refused by its mode, and the answer says so. It refuses every read when it finds itself running as root.
|
||||
|
||||
## What it refuses by name
|
||||
|
||||
Before any read, whatever the file modes would allow (ADR 0310 §5):
|
||||
|
||||
- **The places the mesh keeps a secret on this node**, which the controller composes from the manifests
|
||||
and gives it as `MESH_SECRET_PLACES`: the directory of every module's own secret (its bus credential
|
||||
among them), every credential file and grant directory, the vault's directories, and where the
|
||||
operator-sealed secrets are kept. Without that list the module does not start.
|
||||
- **The known credential files of a machine's programs** (`refuse.go`): an account's `.ssh`, `.gnupg`,
|
||||
`.aws`, `.kube`; the agent's `.credentials.json`, `.netrc`, `.pgpass`, `.git-credentials`; private key
|
||||
names (`id_ed25519`, `*.key`, `ssh_host_*_key`); `/etc/shadow`; `/etc/ssl/private`, `/etc/wireguard`
|
||||
and the network manager's connections.
|
||||
|
||||
A link is judged by where it leads as well as by its name. `stat` and `readlink` say a secret place's owner
|
||||
and mode, and mark it, because those are not its content; `list`, `read` and `search` refuse it.
|
||||
|
||||
## The content rule
|
||||
|
||||
Every answer passes the content rule's secret class, from `mesh-sdk`'s `outward` package (ADR 0310 §4): a
|
||||
line of a file or a matching line shaped like a secret is withheld and its number said, as is a link target
|
||||
or an account's words. A path and an entry's name are not judged: they are what the caller named or what the
|
||||
file system calls a thing, and the rule would take a temporary directory's or a container layer's name for a
|
||||
token.
|
||||
|
||||
## Its verbs and their bounds
|
||||
|
||||
| verb | replaces | bounds |
|
||||
|---|---|---|
|
||||
| `stat` | `stat`, `ls -ld`, `namei -l`, `findmnt -T` | 200 paths, 64 components each; links followed only with `follow` |
|
||||
| `readlink` | `readlink`, `readlink -f`, `realpath` | one path |
|
||||
| `list` | `ls`, `ls -la` | 2000 entries, cut and said |
|
||||
| `read` | `cat`, `head`, `tail`, `less` | 64 KiB, or up to 256 KiB when asked; cut at a line and said; text only |
|
||||
| `search` | `grep`, `grep -r`, `grep -F` | a fixed string; 100 matches (500 asked), 4 deep (8 asked), 5000 files, 4 MiB a file, 20 seconds; the bound met is said |
|
||||
| `account` | `getent passwd`, `getent group`, `id`, `groups` | 2000 accounts and groups; never a password field |
|
||||
|
||||
No answer is larger than 1 MiB. Nothing here writes a file.
|
||||
@@ -0,0 +1,204 @@
|
||||
// mesh-reads (novox/hq ADR 0310, to-be 41 WP8): the holder of node-reads. A node's files and accounts read
|
||||
// through the mesh — the owner and mode of each component of a path, a link, a directory, a text file, a search
|
||||
// by fixed string, an account or a group — so these reads never go through the login shell's execute.
|
||||
//
|
||||
// It runs as an account of its own, mesh-reads, served by the tool runner in a runtime of its own (ADR 0259
|
||||
// §8): in no group that holds a secret, with no login shell and no home of the operator's, so the node's file
|
||||
// modes bound what it sees. On top, it refuses by name the places a secret is kept (refuse.go) and withholds
|
||||
// any piece of an answer shaped like a secret (mesh-sdk's outward). Read-only: nothing here writes a file.
|
||||
//
|
||||
// stdout is the MCP channel; what this module says, it says on stderr.
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
stdio "git.novox.be/novox/mesh-sdk/go"
|
||||
)
|
||||
|
||||
// Seat is the role this module holds.
|
||||
const Seat = "node-reads"
|
||||
|
||||
// PlacesWord is where the controller names this node's secret places (novox/hq ADR 0310 §5).
|
||||
const PlacesWord = "MESH_SECRET_PLACES"
|
||||
|
||||
func main() {
|
||||
places, err := placesFrom(os.Getenv(PlacesWord))
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "[mesh-reads] %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
if err := stdio.Serve("", tools(newReader(places))); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "[mesh-reads] %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
// placesFrom reads the controller's list. Absent is refused: without it, a module's own-secret directory
|
||||
// would be read wherever the account's modes allow, and a refusal the decision promises would be silent.
|
||||
func placesFrom(raw string) ([]string, error) {
|
||||
if strings.TrimSpace(raw) == "" {
|
||||
return nil, fmt.Errorf("%s is not given, so this node's secret places are unknown and nothing would be "+
|
||||
"refused by them (novox/hq ADR 0310 §5): a controller that knows node-reads composes it", PlacesWord)
|
||||
}
|
||||
var places []string
|
||||
if err := json.Unmarshal([]byte(raw), &places); err != nil {
|
||||
return nil, fmt.Errorf("%s is not a list of paths: %v", PlacesWord, err)
|
||||
}
|
||||
return places, nil
|
||||
}
|
||||
|
||||
func str(description string) map[string]any {
|
||||
return map[string]any{"type": "string", "description": description}
|
||||
}
|
||||
|
||||
func arg(a map[string]any, k string) string {
|
||||
switch v := a[k].(type) {
|
||||
case string:
|
||||
return strings.TrimSpace(v)
|
||||
case float64:
|
||||
return strconv.FormatFloat(v, 'f', -1, 64)
|
||||
case bool:
|
||||
return strconv.FormatBool(v)
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func number(a map[string]any, k string) (int, error) {
|
||||
s := arg(a, k)
|
||||
if s == "" {
|
||||
return 0, nil
|
||||
}
|
||||
n, err := strconv.Atoi(s)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("%s is a whole number, and %q was given", k, s)
|
||||
}
|
||||
return n, nil
|
||||
}
|
||||
|
||||
// pathsArg is the paths given, one per line — or, from a caller that sends a list, the list.
|
||||
func pathsArg(a map[string]any) []string {
|
||||
var raw []string
|
||||
switch v := a["paths"].(type) {
|
||||
case string:
|
||||
raw = strings.Split(v, "\n")
|
||||
case []any:
|
||||
for _, p := range v {
|
||||
if s, ok := p.(string); ok {
|
||||
raw = append(raw, s)
|
||||
}
|
||||
}
|
||||
}
|
||||
var out []string
|
||||
for _, p := range raw {
|
||||
if p = strings.TrimSpace(p); p != "" {
|
||||
out = append(out, p)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// bounded is the last bound (novox/hq ADR 0310): whatever a verb made, no answer is larger than MaxAnswer.
|
||||
func bounded(answer any, err error) (any, error) {
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
body, err := json.Marshal(answer)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(body) > MaxAnswer {
|
||||
return nil, fmt.Errorf("the answer is %d bytes, and an answer of node-reads is at most %d: ask for less", len(body), MaxAnswer)
|
||||
}
|
||||
return answer, nil
|
||||
}
|
||||
|
||||
// verb is one of the seat's verbs: listed as `<seat>.<verb>`, so the runtime serves it on the seat's subject.
|
||||
// The descriptions are the controller's (internal/catalogue/reads.go), said again here for a caller of the
|
||||
// module's own address.
|
||||
func verb(name, description string, input map[string]any, run func(a map[string]any) (any, error)) stdio.Tool {
|
||||
return stdio.Tool{Name: Seat + "." + name, Description: description, Input: input,
|
||||
Run: func(a map[string]any) (any, error) { return bounded(run(a)) }}
|
||||
}
|
||||
|
||||
func object(props map[string]any, required ...string) map[string]any {
|
||||
if required == nil {
|
||||
required = []string{}
|
||||
}
|
||||
return map[string]any{"type": "object", "properties": props, "required": required}
|
||||
}
|
||||
|
||||
func tools(r Reader) []stdio.Tool {
|
||||
return []stdio.Tool{
|
||||
verb("stat", "Each component of one path or several, from / down: its type, owner and group (name and "+
|
||||
"number), mode, whether it is a link and to what, and whether it is a mount point and of which file "+
|
||||
"system. A link is not followed unless follow is \"true\". Read as the account mesh-reads, which holds no "+
|
||||
"secret: a component it may not reach says so. At most 200 paths in one call.",
|
||||
object(map[string]any{"paths": str("one absolute path, or several separated by line breaks"),
|
||||
"follow": str("\"true\": also say where the path leads once every link is followed")}, "paths"),
|
||||
func(a map[string]any) (any, error) { return r.Stat(pathsArg(a), arg(a, "follow") == "true") }),
|
||||
verb("readlink", "What one link points to, as written, and where the path leads once every link is "+
|
||||
"followed, and whether that exists.",
|
||||
object(map[string]any{"path": str("an absolute path")}, "path"),
|
||||
func(a map[string]any) (any, error) { return r.Readlink(arg(a, "path")) }),
|
||||
verb("list", "Every entry of one directory with its type, owner, group, mode, size and, for a link, its "+
|
||||
"target. At most 2000 entries; a longer directory is cut and says so. A place a secret is kept is "+
|
||||
"refused by name.",
|
||||
object(map[string]any{"path": str("an absolute path to a directory")}, "path"),
|
||||
func(a map[string]any) (any, error) { return r.List(arg(a, "path")) }),
|
||||
verb("read", "One text file, or a part of it by line: at most 256 KiB in one answer (64 KiB unless more "+
|
||||
"is asked), cut at a line and said so. A negative from counts from the end, as tail does. A binary file "+
|
||||
"is refused; a place a secret is kept, or a known credential file, is refused by name; a line shaped "+
|
||||
"like a secret is withheld and its number said. /proc and /sys are read as far as mesh-reads may.",
|
||||
object(map[string]any{"path": str("an absolute path to a file"),
|
||||
"from": str("the first line to answer, counted from 1; negative: that many lines from the end (optional)"),
|
||||
"lines": str("how many lines to answer (optional)"),
|
||||
"bytes": str("the most bytes to answer, up to 262144 (optional, 65536 unless given)")}, "path"),
|
||||
func(a map[string]any) (any, error) {
|
||||
var ask ReadAsk
|
||||
var err error
|
||||
if ask.From, err = number(a, "from"); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if ask.Lines, err = number(a, "lines"); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if ask.Bytes, err = number(a, "bytes"); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return r.Read(arg(a, "path"), ask)
|
||||
}),
|
||||
verb("search", "Find a fixed string (never a pattern) in one file or under one directory: every matching "+
|
||||
"line with its file and number. Bounded: at most 500 matches (100 unless more is asked), 8 directories "+
|
||||
"deep (4 unless more is asked), 5000 files, 4 MiB per file and 20 seconds; links are not followed and "+
|
||||
"binary files are skipped. A place a secret is kept is skipped and named; a matching line shaped like "+
|
||||
"a secret is withheld. The answer says which bound it met.",
|
||||
object(map[string]any{"text": str("the fixed string to find"),
|
||||
"path": str("an absolute path to a file or a directory"),
|
||||
"depth": str("how many directories deep, up to 8 (optional)"),
|
||||
"matches": str("the most matches to answer, up to 500 (optional)"),
|
||||
"case": str("\"false\": match whatever the case (optional)")}, "text", "path"),
|
||||
func(a map[string]any) (any, error) {
|
||||
text, _ := a["text"].(string) // as given: a space at either end is part of what is found
|
||||
ask := SearchAsk{Text: text, Path: arg(a, "path"), CaseInsensitive: arg(a, "case") == "false"}
|
||||
var err error
|
||||
if ask.Depth, err = number(a, "depth"); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if ask.Matches, err = number(a, "matches"); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return r.Search(ask)
|
||||
}),
|
||||
verb("account", "One account or group by name or number, as this node's account files say: an account's "+
|
||||
"number, group, home, shell and every group it is in; a group's number and members. With neither, every "+
|
||||
"account and group. Never a password field.",
|
||||
object(map[string]any{"name": str("an account's name or number (optional)"),
|
||||
"group": str("a group's name or number (optional)")}),
|
||||
func(a map[string]any) (any, error) { return r.Account(arg(a, "name"), arg(a, "group")) }),
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// ADR 0310 §3: the account the reads run as is the module's own, made by the mesh, with no login shell, no
|
||||
// home of the operator's and no group beside its own, and the module runs as it.
|
||||
func TestTheAccountHoldsNoSecretsGroupAndNoLogin(t *testing.T) {
|
||||
raw, err := os.ReadFile("../../module.json")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var m struct {
|
||||
RunsAs string `json:"runs-as"`
|
||||
SecretsOwner string `json:"secrets-owner"`
|
||||
Claims []map[string]any `json:"claims"`
|
||||
Resources []map[string]any `json:"resources"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &m); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if m.RunsAs != "mesh-reads" || m.SecretsOwner != "mesh-reads" {
|
||||
t.Errorf("runs as %q, secrets owned by %q", m.RunsAs, m.SecretsOwner)
|
||||
}
|
||||
if len(m.Claims) != 1 || m.Claims[0]["name"] != Seat {
|
||||
t.Errorf("claims %v", m.Claims)
|
||||
}
|
||||
found := false
|
||||
for _, r := range m.Resources {
|
||||
if r["type"] != "user" {
|
||||
continue
|
||||
}
|
||||
found = true
|
||||
if r["name"] != m.RunsAs {
|
||||
t.Errorf("a user other than its own: %v", r)
|
||||
}
|
||||
if r["shell"] != "/usr/bin/nologin" {
|
||||
t.Errorf("the account has a login shell: %v", r["shell"])
|
||||
}
|
||||
if home, _ := r["home"].(string); !strings.HasPrefix(home, "/var/lib/") {
|
||||
t.Errorf("the account's home is %q, not one of the mesh's", home)
|
||||
}
|
||||
if _, grouped := r["groups"]; grouped {
|
||||
t.Errorf("the account is put in groups: %v", r["groups"])
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Error("the module makes no account of its own")
|
||||
}
|
||||
for _, r := range m.Resources {
|
||||
if r["type"] != "user" && r["type"] != "directory" {
|
||||
t.Errorf("a read-only holder declares a %v: %v", r["type"], r)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,926 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"io/fs"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
)
|
||||
|
||||
// The bounds (novox/hq ADR 0310, Consequences: set by the build and stated in each verb's description). Every
|
||||
// answer stays under MaxAnswer, which the tools check once more after an answer is made.
|
||||
const (
|
||||
MaxAnswer = 1 << 20 // no answer of this module is larger, whatever the bounds below allow
|
||||
MaxPaths = 200 // paths one stat answers
|
||||
MaxComponents = 64 // components of one path
|
||||
MaxEntries = 2000 // entries one list answers
|
||||
DefaultReadSize = 64 << 10 // bytes a read answers unless more is asked
|
||||
MaxReadSize = 256 << 10 // bytes a read answers at most
|
||||
MaxScan = 64 << 20 // bytes a read passes over to reach the lines asked
|
||||
DefaultMatches = 100
|
||||
MaxMatches = 500
|
||||
DefaultDepth = 4
|
||||
MaxDepth = 8
|
||||
MaxFiles = 5000 // files one search opens
|
||||
MaxFileScan = 4 << 20 // bytes one search reads of one file
|
||||
MaxLine = 300 // characters of one matching line answered
|
||||
SearchTime = 20 * time.Second
|
||||
MaxAccounts = 2000
|
||||
)
|
||||
|
||||
// Reader reads one node's files and accounts as the account it runs as.
|
||||
type Reader struct {
|
||||
Refuser
|
||||
// Passwd, Group and MountInfo are the files accounts, groups and mount points are read from.
|
||||
Passwd, Group, MountInfo string
|
||||
// Euid is the account this runs as; never root (ADR 0310 §3).
|
||||
Euid func() int
|
||||
// Now is the clock a search is bounded by.
|
||||
Now func() time.Time
|
||||
}
|
||||
|
||||
func newReader(places []string) Reader {
|
||||
return Reader{Refuser: Refuser{Places: places}, Passwd: "/etc/passwd", Group: "/etc/group",
|
||||
MountInfo: "/proc/self/mountinfo", Euid: os.Geteuid, Now: time.Now}
|
||||
}
|
||||
|
||||
// asItsOwn refuses every read when this runs as root: what it may see is bounded by its account's file
|
||||
// modes, and root's are none (novox/hq ADR 0310 §3). Loud, never a silent widening.
|
||||
func (r Reader) asItsOwn() error {
|
||||
if r.Euid() == 0 {
|
||||
return errors.New("mesh-reads runs as root here, and its reads are bounded by an account of its own that " +
|
||||
"holds no secret (novox/hq ADR 0310 §3): it answers nothing until it runs as mesh-reads")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func absolute(path string) (string, error) {
|
||||
path = strings.TrimSpace(path)
|
||||
if !strings.HasPrefix(path, "/") {
|
||||
return "", fmt.Errorf("%q is not an absolute path: name it from /", path)
|
||||
}
|
||||
return filepath.Clean(path), nil
|
||||
}
|
||||
|
||||
// why turns a file error into the words an answer says: what stopped the read, never a Go error's text.
|
||||
func why(err error) string {
|
||||
switch {
|
||||
case errors.Is(err, fs.ErrNotExist):
|
||||
return "it does not exist"
|
||||
case errors.Is(err, fs.ErrPermission):
|
||||
return "the account mesh-reads may not reach it (its file modes refuse it)"
|
||||
case errors.Is(err, syscall.ELOOP):
|
||||
return "its links loop"
|
||||
case errors.Is(err, syscall.ENOTDIR):
|
||||
return "a component of it is not a directory"
|
||||
}
|
||||
var pe *fs.PathError
|
||||
if errors.As(err, &pe) {
|
||||
return pe.Err.Error()
|
||||
}
|
||||
return err.Error()
|
||||
}
|
||||
|
||||
// ---- stat ----
|
||||
|
||||
// Component is one component of a path as lstat sees it.
|
||||
type Component struct {
|
||||
Path string `json:"path"`
|
||||
Type string `json:"type,omitempty"`
|
||||
Owner string `json:"owner,omitempty"`
|
||||
UID *int `json:"uid,omitempty"`
|
||||
Group string `json:"group,omitempty"`
|
||||
GID *int `json:"gid,omitempty"`
|
||||
Mode string `json:"mode,omitempty"` // octal, with the setuid, setgid and sticky bits
|
||||
Perms string `json:"perms,omitempty"` // as ls shows them
|
||||
Size *int64 `json:"size,omitempty"`
|
||||
Link string `json:"link,omitempty"` // what a link points to, as written
|
||||
Mount *Mount `json:"mount,omitempty"`
|
||||
Secret string `json:"secret-place,omitempty"` // why a read under it is refused
|
||||
Unreachd string `json:"unreachable,omitempty"`
|
||||
}
|
||||
|
||||
// Mount is a mount point: its file system and source.
|
||||
type Mount struct {
|
||||
FS string `json:"fs"`
|
||||
Source string `json:"source"`
|
||||
}
|
||||
|
||||
// PathStat is one path's components, from / down, and where it leads once every link is followed.
|
||||
type PathStat struct {
|
||||
Path string `json:"path"`
|
||||
Components []Component `json:"components"`
|
||||
LeadsTo string `json:"leads-to,omitempty"`
|
||||
Leads *Component `json:"there,omitempty"`
|
||||
Stopped string `json:"stopped,omitempty"`
|
||||
}
|
||||
|
||||
// StatAnswer is what stat answers.
|
||||
type StatAnswer struct {
|
||||
Paths []PathStat `json:"paths"`
|
||||
Withheld []Withheld `json:"withheld,omitempty"`
|
||||
}
|
||||
|
||||
func (r Reader) Stat(paths []string, follow bool) (StatAnswer, error) {
|
||||
if err := r.asItsOwn(); err != nil {
|
||||
return StatAnswer{}, err
|
||||
}
|
||||
if len(paths) == 0 {
|
||||
return StatAnswer{}, errors.New("name a path to stat")
|
||||
}
|
||||
if len(paths) > MaxPaths {
|
||||
return StatAnswer{}, fmt.Errorf("%d paths asked, and one stat answers at most %d: ask in parts", len(paths), MaxPaths)
|
||||
}
|
||||
names := r.names()
|
||||
mounts := r.mounts()
|
||||
var out StatAnswer
|
||||
for _, raw := range paths {
|
||||
path, err := absolute(raw)
|
||||
if err != nil {
|
||||
return StatAnswer{}, err
|
||||
}
|
||||
ps := PathStat{Path: path}
|
||||
walk := []string{"/"}
|
||||
acc := ""
|
||||
for _, part := range strings.Split(strings.TrimPrefix(path, "/"), "/") {
|
||||
if part == "" {
|
||||
continue
|
||||
}
|
||||
acc += "/" + part
|
||||
walk = append(walk, acc)
|
||||
}
|
||||
if len(walk) > MaxComponents {
|
||||
return StatAnswer{}, fmt.Errorf("%s has %d components, and a stat answers at most %d", path, len(walk), MaxComponents)
|
||||
}
|
||||
for _, p := range walk {
|
||||
c := r.component(p, names, mounts)
|
||||
ps.Components = append(ps.Components, c)
|
||||
if c.Unreachd != "" {
|
||||
ps.Stopped = c.Path + ": " + c.Unreachd
|
||||
break
|
||||
}
|
||||
}
|
||||
if follow && ps.Stopped == "" {
|
||||
if leads, err := filepath.EvalSymlinks(path); err != nil {
|
||||
ps.Stopped = "following its links: " + why(err)
|
||||
} else {
|
||||
ps.LeadsTo = leads
|
||||
if leads != path {
|
||||
c := r.component(leads, names, mounts)
|
||||
ps.Leads = &c
|
||||
}
|
||||
}
|
||||
}
|
||||
out.Paths = append(out.Paths, ps)
|
||||
}
|
||||
out.Withheld = cleanStat(&out)
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (r Reader) component(p string, names accounts, mounts map[string]Mount) Component {
|
||||
c := Component{Path: p}
|
||||
info, err := os.Lstat(p)
|
||||
if err != nil {
|
||||
c.Unreachd = why(err)
|
||||
return c
|
||||
}
|
||||
c.Type = typeOf(info.Mode())
|
||||
c.Perms = info.Mode().String()
|
||||
if st, ok := info.Sys().(*syscall.Stat_t); ok {
|
||||
uid, gid := int(st.Uid), int(st.Gid)
|
||||
c.UID, c.GID = &uid, &gid
|
||||
c.Owner, c.Group = names.user(uid), names.group(gid)
|
||||
c.Mode = fmt.Sprintf("%04o", st.Mode&07777)
|
||||
}
|
||||
if info.Mode().IsRegular() {
|
||||
size := info.Size()
|
||||
c.Size = &size
|
||||
}
|
||||
if info.Mode()&fs.ModeSymlink != 0 {
|
||||
if target, err := os.Readlink(p); err == nil {
|
||||
c.Link = target
|
||||
}
|
||||
}
|
||||
if m, ok := mounts[p]; ok {
|
||||
c.Mount = &m
|
||||
}
|
||||
if why := r.Refused(p); why != "" {
|
||||
c.Secret = why
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
func typeOf(m fs.FileMode) string {
|
||||
switch {
|
||||
case m.IsDir():
|
||||
return "directory"
|
||||
case m&fs.ModeSymlink != 0:
|
||||
return "link"
|
||||
case m.IsRegular():
|
||||
return "file"
|
||||
case m&fs.ModeNamedPipe != 0:
|
||||
return "fifo"
|
||||
case m&fs.ModeSocket != 0:
|
||||
return "socket"
|
||||
case m&fs.ModeCharDevice != 0:
|
||||
return "character device"
|
||||
case m&fs.ModeDevice != 0:
|
||||
return "block device"
|
||||
}
|
||||
return "other"
|
||||
}
|
||||
|
||||
func cleanStat(a *StatAnswer) []Withheld {
|
||||
var w []Withheld
|
||||
cleanC := func(c *Component) {
|
||||
for _, f := range []*string{&c.Link} {
|
||||
if s, what := clean(*f); what != "" {
|
||||
w = append(w, Withheld{At: "a link's target", Class: what})
|
||||
*f = s
|
||||
}
|
||||
}
|
||||
if c.Mount != nil {
|
||||
if s, what := clean(c.Mount.Source); what != "" {
|
||||
w = append(w, Withheld{At: "a mount's source", Class: what})
|
||||
c.Mount.Source = s
|
||||
}
|
||||
}
|
||||
}
|
||||
for i := range a.Paths {
|
||||
p := &a.Paths[i]
|
||||
for j := range p.Components {
|
||||
cleanC(&p.Components[j])
|
||||
}
|
||||
if p.Leads != nil {
|
||||
cleanC(p.Leads)
|
||||
}
|
||||
}
|
||||
return w
|
||||
}
|
||||
|
||||
// ---- readlink ----
|
||||
|
||||
// LinkAnswer is what readlink answers.
|
||||
type LinkAnswer struct {
|
||||
Path string `json:"path"`
|
||||
Link string `json:"link,omitempty"` // as written; empty when the path is not a link
|
||||
IsLink bool `json:"is-link"`
|
||||
LeadsTo string `json:"leads-to,omitempty"`
|
||||
Exists bool `json:"exists"`
|
||||
Stopped string `json:"stopped,omitempty"`
|
||||
Withheld []Withheld `json:"withheld,omitempty"`
|
||||
}
|
||||
|
||||
func (r Reader) Readlink(raw string) (LinkAnswer, error) {
|
||||
if err := r.asItsOwn(); err != nil {
|
||||
return LinkAnswer{}, err
|
||||
}
|
||||
path, err := absolute(raw)
|
||||
if err != nil {
|
||||
return LinkAnswer{}, err
|
||||
}
|
||||
a := LinkAnswer{Path: path}
|
||||
info, err := os.Lstat(path)
|
||||
if err != nil {
|
||||
a.Stopped = why(err)
|
||||
return a, nil
|
||||
}
|
||||
if info.Mode()&fs.ModeSymlink != 0 {
|
||||
a.IsLink = true
|
||||
if a.Link, err = os.Readlink(path); err != nil {
|
||||
a.Stopped = why(err)
|
||||
}
|
||||
}
|
||||
if leads, err := filepath.EvalSymlinks(path); err == nil {
|
||||
a.LeadsTo, a.Exists = leads, true
|
||||
} else if a.Stopped == "" {
|
||||
a.Stopped = "following its links: " + why(err)
|
||||
}
|
||||
for _, f := range []*string{&a.Link} {
|
||||
if s, what := clean(*f); what != "" {
|
||||
a.Withheld = append(a.Withheld, Withheld{At: "a link's target", Class: what})
|
||||
*f = s
|
||||
}
|
||||
}
|
||||
return a, nil
|
||||
}
|
||||
|
||||
// ---- list ----
|
||||
|
||||
// Entry is one entry of a directory.
|
||||
type Entry struct {
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"`
|
||||
Owner string `json:"owner,omitempty"`
|
||||
Group string `json:"group,omitempty"`
|
||||
Mode string `json:"mode,omitempty"`
|
||||
Size *int64 `json:"size,omitempty"`
|
||||
Link string `json:"link,omitempty"`
|
||||
}
|
||||
|
||||
// ListAnswer is what list answers.
|
||||
type ListAnswer struct {
|
||||
Path string `json:"path"`
|
||||
Entries []Entry `json:"entries"`
|
||||
Count int `json:"count"`
|
||||
Cut string `json:"cut,omitempty"`
|
||||
Withheld []Withheld `json:"withheld,omitempty"`
|
||||
}
|
||||
|
||||
// refusedRead is a refusal by name, before any read: the path asked and where it leads are both judged, so a
|
||||
// link to a secret place is refused as the place is.
|
||||
func (r Reader) refusedRead(path string) error {
|
||||
if why := r.Refused(path); why != "" {
|
||||
return fmt.Errorf("%s is refused by name: %s (novox/hq ADR 0310 §5)", path, why)
|
||||
}
|
||||
if leads, err := filepath.EvalSymlinks(path); err == nil && leads != path {
|
||||
if why := r.Refused(leads); why != "" {
|
||||
return fmt.Errorf("%s leads to %s, which is refused by name: %s (novox/hq ADR 0310 §5)", path, leads, why)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (r Reader) List(raw string) (ListAnswer, error) {
|
||||
if err := r.asItsOwn(); err != nil {
|
||||
return ListAnswer{}, err
|
||||
}
|
||||
path, err := absolute(raw)
|
||||
if err != nil {
|
||||
return ListAnswer{}, err
|
||||
}
|
||||
if err := r.refusedRead(path); err != nil {
|
||||
return ListAnswer{}, err
|
||||
}
|
||||
entries, err := os.ReadDir(path)
|
||||
if err != nil {
|
||||
return ListAnswer{}, fmt.Errorf("%s cannot be listed: %s", path, why(err))
|
||||
}
|
||||
names := r.names()
|
||||
a := ListAnswer{Path: path, Count: len(entries), Entries: []Entry{}}
|
||||
if len(entries) > MaxEntries {
|
||||
a.Cut = fmt.Sprintf("cut at %d of %d entries (the bound of one list), in name order", MaxEntries, len(entries))
|
||||
entries = entries[:MaxEntries]
|
||||
}
|
||||
for _, e := range entries {
|
||||
en := Entry{Name: e.Name(), Type: typeOf(e.Type())}
|
||||
if info, err := e.Info(); err == nil {
|
||||
en.Type = typeOf(info.Mode())
|
||||
if st, ok := info.Sys().(*syscall.Stat_t); ok {
|
||||
en.Owner, en.Group = names.user(int(st.Uid)), names.group(int(st.Gid))
|
||||
en.Mode = fmt.Sprintf("%04o", st.Mode&07777)
|
||||
}
|
||||
if info.Mode().IsRegular() {
|
||||
size := info.Size()
|
||||
en.Size = &size
|
||||
}
|
||||
if info.Mode()&fs.ModeSymlink != 0 {
|
||||
en.Link, _ = os.Readlink(filepath.Join(path, e.Name()))
|
||||
}
|
||||
}
|
||||
for _, f := range []*string{&en.Link} {
|
||||
if s, what := clean(*f); what != "" {
|
||||
a.Withheld = append(a.Withheld, Withheld{At: "a link's target", Class: what})
|
||||
*f = s
|
||||
}
|
||||
}
|
||||
a.Entries = append(a.Entries, en)
|
||||
}
|
||||
return a, nil
|
||||
}
|
||||
|
||||
// ---- read ----
|
||||
|
||||
// ReadAnswer is what read answers.
|
||||
type ReadAnswer struct {
|
||||
Path string `json:"path"`
|
||||
Size int64 `json:"size"` // as the file says; 0 for most of /proc and /sys
|
||||
From int `json:"from"` // the first line answered, counted from 1
|
||||
Lines int `json:"lines"`
|
||||
Text string `json:"text"`
|
||||
Cut string `json:"cut,omitempty"`
|
||||
Withheld []int `json:"withheld-lines,omitempty"`
|
||||
Why string `json:"withheld-because,omitempty"`
|
||||
}
|
||||
|
||||
// ReadAsk is what a read asks: the lines from From (1 is the first; a negative From counts from the end, as
|
||||
// tail does), at most Lines of them (0: as many as fit), and at most Bytes bytes.
|
||||
type ReadAsk struct {
|
||||
From, Lines, Bytes int
|
||||
}
|
||||
|
||||
func (r Reader) Read(raw string, ask ReadAsk) (ReadAnswer, error) {
|
||||
if err := r.asItsOwn(); err != nil {
|
||||
return ReadAnswer{}, err
|
||||
}
|
||||
path, err := absolute(raw)
|
||||
if err != nil {
|
||||
return ReadAnswer{}, err
|
||||
}
|
||||
if err := r.refusedRead(path); err != nil {
|
||||
return ReadAnswer{}, err
|
||||
}
|
||||
bound := ask.Bytes
|
||||
switch {
|
||||
case bound <= 0:
|
||||
bound = DefaultReadSize
|
||||
case bound > MaxReadSize:
|
||||
return ReadAnswer{}, fmt.Errorf("%d bytes asked, and one read answers at most %d", bound, MaxReadSize)
|
||||
}
|
||||
if ask.Lines < 0 {
|
||||
return ReadAnswer{}, errors.New("lines is a count: name how many, or none for as many as fit")
|
||||
}
|
||||
// Non-blocking, so a fifo answers rather than waits; and judged by what was opened, not by its name.
|
||||
f, err := os.OpenFile(path, os.O_RDONLY|syscall.O_NONBLOCK, 0)
|
||||
if err != nil {
|
||||
return ReadAnswer{}, fmt.Errorf("%s cannot be read: %s", path, why(err))
|
||||
}
|
||||
defer f.Close()
|
||||
info, err := f.Stat()
|
||||
if err != nil {
|
||||
return ReadAnswer{}, fmt.Errorf("%s cannot be read: %s", path, why(err))
|
||||
}
|
||||
if !info.Mode().IsRegular() {
|
||||
return ReadAnswer{}, fmt.Errorf("%s is a %s, not a file: list a directory, stat anything else", path, typeOf(info.Mode()))
|
||||
}
|
||||
a := ReadAnswer{Path: path, Size: info.Size()}
|
||||
lines, cut, err := pick(io.LimitReader(f, MaxScan), ask, bound)
|
||||
if err != nil {
|
||||
return ReadAnswer{}, fmt.Errorf("%s: %v", path, err)
|
||||
}
|
||||
a.Cut = cut
|
||||
if len(lines) > 0 {
|
||||
a.From = lines[0].n
|
||||
}
|
||||
var text strings.Builder
|
||||
for _, l := range lines {
|
||||
s, what := clean(l.text)
|
||||
if what != "" {
|
||||
a.Withheld = append(a.Withheld, l.n)
|
||||
a.Why = "a line shaped like a secret is withheld by the content rule (novox/hq ADR 0310 §4)"
|
||||
}
|
||||
text.WriteString(s)
|
||||
if l.newline {
|
||||
text.WriteByte('\n')
|
||||
}
|
||||
}
|
||||
a.Text, a.Lines = text.String(), len(lines)
|
||||
return a, nil
|
||||
}
|
||||
|
||||
type line struct {
|
||||
n int
|
||||
text string
|
||||
newline bool
|
||||
size int
|
||||
}
|
||||
|
||||
// pick reads lines and keeps the ones asked, within bound bytes. A NUL byte is a binary file, refused.
|
||||
func pick(in io.Reader, ask ReadAsk, bound int) ([]line, string, error) {
|
||||
br := bufio.NewReaderSize(in, 64<<10)
|
||||
var kept []line
|
||||
size, n := 0, 0
|
||||
from := ask.From
|
||||
if from == 0 {
|
||||
from = 1
|
||||
}
|
||||
for {
|
||||
raw, err := br.ReadString('\n')
|
||||
if raw == "" && err != nil {
|
||||
if err == io.EOF {
|
||||
break
|
||||
}
|
||||
return nil, "", err
|
||||
}
|
||||
n++
|
||||
if strings.IndexByte(raw, 0) >= 0 {
|
||||
return nil, "", errors.New("it is a binary file, and read answers text")
|
||||
}
|
||||
l := line{n: n, text: strings.TrimSuffix(raw, "\n"), newline: strings.HasSuffix(raw, "\n"), size: len(raw)}
|
||||
if from > 0 {
|
||||
if n < from {
|
||||
continue
|
||||
}
|
||||
if ask.Lines > 0 && len(kept) == ask.Lines {
|
||||
break
|
||||
}
|
||||
if size+len(raw) > bound {
|
||||
return kept, fmt.Sprintf("cut at line %d: the answer holds at most %d bytes", n, bound), nil
|
||||
}
|
||||
kept = append(kept, l)
|
||||
size += len(raw)
|
||||
} else {
|
||||
// From the end, as tail does: the last -from lines, kept as they pass.
|
||||
kept = append(kept, l)
|
||||
size += len(raw)
|
||||
for len(kept) > -from || size > bound {
|
||||
size -= kept[0].size
|
||||
kept = kept[1:]
|
||||
}
|
||||
}
|
||||
if err == io.EOF {
|
||||
break
|
||||
}
|
||||
}
|
||||
if from > 0 && n < from && n > 0 {
|
||||
return nil, "", fmt.Errorf("it has %d lines, and line %d was asked", n, from)
|
||||
}
|
||||
if from < 0 && ask.Lines > 0 && len(kept) > ask.Lines {
|
||||
kept = kept[:ask.Lines]
|
||||
}
|
||||
return kept, "", nil
|
||||
}
|
||||
|
||||
// ---- search ----
|
||||
|
||||
// Match is one matching line.
|
||||
type Match struct {
|
||||
File string `json:"file"`
|
||||
Line int `json:"line"`
|
||||
Text string `json:"text"`
|
||||
}
|
||||
|
||||
// SearchAnswer is what search answers.
|
||||
type SearchAnswer struct {
|
||||
Text string `json:"text"`
|
||||
Path string `json:"path"`
|
||||
Matches []Match `json:"matches"`
|
||||
Files int `json:"files-read"`
|
||||
Bound string `json:"bound-met,omitempty"`
|
||||
Skipped []string `json:"refused-by-name,omitempty"`
|
||||
Unreadable int `json:"unreadable,omitempty"`
|
||||
Binary int `json:"binary-skipped,omitempty"`
|
||||
Withheld []Withheld `json:"withheld,omitempty"`
|
||||
}
|
||||
|
||||
// SearchAsk is what a search asks.
|
||||
type SearchAsk struct {
|
||||
Text, Path string
|
||||
Depth, Matches int
|
||||
CaseInsensitive bool
|
||||
}
|
||||
|
||||
var errBound = errors.New("bound met")
|
||||
|
||||
func (r Reader) Search(ask SearchAsk) (SearchAnswer, error) {
|
||||
if err := r.asItsOwn(); err != nil {
|
||||
return SearchAnswer{}, err
|
||||
}
|
||||
if ask.Text == "" {
|
||||
return SearchAnswer{}, errors.New("name the text to find")
|
||||
}
|
||||
if strings.ContainsAny(ask.Text, "\n\x00") {
|
||||
return SearchAnswer{}, errors.New("the text is found within one line: it may not hold a line break")
|
||||
}
|
||||
root, err := absolute(ask.Path)
|
||||
if err != nil {
|
||||
return SearchAnswer{}, err
|
||||
}
|
||||
if err := r.refusedRead(root); err != nil {
|
||||
return SearchAnswer{}, err
|
||||
}
|
||||
depth, matches := ask.Depth, ask.Matches
|
||||
switch {
|
||||
case depth == 0:
|
||||
depth = DefaultDepth
|
||||
case depth < 0 || depth > MaxDepth:
|
||||
return SearchAnswer{}, fmt.Errorf("depth %d asked; a search goes at most %d directories deep", depth, MaxDepth)
|
||||
}
|
||||
switch {
|
||||
case matches == 0:
|
||||
matches = DefaultMatches
|
||||
case matches < 0 || matches > MaxMatches:
|
||||
return SearchAnswer{}, fmt.Errorf("%d matches asked; a search answers at most %d", matches, MaxMatches)
|
||||
}
|
||||
needle := ask.Text
|
||||
if ask.CaseInsensitive {
|
||||
needle = strings.ToLower(needle)
|
||||
}
|
||||
a := SearchAnswer{Text: ask.Text, Path: root, Matches: []Match{}}
|
||||
deadline := r.Now().Add(SearchTime)
|
||||
base := strings.Count(root, "/")
|
||||
if root == "/" {
|
||||
base = 0
|
||||
}
|
||||
err = filepath.WalkDir(root, func(p string, d fs.DirEntry, err error) error {
|
||||
if r.Now().After(deadline) {
|
||||
a.Bound = fmt.Sprintf("time: a search runs at most %s", SearchTime)
|
||||
return errBound
|
||||
}
|
||||
if err != nil {
|
||||
a.Unreadable++
|
||||
if d != nil && d.IsDir() {
|
||||
return fs.SkipDir
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if p != root {
|
||||
if why := r.Refused(p); why != "" {
|
||||
a.Skipped = append(a.Skipped, p)
|
||||
if d.IsDir() {
|
||||
return fs.SkipDir
|
||||
}
|
||||
return nil
|
||||
}
|
||||
}
|
||||
if d.IsDir() {
|
||||
if p != root && strings.Count(p, "/")-base > depth {
|
||||
return fs.SkipDir
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if !d.Type().IsRegular() {
|
||||
return nil // links are not followed; a device, a fifo or a socket is not text
|
||||
}
|
||||
if a.Files >= MaxFiles {
|
||||
a.Bound = fmt.Sprintf("files: a search reads at most %d files", MaxFiles)
|
||||
return errBound
|
||||
}
|
||||
a.Files++
|
||||
return r.searchFile(p, needle, ask.CaseInsensitive, matches, &a)
|
||||
})
|
||||
if err != nil && !errors.Is(err, errBound) {
|
||||
return SearchAnswer{}, fmt.Errorf("%s cannot be searched: %s", root, why(err))
|
||||
}
|
||||
return a, nil
|
||||
}
|
||||
|
||||
func (r Reader) searchFile(p, needle string, fold bool, bound int, a *SearchAnswer) error {
|
||||
f, err := os.OpenFile(p, os.O_RDONLY|syscall.O_NONBLOCK, 0)
|
||||
if err != nil {
|
||||
a.Unreadable++
|
||||
return nil
|
||||
}
|
||||
defer f.Close()
|
||||
br := bufio.NewReaderSize(io.LimitReader(f, MaxFileScan), 64<<10)
|
||||
if head, _ := br.Peek(8 << 10); bytes.IndexByte(head, 0) >= 0 {
|
||||
a.Binary++
|
||||
return nil
|
||||
}
|
||||
n := 0
|
||||
for {
|
||||
raw, err := br.ReadString('\n')
|
||||
if raw == "" && err != nil {
|
||||
return nil
|
||||
}
|
||||
n++
|
||||
hay := raw
|
||||
if fold {
|
||||
hay = strings.ToLower(hay)
|
||||
}
|
||||
if strings.Contains(hay, needle) {
|
||||
if len(a.Matches) >= bound {
|
||||
a.Bound = fmt.Sprintf("matches: %d answered, as asked or the default", bound)
|
||||
return errBound
|
||||
}
|
||||
text := strings.TrimRight(raw, "\r\n")
|
||||
if r := []rune(text); len(r) > MaxLine {
|
||||
text = string(r[:MaxLine]) + "…"
|
||||
}
|
||||
file := p
|
||||
for _, f := range []*string{&text} {
|
||||
if s, what := clean(*f); what != "" {
|
||||
a.Withheld = append(a.Withheld, Withheld{At: "a matching line", Class: what})
|
||||
*f = s
|
||||
}
|
||||
}
|
||||
a.Matches = append(a.Matches, Match{File: file, Line: n, Text: text})
|
||||
}
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ---- accounts ----
|
||||
|
||||
// Account is one account as the account file says it, never its password field.
|
||||
type Account struct {
|
||||
Name string `json:"name"`
|
||||
UID int `json:"uid"`
|
||||
GID int `json:"gid"`
|
||||
Group string `json:"group,omitempty"`
|
||||
Gecos string `json:"gecos,omitempty"`
|
||||
Home string `json:"home"`
|
||||
Shell string `json:"shell"`
|
||||
Groups []string `json:"groups,omitempty"`
|
||||
}
|
||||
|
||||
// GroupEntry is one group.
|
||||
type GroupEntry struct {
|
||||
Name string `json:"name"`
|
||||
GID int `json:"gid"`
|
||||
Members []string `json:"members"`
|
||||
}
|
||||
|
||||
// AccountAnswer is what account answers.
|
||||
type AccountAnswer struct {
|
||||
Accounts []Account `json:"accounts,omitempty"`
|
||||
Groups []GroupEntry `json:"groups,omitempty"`
|
||||
Source string `json:"source"`
|
||||
Cut string `json:"cut,omitempty"`
|
||||
Withheld []Withheld `json:"withheld,omitempty"`
|
||||
}
|
||||
|
||||
type accounts struct {
|
||||
users []Account
|
||||
groups []GroupEntry
|
||||
}
|
||||
|
||||
func (a accounts) user(uid int) string {
|
||||
for _, u := range a.users {
|
||||
if u.UID == uid {
|
||||
return u.Name
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (a accounts) group(gid int) string {
|
||||
for _, g := range a.groups {
|
||||
if g.GID == gid {
|
||||
return g.Name
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// names reads the account files; a file it cannot read gives no names, and a component says only numbers.
|
||||
func (r Reader) names() accounts {
|
||||
var a accounts
|
||||
if raw, err := os.ReadFile(r.Passwd); err == nil {
|
||||
for _, l := range strings.Split(string(raw), "\n") {
|
||||
f := strings.Split(l, ":")
|
||||
if len(f) < 7 || strings.HasPrefix(l, "#") {
|
||||
continue
|
||||
}
|
||||
uid, e1 := strconv.Atoi(f[2])
|
||||
gid, e2 := strconv.Atoi(f[3])
|
||||
if e1 != nil || e2 != nil {
|
||||
continue
|
||||
}
|
||||
a.users = append(a.users, Account{Name: f[0], UID: uid, GID: gid, Gecos: f[4], Home: f[5], Shell: f[6]})
|
||||
}
|
||||
}
|
||||
if raw, err := os.ReadFile(r.Group); err == nil {
|
||||
for _, l := range strings.Split(string(raw), "\n") {
|
||||
f := strings.Split(l, ":")
|
||||
if len(f) < 4 || strings.HasPrefix(l, "#") {
|
||||
continue
|
||||
}
|
||||
gid, err := strconv.Atoi(f[2])
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
members := []string{}
|
||||
for _, m := range strings.Split(f[3], ",") {
|
||||
if m = strings.TrimSpace(m); m != "" {
|
||||
members = append(members, m)
|
||||
}
|
||||
}
|
||||
a.groups = append(a.groups, GroupEntry{Name: f[0], GID: gid, Members: members})
|
||||
}
|
||||
}
|
||||
return a
|
||||
}
|
||||
|
||||
func (r Reader) Account(name, group string) (AccountAnswer, error) {
|
||||
if err := r.asItsOwn(); err != nil {
|
||||
return AccountAnswer{}, err
|
||||
}
|
||||
all := r.names()
|
||||
if len(all.users) == 0 {
|
||||
return AccountAnswer{}, fmt.Errorf("%s gave no account: it is unreadable or empty", r.Passwd)
|
||||
}
|
||||
out := AccountAnswer{Source: r.Passwd + ", " + r.Group + " (an account kept elsewhere, such as a directory " +
|
||||
"service or systemd's dynamic accounts, is not here)"}
|
||||
complete := func(u Account) Account {
|
||||
u.Group = all.group(u.GID)
|
||||
for _, g := range all.groups {
|
||||
if g.GID == u.GID {
|
||||
u.Groups = append(u.Groups, g.Name)
|
||||
continue
|
||||
}
|
||||
for _, m := range g.Members {
|
||||
if m == u.Name {
|
||||
u.Groups = append(u.Groups, g.Name)
|
||||
}
|
||||
}
|
||||
}
|
||||
return u
|
||||
}
|
||||
name, group = strings.TrimSpace(name), strings.TrimSpace(group)
|
||||
switch {
|
||||
case name != "":
|
||||
for _, u := range all.users {
|
||||
if u.Name == name || strconv.Itoa(u.UID) == name {
|
||||
out.Accounts = append(out.Accounts, complete(u))
|
||||
}
|
||||
}
|
||||
if len(out.Accounts) == 0 {
|
||||
return AccountAnswer{}, fmt.Errorf("no account %q on this node", name)
|
||||
}
|
||||
case group != "":
|
||||
for _, g := range all.groups {
|
||||
if g.Name == group || strconv.Itoa(g.GID) == group {
|
||||
// Its members as getent says them, and every account whose own group it is.
|
||||
for _, u := range all.users {
|
||||
if u.GID == g.GID && !contains(g.Members, u.Name) {
|
||||
g.Members = append(g.Members, u.Name)
|
||||
}
|
||||
}
|
||||
out.Groups = append(out.Groups, g)
|
||||
}
|
||||
}
|
||||
if len(out.Groups) == 0 {
|
||||
return AccountAnswer{}, fmt.Errorf("no group %q on this node", group)
|
||||
}
|
||||
default:
|
||||
for _, u := range all.users {
|
||||
out.Accounts = append(out.Accounts, complete(u))
|
||||
}
|
||||
out.Groups = all.groups
|
||||
if len(out.Accounts) > MaxAccounts {
|
||||
out.Cut = fmt.Sprintf("cut at %d of %d accounts", MaxAccounts, len(out.Accounts))
|
||||
out.Accounts = out.Accounts[:MaxAccounts]
|
||||
}
|
||||
if len(out.Groups) > MaxAccounts {
|
||||
out.Cut = strings.TrimPrefix(out.Cut+"; ", "; ") + fmt.Sprintf("cut at %d of %d groups", MaxAccounts, len(out.Groups))
|
||||
out.Groups = out.Groups[:MaxAccounts]
|
||||
}
|
||||
}
|
||||
for i := range out.Accounts {
|
||||
u := &out.Accounts[i]
|
||||
for _, f := range []*string{&u.Gecos, &u.Home, &u.Shell} {
|
||||
if s, what := clean(*f); what != "" {
|
||||
out.Withheld = append(out.Withheld, Withheld{At: "account " + u.Name, Class: what})
|
||||
*f = s
|
||||
}
|
||||
}
|
||||
}
|
||||
sort.SliceStable(out.Accounts, func(i, j int) bool { return out.Accounts[i].UID < out.Accounts[j].UID })
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func contains(list []string, s string) bool {
|
||||
for _, x := range list {
|
||||
if x == s {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// ---- mounts ----
|
||||
|
||||
// mounts reads the mount points this process sees, by path. /proc/self/mountinfo is readable by every account.
|
||||
func (r Reader) mounts() map[string]Mount {
|
||||
out := map[string]Mount{}
|
||||
raw, err := os.ReadFile(r.MountInfo)
|
||||
if err != nil {
|
||||
return out
|
||||
}
|
||||
for _, l := range strings.Split(string(raw), "\n") {
|
||||
f := strings.Fields(l)
|
||||
dash := -1
|
||||
for i, x := range f {
|
||||
if x == "-" {
|
||||
dash = i
|
||||
break
|
||||
}
|
||||
}
|
||||
if len(f) < 5 || dash < 0 || dash+2 >= len(f) {
|
||||
continue
|
||||
}
|
||||
out[unescape(f[4])] = Mount{FS: f[dash+1], Source: unescape(f[dash+2])}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// unescape reads mountinfo's octal escapes (\040 for a space).
|
||||
func unescape(s string) string {
|
||||
if !strings.Contains(s, `\`) {
|
||||
return s
|
||||
}
|
||||
var b strings.Builder
|
||||
for i := 0; i < len(s); i++ {
|
||||
if s[i] == '\\' && i+3 < len(s) {
|
||||
if v, err := strconv.ParseUint(s[i+1:i+4], 8, 8); err == nil {
|
||||
b.WriteByte(byte(v))
|
||||
i += 3
|
||||
continue
|
||||
}
|
||||
}
|
||||
b.WriteByte(s[i])
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
@@ -0,0 +1,452 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"syscall"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// A token shaped as a secret, written into files the tests read.
|
||||
const aToken = "ghp_abcdefghijklmnopqrstuvwxyz0123456789"
|
||||
|
||||
func testReader(t *testing.T, places ...string) (Reader, string) {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
passwd := filepath.Join(dir, "passwd")
|
||||
group := filepath.Join(dir, "group")
|
||||
mountinfo := filepath.Join(dir, "mountinfo")
|
||||
write(t, passwd, "root:x:0:0::/root:/usr/bin/bash\nops:x:1000:1000:The Operator:/home/ops:/usr/bin/zsh\n"+
|
||||
"mesh-reads:x:961:961::/var/lib/mesh-reads:/usr/bin/nologin\n")
|
||||
write(t, group, "root:x:0:\nwheel:x:998:ops\nops:x:1000:\nmesh-reads:x:961:\ndocker:x:970:ops,builder\n")
|
||||
data := filepath.Join(dir, "data")
|
||||
write(t, mountinfo, "22 1 0:21 / / rw,relatime - btrfs /dev/nvme0n1p2 rw\n"+
|
||||
"40 22 0:40 / "+strings.ReplaceAll(data, " ", `\040`)+" rw - nfs4 home-server:/tank/data rw\n")
|
||||
r := Reader{Refuser: Refuser{Places: places}, Passwd: passwd, Group: group, MountInfo: mountinfo,
|
||||
Euid: func() int { return 961 }, Now: time.Now}
|
||||
return r, dir
|
||||
}
|
||||
|
||||
func write(t *testing.T, path, content string) {
|
||||
t.Helper()
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// ADR 0310 §3: run as root, every read is refused, loudly — its bound is an account of its own.
|
||||
func TestEveryReadIsRefusedAsRoot(t *testing.T) {
|
||||
r, dir := testReader(t)
|
||||
r.Euid = func() int { return 0 }
|
||||
write(t, filepath.Join(dir, "f"), "x\n")
|
||||
if _, err := r.Read(filepath.Join(dir, "f"), ReadAsk{}); err == nil || !strings.Contains(err.Error(), "root") {
|
||||
t.Errorf("a read as root: %v", err)
|
||||
}
|
||||
if _, err := r.Stat([]string{dir}, false); err == nil {
|
||||
t.Error("a stat as root was answered")
|
||||
}
|
||||
if _, err := r.Account("ops", ""); err == nil {
|
||||
t.Error("an account lookup as root was answered")
|
||||
}
|
||||
}
|
||||
|
||||
// ADR 0310 §5: without the controller's list of secret places it does not start: a refusal it promises would be
|
||||
// silent.
|
||||
func TestItDoesNotStartWithoutTheNodesSecretPlaces(t *testing.T) {
|
||||
if _, err := placesFrom(""); err == nil || !strings.Contains(err.Error(), PlacesWord) {
|
||||
t.Errorf("no places: %v", err)
|
||||
}
|
||||
if p, err := placesFrom(`["/var/lib/mesh-vault"]`); err != nil || len(p) != 1 {
|
||||
t.Errorf("a list: %v %v", p, err)
|
||||
}
|
||||
if p, err := placesFrom(`[]`); err != nil || len(p) != 0 {
|
||||
t.Errorf("an empty list is a node with no secret place: %v %v", p, err)
|
||||
}
|
||||
}
|
||||
|
||||
// ADR 0310 §5: a path under the vault's directory, a module's own-secret directory and a known credential file
|
||||
// are each refused by name before any read — though the account's modes would allow it — and a link to one is
|
||||
// refused as the place is.
|
||||
func TestSecretPlacesAreRefusedByNameBeforeAnyRead(t *testing.T) {
|
||||
r0, dir := testReader(t)
|
||||
vault := filepath.Join(dir, "var/lib/mesh-vault")
|
||||
own := filepath.Join(dir, "var/lib/desk-channel")
|
||||
r, _ := r0, dir
|
||||
r.Places = []string{vault, own}
|
||||
for _, p := range []string{
|
||||
filepath.Join(vault, "ledger/entries.json"),
|
||||
filepath.Join(own, "broker"),
|
||||
filepath.Join(own, "settings.json"),
|
||||
filepath.Join(dir, "home/ops/.claude/.credentials.json"),
|
||||
filepath.Join(dir, "home/ops/.ssh/config"),
|
||||
filepath.Join(dir, "home/ops/.pgpass"),
|
||||
filepath.Join(dir, "etc/ssh/ssh_host_ed25519_key"),
|
||||
filepath.Join(dir, "etc/shadow"),
|
||||
filepath.Join(dir, "srv/tls/server.key"),
|
||||
} {
|
||||
write(t, p, "harmless words\n")
|
||||
if _, err := r.Read(p, ReadAsk{}); err == nil || !strings.Contains(err.Error(), "refused by name") {
|
||||
t.Errorf("%s was read: %v", p, err)
|
||||
}
|
||||
}
|
||||
if _, err := r.List(vault); err == nil || !strings.Contains(err.Error(), "refused by name") {
|
||||
t.Errorf("the vault's directory was listed: %v", err)
|
||||
}
|
||||
link := filepath.Join(dir, "tmp/innocent")
|
||||
write(t, filepath.Join(dir, "tmp/x"), "")
|
||||
if err := os.Symlink(filepath.Join(own, "broker"), link); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := r.Read(link, ReadAsk{}); err == nil || !strings.Contains(err.Error(), "leads to") {
|
||||
t.Errorf("a link to a secret place was read: %v", err)
|
||||
}
|
||||
// A search under a directory holding a secret place skips it and names it.
|
||||
write(t, filepath.Join(dir, "home/ops/notes.txt"), "harmless words\n")
|
||||
a, err := r.Search(SearchAsk{Text: "harmless", Path: filepath.Join(dir, "home"), Depth: 8})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(a.Matches) != 1 || !strings.HasSuffix(a.Matches[0].File, "notes.txt") || len(a.Skipped) != 3 {
|
||||
t.Errorf("search: matches %v, skipped %v", a.Matches, a.Skipped)
|
||||
}
|
||||
// What a public key or a certificate is, is not refused.
|
||||
for _, p := range []string{filepath.Join(dir, "etc/ssh/ssh_host_ed25519_key.pub"), filepath.Join(dir, "etc/ssl/certs/ca.pem")} {
|
||||
write(t, p, "harmless words\n")
|
||||
if _, err := r.Read(p, ReadAsk{}); err != nil {
|
||||
t.Errorf("%s was refused: %v", p, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ADR 0310 §3: what the account's file modes refuse is refused — the first bound, below the names.
|
||||
func TestAFileTheAccountMayNotReadIsRefusedByItsMode(t *testing.T) {
|
||||
if os.Geteuid() == 0 {
|
||||
t.Skip("run as root, a file's mode refuses nothing")
|
||||
}
|
||||
r, dir := testReader(t)
|
||||
p := filepath.Join(dir, "operators-only")
|
||||
write(t, p, "the operator's words\n")
|
||||
if err := os.Chmod(p, 0o000); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := r.Read(p, ReadAsk{}); err == nil || !strings.Contains(err.Error(), "may not reach it") {
|
||||
t.Errorf("a file of mode 0000: %v", err)
|
||||
}
|
||||
closed := filepath.Join(dir, "closed")
|
||||
write(t, filepath.Join(closed, "inside"), "x")
|
||||
if err := os.Chmod(closed, 0o700&^0o100); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer os.Chmod(closed, 0o755)
|
||||
a, err := r.Stat([]string{filepath.Join(closed, "inside")}, false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := a.Paths[0]; !strings.Contains(got.Stopped, "may not reach") {
|
||||
t.Errorf("a path through a directory it may not search: %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// ADR 0310 §2 and the bounds: a read over the size bound is cut at a line and says so; more than the most is
|
||||
// refused; lines and a tail are answered as asked.
|
||||
func TestAReadIsBounded(t *testing.T) {
|
||||
r, dir := testReader(t)
|
||||
p := filepath.Join(dir, "big")
|
||||
var b strings.Builder
|
||||
for i := 1; i <= 20000; i++ {
|
||||
fmt.Fprintf(&b, "line %d of the file\n", i)
|
||||
}
|
||||
write(t, p, b.String())
|
||||
a, err := r.Read(p, ReadAsk{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(a.Text) > DefaultReadSize || !strings.Contains(a.Cut, fmt.Sprint(DefaultReadSize)) || a.From != 1 {
|
||||
t.Errorf("a default read: %d bytes, cut %q", len(a.Text), a.Cut)
|
||||
}
|
||||
if _, err := r.Read(p, ReadAsk{Bytes: MaxReadSize + 1}); err == nil {
|
||||
t.Error("more than the most a read answers was answered")
|
||||
}
|
||||
a, err = r.Read(p, ReadAsk{From: 10, Lines: 2})
|
||||
if err != nil || a.Text != "line 10 of the file\nline 11 of the file\n" || a.From != 10 || a.Cut != "" {
|
||||
t.Errorf("lines 10 and 11: %+v %v", a, err)
|
||||
}
|
||||
a, err = r.Read(p, ReadAsk{From: -2})
|
||||
if err != nil || a.Text != "line 19999 of the file\nline 20000 of the file\n" || a.From != 19999 {
|
||||
t.Errorf("the last two lines: %q from %d, %v", a.Text, a.From, err)
|
||||
}
|
||||
if _, err := r.Read(p, ReadAsk{From: 30000}); err == nil || !strings.Contains(err.Error(), "20000 lines") {
|
||||
t.Errorf("a line past the end: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// A binary file, a directory and a fifo are refused, and the fifo without waiting for a writer.
|
||||
func TestOnlyATextFileIsRead(t *testing.T) {
|
||||
r, dir := testReader(t)
|
||||
bin := filepath.Join(dir, "bin")
|
||||
write(t, bin, "ELF\x00\x01\x02")
|
||||
if _, err := r.Read(bin, ReadAsk{}); err == nil || !strings.Contains(err.Error(), "binary") {
|
||||
t.Errorf("a binary file: %v", err)
|
||||
}
|
||||
if _, err := r.Read(dir, ReadAsk{}); err == nil || !strings.Contains(err.Error(), "directory") {
|
||||
t.Errorf("a directory: %v", err)
|
||||
}
|
||||
fifo := filepath.Join(dir, "fifo")
|
||||
if err := syscall.Mkfifo(fifo, 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
done := make(chan error, 1)
|
||||
go func() { _, err := r.Read(fifo, ReadAsk{}); done <- err }()
|
||||
select {
|
||||
case err := <-done:
|
||||
if err == nil || !strings.Contains(err.Error(), "fifo") {
|
||||
t.Errorf("a fifo: %v", err)
|
||||
}
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatal("a read of a fifo waits for a writer")
|
||||
}
|
||||
}
|
||||
|
||||
// ADR 0310 §4: a line shaped like a secret is withheld, its number said, and the rest is answered.
|
||||
func TestALineShapedLikeASecretIsWithheld(t *testing.T) {
|
||||
r, dir := testReader(t)
|
||||
p := filepath.Join(dir, "app.conf")
|
||||
write(t, p, "listen = /run/app.sock\ntoken = "+aToken+"\nuser = app\n")
|
||||
a, err := r.Read(p, ReadAsk{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if strings.Contains(a.Text, aToken) || !strings.Contains(a.Text, "user = app") ||
|
||||
!strings.Contains(a.Text, "listen = /run/app.sock") || fmt.Sprint(a.Withheld) != "[2]" {
|
||||
t.Errorf("read: %q withheld %v", a.Text, a.Withheld)
|
||||
}
|
||||
m, err := r.Search(SearchAsk{Text: "token", Path: dir})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
body, _ := json.Marshal(m)
|
||||
if strings.Contains(string(body), aToken) || len(m.Matches) != 1 || len(m.Withheld) != 1 {
|
||||
t.Errorf("search: %s", body)
|
||||
}
|
||||
}
|
||||
|
||||
// The bounds of a search: matches, depth, the most asked, and time — each met is said.
|
||||
func TestASearchIsBounded(t *testing.T) {
|
||||
r, dir := testReader(t)
|
||||
for i := 0; i < 30; i++ {
|
||||
write(t, filepath.Join(dir, fmt.Sprintf("f%02d", i)), "needle\nneedle\n")
|
||||
}
|
||||
a, err := r.Search(SearchAsk{Text: "needle", Path: dir, Matches: 7})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(a.Matches) != 7 || !strings.HasPrefix(a.Bound, "matches") {
|
||||
t.Errorf("seven matches asked: %d, bound %q", len(a.Matches), a.Bound)
|
||||
}
|
||||
if _, err := r.Search(SearchAsk{Text: "needle", Path: dir, Matches: MaxMatches + 1}); err == nil {
|
||||
t.Error("more matches than the most were asked and answered")
|
||||
}
|
||||
if _, err := r.Search(SearchAsk{Text: "needle", Path: dir, Depth: MaxDepth + 1}); err == nil {
|
||||
t.Error("deeper than the most was asked and answered")
|
||||
}
|
||||
deep := t.TempDir()
|
||||
write(t, filepath.Join(deep, "a/b/c/shallow"), "needle\n")
|
||||
write(t, filepath.Join(deep, "a/b/c/d/e/deep"), "needle\n")
|
||||
a, err = r.Search(SearchAsk{Text: "needle", Path: deep, Depth: 3})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(a.Matches) != 1 || !strings.HasSuffix(a.Matches[0].File, "shallow") {
|
||||
t.Errorf("three deep: %+v", a.Matches)
|
||||
}
|
||||
// Case: as asked.
|
||||
write(t, filepath.Join(deep, "case"), "NEEDLE\n")
|
||||
if a, _ := r.Search(SearchAsk{Text: "needle", Path: filepath.Join(deep, "case")}); len(a.Matches) != 0 {
|
||||
t.Error("a match whatever the case, unasked")
|
||||
}
|
||||
if a, _ := r.Search(SearchAsk{Text: "needle", Path: filepath.Join(deep, "case"), CaseInsensitive: true}); len(a.Matches) != 1 {
|
||||
t.Error("no match whatever the case, asked")
|
||||
}
|
||||
// Time: a clock past the bound stops the walk and says so.
|
||||
start := time.Now()
|
||||
calls := 0
|
||||
r.Now = func() time.Time { calls++; return start.Add(time.Duration(calls) * SearchTime / 3) }
|
||||
a, err = r.Search(SearchAsk{Text: "needle", Path: dir})
|
||||
if err != nil || !strings.HasPrefix(a.Bound, "time") {
|
||||
t.Errorf("the time bound: %q %v", a.Bound, err)
|
||||
}
|
||||
// A fixed string: a pattern's characters are themselves.
|
||||
write(t, filepath.Join(deep, "pattern"), "a.c\nabc\n")
|
||||
if a, _ := r.Search(SearchAsk{Text: "a.c", Path: filepath.Join(deep, "pattern")}); len(a.Matches) != 1 || a.Matches[0].Line != 1 {
|
||||
t.Errorf("a.c as a fixed string: %+v", a.Matches)
|
||||
}
|
||||
}
|
||||
|
||||
// The survey issue 418 was asked for: each component of a path with its owner, group, mode, whether it is a
|
||||
// link and to what, and whether it is a mount point — the link not followed unless asked.
|
||||
func TestStatSaysEachComponent(t *testing.T) {
|
||||
r, dir := testReader(t)
|
||||
data := filepath.Join(dir, "data")
|
||||
write(t, filepath.Join(data, "real/file"), "x")
|
||||
if err := os.Chmod(filepath.Join(data, "real"), os.ModeSetgid|0o750); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.Symlink("real", filepath.Join(data, "link")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
a, err := r.Stat([]string{filepath.Join(data, "link"), filepath.Join(data, "real/file")}, false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
link := a.Paths[0]
|
||||
last := link.Components[len(link.Components)-1]
|
||||
if link.Components[0].Path != "/" || last.Type != "link" || last.Link != "real" || link.LeadsTo != "" {
|
||||
t.Errorf("the link, not followed: %+v", link)
|
||||
}
|
||||
var dataC, realC Component
|
||||
for _, c := range a.Paths[1].Components {
|
||||
switch c.Path {
|
||||
case data:
|
||||
dataC = c
|
||||
case filepath.Join(data, "real"):
|
||||
realC = c
|
||||
}
|
||||
}
|
||||
if dataC.Mount == nil || dataC.Mount.FS != "nfs4" {
|
||||
t.Errorf("the mount point is not said: %+v", dataC)
|
||||
}
|
||||
if realC.Mode != "2750" || realC.Type != "directory" || realC.UID == nil || realC.Perms == "" {
|
||||
t.Errorf("a component's mode: %+v", realC)
|
||||
}
|
||||
if uid := *realC.UID; uid == 1000 && realC.Owner != "ops" {
|
||||
t.Errorf("owner %d named %q", uid, realC.Owner)
|
||||
}
|
||||
a, err = r.Stat([]string{filepath.Join(data, "link")}, true)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if a.Paths[0].LeadsTo != filepath.Join(data, "real") || a.Paths[0].Leads == nil || a.Paths[0].Leads.Type != "directory" {
|
||||
t.Errorf("followed: %+v", a.Paths[0])
|
||||
}
|
||||
a, err = r.Stat([]string{filepath.Join(dir, "missing/deeper")}, false)
|
||||
if err != nil || !strings.Contains(a.Paths[0].Stopped, "does not exist") {
|
||||
t.Errorf("a missing path: %+v %v", a, err)
|
||||
}
|
||||
if _, err := r.Stat([]string{"relative/path"}, false); err == nil {
|
||||
t.Error("a relative path was answered")
|
||||
}
|
||||
many := make([]string, MaxPaths+1)
|
||||
for i := range many {
|
||||
many[i] = "/"
|
||||
}
|
||||
if _, err := r.Stat(many, false); err == nil {
|
||||
t.Error("more paths than the most were answered")
|
||||
}
|
||||
}
|
||||
|
||||
func TestReadlinkSaysTheTargetAndWhereItLeads(t *testing.T) {
|
||||
r, dir := testReader(t)
|
||||
write(t, filepath.Join(dir, "target"), "")
|
||||
if err := os.Symlink("target", filepath.Join(dir, "l1")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.Symlink("l1", filepath.Join(dir, "l2")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.Symlink("nowhere", filepath.Join(dir, "dangling")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
a, err := r.Readlink(filepath.Join(dir, "l2"))
|
||||
if err != nil || !a.IsLink || a.Link != "l1" || a.LeadsTo != filepath.Join(dir, "target") || !a.Exists {
|
||||
t.Errorf("l2: %+v %v", a, err)
|
||||
}
|
||||
a, err = r.Readlink(filepath.Join(dir, "dangling"))
|
||||
if err != nil || !a.IsLink || a.Exists || !strings.Contains(a.Stopped, "does not exist") {
|
||||
t.Errorf("dangling: %+v %v", a, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestListSaysEveryEntry(t *testing.T) {
|
||||
r, dir := testReader(t)
|
||||
write(t, filepath.Join(dir, "d/a"), "abc")
|
||||
if err := os.Symlink("a", filepath.Join(dir, "d/b")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
a, err := r.List(filepath.Join(dir, "d"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if a.Count != 2 || a.Entries[0].Name != "a" || *a.Entries[0].Size != 3 || a.Entries[1].Type != "link" || a.Entries[1].Link != "a" {
|
||||
t.Errorf("list: %+v", a)
|
||||
}
|
||||
for i := 0; i < MaxEntries+5; i++ {
|
||||
write(t, filepath.Join(dir, "many", fmt.Sprintf("e%05d", i)), "")
|
||||
}
|
||||
a, err = r.List(filepath.Join(dir, "many"))
|
||||
if err != nil || len(a.Entries) != MaxEntries || a.Count != MaxEntries+5 || a.Cut == "" {
|
||||
t.Errorf("a long directory: %d of %d, cut %q, %v", len(a.Entries), a.Count, a.Cut, err)
|
||||
}
|
||||
}
|
||||
|
||||
// getent passwd, getent group and id: an account with every group it is in, a group with its members, never a
|
||||
// password field.
|
||||
func TestAccountAndGroupLookup(t *testing.T) {
|
||||
r, _ := testReader(t)
|
||||
a, err := r.Account("ops", "")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
u := a.Accounts[0]
|
||||
sort.Strings(u.Groups)
|
||||
if u.UID != 1000 || u.Group != "ops" || u.Shell != "/usr/bin/zsh" || fmt.Sprint(u.Groups) != "[docker ops wheel]" {
|
||||
t.Errorf("ops: %+v", u)
|
||||
}
|
||||
if a, err := r.Account("961", ""); err != nil || a.Accounts[0].Name != "mesh-reads" {
|
||||
t.Errorf("by number: %+v %v", a, err)
|
||||
}
|
||||
g, err := r.Account("", "docker")
|
||||
if err != nil || g.Groups[0].GID != 970 || fmt.Sprint(g.Groups[0].Members) != "[ops builder]" {
|
||||
t.Errorf("docker: %+v %v", g, err)
|
||||
}
|
||||
if _, err := r.Account("nobody-here", ""); err == nil {
|
||||
t.Error("a missing account was answered")
|
||||
}
|
||||
all, err := r.Account("", "")
|
||||
if err != nil || len(all.Accounts) != 3 || len(all.Groups) != 5 {
|
||||
t.Errorf("all: %+v %v", all, err)
|
||||
}
|
||||
body, _ := json.Marshal(all)
|
||||
if strings.Contains(string(body), `"x"`) || strings.Contains(string(body), "password") {
|
||||
t.Errorf("a password field is answered: %s", body)
|
||||
}
|
||||
}
|
||||
|
||||
// No answer is larger than the bound, whatever a verb made.
|
||||
func TestNoAnswerIsLargerThanTheBound(t *testing.T) {
|
||||
if _, err := bounded(strings.Repeat("x", MaxAnswer+1), nil); err == nil {
|
||||
t.Error("an answer over the bound was given")
|
||||
}
|
||||
if _, err := bounded("small", nil); err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
}
|
||||
|
||||
// The seat's six verbs, as the controller defines them (mesh-controller internal/catalogue/reads.go).
|
||||
func TestItServesTheSeatsVerbs(t *testing.T) {
|
||||
r, _ := testReader(t)
|
||||
var names []string
|
||||
for _, tool := range tools(r) {
|
||||
names = append(names, tool.Name)
|
||||
}
|
||||
if fmt.Sprint(names) != "[node-reads.stat node-reads.readlink node-reads.list node-reads.read node-reads.search node-reads.account]" {
|
||||
t.Errorf("verbs: %v", names)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,109 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/outward"
|
||||
)
|
||||
|
||||
// What is refused by name, before any read (novox/hq ADR 0310 §5), and what an answer withholds (§4).
|
||||
//
|
||||
// **Two lists of places.** The places the mesh knows a secret is kept on this node — every module's own-secret
|
||||
// directory, every credential file and grant directory, the vault's directories — come from the controller,
|
||||
// which composes them from the manifests (MESH_SECRET_PLACES). The known credential files of the programs
|
||||
// on a machine are this module's list, below: the mesh does not write them, so nothing else names them. Both
|
||||
// hold whatever the account's file modes would allow: the file modes are the first bound, the names the second.
|
||||
|
||||
// knownDirs are directories that hold nothing but credentials, wherever they are: an account's keys.
|
||||
var knownDirs = map[string]bool{
|
||||
".ssh": true, ".gnupg": true, ".password-store": true, ".aws": true, ".kube": true, ".azure": true,
|
||||
}
|
||||
|
||||
// knownNames are file names that are a credential whatever directory they are in.
|
||||
var knownNames = map[string]bool{
|
||||
".credentials.json": true, // the agent's credentials (Claude Code)
|
||||
".netrc": true, ".pgpass": true, ".git-credentials": true, ".npmrc": true, ".pypirc": true,
|
||||
".my.cnf": true, ".vault-token": true, "credentials": true, "credentials.json": true,
|
||||
"id_rsa": true, "id_dsa": true, "id_ecdsa": true, "id_ed25519": true, "id_ecdsa_sk": true, "id_ed25519_sk": true,
|
||||
"shadow": true, "gshadow": true, "shadow-": true, "gshadow-": true, "opasswd": true,
|
||||
"broker": true, // the mesh's name for a module's bus credential
|
||||
}
|
||||
|
||||
// knownSuffixes are file name endings a credential has.
|
||||
var knownSuffixes = []string{".key", ".p12", ".pfx", ".jks", ".keystore", ".creds", ".nk", ".kdbx", "_key"}
|
||||
|
||||
// knownPaths are the places on a machine that hold credentials and are not a module's.
|
||||
var knownPaths = []string{
|
||||
"/etc/ssl/private", "/etc/wireguard", "/etc/NetworkManager/system-connections", "/etc/ipsec.d/private",
|
||||
"/var/lib/iwd", "/etc/docker/key.json", "/root/.docker/config.json", "/var/lib/sss/secrets",
|
||||
}
|
||||
|
||||
// Refuser says whether a path is a place a secret is kept.
|
||||
type Refuser struct {
|
||||
// Places are the mesh's, from the controller: refused with everything beneath them.
|
||||
Places []string
|
||||
}
|
||||
|
||||
// Refused says why a path may not be read, or "" when it may. The path is clean and absolute.
|
||||
func (r Refuser) Refused(path string) string {
|
||||
path = filepath.Clean(path)
|
||||
for _, p := range r.Places {
|
||||
if within(path, p) {
|
||||
return "it is a place the mesh keeps a secret (" + p + ")"
|
||||
}
|
||||
}
|
||||
for _, p := range knownPaths {
|
||||
if within(path, p) {
|
||||
return "it is a place a machine keeps credentials (" + p + ")"
|
||||
}
|
||||
}
|
||||
parts := strings.Split(strings.TrimPrefix(path, "/"), "/")
|
||||
for i, part := range parts {
|
||||
if knownDirs[part] {
|
||||
return "it is in " + part + ", which holds an account's credentials"
|
||||
}
|
||||
if i == len(parts)-1 {
|
||||
if knownNames[part] {
|
||||
return part + " is a known credential file"
|
||||
}
|
||||
for _, s := range knownSuffixes {
|
||||
if strings.HasSuffix(part, s) && len(part) > len(s) {
|
||||
return part + " is named as a key or a credential (" + s + ")"
|
||||
}
|
||||
}
|
||||
if strings.HasSuffix(part, ".pem") && (strings.Contains(part, "key") || strings.Contains(part, "private")) {
|
||||
return part + " is named as a private key"
|
||||
}
|
||||
if part == "config.json" && i > 0 && parts[i-1] == ".docker" {
|
||||
return "a container registry login is kept there"
|
||||
}
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func within(path, place string) bool {
|
||||
return path == place || strings.HasPrefix(path, strings.TrimSuffix(place, "/")+"/")
|
||||
}
|
||||
|
||||
// Withheld is one piece of an answer the content rule kept back: where it was, and the class it carried.
|
||||
type Withheld struct {
|
||||
At string `json:"at"`
|
||||
Class string `json:"why"`
|
||||
}
|
||||
|
||||
// clean is the content rule's secret class (mesh-sdk outward, novox/hq ADR 0310 §4) over one piece of an
|
||||
// answer: the piece, or a mark in its place when it is shaped like a secret.
|
||||
//
|
||||
// **What a file holds, not what the file system names.** It judges a file's lines, a matching line, a link's
|
||||
// target, a mount's source and an account's words. A path and an entry's name are not judged: they are what
|
||||
// the caller named or the file system calls a thing, and the rule's random-run test takes a temporary
|
||||
// directory's or a container layer's name for a token, which would withhold the very survey asked for. A
|
||||
// place where a secret sits by name is refused before any read instead (Refuser).
|
||||
func clean(s string) (string, string) {
|
||||
if refusal, ok := outward.CheckSecret(s); !ok {
|
||||
return "‹withheld: " + refusal.What + "›", refusal.What
|
||||
}
|
||||
return s, ""
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
module mesh-reads
|
||||
|
||||
go 1.25.0
|
||||
|
||||
require git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261012115257-4dc5f29d3e23
|
||||
@@ -0,0 +1,2 @@
|
||||
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261012115257-4dc5f29d3e23 h1:/F4NwABPAuFZlBx1yK4rdj6sVL5K1XvRlOnZZfNnlsQ=
|
||||
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261012115257-4dc5f29d3e23/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
|
||||
@@ -0,0 +1,54 @@
|
||||
{
|
||||
"module": "mesh-reads",
|
||||
"version": "1",
|
||||
"runs-as": "mesh-reads",
|
||||
"claims": [
|
||||
{
|
||||
"name": "node-reads",
|
||||
"scope": "node",
|
||||
"serves": [
|
||||
"stat",
|
||||
"readlink",
|
||||
"list",
|
||||
"read",
|
||||
"search",
|
||||
"account"
|
||||
]
|
||||
}
|
||||
],
|
||||
"own-secrets": {
|
||||
"broker": "${dir:state}/broker"
|
||||
},
|
||||
"secrets-owner": "mesh-reads",
|
||||
"resources": [
|
||||
{
|
||||
"id": "account",
|
||||
"type": "user",
|
||||
"name": "mesh-reads",
|
||||
"shell": "/usr/bin/nologin",
|
||||
"home": "/var/lib/mesh-reads"
|
||||
},
|
||||
{
|
||||
"id": "state",
|
||||
"type": "directory",
|
||||
"mode": "0700",
|
||||
"owner": "mesh-reads",
|
||||
"place": "."
|
||||
}
|
||||
],
|
||||
"build": {
|
||||
"artifacts": [
|
||||
{
|
||||
"name": "tools",
|
||||
"kind": "bundle",
|
||||
"language": "go",
|
||||
"system": "arch",
|
||||
"from": "cmd/mesh-reads",
|
||||
"binary": "mesh-reads",
|
||||
"loads": [
|
||||
"mesh-reads"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -16,15 +16,20 @@ channel's client and no channel's secret.
|
||||
its sender) and `desk-channel` (kind `desktop`, the machines' notifiers, proves nothing). Which kinds exist
|
||||
and what each promises is the controller's word, in this module's membership; whether each can send now
|
||||
is the channel's own, said on `intake.standing`.
|
||||
- **Keeps its state on the bus** (ADR 0201): `open` and `sent` (the messages), `asks` (every ask, closed ones
|
||||
for 30 days), `identities` (the operator's linked accounts, one per kind, and a link asked for).
|
||||
- **Keeps its state on the bus** (ADR 0201): `open` and `sent` (the messages, and the restart passes made),
|
||||
`asks` (every ask, closed ones for 30 days), `identities` (the operator's linked accounts, one per kind, and a
|
||||
link asked for), `walks` (the delivery walks open now, each with what it holds).
|
||||
- **Lost answers are said again from here, never from a channel's memory** (issue 384,
|
||||
`cmd/messenger/restart.go`): a channel whose messages' answers were lost — the desk's notifier started
|
||||
again — says when in its standing (`answers-lost-at`). The router says again on that kind, directly and not
|
||||
through the outbox, each message that is still open, asked there and last said before that time: one per
|
||||
problem, under its handle, with the words and answers of now, so it replaces the message in place. What is
|
||||
silenced is edited to say so, with no answers and no urgency; what ended never comes back. A kind that cannot
|
||||
send waits; it never goes to another channel. Telegram's answers are never lost and it says no time.
|
||||
through the outbox, each message that is still open, asked there, never answered and last said before that
|
||||
time: one per problem, under its handle, with the words and answers of now, so it replaces the message in
|
||||
place. **A restart says nothing it has said** (novox/hq ADR 0308 §6, issue 531): a silenced message, one
|
||||
withheld for its content, one answered and one a walk holds are left as they are, and what ended never comes
|
||||
back. The standings of one restart (a notifier on each machine, seconds apart) are one pass: a loss is acted
|
||||
on once it stood 10 s, and the pass made is kept in the state (`sent`, key `passes`), so a router that starts
|
||||
again does not make it again. A kind that cannot send waits; it never goes to another channel. Telegram's
|
||||
answers are never lost and it says no time.
|
||||
|
||||
## Asks, answers and warrants (novox/hq ADR 0259)
|
||||
|
||||
@@ -165,10 +170,36 @@ ends takes them away everywhere; its expiry is said with its day when it is not
|
||||
history. An event that cannot be read
|
||||
is refused by name, counted, and told to the operator once an hour.
|
||||
|
||||
## While a delivery walks (novox/hq ADR 0308)
|
||||
|
||||
The controller's `plan-moved` says each walk's start and end: `walk`, `phase` (`started`, `ended`), `touches`
|
||||
(`[{machine, modules}]`: what it moves and what a moved tool runner or node-engine carries), `outcome`
|
||||
(`delivered`, `failed`, `stopped`), `bound_seconds` and `budget_class` (`cmd/messenger/walks.go`).
|
||||
|
||||
- **Held:** while a walk is open and within its bound, a message whose subject is a machine it touches, or a
|
||||
module it touches (on a machine it touches, when the subject names one), is held: kept on the walk with when
|
||||
and what, and the history says `held`. A message several open walks touch is told when the last ends or passes
|
||||
its bound. At a walk's end, a later open walk that touches a message takes it over only as a first hold would:
|
||||
still open, not silenced, not said since, and not one that is never held.
|
||||
- **Never held:** an ask waiting for the operator (and a condition an open ask is about), a secret given or
|
||||
replaced, a subject no open walk touches, and a walk's own failure (a plan, walk, delivery or build).
|
||||
- **The end is one message:** the outcome, the modules and machines reached, how long; each held message still
|
||||
open, one line each; the ones that cleared counted. Delivered with nothing held and nothing open: one line —
|
||||
and nothing on a kind listed in the setting `only-what-needs-me`. What it carries is marked told on a kind
|
||||
only once the message was sent there. Until then each is kept owed in the state (`owed`): a message lost with a
|
||||
restart before it was sent leaves them untold and owed, and the restarted router says each on its own.
|
||||
- **The bound:** at its start plus `bound_seconds` (900 when not given), what is held and still open is said,
|
||||
with "The walk … is still running, since HH:MM", and the cleared counted; the walk holds nothing more. A walk
|
||||
whose end never comes is forgotten a day after it started. A start heard after its end opens nothing: the
|
||||
controller's moments are compared on its own clock, never with one capped at the router's now.
|
||||
- **Kept in the state** (`walks`): a router that starts again mid-walk holds and tells as before.
|
||||
|
||||
## What the operator gives
|
||||
|
||||
1. **The time zone**, as a setting: `{"time-zone": "<an IANA zone name>"}`. Every time a message says is
|
||||
said in it; not given, the zone of the machine this runs on.
|
||||
**The kinds that say only what needs the operator**, as a setting: `{"only-what-needs-me": ["<kind>"]}`. A
|
||||
walk that delivered with nothing held and nothing open is not told on them (ADR 0308 §4); empty by default.
|
||||
2. **The channels**: assign `telegram` and `desk-channel` (each says in its README what it needs). Linking
|
||||
an account as the operator is done on the channel, never by a setting.
|
||||
|
||||
|
||||
@@ -154,6 +154,11 @@ type Record struct {
|
||||
Subject string `json:"subject"`
|
||||
Severity string `json:"severity"`
|
||||
Summary string `json:"summary"`
|
||||
// Scope, SubjectID and Machine are the condition's subject as the controller gave it, which a walk's touches
|
||||
// are matched against (ADR 0308 §2); empty for a record kept before, whose key names them.
|
||||
Scope string `json:"scope,omitempty"`
|
||||
SubjectID string `json:"subject_id,omitempty"`
|
||||
Machine string `json:"machine,omitempty"`
|
||||
// Headline, Explanation and Resolved are the plain words the operator reads (novox/hq ADR 0253):
|
||||
// the controller's, or a module's through notify; empty, the message is said from the summary.
|
||||
Headline string `json:"headline,omitempty"`
|
||||
@@ -195,14 +200,17 @@ type Record struct {
|
||||
// ClearHeld is what was waiting to be said of it when it cleared (a reminder, an escalation): said after
|
||||
// all if it comes back before its clearing was said. Blips counts those comebacks, never said, so a
|
||||
// condition that flaps is known as one and its message's "again, Nth time" counts only what was said.
|
||||
ClearHeld string `json:"clear_held,omitempty"`
|
||||
Blips int `json:"blips,omitempty"`
|
||||
Reminded bool `json:"reminded,omitempty"`
|
||||
Pending string `json:"pending,omitempty"` // what could not be said yet: raised, reminder, …
|
||||
Folded bool `json:"folded,omitempty"` // held by the cap
|
||||
Cleared time.Time `json:"cleared,omitempty"`
|
||||
Count int `json:"count"`
|
||||
Refused string `json:"refused,omitempty"`
|
||||
ClearHeld string `json:"clear_held,omitempty"`
|
||||
Blips int `json:"blips,omitempty"`
|
||||
Reminded bool `json:"reminded,omitempty"`
|
||||
Pending string `json:"pending,omitempty"` // what could not be said yet: raised, reminder, …
|
||||
// Owed is what a walk held of it and has not told yet (ADR 0308 §4, §5): kept in the state until a message
|
||||
// carrying it was sent, so a router that restarts before says it on its own (issue 538), never history.
|
||||
Owed string `json:"owed,omitempty"`
|
||||
Folded bool `json:"folded,omitempty"` // held by the cap
|
||||
Cleared time.Time `json:"cleared,omitempty"`
|
||||
Count int `json:"count"`
|
||||
Refused string `json:"refused,omitempty"`
|
||||
// History: known from the state or from an old event, and never said — not news.
|
||||
History bool `json:"history,omitempty"`
|
||||
// Seen is when this holder last took word of it; the open set read after it is authoritative.
|
||||
@@ -211,7 +219,8 @@ type Record struct {
|
||||
|
||||
// recordOf is a condition as this holder keeps it.
|
||||
func recordOf(c Condition) Record {
|
||||
return Record{Key: c.Key, Kind: c.Kind, Subject: c.SubjectWords(), Severity: c.Severity, Summary: c.Summary,
|
||||
return Record{Key: c.Key, Kind: c.Kind, Subject: c.SubjectWords(), Scope: c.Scope, SubjectID: c.ID, Machine: c.Machine,
|
||||
Severity: c.Severity, Summary: c.Summary,
|
||||
Headline: c.Headline, Explanation: c.Explanation, Resolved: c.Resolved, Needs: c.Needs, Actions: c.Actions,
|
||||
Origin: "condition",
|
||||
More: "conditions show " + c.Key, Raised: c.Raised, SilencedTill: c.SilencedTill, SilencedWhy: c.SilencedWhy}
|
||||
@@ -232,6 +241,13 @@ func (r *Record) takeWords(from Record) {
|
||||
}
|
||||
}
|
||||
|
||||
// takeSubject keeps the subject as the controller says it now, when it says one.
|
||||
func (r *Record) takeSubject(from Record) {
|
||||
if from.Scope != "" || from.SubjectID != "" || from.Machine != "" {
|
||||
r.Scope, r.SubjectID, r.Machine = from.Scope, from.SubjectID, from.Machine
|
||||
}
|
||||
}
|
||||
|
||||
func (r *Record) silenced(now time.Time) bool {
|
||||
return !r.SilencedTill.IsZero() && now.Before(r.SilencedTill)
|
||||
}
|
||||
@@ -292,6 +308,9 @@ type Holder struct {
|
||||
Zone func() *time.Location
|
||||
// Tracking asks the issue tracker which issues track which conditions; nil asks it through Ask (tracking.go).
|
||||
Tracking func() (map[string][]TrackedIssue, error)
|
||||
// NeedsOnly says whether a kind is set to say only what needs the operator (the setting
|
||||
// `only-what-needs-me`, ADR 0308 §4); nil, none is.
|
||||
NeedsOnly func(kind string) bool
|
||||
|
||||
work sync.Mutex // one event, call or tick at a time
|
||||
mu sync.Mutex // what the status reads
|
||||
@@ -329,8 +348,14 @@ type Holder struct {
|
||||
lastUnlink map[string]Unlinked
|
||||
// rootWord is the controller's last word on whether an agent can become root, by the machines asked.
|
||||
rootWord map[string]rootWord
|
||||
// shows is, by kind, the AnswersLostAt this router last acted on in full (restart.go).
|
||||
shows map[string]time.Time
|
||||
// passes is, by kind, the restart pass this router last made in full, kept in its state (restart.go); lost
|
||||
// the loss of answers it has yet to act on, by kind.
|
||||
passes map[string]Pass
|
||||
lost map[string]time.Time
|
||||
// walks are the walks open now, by id, each with what it holds (walks.go, ADR 0308).
|
||||
walks map[string]*Walk
|
||||
// ended are the walks whose end was heard, by id: what a start heard after its end is matched against.
|
||||
ended map[string]*Walk
|
||||
// track is the issue tracker's last word on which issues track which conditions (tracking.go).
|
||||
track trackWord
|
||||
}
|
||||
@@ -348,7 +373,10 @@ func (h *Holder) init() {
|
||||
h.chanOK = map[string]time.Time{}
|
||||
h.saidOnce = map[string]time.Time{}
|
||||
h.heard = map[string]int{}
|
||||
h.shows = map[string]time.Time{}
|
||||
h.passes = map[string]Pass{}
|
||||
h.lost = map[string]time.Time{}
|
||||
h.walks = map[string]*Walk{}
|
||||
h.ended = map[string]*Walk{}
|
||||
}
|
||||
if h.Now == nil {
|
||||
h.Now = time.Now
|
||||
@@ -382,11 +410,20 @@ func (h *Holder) Load() error {
|
||||
r.History = true
|
||||
}
|
||||
}
|
||||
if r.Owed != "" && r.Cleared.IsZero() {
|
||||
// Held from the operator on purpose and not told yet: said on its own, as the next tick tries what is
|
||||
// unsent (issue 538). What the operator could have heard is history; this they could not.
|
||||
r.Pending, r.History = r.Owed, false
|
||||
}
|
||||
h.open[r.Key] = &r
|
||||
}
|
||||
if sent, err := h.Store.Recent(); err == nil {
|
||||
h.recent = sent
|
||||
}
|
||||
if err := h.loadWalks(); err != nil {
|
||||
h.storeErr = err.Error()
|
||||
return err
|
||||
}
|
||||
h.mu.Unlock()
|
||||
err = h.loadAsks()
|
||||
h.mu.Lock()
|
||||
@@ -485,6 +522,7 @@ func (h *Holder) raised(rec Record, quiet bool) {
|
||||
wasQuiet := needsNothing(old)
|
||||
wasSilenced := old.silenced(now)
|
||||
old.Summary, old.Subject, old.Kind = rec.Summary, rec.Subject, rec.Kind
|
||||
old.takeSubject(rec)
|
||||
old.SilencedTill, old.SilencedWhy = rec.SilencedTill, rec.SilencedWhy
|
||||
old.takeWords(rec)
|
||||
if rec.Severity != "" {
|
||||
@@ -584,6 +622,7 @@ func (h *Holder) changed(rec Record, quiet bool) {
|
||||
wasQuiet := needsNothing(old)
|
||||
wasSilenced := old.silenced(h.Now())
|
||||
old.Summary, old.Subject = rec.Summary, rec.Subject
|
||||
old.takeSubject(rec)
|
||||
old.SilencedTill, old.SilencedWhy = rec.SilencedTill, rec.SilencedWhy
|
||||
old.takeWords(rec)
|
||||
if rec.Severity != "" {
|
||||
@@ -710,6 +749,8 @@ func (h *Holder) Tick() {
|
||||
sort.Slice(recs, func(i, j int) bool { return recs[i].Raised.Before(recs[j].Raised) })
|
||||
h.sweepAsks(now)
|
||||
h.breakGlass()
|
||||
h.boundWalks(now)
|
||||
h.duePasses(now)
|
||||
for _, r := range recs {
|
||||
switch {
|
||||
case !r.Cleared.IsZero():
|
||||
@@ -775,6 +816,7 @@ func (h *Holder) Flush() {
|
||||
h.init()
|
||||
now := h.Now()
|
||||
h.mu.Unlock()
|
||||
h.duePasses(now)
|
||||
h.flush(now, false)
|
||||
}
|
||||
|
||||
|
||||
@@ -66,6 +66,9 @@ func (f *fakeChannel) Edit(id string, m Message) error {
|
||||
type memStore struct {
|
||||
recs map[string]Record
|
||||
recent []Sent
|
||||
// walks and passes are what ADR 0308 keeps: the open walks and the restart passes.
|
||||
walks map[string]Walk
|
||||
passes map[string]Pass
|
||||
// puts counts the writes, so a write of bytes identical to what is stored is still seen (issue 422's review).
|
||||
puts int
|
||||
}
|
||||
@@ -87,7 +90,24 @@ func (m *memStore) All() ([]Record, error) {
|
||||
return out, nil
|
||||
}
|
||||
func (m *memStore) PutRecent(s []Sent) error { m.recent = s; return nil }
|
||||
func (m *memStore) Recent() ([]Sent, error) { return m.recent, nil }
|
||||
func (m *memStore) PutWalk(w Walk) error {
|
||||
if m.walks == nil {
|
||||
m.walks = map[string]Walk{}
|
||||
}
|
||||
m.walks[w.ID] = w
|
||||
return nil
|
||||
}
|
||||
func (m *memStore) DeleteWalk(id string) error { delete(m.walks, id); return nil }
|
||||
func (m *memStore) Walks() ([]Walk, error) {
|
||||
var out []Walk
|
||||
for _, w := range m.walks {
|
||||
out = append(out, w)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
func (m *memStore) PutPasses(p map[string]Pass) error { m.passes = p; return nil }
|
||||
func (m *memStore) Passes() (map[string]Pass, error) { return m.passes, nil }
|
||||
func (m *memStore) Recent() ([]Sent, error) { return m.recent, nil }
|
||||
|
||||
type clock struct{ t time.Time }
|
||||
|
||||
|
||||
@@ -29,6 +29,9 @@ type Settings struct {
|
||||
// TimeZone is the operator's, an IANA name such as Europe/Brussels: every time a message says is said
|
||||
// in it. Empty is this machine's own zone.
|
||||
TimeZone string `json:"time-zone"`
|
||||
// OnlyWhatNeedsMe are the channel kinds set to say only what needs the operator: a walk that delivered
|
||||
// with nothing held and nothing open is not told there at all (novox/hq ADR 0308 §4).
|
||||
OnlyWhatNeedsMe []string `json:"only-what-needs-me"`
|
||||
}
|
||||
|
||||
// zoneOf is the location a time-zone setting names, or this machine's own when it names none or one
|
||||
@@ -60,6 +63,19 @@ func readSettings(path string) (Settings, error) {
|
||||
if tz, ok := loose["time-zone"].(string); ok {
|
||||
s.TimeZone = strings.TrimSpace(tz)
|
||||
}
|
||||
switch v := loose["only-what-needs-me"].(type) {
|
||||
case nil:
|
||||
case []any:
|
||||
for _, k := range v {
|
||||
kind, ok := k.(string)
|
||||
if !ok || strings.TrimSpace(kind) == "" {
|
||||
return s, fmt.Errorf("only-what-needs-me lists something that is not a channel kind: %v", k)
|
||||
}
|
||||
s.OnlyWhatNeedsMe = append(s.OnlyWhatNeedsMe, strings.TrimSpace(kind))
|
||||
}
|
||||
default:
|
||||
return s, fmt.Errorf("only-what-needs-me is not a list of channel kinds")
|
||||
}
|
||||
return s, nil
|
||||
}
|
||||
|
||||
@@ -109,6 +125,50 @@ func (stateStore) All() ([]Record, error) {
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// The walks open now, one key each in the state `walks`, and the restart passes under one key of `sent`
|
||||
// (novox/hq ADR 0308): a router that starts again holds, tells and passes as if it had not.
|
||||
func (stateStore) PutWalk(w Walk) error {
|
||||
_, err := stdio.State("walks").Put(kvKey(w.ID), w)
|
||||
return err
|
||||
}
|
||||
func (stateStore) DeleteWalk(id string) error { return stdio.State("walks").Delete(kvKey(id)) }
|
||||
func (stateStore) Walks() ([]Walk, error) {
|
||||
keys, err := stdio.State("walks").Keys()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var out []Walk
|
||||
for _, k := range keys {
|
||||
e, err := stdio.State("walks").Get(k)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if e == nil {
|
||||
continue
|
||||
}
|
||||
var w Walk
|
||||
if err := json.Unmarshal(e.Value, &w); err != nil {
|
||||
logf("[messenger] the walk kept as %s cannot be read (%v); left as it is", k, err)
|
||||
continue
|
||||
}
|
||||
out = append(out, w)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
func (stateStore) PutPasses(p map[string]Pass) error {
|
||||
_, err := stdio.State("sent").Put("passes", p)
|
||||
return err
|
||||
}
|
||||
func (stateStore) Passes() (map[string]Pass, error) {
|
||||
e, err := stdio.State("sent").Get("passes")
|
||||
if err != nil || e == nil {
|
||||
return nil, err
|
||||
}
|
||||
var p map[string]Pass
|
||||
return p, json.Unmarshal(e.Value, &p)
|
||||
}
|
||||
|
||||
func (stateStore) PutRecent(s []Sent) error {
|
||||
_, err := stdio.State("sent").Put("recent", s)
|
||||
return err
|
||||
@@ -271,6 +331,14 @@ func main() {
|
||||
return loc
|
||||
},
|
||||
Emit: func(event string, body any) error { return stdio.Emit(event, body) },
|
||||
NeedsOnly: func(kind string) bool {
|
||||
for _, k := range settings().OnlyWhatNeedsMe {
|
||||
if k == kind {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
},
|
||||
}
|
||||
bench.To = func(kind string) string {
|
||||
if ids := h.identitiesOf(kind); len(ids) > 0 {
|
||||
@@ -326,6 +394,18 @@ func run(h *Holder, l *listening) {
|
||||
}
|
||||
}()
|
||||
handle := func(e stdio.Envelope) error {
|
||||
if e.Key == ControllerSeat+"."+EventPlanMoved || e.Key == ControllerSeat+"."+EventWalk {
|
||||
// A walk's start or end (novox/hq ADR 0308 §1); any other move of a plan is not read here.
|
||||
ev, isWalk, err := DecodeWalk(e.Body)
|
||||
if err != nil {
|
||||
h.Unreadable(e.Key, err)
|
||||
return nil
|
||||
}
|
||||
if isWalk {
|
||||
h.Walked(ev)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
event, ok := eventOf(e.Key)
|
||||
if !ok {
|
||||
return nil
|
||||
|
||||
@@ -81,7 +81,7 @@ func TestItDeclaresTheSeatsOfAnAskAndHoldsTheOperatorChannel(t *testing.T) {
|
||||
}
|
||||
if !reflect.DeepEqual(m.Consumes, []string{
|
||||
ControllerSeat + "." + EventRaised, ControllerSeat + "." + EventChanged, ControllerSeat + "." + EventCleared,
|
||||
"intake.choice", "intake.link", "intake.failed", "intake.standing"}) {
|
||||
ControllerSeat + "." + EventPlanMoved, "intake.choice", "intake.link", "intake.failed", "intake.standing"}) {
|
||||
t.Fatalf("consumes %v", m.Consumes)
|
||||
}
|
||||
// It performs nothing: the verbs it calls read the open conditions, whether root is free, and which issues
|
||||
@@ -100,7 +100,7 @@ func TestItDeclaresTheSeatsOfAnAskAndHoldsTheOperatorChannel(t *testing.T) {
|
||||
}
|
||||
state = append(state, name)
|
||||
}
|
||||
if !reflect.DeepEqual(state, []string{"open", "sent", "asks", "identities"}) {
|
||||
if !reflect.DeepEqual(state, []string{"open", "sent", "asks", "identities", "walks"}) {
|
||||
t.Fatalf("state %v", state)
|
||||
}
|
||||
// It holds no channel's secret: only its own bus account, read by its own account alone (ADR 0259 §8).
|
||||
|
||||
@@ -18,6 +18,9 @@ type item struct {
|
||||
Severity string
|
||||
At time.Time
|
||||
Msg *Message
|
||||
// Carries are the records a walk's end or bound message tells (ADR 0308 §4): each is marked told on a
|
||||
// channel only once that message was sent there (issue 538 A), never when it is held to be sent.
|
||||
Carries []string
|
||||
}
|
||||
|
||||
func (i item) urgent() bool { return i.Severity == Urgent && i.What != "cleared" }
|
||||
@@ -32,6 +35,10 @@ func (h *Holder) deliver(r *Record, what string) {
|
||||
h.record(Sent{At: h.Now(), Channel: "-", Key: r.Key, What: what, Outcome: "quiet: nothing for the operator to do"})
|
||||
return
|
||||
}
|
||||
if h.hold(r, what) {
|
||||
// An open walk touches it: told when the walk ends or passes its bound (ADR 0308 §2).
|
||||
return
|
||||
}
|
||||
var to []Channel
|
||||
if kinds := h.askKinds(r); len(kinds) > 0 {
|
||||
// An ask goes to every channel that carries one of its answers (novox/hq ADR 0259): the desk for an
|
||||
@@ -118,6 +125,8 @@ func (h *Holder) queuedWhat(key string) string {
|
||||
return ""
|
||||
}
|
||||
|
||||
// queued says whether something is held to be said of a record: its own message, or a walk's message that
|
||||
// carries it.
|
||||
func (h *Holder) queued(key string) bool {
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
@@ -126,6 +135,11 @@ func (h *Holder) queued(key string) bool {
|
||||
if it.Msg == nil && it.Key == key {
|
||||
return true
|
||||
}
|
||||
for _, k := range it.Carries {
|
||||
if k == key {
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return false
|
||||
@@ -135,6 +149,9 @@ func (h *Holder) queued(key string) bool {
|
||||
// edit fails, or that said it inside a digest, is given a new message instead — held like any other.
|
||||
// silentOnly: only where an edit notifies nobody; elsewhere nothing.
|
||||
func (h *Holder) edit(r *Record, what string, silentOnly bool) {
|
||||
if h.hold(r, what) {
|
||||
return // an open walk touches it (ADR 0308 §2); a clearing or a silence is never held
|
||||
}
|
||||
h.mu.Lock()
|
||||
sent := map[string]string{}
|
||||
for k, v := range r.Sent {
|
||||
@@ -225,12 +242,23 @@ func (h *Holder) due(ch Channel, list []item, now time.Time) bool {
|
||||
if now.Sub(oldest) < BurstWindow {
|
||||
return false
|
||||
}
|
||||
if ch == h.Desk && !lastWarn.IsZero() && now.Sub(lastWarn) < DesktopWarningEvery {
|
||||
if ch == h.Desk && !lastWarn.IsZero() && now.Sub(lastWarn) < DesktopWarningEvery && !carriesWalk(list) {
|
||||
// A walk's message is what the rhythm would have spared the operator many of: not held to it.
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// carriesWalk says whether held items carry a walk's end or bound message.
|
||||
func carriesWalk(list []item) bool {
|
||||
for _, it := range list {
|
||||
if it.What == WalkEndWhat {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// line is what an item says, resolved against the record now: nil when it is no longer to be said.
|
||||
func (h *Holder) resolve(it item, now time.Time, kind string) (*Record, *Message) {
|
||||
if it.Msg != nil {
|
||||
@@ -310,7 +338,9 @@ func (h *Holder) flushChannel(ch Channel, now time.Time, force bool) {
|
||||
for _, e := range entries {
|
||||
// A message said again about a record shown here goes alone too: it replaces that record's message,
|
||||
// so a reminder is never a digest line beside the message it follows (issue 373).
|
||||
if len(e.msg.Actions) > 1 || (len(e.msg.Actions) > 0 && e.r != nil && e.r.Ask != "") || h.replacing(ch, e) != "" {
|
||||
// A walk's end or bound message goes alone too: its lines are what it says (ADR 0308 §4).
|
||||
if len(e.msg.Actions) > 1 || (len(e.msg.Actions) > 0 && e.r != nil && e.r.Ask != "") || h.replacing(ch, e) != "" ||
|
||||
e.it.What == WalkEndWhat {
|
||||
alone = append(alone, e)
|
||||
} else {
|
||||
rest = append(rest, e)
|
||||
@@ -400,6 +430,7 @@ func (h *Holder) sendEntries(ch Channel, entries []entry, now time.Time) bool {
|
||||
}
|
||||
r.Pending, r.Folded, r.History = "", false, false
|
||||
if e.it.What != "cleared" {
|
||||
r.Owed = ""
|
||||
if r.Sent == nil {
|
||||
r.Sent = map[string]string{}
|
||||
}
|
||||
@@ -415,7 +446,39 @@ func (h *Holder) sendEntries(ch Channel, entries []entry, now time.Time) bool {
|
||||
r.LastSaid = now
|
||||
}
|
||||
}
|
||||
type carried struct {
|
||||
r *Record
|
||||
walk string
|
||||
}
|
||||
var told []carried
|
||||
for _, e := range entries {
|
||||
for _, key := range e.it.Carries {
|
||||
r := h.open[key]
|
||||
if r == nil {
|
||||
continue
|
||||
}
|
||||
if r.Sent == nil {
|
||||
r.Sent = map[string]string{}
|
||||
}
|
||||
if _, has := r.Sent[name]; !has {
|
||||
r.Sent[name] = ""
|
||||
}
|
||||
if r.FirstSent.IsZero() {
|
||||
r.FirstSent = now
|
||||
}
|
||||
r.LastSaid, r.Pending, r.History, r.Owed = now, "", false, ""
|
||||
if !r.Cleared.IsZero() && !r.ClearUnsaid {
|
||||
// It cleared while the message saying it open waited: its clearing is said too.
|
||||
r.ClearUnsaid = true
|
||||
}
|
||||
told = append(told, carried{r, strings.TrimPrefix(e.it.Key, "walk.")})
|
||||
}
|
||||
}
|
||||
h.mu.Unlock()
|
||||
for _, c := range told {
|
||||
h.record(Sent{At: now, Channel: name, Key: c.r.Key, What: "released", Outcome: "told in the message of walk " + c.walk})
|
||||
h.persist(c.r)
|
||||
}
|
||||
key, what := "(digest)", fmt.Sprintf("digest of %d", len(entries))
|
||||
if len(entries) == 1 {
|
||||
key, what = entries[0].it.Key, entries[0].it.What
|
||||
|
||||
@@ -18,6 +18,22 @@ package main
|
||||
// Whether a record is said again is its own: last said before the answers were lost. So a router that started
|
||||
// after the loss still says it, and a standing that repeats the time says nothing more (the record was said
|
||||
// since). A kind that cannot send waits: what is open there is said once it can.
|
||||
//
|
||||
// **A restart says nothing it has said** (novox/hq ADR 0308 §6, issue 531). On 2026-10-12 every delivery of the
|
||||
// tool runner made this pass twice, a second apart, each time editing the four silenced messages on the desk,
|
||||
// which showed them again — one as "A mesh message was withheld". Three causes, each closed here:
|
||||
//
|
||||
// - **Twice:** the desk's standing says the latest start of any machine's notifier, and the notifiers of two
|
||||
// machines started a second apart, so two standings with two times came, and each was acted on at once.
|
||||
// Now a loss is acted on once it stands for LostSettle: the standings of one restart are one pass.
|
||||
// - **Again after the router's own restart:** which loss was acted on was kept in memory, so a router that
|
||||
// started again acted on the standing it had already acted on. Now it is kept in the module's state
|
||||
// (`sent`, key `passes`), read back on start.
|
||||
// - **What it said:** a silenced message was edited to its silenced words, and a message withheld for its
|
||||
// content was said again as withheld. Now the pass says again only what is open, asked there and never
|
||||
// answered, whose words may leave; a silenced, a withheld or an answered message is left as it is. Being
|
||||
// held by a walk is no reason: a message with an open ask is never held (ADR 0308 §3), even one a walk took
|
||||
// before its ask came, so its answers are made to work again like any other's.
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
@@ -25,7 +41,17 @@ import (
|
||||
"time"
|
||||
)
|
||||
|
||||
// Stand takes what a kind said of itself, and acts on answers lost there since this router last acted.
|
||||
// LostSettle: a loss of answers is acted on once no later loss came for this long — the standings of one
|
||||
// restart, a notifier on each machine, are one pass.
|
||||
const LostSettle = 10 * time.Second
|
||||
|
||||
// Pass is the restart pass made in full on a kind: the loss it answered and when it was made.
|
||||
type Pass struct {
|
||||
Lost time.Time `json:"lost"`
|
||||
At time.Time `json:"at"`
|
||||
}
|
||||
|
||||
// Stand takes what a kind said of itself, and notes answers lost there since this router last acted.
|
||||
func (h *Holder) Stand(kind string, s Standing) {
|
||||
if h.Bench != nil {
|
||||
h.Bench.Stand(kind, s)
|
||||
@@ -33,28 +59,73 @@ func (h *Holder) Stand(kind string, s Standing) {
|
||||
if s.AnswersLostAt.IsZero() {
|
||||
return
|
||||
}
|
||||
h.work.Lock()
|
||||
defer h.work.Unlock()
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
h.init()
|
||||
done := h.shows[kind].Equal(s.AnswersLostAt)
|
||||
if !s.AnswersLostAt.After(h.passes[kind].Lost) || !s.AnswersLostAt.After(h.lost[kind]) {
|
||||
return // acted on, or noted already
|
||||
}
|
||||
h.lost[kind] = s.AnswersLostAt
|
||||
}
|
||||
|
||||
// duePasses makes the pass of each kind whose loss of answers stood for LostSettle, once per loss.
|
||||
func (h *Holder) duePasses(now time.Time) {
|
||||
h.mu.Lock()
|
||||
due := map[string]time.Time{}
|
||||
for kind, lost := range h.lost {
|
||||
if now.Sub(lost) >= LostSettle {
|
||||
due[kind] = lost
|
||||
}
|
||||
}
|
||||
h.mu.Unlock()
|
||||
if done {
|
||||
return
|
||||
}
|
||||
ch := h.channelOf(kind)
|
||||
if ch == nil {
|
||||
return // the controller's word on the kinds is not read yet: acted on at a later standing
|
||||
}
|
||||
if err := ch.Ready(); err != nil {
|
||||
h.Logf("[messenger] %s lost its answers at %s and cannot send (%v): what is open there is said again once it can",
|
||||
kind, s.AnswersLostAt.UTC().Format(time.RFC3339), err)
|
||||
return
|
||||
}
|
||||
if h.reoffer(ch, s.AnswersLostAt) {
|
||||
for kind, lost := range due {
|
||||
ch := h.channelOf(kind)
|
||||
if ch == nil {
|
||||
continue // the controller's word on the kinds is not read yet: acted on later
|
||||
}
|
||||
if err := ch.Ready(); err != nil {
|
||||
h.mu.Lock()
|
||||
said := h.saidOnce["lost."+kind]
|
||||
h.mu.Unlock()
|
||||
if said.IsZero() || now.Sub(said) >= 10*time.Minute {
|
||||
h.Logf("[messenger] %s lost its answers at %s and cannot send (%v): what is open there is said again once it can",
|
||||
kind, lost.UTC().Format(time.RFC3339), err)
|
||||
h.mu.Lock()
|
||||
h.saidOnce["lost."+kind] = now
|
||||
h.mu.Unlock()
|
||||
}
|
||||
continue
|
||||
}
|
||||
h.mu.Lock()
|
||||
h.shows[kind] = s.AnswersLostAt
|
||||
tried := h.saidOnce["lost-try."+kind]
|
||||
h.mu.Unlock()
|
||||
if !tried.IsZero() && now.Sub(tried) < RetryEvery {
|
||||
continue
|
||||
}
|
||||
if !h.reoffer(ch, lost) {
|
||||
h.mu.Lock()
|
||||
h.saidOnce["lost-try."+kind] = now // what failed is tried again in a minute
|
||||
h.mu.Unlock()
|
||||
continue
|
||||
}
|
||||
h.mu.Lock()
|
||||
delete(h.saidOnce, "lost-try."+kind)
|
||||
h.mu.Unlock()
|
||||
h.mu.Lock()
|
||||
h.passes[kind] = Pass{Lost: lost, At: now}
|
||||
if h.lost[kind].Equal(lost) {
|
||||
delete(h.lost, kind)
|
||||
}
|
||||
passes := map[string]Pass{}
|
||||
for k, p := range h.passes {
|
||||
passes[k] = p
|
||||
}
|
||||
h.mu.Unlock()
|
||||
if k, ok := h.Store.(walkKeeper); ok {
|
||||
if err := k.PutPasses(passes); err != nil {
|
||||
h.noteStore(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -68,9 +139,9 @@ func (h *Holder) channelOf(kind string) Channel {
|
||||
return nil
|
||||
}
|
||||
|
||||
// reoffer says again on one channel, whose answers were lost at `lost`, what is still open and asked there,
|
||||
// edits what is silenced to say so, and leaves what ended alone. It answers whether every message to be
|
||||
// said again was.
|
||||
// reoffer says again on one channel, whose answers were lost at `lost`, what is still open, asked there and
|
||||
// never answered, and leaves everything else as it is: what ended, what is silenced, what was withheld for its
|
||||
// content, and what was answered (ADR 0308 §6). It answers whether every message to be said again was.
|
||||
func (h *Holder) reoffer(ch Channel, lost time.Time) bool {
|
||||
kind := ch.Name()
|
||||
rp, canReplace := ch.(replacer)
|
||||
@@ -82,13 +153,14 @@ func (h *Holder) reoffer(ch Channel, lost time.Time) bool {
|
||||
}
|
||||
h.mu.Unlock()
|
||||
sort.Slice(recs, func(i, j int) bool { return recs[i].Raised.Before(recs[j].Raised) })
|
||||
again, silenced, ended, failed := 0, 0, 0, 0
|
||||
again, left, ended, failed := 0, 0, 0, 0
|
||||
for _, r := range recs {
|
||||
h.mu.Lock()
|
||||
id, shown := r.Sent[kind]
|
||||
open := r.Cleared.IsZero()
|
||||
before := lastSaid(r).Before(lost)
|
||||
reminded := r.Reminded
|
||||
withheld, answered := r.Refused != "", r.Answered != ""
|
||||
h.mu.Unlock()
|
||||
if !shown || id == "" {
|
||||
continue // never a message of its own there: nothing to say again under
|
||||
@@ -96,13 +168,8 @@ func (h *Holder) reoffer(ch Channel, lost time.Time) bool {
|
||||
switch {
|
||||
case !open:
|
||||
ended++ // its end is said where its message is; nothing comes back
|
||||
case r.silenced(now):
|
||||
h.mu.Lock()
|
||||
r.SilenceSaid = true
|
||||
h.mu.Unlock()
|
||||
h.editOn(ch, r, "silenced")
|
||||
h.persist(r)
|
||||
silenced++
|
||||
case r.silenced(now), withheld, answered:
|
||||
left++ // a silence, a withheld message or an answer is never said again for a restart
|
||||
case h.askKinds(r)[kind] && before && canReplace:
|
||||
what := "raised"
|
||||
if reminded {
|
||||
@@ -125,28 +192,10 @@ func (h *Holder) reoffer(ch Channel, lost time.Time) bool {
|
||||
}
|
||||
}
|
||||
h.record(Sent{At: now, Channel: kind, Key: "(answers lost)", What: "at " + lost.UTC().Format(time.RFC3339),
|
||||
Outcome: fmt.Sprintf("%d open and asked said again, %d silenced said so, %d ended left ended, %d failed",
|
||||
again, silenced, ended, failed)})
|
||||
h.Logf("[messenger] %s lost its answers at %s: %d open and asked said again in place, %d silenced said so, %d ended "+
|
||||
"left ended, %d failed; nothing shown from the channel's memory", kind, lost.UTC().Format(time.RFC3339), again,
|
||||
silenced, ended, failed)
|
||||
Outcome: fmt.Sprintf("%d open and asked said again, %d silenced, withheld or answered left as they are, "+
|
||||
"%d ended left ended, %d failed", again, left, ended, failed)})
|
||||
h.Logf("[messenger] %s lost its answers at %s: %d open and asked said again in place, %d silenced, withheld or "+
|
||||
"answered left as they are, %d ended left ended, %d failed; nothing shown from the channel's memory", kind,
|
||||
lost.UTC().Format(time.RFC3339), again, left, ended, failed)
|
||||
return failed == 0
|
||||
}
|
||||
|
||||
// editOn edits a record's message on one channel, where it is shown as a message of its own.
|
||||
func (h *Holder) editOn(ch Channel, r *Record, what string) {
|
||||
kind := ch.Name()
|
||||
h.mu.Lock()
|
||||
id := r.Sent[kind]
|
||||
h.mu.Unlock()
|
||||
if id == "" || !ch.CanEdit() {
|
||||
return
|
||||
}
|
||||
err := ch.Edit(id, h.sayOn(r, what, kind))
|
||||
h.noteChannel(kind, err)
|
||||
if err != nil {
|
||||
h.Logf("[messenger] could not edit the message for %s on %s (%v)", r.Key, kind, err)
|
||||
return
|
||||
}
|
||||
h.record(Sent{At: h.Now(), Channel: kind, Key: r.Key, What: what, Outcome: "edited"})
|
||||
}
|
||||
|
||||
@@ -12,8 +12,8 @@ import (
|
||||
// resolved, silenced and cancelled ones among them. Now the desk says in its standing when the answers were
|
||||
// lost, and this router says again only what is still open and unanswered: with the desk showing one open
|
||||
// message with an answer, one silenced, one open asking nothing and one resolved, one is said again under its
|
||||
// handle (in place of its message), with its answers; one is edited to the silenced words, with no answers
|
||||
// and no urgency; nothing is said for the others. A standing that repeats the time says nothing more; its
|
||||
// handle (in place of its message), with its answers; nothing is said for the others — the silenced one
|
||||
// included (novox/hq ADR 0308 §6, issue 531: its edit showed it on the desk again). A standing that repeats the time says nothing more; its
|
||||
// ticket is one-time still, and the channel that reaches the operator away hears nothing of it.
|
||||
func TestLostAnswersGetOnlyWhatIsOpenAndUnansweredSaidAgain(t *testing.T) {
|
||||
h, tg, dt, c, _, w := newRouter(t)
|
||||
@@ -77,8 +77,7 @@ func TestLostAnswersGetOnlyWhatIsOpenAndUnansweredSaidAgain(t *testing.T) {
|
||||
if e := dt.edits["g14=4"]; !strings.HasPrefix(e.Title, "Resolved: ") {
|
||||
t.Fatalf("the clearing was not said: %+v", e)
|
||||
}
|
||||
silencedBefore := dt.edits["g14=2"]
|
||||
delete(dt.edits, "g14=2") // so the edit said again is seen as such
|
||||
delete(dt.edits, "g14=2") // so an edit said again would be seen as such
|
||||
sends, edits, replaced, tgSaid := len(dt.sends), len(dt.edits), len(dt.replaced), saidOn(tg)
|
||||
// The desk's notifier started again: its standing says when the answers were lost.
|
||||
c.pass(time.Minute)
|
||||
@@ -95,12 +94,11 @@ func TestLostAnswersGetOnlyWhatIsOpenAndUnansweredSaidAgain(t *testing.T) {
|
||||
if ticketFor(again, "Silence for a week") == "" || !strings.Contains(again.Title, headlineOf(open.Key)) {
|
||||
t.Fatalf("said again without its answer or its words: %+v", again)
|
||||
}
|
||||
if e, said := dt.edits["g14=2"]; !said || !strings.HasPrefix(e.Title, "Silenced until") || len(e.Actions) != 0 || e.Urgent ||
|
||||
!e.Quiet || e.Title != silencedBefore.Title {
|
||||
t.Fatalf("the silenced one was not edited to the silenced words again: %+v (said %v)", e, said)
|
||||
if e, said := dt.edits["g14=2"]; said {
|
||||
t.Fatalf("the silenced one was said again for the restart: %+v", e)
|
||||
}
|
||||
if len(dt.edits) != edits+1 {
|
||||
t.Fatalf("an edit reached a message that is not silenced: %+v", dt.edits)
|
||||
if len(dt.edits) != edits {
|
||||
t.Fatalf("an edit reached the desk for the restart: %+v", dt.edits)
|
||||
}
|
||||
if e := dt.edits["g14=4"]; !strings.HasPrefix(e.Title, "Resolved: ") {
|
||||
t.Fatalf("the resolved one was changed: %+v", e)
|
||||
@@ -179,8 +177,13 @@ func TestLostAnswersAreJudgedPerMessageAndWaitForTheDesk(t *testing.T) {
|
||||
if len(dt.replaced) != replaced+1 || saidOn(tg) != tgSaid {
|
||||
t.Fatalf("once the desk can send: desk %+v, telegram %+v", dt.replaced[replaced:], tg.sends)
|
||||
}
|
||||
// The router started again, remembering no standing it acted on: the message was said since, so nothing.
|
||||
h.shows = map[string]time.Time{}
|
||||
// The router started again, reading back the pass it made from its state: nothing.
|
||||
h.mu.Lock()
|
||||
h.passes, h.lost = map[string]Pass{}, map[string]time.Time{}
|
||||
if err := h.loadWalks(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
h.mu.Unlock()
|
||||
h.Stand("desktop", Standing{Ready: true, At: c.now(), AnswersLostAt: lost})
|
||||
runFor(h, c, time.Minute)
|
||||
if len(dt.replaced) != replaced+1 {
|
||||
@@ -191,12 +194,12 @@ func TestLostAnswersAreJudgedPerMessageAndWaitForTheDesk(t *testing.T) {
|
||||
lost = c.now()
|
||||
dt.fail = errorf("no session answers")
|
||||
h.Stand("desktop", Standing{Ready: true, At: c.now(), AnswersLostAt: lost})
|
||||
runFor(h, c, 30*time.Second)
|
||||
dt.fail = nil
|
||||
runFor(h, c, time.Minute)
|
||||
if len(dt.replaced) != replaced+1 || saidOn(tg) != tgSaid {
|
||||
t.Fatalf("after a failed send: desk %+v, telegram %+v", dt.replaced[replaced:], tg.sends)
|
||||
}
|
||||
h.Stand("desktop", Standing{Ready: true, At: c.now(), AnswersLostAt: lost})
|
||||
runFor(h, c, time.Minute)
|
||||
if len(dt.replaced) != replaced+2 || saidOn(tg) != tgSaid {
|
||||
t.Fatalf("the failed send was not tried again on the desk: desk %+v, telegram %+v", dt.replaced[replaced:], tg.sends)
|
||||
}
|
||||
|
||||
@@ -113,6 +113,9 @@ func (h *Holder) Status(listening string) Status {
|
||||
"deduplicated by the condition's key; reminded once after 1 h (urgent) or 12 h (warning); cleared by editing the first message",
|
||||
"at most 20 messages an hour per channel, a last line of defence: reached, it is said once and the rest held for one digest",
|
||||
"refused: anything carrying an address, a path or a secret; channel-refused is sent with the words withheld",
|
||||
"while a delivery walks, a message about a machine or module it touches is held, and the walk's end is one message: its outcome, what it reached, how long, what is still open; past the walk's bound (twice its largest delivery budget, 15 min with none) what is held and open is said (novox/hq ADR 0308)",
|
||||
"an ask, a secret given or replaced, and a walk's own failure are never held",
|
||||
"a channel that lost its answers is said again only what is open, asked there and never answered — never a silenced or withheld message — once per restart, and the pass is kept in this module's state",
|
||||
"a silenced condition's message says it is silenced, until when and why, and nothing more is said of it while the silence lasts; a silence is the controller's, asked of the operator or set through the mesh",
|
||||
}
|
||||
anyCan := false
|
||||
@@ -152,9 +155,11 @@ type OpenMessage struct {
|
||||
Reminded bool `json:"reminded"`
|
||||
Unsent string `json:"unsent,omitempty"`
|
||||
Held bool `json:"held,omitempty"`
|
||||
History bool `json:"never_said,omitempty"`
|
||||
Refused string `json:"refused,omitempty"`
|
||||
Count int `json:"times_opened"`
|
||||
// HeldBy are the walks holding it until they end or pass their bound (novox/hq ADR 0308).
|
||||
HeldBy []string `json:"held_by_walks,omitempty"`
|
||||
History bool `json:"never_said,omitempty"`
|
||||
Refused string `json:"refused,omitempty"`
|
||||
Count int `json:"times_opened"`
|
||||
}
|
||||
|
||||
// Open is what is unresolved now, urgent first, oldest first.
|
||||
@@ -180,6 +185,7 @@ func (h *Holder) Open() []OpenMessage {
|
||||
o.SentOn = append(o.SentOn, ch)
|
||||
}
|
||||
sort.Strings(o.SentOn)
|
||||
o.HeldBy = h.heldBy(r.Key, now)
|
||||
if r.silenced(now) {
|
||||
o.Silenced = r.SilencedTill.UTC().Format(time.RFC3339)
|
||||
}
|
||||
|
||||
@@ -103,7 +103,9 @@ func (h *Holder) Sync(open []Condition, asked time.Time) SyncResult {
|
||||
switch {
|
||||
case told || silenced || h.queued(r.Key):
|
||||
case r.Severity == Urgent:
|
||||
urgent = append(urgent, r)
|
||||
if !h.hold(r, "raised") { // an open walk touches it: told at its end or bound (ADR 0308 §2)
|
||||
urgent = append(urgent, r)
|
||||
}
|
||||
case !known && now.Sub(r.Raised) <= FreshFor:
|
||||
fresh = append(fresh, r)
|
||||
case wasQuiet && !needsNothing(r):
|
||||
|
||||
@@ -0,0 +1,906 @@
|
||||
package main
|
||||
|
||||
// While a delivery walks, what it touches is held from the operator and told once, with the walk's result,
|
||||
// when it ends (novox/hq ADR 0308, issue 532).
|
||||
//
|
||||
// **What this reads.** The controller says each walk's start and end on its `plan-moved` event (the plan
|
||||
// whole, ADR 0282 §6), and on those two moments the body carries beside the plan (ADR 0308 §1):
|
||||
//
|
||||
// - `walk`: the plan's id (`id` is read when it is missing);
|
||||
// - `phase`: `started` or `ended`; a plan-moved without it is a move inside a walk and is not read here;
|
||||
// - `touches`: `[{machine, modules}]`, every module the walk moves on every machine it sends it to, and every
|
||||
// module a moved tool runner or node-engine carries there;
|
||||
// - `outcome`: `delivered`, `failed` or `stopped`, on `ended`;
|
||||
// - `bound_seconds`: twice the largest delivery budget among its deliveries, or 900 when none has one (§5),
|
||||
// on `started`; missing, 900 is taken;
|
||||
// - `budget_class`: `core`, `leaf` or `none`;
|
||||
// - `started_at` on the start and `ended_at` on the end (else `at`, else the plan's `updated`): when it
|
||||
// happened. A start older than its bound is history and opens nothing; an end older than FreshFor with
|
||||
// nothing held closes quietly.
|
||||
//
|
||||
// A touch may name `carried` beside `modules` (what a moved tool runner or node-engine carries); the
|
||||
// controller lists those in `modules` too, which is what is read.
|
||||
//
|
||||
// An event of its own named `walk`, with the same body, is read the same way, should the controller say it
|
||||
// there instead.
|
||||
//
|
||||
// **What is held (§2).** While a walk is open and within its bound, a message whose subject is a machine the
|
||||
// walk touches, or a module it touches, is held instead of sent: kept on the walk, with when it was held and
|
||||
// what was to be said, and the channel's history records it with the outcome `held`. A message touched by
|
||||
// several open walks is kept on each, and told when the last of them ends or passes its bound.
|
||||
//
|
||||
// **What never is (§3):** an ask waiting for the operator's answer (an ask's own message, and a condition an
|
||||
// open ask is about), a secret given or replaced, a message about a machine or module no open walk touches,
|
||||
// and a walk's own failure: anything about a plan, a walk, a delivery or a build.
|
||||
//
|
||||
// **The end (§4).** One message: its outcome, the modules and machines it reached and how long it took; then
|
||||
// each held message whose condition is still open, one line each; the ones that cleared meanwhile only
|
||||
// counted. A walk that delivered with nothing held and nothing open is one line, and nothing at all on a kind
|
||||
// the operator set to say only what needs them (the setting `only-what-needs-me`).
|
||||
//
|
||||
// **The bound (§5).** At the walk's start plus its bound, what it holds that is still open is said, with a line
|
||||
// that the walk is still running and since when, and the cleared ones counted; it holds nothing more after
|
||||
// that. A walk whose end never comes is forgotten a day after it started.
|
||||
//
|
||||
// **Kept in the module's state** (the bucket `walks`, one key per walk), so a router that restarts mid-walk
|
||||
// holds and tells as if it had not.
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// The walk events, by their local names under the controller's seat.
|
||||
const (
|
||||
EventPlanMoved = "plan-moved"
|
||||
EventWalk = "walk"
|
||||
|
||||
WalkStarted = "started"
|
||||
WalkEnded = "ended"
|
||||
|
||||
WalkDelivered = "delivered"
|
||||
WalkFailed = "failed"
|
||||
WalkStopped = "stopped"
|
||||
|
||||
// DefaultWalkBound is a walk's bound when the controller gives none: the bound of a walk none of whose
|
||||
// deliveries has a budget (ADR 0308 §5).
|
||||
DefaultWalkBound = 15 * time.Minute
|
||||
// WalkForget: a walk whose end never came is forgotten this long after it started.
|
||||
WalkForget = 24 * time.Hour
|
||||
// WalkEndWhat is the outbox's name for a walk's end or bound message: it goes alone, never in a digest.
|
||||
WalkEndWhat = "walk-end"
|
||||
// MostNamed: an end message names at most this many modules or machines, then counts the rest.
|
||||
MostNamed = 6
|
||||
)
|
||||
|
||||
// Touch is what a walk moves on one machine.
|
||||
type Touch struct {
|
||||
Machine string `json:"machine"`
|
||||
Modules []string `json:"modules"`
|
||||
}
|
||||
|
||||
// WalkEvent is a walk's start or end as the controller says it.
|
||||
type WalkEvent struct {
|
||||
Walk string
|
||||
Phase string
|
||||
Outcome string
|
||||
Class string
|
||||
Touches []Touch
|
||||
BoundSeconds int
|
||||
At time.Time
|
||||
// StartedAt is the walk's own start, on both events (`started_at`): the bound runs from it, so a router
|
||||
// that hears a start late, or starts again, ends the hold on time.
|
||||
StartedAt time.Time
|
||||
}
|
||||
|
||||
// HeldMessage is one message a walk holds: when it was first held and what was to be said.
|
||||
type HeldMessage struct {
|
||||
At time.Time `json:"at"`
|
||||
What string `json:"what"`
|
||||
}
|
||||
|
||||
// Walk is one open walk as this router keeps it.
|
||||
type Walk struct {
|
||||
ID string `json:"id"`
|
||||
Started time.Time `json:"started"`
|
||||
BoundSeconds int `json:"bound_seconds"`
|
||||
Class string `json:"budget_class,omitempty"`
|
||||
Touches []Touch `json:"touches"`
|
||||
Held map[string]HeldMessage `json:"held,omitempty"`
|
||||
// BoundSaid: its bound passed and what it held was said; it holds nothing more.
|
||||
BoundSaid bool `json:"bound_said,omitempty"`
|
||||
// Ended: its end was heard, at this moment. Kept a day, so a start heard after its end opens nothing.
|
||||
Ended time.Time `json:"ended,omitempty"`
|
||||
// EndedSaid is the end's moment as the controller said it, by the controller's clock and never capped at
|
||||
// this router's now: a start heard after its end is compared with it on that one clock (issue 538 C).
|
||||
EndedSaid time.Time `json:"ended_said,omitempty"`
|
||||
}
|
||||
|
||||
func (w *Walk) bound() time.Duration {
|
||||
if w.BoundSeconds <= 0 {
|
||||
return DefaultWalkBound
|
||||
}
|
||||
return time.Duration(w.BoundSeconds) * time.Second
|
||||
}
|
||||
|
||||
// holding says whether the walk still holds at a moment: open, and within its bound.
|
||||
func (w *Walk) holding(now time.Time) bool {
|
||||
return !w.BoundSaid && now.Before(w.Started.Add(w.bound()))
|
||||
}
|
||||
|
||||
// touches says whether the walk touches a subject: a machine it sends to, or — when the subject names no
|
||||
// machine of its own — a module it moves anywhere.
|
||||
func (w *Walk) touches(s subject) bool {
|
||||
for _, t := range w.Touches {
|
||||
for _, m := range s.machines {
|
||||
if m != "" && m == t.Machine {
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
if s.machineNamed {
|
||||
return false
|
||||
}
|
||||
for _, t := range w.Touches {
|
||||
for _, mod := range t.Modules {
|
||||
for _, m := range s.modules {
|
||||
if m != "" && m == mod {
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// walkKeeper is a store that keeps the walks and the restart passes too (the module's state, ADR 0201).
|
||||
type walkKeeper interface {
|
||||
PutWalk(w Walk) error
|
||||
DeleteWalk(id string) error
|
||||
Walks() ([]Walk, error)
|
||||
PutPasses(map[string]Pass) error
|
||||
Passes() (map[string]Pass, error)
|
||||
}
|
||||
|
||||
// DecodeWalk reads a plan-moved or walk event's body: false when it says no walk's start or end.
|
||||
func DecodeWalk(body []byte) (WalkEvent, bool, error) {
|
||||
var raw map[string]json.RawMessage
|
||||
if err := json.Unmarshal(body, &raw); err != nil || raw == nil {
|
||||
return WalkEvent{}, false, fmt.Errorf("a walk event's body is not a JSON object")
|
||||
}
|
||||
str := func(names ...string) string {
|
||||
for _, n := range names {
|
||||
var s string
|
||||
if v, ok := raw[n]; ok && json.Unmarshal(v, &s) == nil && strings.TrimSpace(s) != "" {
|
||||
return strings.TrimSpace(s)
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
var ev WalkEvent
|
||||
ev.Phase = str("phase")
|
||||
if ev.Phase == "" {
|
||||
return WalkEvent{}, false, nil
|
||||
}
|
||||
ev.Walk = str("walk", "id")
|
||||
if ev.Walk == "" {
|
||||
return WalkEvent{}, false, fmt.Errorf("a walk's %s names no walk", ev.Phase)
|
||||
}
|
||||
switch ev.Phase {
|
||||
case WalkStarted, WalkEnded:
|
||||
default:
|
||||
return WalkEvent{}, false, fmt.Errorf("walk %s: phase %q is neither started nor ended", ev.Walk, ev.Phase)
|
||||
}
|
||||
ev.Outcome = str("outcome")
|
||||
if ev.Phase == WalkEnded {
|
||||
switch ev.Outcome {
|
||||
case WalkDelivered, WalkFailed, WalkStopped:
|
||||
default:
|
||||
return WalkEvent{}, false, fmt.Errorf("walk %s ended with outcome %q: neither delivered, failed nor stopped", ev.Walk, ev.Outcome)
|
||||
}
|
||||
}
|
||||
ev.Class = str("budget_class", "budget-class")
|
||||
for _, n := range []string{"bound_seconds", "bound-seconds"} {
|
||||
if v, ok := raw[n]; ok && string(v) != "null" {
|
||||
var f float64
|
||||
if json.Unmarshal(v, &f) != nil || f < 0 {
|
||||
return WalkEvent{}, false, fmt.Errorf("walk %s: %s is not a number of seconds", ev.Walk, n)
|
||||
}
|
||||
ev.BoundSeconds = int(f)
|
||||
break
|
||||
}
|
||||
}
|
||||
if v, ok := raw["touches"]; ok && string(v) != "null" {
|
||||
var list []struct {
|
||||
Machine string `json:"machine"`
|
||||
Node string `json:"node"`
|
||||
Modules []string `json:"modules"`
|
||||
}
|
||||
if json.Unmarshal(v, &list) != nil {
|
||||
return WalkEvent{}, false, fmt.Errorf("walk %s: touches is not a list of {machine, modules}", ev.Walk)
|
||||
}
|
||||
for _, t := range list {
|
||||
m := t.Machine
|
||||
if m == "" {
|
||||
m = t.Node
|
||||
}
|
||||
if m != "" {
|
||||
ev.Touches = append(ev.Touches, Touch{Machine: m, Modules: t.Modules})
|
||||
}
|
||||
}
|
||||
}
|
||||
if s := str("started_at"); s != "" {
|
||||
if t, err := time.Parse(time.RFC3339Nano, s); err == nil {
|
||||
ev.StartedAt = t
|
||||
}
|
||||
}
|
||||
when := []string{"started_at", "at", "updated"}
|
||||
if ev.Phase == WalkEnded {
|
||||
when = []string{"ended_at", "at", "updated"}
|
||||
}
|
||||
for _, n := range when {
|
||||
if s := str(n); s != "" {
|
||||
if t, err := time.Parse(time.RFC3339Nano, s); err == nil {
|
||||
ev.At = t
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
return ev, true, nil
|
||||
}
|
||||
|
||||
// mergeTouches is what two sayings of a walk's touches name together.
|
||||
func mergeTouches(a, b []Touch) []Touch {
|
||||
by := map[string]map[string]bool{}
|
||||
var order []string
|
||||
for _, list := range [][]Touch{a, b} {
|
||||
for _, t := range list {
|
||||
if by[t.Machine] == nil {
|
||||
by[t.Machine] = map[string]bool{}
|
||||
order = append(order, t.Machine)
|
||||
}
|
||||
for _, m := range t.Modules {
|
||||
by[t.Machine][m] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
sort.Strings(order)
|
||||
var out []Touch
|
||||
for _, machine := range order {
|
||||
var mods []string
|
||||
for m := range by[machine] {
|
||||
mods = append(mods, m)
|
||||
}
|
||||
sort.Strings(mods)
|
||||
out = append(out, Touch{Machine: machine, Modules: mods})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Walked takes a walk's start or end.
|
||||
func (h *Holder) Walked(ev WalkEvent) {
|
||||
h.work.Lock()
|
||||
defer h.work.Unlock()
|
||||
h.mu.Lock()
|
||||
h.init()
|
||||
now := h.Now()
|
||||
h.mu.Unlock()
|
||||
at := ev.At
|
||||
if at.IsZero() || at.After(now) {
|
||||
at = now
|
||||
}
|
||||
switch ev.Phase {
|
||||
case WalkStarted:
|
||||
w := &Walk{ID: ev.Walk, Started: at, BoundSeconds: ev.BoundSeconds, Class: ev.Class}
|
||||
h.mu.Lock()
|
||||
end := h.ended[ev.Walk]
|
||||
h.mu.Unlock()
|
||||
if end != nil && startOfEnded(ev, at, end) {
|
||||
// A short walk's two events can come in either order: this start is of the walk that ended.
|
||||
h.Logf("[messenger] walk %s: its start came after its end, which was heard at %s: nothing held", ev.Walk,
|
||||
end.Ended.UTC().Format(time.RFC3339))
|
||||
return
|
||||
}
|
||||
if now.Sub(at) >= w.bound() {
|
||||
h.Logf("[messenger] walk %s started %s ago, past its bound of %s: history, nothing held", ev.Walk,
|
||||
roughly(now.Sub(at)), w.bound())
|
||||
return
|
||||
}
|
||||
h.mu.Lock()
|
||||
if known := h.walks[ev.Walk]; known != nil {
|
||||
w = known // said again: the same walk, its touches as both sayings name them
|
||||
} else {
|
||||
w.Held = map[string]HeldMessage{}
|
||||
h.walks[ev.Walk] = w
|
||||
}
|
||||
w.Touches = mergeTouches(w.Touches, ev.Touches)
|
||||
h.mu.Unlock()
|
||||
h.keepWalk(w)
|
||||
h.record(Sent{At: now, Channel: "-", Key: "walk." + ev.Walk, What: "started: " + touchWords(w.Touches),
|
||||
Outcome: "holding what it touches for at most " + roughly(w.bound())})
|
||||
case WalkEnded:
|
||||
h.mu.Lock()
|
||||
w := h.walks[ev.Walk]
|
||||
if w != nil && !ev.StartedAt.IsZero() && w.Started.After(ev.StartedAt) {
|
||||
// The end of an earlier walk under the same id, heard after the next one started: that one stays open.
|
||||
h.mu.Unlock()
|
||||
h.Logf("[messenger] walk %s: an end of its earlier walk (started %s) came late: the open one stays", ev.Walk,
|
||||
ev.StartedAt.UTC().Format(time.RFC3339))
|
||||
return
|
||||
}
|
||||
if w != nil {
|
||||
delete(h.walks, ev.Walk)
|
||||
w.Touches = mergeTouches(w.Touches, ev.Touches)
|
||||
if !ev.StartedAt.IsZero() {
|
||||
w.Started = ev.StartedAt
|
||||
}
|
||||
}
|
||||
tomb := &Walk{ID: ev.Walk, Started: ev.StartedAt, Ended: at, EndedSaid: ev.At}
|
||||
if prev := h.ended[ev.Walk]; prev != nil && prev.Ended.After(at) {
|
||||
tomb = prev
|
||||
}
|
||||
h.ended[ev.Walk] = tomb
|
||||
h.mu.Unlock()
|
||||
h.keepWalk(tomb) // in place of the open walk: a start heard after this end opens nothing
|
||||
if w == nil {
|
||||
h.Logf("[messenger] walk %s %s, and its start was not heard here: nothing held, nothing said", ev.Walk, ev.Outcome)
|
||||
return
|
||||
}
|
||||
h.endWalk(w, ev.Outcome, at, now)
|
||||
}
|
||||
}
|
||||
|
||||
// startOfEnded says whether a start is of a walk whose end was heard already: of that walk, or of an earlier one
|
||||
// under its id. The controller's own moments are compared on its clock, never one capped at this router's now
|
||||
// against one that is not (issue 538 C: a controller's clock ahead of the router's would open a walk after its
|
||||
// end). Only when the controller said neither moment are the moments as heard here compared.
|
||||
func startOfEnded(ev WalkEvent, at time.Time, end *Walk) bool {
|
||||
said := ev.StartedAt
|
||||
if said.IsZero() {
|
||||
said = ev.At
|
||||
}
|
||||
switch {
|
||||
case !said.IsZero() && !end.Started.IsZero():
|
||||
return !said.After(end.Started)
|
||||
case !said.IsZero() && !end.EndedSaid.IsZero():
|
||||
return !said.After(end.EndedSaid)
|
||||
}
|
||||
return !at.After(end.Ended)
|
||||
}
|
||||
|
||||
// release is what a walk held that no other open walk holds or touches: each open message, oldest first, with
|
||||
// what was to be said of it, and how many cleared meanwhile. A message said meanwhile on a path that is never
|
||||
// held, or silenced meanwhile, is neither. One that another open walk touches — even one that started after it
|
||||
// was held — is held on by that walk, and told at its end or bound (ADR 0308 §2).
|
||||
//
|
||||
// A message is handed on under the rules of its first hold (issue 538 B): still open, not silenced, not said
|
||||
// since it was held, and not one ADR 0308 §3 never holds (an open ask about it, a walk's own failure).
|
||||
func (h *Holder) release(w *Walk, now time.Time) (open []*Record, whats map[string]string, cleared int) {
|
||||
// What may be handed on is judged first: neverHeld asks the asks and the walks under the lock itself.
|
||||
h.mu.Lock()
|
||||
recs := map[string]*Record{}
|
||||
for key := range w.Held {
|
||||
if r := h.open[key]; r != nil && r.Cleared.IsZero() && !r.silenced(now) && !lastSaid(r).After(w.Held[key].At) {
|
||||
recs[key] = r
|
||||
}
|
||||
}
|
||||
h.mu.Unlock()
|
||||
handable := map[string]bool{}
|
||||
for key, r := range recs {
|
||||
handable[key] = !h.neverHeld(r)
|
||||
}
|
||||
h.mu.Lock()
|
||||
whats = map[string]string{}
|
||||
taken := map[string]*Walk{}
|
||||
var handed []string
|
||||
for key, hm := range w.Held {
|
||||
r := h.open[key]
|
||||
live := r != nil && r.Cleared.IsZero() && handable[key]
|
||||
var s subject
|
||||
if live {
|
||||
s = subjectOf(r)
|
||||
}
|
||||
elsewhere := false
|
||||
for _, o := range h.walks {
|
||||
if o.ID == w.ID || !o.holding(now) {
|
||||
continue
|
||||
}
|
||||
if _, ok := o.Held[key]; ok {
|
||||
elsewhere = true
|
||||
continue
|
||||
}
|
||||
if live && o.touches(s) {
|
||||
if o.Held == nil {
|
||||
o.Held = map[string]HeldMessage{}
|
||||
}
|
||||
o.Held[key] = hm
|
||||
taken[o.ID] = o
|
||||
handed = append(handed, key+" to walk "+o.ID)
|
||||
elsewhere = true
|
||||
}
|
||||
}
|
||||
if elsewhere {
|
||||
continue
|
||||
}
|
||||
switch {
|
||||
case r == nil || !r.Cleared.IsZero():
|
||||
cleared++
|
||||
case r.silenced(now) || lastSaid(r).After(hm.At):
|
||||
default:
|
||||
open = append(open, r)
|
||||
whats[key] = hm.What
|
||||
}
|
||||
}
|
||||
sort.Slice(open, func(i, j int) bool {
|
||||
if !open[i].Raised.Equal(open[j].Raised) {
|
||||
return open[i].Raised.Before(open[j].Raised)
|
||||
}
|
||||
return open[i].Key < open[j].Key
|
||||
})
|
||||
h.mu.Unlock()
|
||||
for _, o := range taken {
|
||||
h.keepWalk(o)
|
||||
}
|
||||
sort.Strings(handed)
|
||||
for _, x := range handed {
|
||||
key, _, _ := strings.Cut(x, " to walk ")
|
||||
h.record(Sent{At: now, Channel: "-", Key: key, What: "held on: walk " + w.ID + " ended, " + x[len(key)+1:] + " touches it",
|
||||
Outcome: "held"})
|
||||
}
|
||||
return open, whats, cleared
|
||||
}
|
||||
|
||||
// endWalk tells a walk's end once (ADR 0308 §4).
|
||||
func (h *Holder) endWalk(w *Walk, outcome string, ended, now time.Time) {
|
||||
open, whats, cleared := h.release(w, now)
|
||||
if now.Sub(ended) > FreshFor && len(open) == 0 {
|
||||
h.Logf("[messenger] walk %s %s %s ago, heard late: history, nothing said (%d held cleared meanwhile)",
|
||||
w.ID, outcome, roughly(now.Sub(ended)), cleared)
|
||||
h.record(Sent{At: now, Channel: "-", Key: "walk." + w.ID, What: "ended " + outcome, Outcome: "quiet: heard late"})
|
||||
return
|
||||
}
|
||||
took := ended.Sub(w.Started)
|
||||
head := map[string]string{WalkDelivered: "Delivered", WalkFailed: "Delivery failed", WalkStopped: "Delivery stopped"}[outcome]
|
||||
if head == "" {
|
||||
head = "Delivery ended"
|
||||
}
|
||||
reached := touchWords(w.Touches)
|
||||
oneLine := outcome == WalkDelivered && len(w.Held) == 0 && len(open) == 0
|
||||
var m Message
|
||||
if oneLine {
|
||||
m = Message{Title: head + ": " + reached + ", in " + roughly(took), Quiet: true}
|
||||
} else {
|
||||
lines := []string{"It took " + roughly(took) + "."}
|
||||
lines = append(lines, h.heldLines(open, whats, cleared)...)
|
||||
m = Message{Title: head + ": " + reached, Body: strings.Join(lines, "\n")}
|
||||
}
|
||||
h.tellWalk(w, m, open, whats, oneLine, "ended "+outcome, now)
|
||||
}
|
||||
|
||||
// heldLines are the lines a walk's end or bound says of what it held: each still open, then the cleared count.
|
||||
func (h *Holder) heldLines(open []*Record, whats map[string]string, cleared int) []string {
|
||||
var lines []string
|
||||
if len(open) > 0 {
|
||||
lines = append(lines, "Still open after it:")
|
||||
for _, r := range open {
|
||||
what := whats[r.Key]
|
||||
if what == "" {
|
||||
what = "raised"
|
||||
}
|
||||
lines = append(lines, "- "+h.say(r, what).Title)
|
||||
}
|
||||
}
|
||||
switch {
|
||||
case cleared == 1:
|
||||
lines = append(lines, "1 problem it raised resolved during it.")
|
||||
case cleared > 1:
|
||||
lines = append(lines, fmt.Sprintf("%d problems it raised resolved during it.", cleared))
|
||||
}
|
||||
return lines
|
||||
}
|
||||
|
||||
// boundWalks says, of every walk past its bound, what it holds that is still open (ADR 0308 §5), then forgets
|
||||
// the walks whose end never came, and the ends heard a day ago. What a walk holds is said before it is
|
||||
// forgotten: a router back after a long while says it, then forgets the walk.
|
||||
func (h *Holder) boundWalks(now time.Time) {
|
||||
h.mu.Lock()
|
||||
var due, gone []*Walk
|
||||
var stale []string
|
||||
for _, w := range h.walks {
|
||||
if !w.BoundSaid && !now.Before(w.Started.Add(w.bound())) {
|
||||
due = append(due, w)
|
||||
}
|
||||
if now.Sub(w.Started) >= WalkForget {
|
||||
gone = append(gone, w)
|
||||
}
|
||||
}
|
||||
for id, e := range h.ended {
|
||||
if now.Sub(e.Ended) >= WalkForget {
|
||||
stale = append(stale, id)
|
||||
}
|
||||
}
|
||||
for _, id := range stale {
|
||||
delete(h.ended, id)
|
||||
}
|
||||
h.mu.Unlock()
|
||||
for _, id := range stale {
|
||||
h.mu.Lock()
|
||||
reopened := h.walks[id] != nil
|
||||
h.mu.Unlock()
|
||||
if !reopened {
|
||||
h.forgetWalk(id)
|
||||
}
|
||||
}
|
||||
defer func() {
|
||||
for _, w := range gone {
|
||||
h.mu.Lock()
|
||||
delete(h.walks, w.ID)
|
||||
h.mu.Unlock()
|
||||
h.Logf("[messenger] walk %s started %s ago and its end never came: forgotten", w.ID, roughly(now.Sub(w.Started)))
|
||||
h.forgetWalk(w.ID)
|
||||
}
|
||||
}()
|
||||
sort.Slice(due, func(i, j int) bool { return due[i].Started.Before(due[j].Started) })
|
||||
for _, w := range due {
|
||||
h.mu.Lock()
|
||||
w.BoundSaid = true
|
||||
h.mu.Unlock()
|
||||
open, whats, cleared := h.release(w, now)
|
||||
h.mu.Lock()
|
||||
w.Held = map[string]HeldMessage{}
|
||||
h.mu.Unlock()
|
||||
h.keepWalk(w)
|
||||
if len(open) == 0 && cleared == 0 {
|
||||
h.record(Sent{At: now, Channel: "-", Key: "walk." + w.ID, What: "past its bound", Outcome: "nothing held"})
|
||||
continue
|
||||
}
|
||||
since := w.Started.In(h.zone()).Format("15:04")
|
||||
lines := []string{"The walk " + w.ID + " is still running, since " + since + "."}
|
||||
if _, ok := Check(lines[0]); !ok {
|
||||
lines[0] = "The walk is still running, since " + since + "."
|
||||
}
|
||||
lines = append(lines, h.heldLines(open, whats, cleared)...)
|
||||
m := Message{Title: "Still running: the delivery to " + touchWords(w.Touches), Body: strings.Join(lines, "\n")}
|
||||
h.tellWalk(w, m, open, whats, false, "past its bound", now)
|
||||
}
|
||||
}
|
||||
|
||||
// tellWalk holds a walk's message on the kinds it goes to, alone, and marks what it carries as said there. When
|
||||
// no channel can send, or the message cannot carry what it held, each held message is left to its own path:
|
||||
// unsent, and tried again every minute as any other (outbox.go), or said on its own.
|
||||
func (h *Holder) tellWalk(w *Walk, m Message, open []*Record, whats map[string]string, oneLine bool, what string, now time.Time) {
|
||||
for _, r := range open {
|
||||
m.Urgent = m.Urgent || r.Severity == Urgent
|
||||
}
|
||||
var alone []*Record // said on their own, after the walk's message
|
||||
if _, ok := Check(m.Text()); !ok {
|
||||
m = Message{Title: "A delivery ended", Body: "What it reached could not be said here: the delivery's walks say it.", Urgent: m.Urgent}
|
||||
alone, open = open, nil
|
||||
}
|
||||
// Owed until a message carrying it was sent, in the state: a router that restarts first says it on its own
|
||||
// (issue 538), whichever path below it takes.
|
||||
h.mu.Lock()
|
||||
for _, r := range append(append([]*Record(nil), open...), alone...) {
|
||||
say := whats[r.Key]
|
||||
if say == "" {
|
||||
say = "raised"
|
||||
}
|
||||
r.Owed = say
|
||||
}
|
||||
h.mu.Unlock()
|
||||
var to []Channel
|
||||
if m.Urgent {
|
||||
for _, ch := range h.channels() {
|
||||
if ch.Ready() == nil {
|
||||
to = append(to, ch)
|
||||
}
|
||||
}
|
||||
} else {
|
||||
switch {
|
||||
case h.Desk != nil && h.Desk.Ready() == nil:
|
||||
to = []Channel{h.Desk}
|
||||
case h.Away != nil && h.Away.Ready() == nil:
|
||||
to = []Channel{h.Away}
|
||||
}
|
||||
}
|
||||
var kept []Channel
|
||||
for _, ch := range to {
|
||||
if oneLine && h.NeedsOnly != nil && h.NeedsOnly(ch.Name()) {
|
||||
h.record(Sent{At: now, Channel: ch.Name(), Key: "walk." + w.ID, What: what,
|
||||
Outcome: "quiet: this kind says only what needs the operator"})
|
||||
continue
|
||||
}
|
||||
kept = append(kept, ch)
|
||||
}
|
||||
if len(to) == 0 {
|
||||
h.Logf("[messenger] walk %s %s: no channel can send now; what it held is tried again every minute: %s", w.ID, what, h.whyNot())
|
||||
for _, r := range append(open, alone...) {
|
||||
say := whats[r.Key]
|
||||
if say == "" {
|
||||
say = "raised"
|
||||
}
|
||||
h.mu.Lock()
|
||||
r.Pending = say
|
||||
h.mu.Unlock()
|
||||
h.record(Sent{At: now, Channel: "-", Key: r.Key, What: "released by walk " + w.ID + " " + what,
|
||||
Outcome: "unsent: no channel can send; tried again every minute"})
|
||||
h.persist(r)
|
||||
}
|
||||
return
|
||||
}
|
||||
sev := Warning
|
||||
if m.Urgent {
|
||||
sev = Urgent
|
||||
}
|
||||
// What it carries is marked told on a channel once the message was sent there (outbox.go), not now: a
|
||||
// message lost with a restart before it was sent leaves them untold and owed, and the restarted router says
|
||||
// each on its own (issue 538 A).
|
||||
var carries []string
|
||||
for _, r := range open {
|
||||
carries = append(carries, r.Key)
|
||||
}
|
||||
for _, ch := range kept {
|
||||
msg := m
|
||||
h.enqueue(ch.Name(), item{Key: "walk." + w.ID, What: WalkEndWhat, Severity: sev, At: now, Msg: &msg, Carries: carries})
|
||||
}
|
||||
for _, r := range open {
|
||||
if len(kept) > 0 {
|
||||
h.record(Sent{At: now, Channel: "-", Key: r.Key, What: "released",
|
||||
Outcome: "to be told in the message of walk " + w.ID + " " + what})
|
||||
}
|
||||
h.persist(r) // owed, in the state, until the message was sent
|
||||
}
|
||||
h.flush(now, false)
|
||||
for _, r := range alone {
|
||||
say := whats[r.Key]
|
||||
if say == "" {
|
||||
say = "raised"
|
||||
}
|
||||
h.record(Sent{At: now, Channel: "-", Key: r.Key, What: "released", Outcome: "said on its own: walk " + w.ID + "'s message could not carry it"})
|
||||
h.deliver(r, say)
|
||||
h.persist(r)
|
||||
}
|
||||
}
|
||||
|
||||
// touchWords names what a walk reaches: its modules, then the machines.
|
||||
func touchWords(touches []Touch) string {
|
||||
mods := map[string]bool{}
|
||||
var machines []string
|
||||
for _, t := range touches {
|
||||
machines = append(machines, t.Machine)
|
||||
for _, m := range t.Modules {
|
||||
mods[m] = true
|
||||
}
|
||||
}
|
||||
var names []string
|
||||
for m := range mods {
|
||||
names = append(names, m)
|
||||
}
|
||||
sort.Strings(names)
|
||||
sort.Strings(machines)
|
||||
if len(names) == 0 {
|
||||
return "nothing moved"
|
||||
}
|
||||
if len(machines) == 0 {
|
||||
return listed(names)
|
||||
}
|
||||
return listed(names) + " on " + listed(machines)
|
||||
}
|
||||
|
||||
// listed is at most MostNamed names, then how many more.
|
||||
func listed(names []string) string {
|
||||
if len(names) <= MostNamed {
|
||||
return strings.Join(names, ", ")
|
||||
}
|
||||
return strings.Join(names[:MostNamed], ", ") + fmt.Sprintf(" and %d more", len(names)-MostNamed)
|
||||
}
|
||||
|
||||
// subject is what a message is about, as a walk's touches are matched against it.
|
||||
type subject struct {
|
||||
machines, modules []string
|
||||
// machineNamed: the subject names its machine, so a module it names is matched on that machine alone.
|
||||
machineNamed bool
|
||||
// tokens are every name it carries, to tell a message about a walk itself.
|
||||
tokens []string
|
||||
}
|
||||
|
||||
// subjectOf is a record's subject: a condition's scope and id (`<scope>.<id>.<kind>` when the record kept no
|
||||
// subject of its own), a notice's subject words.
|
||||
func subjectOf(r *Record) subject {
|
||||
scope, id := r.Scope, r.SubjectID
|
||||
parts := strings.Split(r.Key, ".")
|
||||
if scope == "" && len(parts) > 0 {
|
||||
scope = parts[0]
|
||||
}
|
||||
if id == "" && len(parts) >= 3 {
|
||||
id = strings.Join(parts[1:len(parts)-1], ".")
|
||||
}
|
||||
var s subject
|
||||
tokens := strings.Split(id, ".")
|
||||
if r.Origin == "notify" {
|
||||
tokens = append(strings.FieldsFunc(r.Subject, func(c rune) bool { return c == ' ' || c == ',' || c == '/' }), parts[1:]...)
|
||||
scope = "notify"
|
||||
}
|
||||
s.tokens = tokens
|
||||
if r.Machine != "" {
|
||||
s.machines, s.machineNamed = append(s.machines, r.Machine), true
|
||||
}
|
||||
switch scope {
|
||||
case "machine", "node", "bus":
|
||||
s.machines, s.machineNamed = append(s.machines, tokens[0]), true
|
||||
if len(tokens) > 1 {
|
||||
s.modules = append(s.modules, tokens[1])
|
||||
}
|
||||
case "module", "core", "seat":
|
||||
s.modules = append(s.modules, tokens[0])
|
||||
if len(tokens) > 1 {
|
||||
s.machines, s.machineNamed = append(s.machines, tokens[1]), true
|
||||
}
|
||||
default:
|
||||
// Its names, each matched as a machine and as a module.
|
||||
s.machines = append(s.machines, tokens...)
|
||||
s.modules = append(s.modules, tokens...)
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// neverHeldScopes are the subjects of a walk's own failure (ADR 0308 §3): a plan, a walk, a delivery, a build.
|
||||
var neverHeldScopes = map[string]bool{"plan": true, "walk": true, "delivery": true, "build": true, "release": true}
|
||||
|
||||
// neverHeldKinds are what is sent at once whatever a walk touches: a secret given or replaced (ADR 0308 §3).
|
||||
var neverHeldKinds = map[string]bool{"secret-given": true, "secret-replaced": true}
|
||||
|
||||
// neverHeld says whether a record is one ADR 0308 §3 sends at once.
|
||||
func (h *Holder) neverHeld(r *Record) bool {
|
||||
if r.Origin == "ask" || h.openAsk(r) != nil {
|
||||
return true
|
||||
}
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
parts := strings.Split(r.Key, ".")
|
||||
kind := r.Kind
|
||||
if kind == "" || kind == "notice" {
|
||||
kind = parts[len(parts)-1]
|
||||
}
|
||||
scope := r.Scope
|
||||
if scope == "" {
|
||||
scope = parts[0]
|
||||
}
|
||||
if neverHeldKinds[kind] || neverHeldScopes[scope] {
|
||||
return true
|
||||
}
|
||||
for _, t := range subjectOf(r).tokens {
|
||||
if h.walks[t] != nil {
|
||||
return true // about a walk itself
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// hold holds a record's message when an open walk touches its subject (ADR 0308 §2): kept on every such walk,
|
||||
// with when it was first held and what was to be said, and said held in the channel's history.
|
||||
func (h *Holder) hold(r *Record, what string) bool {
|
||||
switch what {
|
||||
case "cleared", "silenced", "asked":
|
||||
return false
|
||||
}
|
||||
h.mu.Lock()
|
||||
h.init()
|
||||
walking := len(h.walks) > 0
|
||||
h.mu.Unlock()
|
||||
if !walking || h.neverHeld(r) {
|
||||
return false
|
||||
}
|
||||
now := h.Now()
|
||||
h.mu.Lock()
|
||||
s := subjectOf(r)
|
||||
var by []*Walk
|
||||
var ids []string
|
||||
fresh := false
|
||||
for _, w := range h.walks {
|
||||
if !w.holding(now) || !w.touches(s) {
|
||||
continue
|
||||
}
|
||||
prev, had := w.Held[r.Key]
|
||||
hm := HeldMessage{At: now, What: what}
|
||||
if had {
|
||||
hm.At = prev.At
|
||||
if !r.told() && prev.What == "raised" {
|
||||
hm.What = prev.What // never said yet: it is a new problem to the operator, whatever came since
|
||||
}
|
||||
}
|
||||
if !had || hm.What != prev.What {
|
||||
fresh = true
|
||||
}
|
||||
if w.Held == nil {
|
||||
w.Held = map[string]HeldMessage{}
|
||||
}
|
||||
w.Held[r.Key] = hm
|
||||
by = append(by, w)
|
||||
ids = append(ids, w.ID)
|
||||
}
|
||||
h.mu.Unlock()
|
||||
if len(by) == 0 {
|
||||
return false
|
||||
}
|
||||
sort.Strings(ids)
|
||||
for _, w := range by {
|
||||
h.keepWalk(w)
|
||||
}
|
||||
if fresh {
|
||||
h.record(Sent{At: now, Channel: "-", Key: r.Key, What: what + ", while walk " + strings.Join(ids, ", ") + " touches it",
|
||||
Outcome: "held"})
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// heldBy is the walks holding a record now.
|
||||
func (h *Holder) heldBy(key string, now time.Time) []string {
|
||||
var ids []string
|
||||
for _, w := range h.walks {
|
||||
if _, ok := w.Held[key]; ok && w.holding(now) {
|
||||
ids = append(ids, w.ID)
|
||||
}
|
||||
}
|
||||
sort.Strings(ids)
|
||||
return ids
|
||||
}
|
||||
|
||||
func (h *Holder) keepWalk(w *Walk) {
|
||||
k, ok := h.Store.(walkKeeper)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
h.mu.Lock()
|
||||
c := *w
|
||||
c.Held = map[string]HeldMessage{}
|
||||
for key, v := range w.Held {
|
||||
c.Held[key] = v
|
||||
}
|
||||
h.mu.Unlock()
|
||||
if err := k.PutWalk(c); err != nil {
|
||||
h.noteStore(err)
|
||||
}
|
||||
}
|
||||
|
||||
func (h *Holder) forgetWalk(id string) {
|
||||
if k, ok := h.Store.(walkKeeper); ok {
|
||||
if err := k.DeleteWalk(id); err != nil {
|
||||
h.noteStore(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// loadWalks reads back the open walks and the restart passes; called with h.mu held.
|
||||
func (h *Holder) loadWalks() error {
|
||||
k, ok := h.Store.(walkKeeper)
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
walks, err := k.Walks()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for i := range walks {
|
||||
w := walks[i]
|
||||
if !w.Ended.IsZero() {
|
||||
h.ended[w.ID] = &w
|
||||
continue
|
||||
}
|
||||
if w.Held == nil {
|
||||
w.Held = map[string]HeldMessage{}
|
||||
}
|
||||
h.walks[w.ID] = &w
|
||||
}
|
||||
passes, err := k.Passes()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for kind, p := range passes {
|
||||
h.passes[kind] = p
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,770 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// The tests of novox/hq ADR 0308 "How it is checked" §2–§6, with the controller's walk events given by a fake
|
||||
// source: the body the controller's plan-moved carries at a walk's start and end (ADR 0308 §1).
|
||||
|
||||
// walkBody is a plan-moved body as the controller says a walk's start or end (mesh-controller pull request 247).
|
||||
func walkBody(t *testing.T, id, phase, outcome string, at time.Time, bound int, touches ...Touch) []byte {
|
||||
t.Helper()
|
||||
b := map[string]any{"id": id, "state": "walking", "walk": id, "phase": phase, "touches": touches,
|
||||
"updated": at.Format(time.RFC3339Nano), "budget_class": "leaf"}
|
||||
if phase == WalkStarted {
|
||||
b["started_at"] = at.Format(time.RFC3339Nano)
|
||||
} else {
|
||||
b["ended_at"] = at.Format(time.RFC3339Nano)
|
||||
}
|
||||
if outcome != "" {
|
||||
b["outcome"] = outcome
|
||||
}
|
||||
if bound > 0 {
|
||||
b["bound_seconds"] = bound
|
||||
}
|
||||
raw, err := json.Marshal(b)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return raw
|
||||
}
|
||||
|
||||
// walk says a walk's start or end to the router, through the decoder the bus's events go through.
|
||||
func walk(t *testing.T, h *Holder, id, phase, outcome string, at time.Time, bound int, touches ...Touch) {
|
||||
t.Helper()
|
||||
ev, ok, err := DecodeWalk(walkBody(t, id, phase, outcome, at, bound, touches...))
|
||||
if err != nil || !ok {
|
||||
t.Fatalf("the walk event was not read: %v %v", ok, err)
|
||||
}
|
||||
h.Walked(ev)
|
||||
}
|
||||
|
||||
func novox(modules ...string) Touch { return Touch{Machine: "novox", Modules: modules} }
|
||||
|
||||
// heard says whether any message on the channels carries the words: sent, edited or said again in place.
|
||||
func heard(words string, chs ...*fakeChannel) bool {
|
||||
for _, ch := range chs {
|
||||
for _, m := range ch.sends {
|
||||
if strings.Contains(m.Text(), words) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
for _, m := range ch.edits {
|
||||
if strings.Contains(m.Text(), words) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
for _, r := range ch.replaced {
|
||||
if strings.Contains(r.m.Text(), words) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// outcomes are the history's outcomes for a key, oldest first.
|
||||
func outcomes(h *Holder, key string) []string {
|
||||
var out []string
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
for _, s := range h.recent {
|
||||
if s.Key == key {
|
||||
out = append(out, s.Outcome)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func has(list []string, want string) bool {
|
||||
for _, s := range list {
|
||||
if s == want {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// §2: during a walk, a message about a machine or a module it touches is held, not sent, and the channel's
|
||||
// history says it was held — a module's condition, a machine's, and a module's notice alike.
|
||||
func TestDuringAWalkWhatItTouchesIsHeldAndTheHistorySaysSo(t *testing.T) {
|
||||
h, tg, dt, c, _ := newHolder(t)
|
||||
walk(t, h, "plan-1", WalkStarted, "", c.now(), 600, novox("mesh-tools", "node-tools"))
|
||||
mod := cond("module.node-tools.novox.unhealthy", Urgent, "the tool runner on novox does not answer")
|
||||
mod.Machine, mod.Raised = "novox", c.now()
|
||||
h.Condition(EventRaised, mod)
|
||||
machine := cond("machine.novox.claude-code.agent-home.data-shrank", Urgent, "the agent's home shrank")
|
||||
machine.Raised = c.now()
|
||||
h.Condition(EventRaised, machine)
|
||||
if _, err := h.Notify("tools.slow", Urgent, "tools answer slowly", "node-tools", false, "tools slow",
|
||||
"Needs you: look at the tools."); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
runFor(h, c, 2*time.Minute)
|
||||
if saidOn(tg, dt) != 0 {
|
||||
t.Fatalf("held messages were said: telegram %+v, desk %+v", tg.sends, dt.sends)
|
||||
}
|
||||
for _, key := range []string{mod.Key, machine.Key, "notify.tools.slow"} {
|
||||
if !has(outcomes(h, key), "held") {
|
||||
t.Errorf("the history does not say %s was held: %v", key, outcomes(h, key))
|
||||
}
|
||||
}
|
||||
if len(h.Open()) != 3 {
|
||||
t.Fatalf("held messages are not kept open: %+v", h.Open())
|
||||
}
|
||||
}
|
||||
|
||||
// §2: a message touched by two overlapping walks is held until the second of them ends; the first's end says
|
||||
// nothing of it.
|
||||
func TestAMessageTouchedByTwoOverlappingWalksIsHeldUntilTheSecondEnds(t *testing.T) {
|
||||
h, tg, dt, c, _ := newHolder(t)
|
||||
walk(t, h, "plan-a", WalkStarted, "", c.now(), 1200, novox("mesh-tools"))
|
||||
c.pass(time.Minute)
|
||||
walk(t, h, "plan-b", WalkStarted, "", c.now(), 1200, novox("node-tools"), Touch{Machine: "ace", Modules: []string{"node-tools"}})
|
||||
k := cond("machine.novox.network", Urgent, "novox lost its default route")
|
||||
k.Raised = c.now()
|
||||
h.Condition(EventRaised, k)
|
||||
runFor(h, c, time.Minute)
|
||||
walk(t, h, "plan-a", WalkEnded, WalkDelivered, c.now(), 0, novox("mesh-tools"))
|
||||
runFor(h, c, time.Minute)
|
||||
if heard(headlineOf(k.Key), tg, dt) {
|
||||
t.Fatalf("said at the first walk's end, while the second still holds it: %+v %+v", tg.sends, dt.sends)
|
||||
}
|
||||
if !heard("Delivered: mesh-tools on novox", tg, dt) {
|
||||
t.Fatalf("the first walk's end was not told: %+v", dt.sends)
|
||||
}
|
||||
walk(t, h, "plan-b", WalkEnded, WalkDelivered, c.now(), 0)
|
||||
runFor(h, c, time.Minute)
|
||||
if !heard("Still open after it:\n- Urgent: "+headlineOf(k.Key), tg, dt) {
|
||||
t.Fatalf("the second walk's end did not tell it: telegram %+v, desk %+v", tg.sends, dt.sends)
|
||||
}
|
||||
}
|
||||
|
||||
// §3: an ask, a secret given, a message about a machine no open walk touches and a walk's own failure are
|
||||
// sent at once, while a walk touches the machine they are on.
|
||||
func TestWhatAWalkNeverHoldsIsSentAtOnce(t *testing.T) {
|
||||
h, tg, dt, c, _, _ := newRouter(t)
|
||||
walk(t, h, "plan-9", WalkStarted, "", c.now(), 1200, novox("telegram", "mesh-tools"))
|
||||
for _, key := range []string{
|
||||
"machine.novox.telegram.telegram-token.secret-given", // a secret given, on a touched machine
|
||||
"machine.novox.mounts.smb.secret-replaced", // a secret replaced
|
||||
"machine.ace.silent", // a machine no walk touches
|
||||
"delivery.leaf.over-budget", // the walk's own delivery
|
||||
"plan.plan-9.failed", // the walk itself
|
||||
} {
|
||||
k := cond(key, Urgent, "it happened")
|
||||
k.Raised = c.now()
|
||||
if !strings.HasPrefix(key, "machine.") {
|
||||
k.Machine = "novox" // a walk's own failure, on a machine it touches
|
||||
}
|
||||
h.Condition(EventRaised, k)
|
||||
runFor(h, c, time.Minute)
|
||||
stand(h, c)
|
||||
if !heard(headlineOf(key), tg, dt) {
|
||||
t.Errorf("%s was not sent at once during the walk: %v", key, outcomes(h, key))
|
||||
}
|
||||
}
|
||||
// An ask's own message, and an ask about a held condition: both offered at once.
|
||||
submit(t, h, ConditionOwner, ackAsk(c, "own"))
|
||||
runFor(h, c, time.Minute)
|
||||
if !has(outcomes(h, "ask."+ConditionOwner+".own"), "sent") {
|
||||
t.Errorf("an ask waiting for the operator was held: %v", outcomes(h, "ask."+ConditionOwner+".own"))
|
||||
}
|
||||
held := cond("machine.novox.units", Urgent, "2 failed services on novox")
|
||||
held.Raised = c.now()
|
||||
h.Condition(EventRaised, held)
|
||||
runFor(h, c, time.Minute)
|
||||
if heard(headlineOf(held.Key), tg, dt) {
|
||||
t.Fatalf("a condition about a touched machine was not held")
|
||||
}
|
||||
about := ackAsk(c, "about")
|
||||
about.About = held.Key
|
||||
submit(t, h, ConditionOwner, about)
|
||||
runFor(h, c, time.Minute)
|
||||
if !heard(headlineOf(held.Key), dt) {
|
||||
t.Fatalf("the ask about a held condition was held: %v", outcomes(h, held.Key))
|
||||
}
|
||||
}
|
||||
|
||||
// §4: a walk's end is one message — its outcome, what it reached and how long it took — then each held message
|
||||
// still open, one line each; the one that cleared is only counted, and nothing more is said of them after.
|
||||
func TestAWalksEndIsToldOnceWithWhatIsStillOpen(t *testing.T) {
|
||||
h, tg, dt, c, _ := newHolder(t)
|
||||
walk(t, h, "plan-3", WalkStarted, "", c.now(), 600, novox("mesh-tools"))
|
||||
still := cond("machine.novox.tools-silent", Warning, "the tools on novox are silent")
|
||||
still.Raised = c.now()
|
||||
h.Condition(EventRaised, still)
|
||||
gone := cond("module.mesh-tools.novox.unhealthy", Warning, "mesh-tools is unhealthy on novox")
|
||||
gone.Raised = c.now()
|
||||
h.Condition(EventRaised, gone)
|
||||
c.pass(time.Second)
|
||||
other := cond("machine.novox.core-behind", Urgent, "novox runs an older core")
|
||||
other.Raised = c.now()
|
||||
h.Condition(EventRaised, other)
|
||||
runFor(h, c, 2*time.Minute)
|
||||
h.Condition(EventCleared, Condition{Key: gone.Key, At: c.now()})
|
||||
runFor(h, c, 5*time.Minute)
|
||||
if saidOn(tg, dt) != 0 {
|
||||
t.Fatalf("said during the walk: %+v %+v", tg.sends, dt.sends)
|
||||
}
|
||||
walk(t, h, "plan-3", WalkEnded, WalkFailed, c.now(), 0)
|
||||
runFor(h, c, time.Minute)
|
||||
// Urgent among what it holds: on every kind, one message each.
|
||||
if len(dt.sends) != 1 || len(tg.sends) != 1 {
|
||||
t.Fatalf("the end was not one message per kind: desk %+v, telegram %+v", dt.sends, tg.sends)
|
||||
}
|
||||
m := dt.sends[0]
|
||||
if m.Title != "Delivery failed: mesh-tools on novox" || !strings.Contains(m.Body, "It took 7 min.") {
|
||||
t.Fatalf("the end does not say its outcome, what it reached and how long: %q", m.Text())
|
||||
}
|
||||
want := "Still open after it:\n- " + headlineOf(still.Key) + "\n- Urgent: " + headlineOf(other.Key) +
|
||||
"\n1 problem it raised resolved during it."
|
||||
if !strings.Contains(m.Body, want) || strings.Contains(m.Text(), headlineOf(gone.Key)) {
|
||||
t.Fatalf("the end message %q does not say %q", m.Body, want)
|
||||
}
|
||||
keyless(t, m)
|
||||
runFor(h, c, 20*time.Minute)
|
||||
if len(dt.sends) != 1 || len(tg.sends) != 1 || len(dt.replaced)+len(tg.replaced) != 0 {
|
||||
t.Fatalf("what the end told was said again: desk %+v, telegram %+v", dt.sends, tg.sends)
|
||||
}
|
||||
}
|
||||
|
||||
// §4: a walk that delivered with nothing held and nothing open is one line, and nothing at all on a kind set
|
||||
// to say only what needs the operator.
|
||||
func TestADeliveredWalkWithNothingHeldIsOneLineOrNothing(t *testing.T) {
|
||||
h, tg, dt, c, _ := newHolder(t)
|
||||
walk(t, h, "plan-4", WalkStarted, "", c.now(), 600, novox("mesh-tools"), Touch{Machine: "ace", Modules: []string{"mesh-tools"}})
|
||||
runFor(h, c, 3*time.Minute)
|
||||
walk(t, h, "plan-4", WalkEnded, WalkDelivered, c.now(), 0)
|
||||
runFor(h, c, time.Minute)
|
||||
if len(dt.sends) != 1 || len(tg.sends) != 0 {
|
||||
t.Fatalf("desk %+v, telegram %+v", dt.sends, tg.sends)
|
||||
}
|
||||
if m := dt.sends[0]; m.Title != "Delivered: mesh-tools on ace, novox, in 3 min" || m.Body != "" {
|
||||
t.Fatalf("not one line: %q", m.Text())
|
||||
}
|
||||
h.NeedsOnly = func(kind string) bool { return kind == "desktop" }
|
||||
walk(t, h, "plan-5", WalkStarted, "", c.now(), 600, novox("mesh-tools"))
|
||||
runFor(h, c, 3*time.Minute)
|
||||
walk(t, h, "plan-5", WalkEnded, WalkDelivered, c.now(), 0)
|
||||
runFor(h, c, time.Minute)
|
||||
if len(dt.sends) != 1 || len(tg.sends) != 0 {
|
||||
t.Fatalf("told on a kind that says only what needs the operator: desk %+v, telegram %+v", dt.sends, tg.sends)
|
||||
}
|
||||
if !has(outcomes(h, "walk.plan-5"), "quiet: this kind says only what needs the operator") {
|
||||
t.Fatalf("the history does not say why nothing was told: %v", outcomes(h, "walk.plan-5"))
|
||||
}
|
||||
}
|
||||
|
||||
// §5: the bound is the controller's (bound_seconds: twice the largest budget of the walk's deliveries; none
|
||||
// given, 15 minutes). At it, what is held and still open is sent with the still-running line, and what
|
||||
// cleared is counted; before it, nothing; after it, the walk holds nothing more.
|
||||
func TestAHoldEndsAtTheWalksBound(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
given int
|
||||
bound time.Duration
|
||||
}{
|
||||
{"core: twice ten minutes", 1200, 20 * time.Minute},
|
||||
{"leaf at five minutes", 600, 10 * time.Minute},
|
||||
{"leaf at three minutes", 360, 6 * time.Minute},
|
||||
{"no budget", 0, 15 * time.Minute},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
h, tg, dt, c, _ := newHolder(t)
|
||||
started := c.now()
|
||||
walk(t, h, "plan-7", WalkStarted, "", started, tc.given, novox("node-tools"))
|
||||
open := cond("machine.novox.tools-silent", Urgent, "the tools on novox are silent")
|
||||
open.Raised = c.now()
|
||||
h.Condition(EventRaised, open)
|
||||
gone := cond("module.node-tools.novox.unhealthy", Urgent, "node-tools is unhealthy")
|
||||
gone.Raised = c.now()
|
||||
h.Condition(EventRaised, gone)
|
||||
c.pass(30 * time.Second)
|
||||
h.Condition(EventCleared, Condition{Key: gone.Key, At: c.now()})
|
||||
runFor(h, c, tc.bound-time.Minute-30*time.Second)
|
||||
if saidOn(tg, dt) != 0 {
|
||||
t.Fatalf("said before the bound: %+v %+v", tg.sends, dt.sends)
|
||||
}
|
||||
runFor(h, c, 2*time.Minute)
|
||||
if len(dt.sends) != 1 || len(tg.sends) != 1 {
|
||||
t.Fatalf("at the bound: desk %+v, telegram %+v", dt.sends, tg.sends)
|
||||
}
|
||||
m := dt.sends[0]
|
||||
since := started.In(h.zone()).Format("15:04")
|
||||
for _, want := range []string{"is still running, since " + since + ".", "- Urgent: " + headlineOf(open.Key),
|
||||
"1 problem it raised resolved during it."} {
|
||||
if !strings.Contains(m.Body, want) {
|
||||
t.Fatalf("the bound's message %q does not say %q", m.Text(), want)
|
||||
}
|
||||
}
|
||||
// Past its bound the walk holds nothing: a new message about what it touches goes at once.
|
||||
late := cond("machine.novox.disk", Urgent, "a disk fails on novox")
|
||||
late.Raised = c.now()
|
||||
h.Condition(EventRaised, late)
|
||||
runFor(h, c, time.Minute)
|
||||
if !heard(headlineOf(late.Key), dt) {
|
||||
t.Fatalf("held past the bound: %v", outcomes(h, late.Key))
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// §2, kept: what a walk holds survives the router's restart, and is told at the walk's end.
|
||||
func TestHeldMessagesSurviveARouterRestart(t *testing.T) {
|
||||
h, tg, dt, c, st := newHolder(t)
|
||||
walk(t, h, "plan-8", WalkStarted, "", c.now(), 1200, novox("mesh-tools"))
|
||||
k := cond("machine.novox.tools-silent", Urgent, "the tools on novox are silent")
|
||||
k.Raised = c.now()
|
||||
h.Condition(EventRaised, k)
|
||||
runFor(h, c, time.Minute)
|
||||
again := &Holder{Away: tg, Desk: dt, Store: st, Now: c.now, Logf: t.Logf}
|
||||
if err := again.Load(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
again.Start()
|
||||
c.pass(2 * time.Minute)
|
||||
walk(t, again, "plan-8", WalkEnded, WalkDelivered, c.now(), 0)
|
||||
runFor(again, c, time.Minute)
|
||||
if !heard("- Urgent: "+headlineOf(k.Key), dt) || saidOn(tg, dt) != 2 {
|
||||
t.Fatalf("what was held before the restart was not told at the walk's end: desk %+v, telegram %+v", dt.sends, tg.sends)
|
||||
}
|
||||
}
|
||||
|
||||
// §6, issue 531: the desk's notifiers on two machines start again a second apart, with four silenced messages
|
||||
// shown there — one of them withheld for its content, a file path — and a fifth open, asked there, and
|
||||
// withheld. Nothing is said again: no silenced, no withheld message; the pass runs once; and a router that
|
||||
// starts again reads it back from its state and makes it no more.
|
||||
func TestARestartSaysNothingItHasSaid(t *testing.T) {
|
||||
h, tg, dt, c, _, _ := newRouter(t)
|
||||
h.Start()
|
||||
c.pass(2 * time.Minute)
|
||||
keys := []string{"machine.novox.telegram.telegram-token.secret-given", "machine.novox.claude-code.agent-home.data-shrank",
|
||||
"machine.shanks.mounts.smb-password-recalbox.secret-given", "machine.ace.nzbgeek.api-key.secret-given"}
|
||||
var silenced []Condition
|
||||
for i, key := range keys {
|
||||
k := cond(key, Urgent, "it happened")
|
||||
if i == 1 {
|
||||
k.Explanation = "Needs you: look at /var/lib/agent-home, which shrank."
|
||||
}
|
||||
k.Raised = c.now()
|
||||
h.Condition(EventRaised, k)
|
||||
runFor(h, c, time.Minute)
|
||||
stand(h, c)
|
||||
silenced = append(silenced, k)
|
||||
}
|
||||
withheldOpen := cond("machine.novox.units", Urgent, "2 failed services")
|
||||
withheldOpen.Explanation = "Needs you: read /etc/systemd/system/x.service."
|
||||
withheldOpen.Raised = c.now()
|
||||
h.Condition(EventRaised, withheldOpen)
|
||||
runFor(h, c, time.Minute)
|
||||
ask := ackAsk(c, "units")
|
||||
ask.About = withheldOpen.Key
|
||||
submit(t, h, ConditionOwner, ask)
|
||||
runFor(h, c, time.Minute)
|
||||
for _, k := range silenced {
|
||||
k.SilencedTill, k.SilencedWhy, k.At = c.now().Add(7*24*time.Hour), "known", c.now()
|
||||
h.Condition(EventChanged, k)
|
||||
}
|
||||
runFor(h, c, time.Minute)
|
||||
for _, key := range []string{silenced[1].Key, withheldOpen.Key} {
|
||||
if r := h.open[key]; r == nil || r.Refused == "" {
|
||||
t.Fatalf("the test's %s was not withheld for its content: %+v", key, r)
|
||||
}
|
||||
}
|
||||
sends, edits, replaced, tgSaid := len(dt.sends), len(dt.edits), len(dt.replaced), saidOn(tg)
|
||||
dt.edits = map[string]Message{}
|
||||
passes := func(h *Holder) int { return len(outcomes(h, "(answers lost)")) }
|
||||
// The notifiers on two machines started again a second apart: two standings, two times.
|
||||
c.pass(time.Minute)
|
||||
first := c.now()
|
||||
h.Stand("desktop", Standing{Ready: true, At: c.now(), AnswersLostAt: first})
|
||||
c.pass(time.Second)
|
||||
h.Flush() // the runtime's tick, between the two standings
|
||||
second := c.now()
|
||||
h.Stand("desktop", Standing{Ready: true, At: c.now(), AnswersLostAt: second})
|
||||
runFor(h, c, time.Minute)
|
||||
if len(dt.sends) != sends || len(dt.edits) != 0 || len(dt.replaced) != replaced || saidOn(tg) != tgSaid {
|
||||
t.Fatalf("the restart said again: sends %+v, edits %+v (was %d), replaced %+v", dt.sends[sends:], dt.edits, edits,
|
||||
dt.replaced[replaced:])
|
||||
}
|
||||
if passes(h) != 1 {
|
||||
t.Fatalf("the pass ran %d times for one restart", passes(h))
|
||||
}
|
||||
// The router starts again; the desk repeats its standing.
|
||||
again := &Holder{Away: tg, Desk: dt, Store: h.Store, Asks: h.Asks, Bench: h.Bench, Publish: h.Publish, Now: c.now,
|
||||
Logf: t.Logf, Zone: h.Zone, RootFree: h.RootFree, Node: h.Node}
|
||||
if err := again.Load(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
again.Start()
|
||||
again.Stand("desktop", Standing{Ready: true, At: c.now(), AnswersLostAt: second})
|
||||
runFor(again, c, 2*time.Minute)
|
||||
if len(dt.sends) != sends || len(dt.edits) != 0 || len(dt.replaced) != replaced || saidOn(tg) != tgSaid || passes(again) != 1 {
|
||||
t.Fatalf("the router's restart made the pass again: %d passes, sends %+v, edits %+v", passes(again),
|
||||
dt.sends[sends:], dt.edits)
|
||||
}
|
||||
}
|
||||
|
||||
// The controller's walk events are read as ADR 0308 §1 says them; any other plan-moved is not a walk's.
|
||||
func TestTheWalkEventsAreReadAsTheControllerSaysThem(t *testing.T) {
|
||||
at := time.Date(2026, 10, 12, 11, 12, 0, 0, time.UTC)
|
||||
ev, ok, err := DecodeWalk(walkBody(t, "plan-1", WalkStarted, "", at, 1200, novox("mesh-tools", "node-tools")))
|
||||
if err != nil || !ok || ev.Walk != "plan-1" || ev.BoundSeconds != 1200 || ev.Class != "leaf" || !ev.At.Equal(at) ||
|
||||
len(ev.Touches) != 1 || ev.Touches[0].Machine != "novox" || len(ev.Touches[0].Modules) != 2 {
|
||||
t.Fatalf("%+v %v %v", ev, ok, err)
|
||||
}
|
||||
// As mesh-controller's walk events (its pull request 247) say an end: ended_at, and carried beside modules.
|
||||
end, ok, err := DecodeWalk([]byte(`{"id":"plan-1","walk":"plan-1","phase":"ended","outcome":"stopped",` +
|
||||
`"touches":[{"machine":"novox","modules":["node-tools","messenger"],"carried":["messenger"]}],` +
|
||||
`"started_at":"2026-10-12T11:12:00Z","ended_at":"2026-10-12T11:19:00Z","updated":"2026-10-12T11:19:01Z"}`))
|
||||
if err != nil || !ok || end.Outcome != WalkStopped || !end.At.Equal(at.Add(7*time.Minute)) || len(end.Touches[0].Modules) != 2 {
|
||||
t.Fatalf("%+v %v %v", end, ok, err)
|
||||
}
|
||||
if _, ok, err := DecodeWalk([]byte(`{"id":"plan-1","state":"walking","tier":1}`)); ok || err != nil {
|
||||
t.Fatalf("a move inside a walk read as its start or end: %v %v", ok, err)
|
||||
}
|
||||
if _, _, err := DecodeWalk([]byte(`{"walk":"plan-1","phase":"ended"}`)); err == nil {
|
||||
t.Fatalf("an end without its outcome was read")
|
||||
}
|
||||
w := Walk{}
|
||||
if w.bound() != DefaultWalkBound {
|
||||
t.Fatalf("a walk without a bound is bounded at %s", w.bound())
|
||||
}
|
||||
// A start older than its bound is history: nothing is held.
|
||||
h, _, _, c, _ := newHolder(t)
|
||||
walk(t, h, "plan-old", WalkStarted, "", c.now().Add(-time.Hour), 1200, novox("mesh-tools"))
|
||||
if len(h.walks) != 0 {
|
||||
t.Fatalf("an old start opened a walk: %+v", h.walks)
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
// Review of pull request 230, 1: no channel can send at a walk's end or bound. What it held is not lost with the
|
||||
// walk: it is unsent, and said on its own once a channel can send.
|
||||
func TestWhatAWalkHeldIsSaidOnceAChannelCanSendAgain(t *testing.T) {
|
||||
for _, at := range []string{"end", "bound"} {
|
||||
t.Run(at, func(t *testing.T) {
|
||||
h, tg, dt, c, _ := newHolder(t)
|
||||
walk(t, h, "plan-c", WalkStarted, "", c.now(), 600, novox("mesh-tools"))
|
||||
k := cond("machine.novox.tools-silent", Urgent, "the tools on novox are silent")
|
||||
k.Raised = c.now()
|
||||
h.Condition(EventRaised, k)
|
||||
runFor(h, c, 2*time.Minute)
|
||||
down := errors.New("no session")
|
||||
tg.notReady, dt.notReady = down, down
|
||||
if at == "end" {
|
||||
walk(t, h, "plan-c", WalkEnded, WalkFailed, c.now(), 0)
|
||||
runFor(h, c, time.Minute)
|
||||
} else {
|
||||
runFor(h, c, 10*time.Minute)
|
||||
}
|
||||
if saidOn(tg, dt) != 0 {
|
||||
t.Fatalf("said on a channel that cannot send: %+v %+v", tg.sends, dt.sends)
|
||||
}
|
||||
if r := h.open[k.Key]; r == nil || r.Pending == "" {
|
||||
t.Fatalf("what the walk held is not left unsent: %+v", r)
|
||||
}
|
||||
tg.notReady, dt.notReady = nil, nil
|
||||
runFor(h, c, 2*time.Minute)
|
||||
if !heard(headlineOf(k.Key), dt) {
|
||||
t.Fatalf("what the walk held was never said: %v", outcomes(h, k.Key))
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Review of pull request 230, 2: a message held by walk A, about a machine walk B also touches, is held on at
|
||||
// A's end while B — started after it was held — is still open, and told at B's end.
|
||||
func TestAMessageIsHeldOnByALaterWalkThatTouchesIt(t *testing.T) {
|
||||
h, tg, dt, c, _ := newHolder(t)
|
||||
walk(t, h, "plan-a", WalkStarted, "", c.now(), 1200, novox("mesh-tools"))
|
||||
k := cond("machine.novox.network", Urgent, "novox lost its default route")
|
||||
k.Raised = c.now()
|
||||
h.Condition(EventRaised, k)
|
||||
runFor(h, c, time.Minute)
|
||||
walk(t, h, "plan-b", WalkStarted, "", c.now(), 1200, novox("node-tools"))
|
||||
runFor(h, c, time.Minute)
|
||||
walk(t, h, "plan-a", WalkEnded, WalkDelivered, c.now(), 0)
|
||||
runFor(h, c, time.Minute)
|
||||
if heard(headlineOf(k.Key), tg, dt) {
|
||||
t.Fatalf("told at the first walk's end while the second touches it: %+v %+v", tg.sends, dt.sends)
|
||||
}
|
||||
if got := h.heldBy(k.Key, c.now()); len(got) != 1 || got[0] != "plan-b" {
|
||||
t.Fatalf("the second walk does not hold it on: %v", got)
|
||||
}
|
||||
walk(t, h, "plan-b", WalkEnded, WalkDelivered, c.now(), 0)
|
||||
runFor(h, c, time.Minute)
|
||||
if !heard("Still open after it:\n- Urgent: "+headlineOf(k.Key), tg, dt) {
|
||||
t.Fatalf("the second walk's end did not tell it: %+v %+v", tg.sends, dt.sends)
|
||||
}
|
||||
}
|
||||
|
||||
// Review of pull request 230, 3: a router back after more than a day reads back a walk whose end never came;
|
||||
// what it held is said, with the still-running line, before the walk is forgotten.
|
||||
func TestAWalkReadBackADayLateSaysWhatItHeldBeforeItIsForgotten(t *testing.T) {
|
||||
h, tg, dt, c, st := newHolder(t)
|
||||
walk(t, h, "plan-d", WalkStarted, "", c.now(), 600, novox("mesh-tools"))
|
||||
k := cond("machine.novox.tools-silent", Urgent, "the tools on novox are silent")
|
||||
k.Raised = c.now()
|
||||
h.Condition(EventRaised, k)
|
||||
runFor(h, c, time.Minute)
|
||||
c.pass(25 * time.Hour)
|
||||
again := &Holder{Away: tg, Desk: dt, Store: st, Now: c.now, Logf: t.Logf}
|
||||
if err := again.Load(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
again.Start()
|
||||
runFor(again, c, 2*time.Minute)
|
||||
if !heard("is still running", dt) || !heard("- Urgent: "+headlineOf(k.Key), dt) {
|
||||
t.Fatalf("what the walk held was forgotten unsaid: desk %+v, %v", dt.sends, outcomes(again, k.Key))
|
||||
}
|
||||
if len(again.walks) != 0 {
|
||||
t.Fatalf("the walk was not forgotten: %+v", again.walks)
|
||||
}
|
||||
if ws, _ := st.Walks(); len(ws) != 0 {
|
||||
t.Fatalf("the walk is still kept: %+v", ws)
|
||||
}
|
||||
}
|
||||
|
||||
// Review of pull request 230, 4 (and of mesh-controller pull request 247): a short walk's end can come before
|
||||
// its start. The start of a walk already ended opens nothing — also after the router's restart — while the
|
||||
// next walk under the same id, started after that end, does.
|
||||
func TestAStartHeardAfterItsEndOpensNothing(t *testing.T) {
|
||||
h, tg, dt, c, st := newHolder(t)
|
||||
started := c.now()
|
||||
c.pass(20 * time.Second)
|
||||
end, _, err := DecodeWalk([]byte(`{"walk":"plan-e","phase":"ended","outcome":"delivered","touches":[{"machine":"novox","modules":["mesh-tools"]}],` +
|
||||
`"started_at":"` + started.Format(time.RFC3339Nano) + `","ended_at":"` + c.now().Format(time.RFC3339Nano) + `"}`))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
h.Walked(end)
|
||||
c.pass(time.Second)
|
||||
walk(t, h, "plan-e", WalkStarted, "", started, 600, novox("mesh-tools"))
|
||||
if len(h.walks) != 0 {
|
||||
t.Fatalf("a start heard after its end opened a walk: %+v", h.walks)
|
||||
}
|
||||
again := &Holder{Away: tg, Desk: dt, Store: st, Now: c.now, Logf: t.Logf}
|
||||
if err := again.Load(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
again.Start()
|
||||
walk(t, again, "plan-e", WalkStarted, "", started, 600, novox("mesh-tools"))
|
||||
if len(again.walks) != 0 {
|
||||
t.Fatalf("after a restart, a start heard after its end opened a walk: %+v", again.walks)
|
||||
}
|
||||
k := cond("machine.novox.tools-silent", Urgent, "the tools on novox are silent")
|
||||
k.Raised = c.now()
|
||||
again.Condition(EventRaised, k)
|
||||
runFor(again, c, time.Minute)
|
||||
if !heard(headlineOf(k.Key), dt) {
|
||||
t.Fatalf("held by a walk that had ended: %v", outcomes(again, k.Key))
|
||||
}
|
||||
// Retried: the same id walks again, started after the end.
|
||||
walk(t, again, "plan-e", WalkStarted, "", c.now(), 600, novox("mesh-tools"))
|
||||
if len(again.walks) != 1 {
|
||||
t.Fatalf("a walk started after the last end did not open: %+v", again.walks)
|
||||
}
|
||||
}
|
||||
|
||||
// Review of pull request 230, non-blocking: a message whose answers became asked while a walk held it is never
|
||||
// held (ADR 0308 §3); when the desk's answers are lost, it is said again in place like any other, so its
|
||||
// buttons work again.
|
||||
func TestARestartSaysAgainAnAskedMessageAWalkTookBeforeItsAsk(t *testing.T) {
|
||||
h, _, dt, c, _, _ := newRouter(t)
|
||||
h.Start()
|
||||
stand(h, c)
|
||||
walk(t, h, "plan-f", WalkStarted, "", c.now(), 1200, novox("mesh-tools"))
|
||||
k := cond("machine.novox.units", Urgent, "2 failed services on novox")
|
||||
k.Raised = c.now()
|
||||
h.Condition(EventRaised, k)
|
||||
runFor(h, c, time.Minute)
|
||||
ask := ackAsk(c, "units")
|
||||
ask.About = k.Key
|
||||
submit(t, h, ConditionOwner, ask)
|
||||
runFor(h, c, time.Minute)
|
||||
stand(h, c)
|
||||
if !heard(headlineOf(k.Key), dt) {
|
||||
t.Fatalf("the ask about it was not offered: %v", outcomes(h, k.Key))
|
||||
}
|
||||
replaced := len(dt.replaced)
|
||||
c.pass(time.Minute)
|
||||
h.Stand("desktop", Standing{Ready: true, At: c.now(), AnswersLostAt: c.now()})
|
||||
runFor(h, c, time.Minute)
|
||||
if len(dt.replaced) != replaced+1 {
|
||||
t.Fatalf("its answers were not made to work again: replaced %+v, %v", dt.replaced[replaced:], outcomes(h, "(answers lost)"))
|
||||
}
|
||||
}
|
||||
|
||||
// Review of pull request 230, non-blocking: a walk's end that cannot be said as it is (here a module named by a
|
||||
// file path) is said in its plain form, and what it held, still open, is said on its own, not marked told.
|
||||
func TestWhatAWalksPlainEndCannotCarryIsSaidOnItsOwn(t *testing.T) {
|
||||
h, tg, dt, c, _ := newHolder(t)
|
||||
walk(t, h, "plan-g", WalkStarted, "", c.now(), 600, novox("/var/lib/mesh-tools"))
|
||||
k := cond("machine.novox.tools-silent", Urgent, "the tools on novox are silent")
|
||||
k.Raised = c.now()
|
||||
h.Condition(EventRaised, k)
|
||||
runFor(h, c, time.Minute)
|
||||
walk(t, h, "plan-g", WalkEnded, WalkFailed, c.now(), 0)
|
||||
runFor(h, c, time.Minute)
|
||||
if !heard("A delivery ended", dt) {
|
||||
t.Fatalf("the plain end was not said: %+v", dt.sends)
|
||||
}
|
||||
if !heard(headlineOf(k.Key), tg, dt) {
|
||||
t.Fatalf("what the walk held was marked told and never said: %v", outcomes(h, k.Key))
|
||||
}
|
||||
}
|
||||
|
||||
// Issue 538 A: what a walk held is marked told only once the message carrying it was sent. When that message
|
||||
// could not be sent and the router restarts before its retry, the message is gone with it: what it held is not
|
||||
// marked told, it stays owed in the state, and the restarted router says it — an urgent one and a warning alike,
|
||||
// once each, never left as history.
|
||||
func TestAHeldMessageIsToldOnlyOnceTheWalksMessageWasSent(t *testing.T) {
|
||||
for _, then := range []string{"sent", "restart urgent", "restart warning"} {
|
||||
t.Run(then, func(t *testing.T) {
|
||||
h, tg, dt, c, st := newHolder(t)
|
||||
sev := Urgent
|
||||
if then == "restart warning" {
|
||||
sev = Warning
|
||||
}
|
||||
walk(t, h, "plan-h", WalkStarted, "", c.now(), 1200, novox("mesh-tools"))
|
||||
k := cond("machine.novox.tools-silent", sev, "the tools on novox are silent")
|
||||
k.Raised = c.now()
|
||||
h.Condition(EventRaised, k)
|
||||
runFor(h, c, time.Minute)
|
||||
down := errors.New("cannot connect")
|
||||
tg.fail, dt.fail = down, down
|
||||
walk(t, h, "plan-h", WalkEnded, WalkFailed, c.now(), 0)
|
||||
runFor(h, c, time.Minute)
|
||||
if saidOn(tg, dt) != 0 {
|
||||
t.Fatalf("said on channels that cannot send: %+v %+v", tg.sends, dt.sends)
|
||||
}
|
||||
if r := h.open[k.Key]; r == nil || r.told() {
|
||||
t.Fatalf("marked told while the message carrying it was not sent: %+v", r)
|
||||
}
|
||||
tg.fail, dt.fail = nil, nil
|
||||
if then == "sent" {
|
||||
runFor(h, c, 2*time.Minute)
|
||||
if !heard("Still open after it:\n- Urgent: "+headlineOf(k.Key), tg, dt) || saidOn(tg, dt) != 2 {
|
||||
t.Fatalf("not told once in the walk's message: telegram %+v, desk %+v", tg.sends, dt.sends)
|
||||
}
|
||||
if r := h.open[k.Key]; r == nil || !r.told() || r.Owed != "" || r.Sent["telegram"] != "" || r.Sent["desktop"] != "" {
|
||||
t.Fatalf("not marked told on both channels once sent: %+v", r)
|
||||
}
|
||||
runFor(h, c, 10*time.Minute)
|
||||
if saidOn(tg, dt) != 2 {
|
||||
t.Fatalf("said again after the walk's message: telegram %+v, desk %+v", tg.sends, dt.sends)
|
||||
}
|
||||
return
|
||||
}
|
||||
again := &Holder{Away: tg, Desk: dt, Store: st, Now: c.now, Logf: t.Logf}
|
||||
if err := again.Load(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
again.Start()
|
||||
again.Sync([]Condition{k}, c.now())
|
||||
runFor(again, c, 5*time.Minute)
|
||||
if !heard(headlineOf(k.Key), tg, dt) {
|
||||
t.Fatalf("what the lost message carried was not said after the restart: %v", outcomes(again, k.Key))
|
||||
}
|
||||
said := saidOn(tg, dt)
|
||||
if r := again.open[k.Key]; r == nil || !r.told() || r.Owed != "" || r.Pending != "" {
|
||||
t.Fatalf("said, and still owed or not marked told: %+v", r)
|
||||
}
|
||||
runFor(again, c, 20*time.Minute) // within its reminder's bound: nothing more is due
|
||||
again.Sync([]Condition{k}, c.now())
|
||||
runFor(again, c, time.Minute)
|
||||
if saidOn(tg, dt) != said {
|
||||
t.Fatalf("said again: telegram %+v, desk %+v", tg.sends, dt.sends)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Issue 538 B: at a walk's end, a later walk that touches what it held takes over only what a first hold would
|
||||
// take: not a message silenced meanwhile, nor one an ask waiting for the operator is about (ADR 0308 §3).
|
||||
func TestAWalksEndHandsOnOnlyWhatALaterWalkWouldHold(t *testing.T) {
|
||||
h, tg, dt, c, _, _ := newRouter(t)
|
||||
h.Start()
|
||||
stand(h, c)
|
||||
walk(t, h, "plan-a", WalkStarted, "", c.now(), 1200, novox("mesh-tools"))
|
||||
quiet := cond("machine.novox.network", Urgent, "novox lost its default route")
|
||||
quiet.Raised = c.now()
|
||||
h.Condition(EventRaised, quiet)
|
||||
asked := cond("machine.novox.units", Urgent, "2 failed services on novox")
|
||||
asked.Raised = c.now()
|
||||
h.Condition(EventRaised, asked)
|
||||
runFor(h, c, time.Minute)
|
||||
stand(h, c)
|
||||
if heard(headlineOf(quiet.Key), tg, dt) || heard(headlineOf(asked.Key), tg, dt) {
|
||||
t.Fatalf("the test's messages were not held")
|
||||
}
|
||||
quiet.SilencedTill, quiet.SilencedWhy, quiet.At = c.now().Add(7*24*time.Hour), "known", c.now()
|
||||
h.Condition(EventChanged, quiet)
|
||||
ask := ackAsk(c, "units")
|
||||
ask.About = asked.Key
|
||||
submit(t, h, ConditionOwner, ask)
|
||||
runFor(h, c, time.Minute)
|
||||
stand(h, c)
|
||||
walk(t, h, "plan-b", WalkStarted, "", c.now(), 1200, novox("node-tools"))
|
||||
runFor(h, c, time.Minute)
|
||||
stand(h, c)
|
||||
helds := func(key string) int {
|
||||
n := 0
|
||||
for _, o := range outcomes(h, key) {
|
||||
if o == "held" {
|
||||
n++
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
before := map[string]int{quiet.Key: helds(quiet.Key), asked.Key: helds(asked.Key)}
|
||||
walk(t, h, "plan-a", WalkEnded, WalkDelivered, c.now(), 0)
|
||||
runFor(h, c, time.Minute)
|
||||
for _, key := range []string{quiet.Key, asked.Key} {
|
||||
if got := h.heldBy(key, c.now()); len(got) != 0 {
|
||||
t.Errorf("%s was handed on to %v, which a first hold would never hold", key, got)
|
||||
}
|
||||
if helds(key) != before[key] {
|
||||
t.Errorf("the history says %s held again: %v", key, outcomes(h, key))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Issue 538 C: the controller's clock ahead of the router's. A short walk's end, heard before its start, opens
|
||||
// nothing when its start comes, whether the end said the walk's start or only its own moment; the next walk
|
||||
// under its id, started after that end, still opens.
|
||||
func TestAStartHeardAfterItsEndOpensNothingWhenTheControllersClockIsAhead(t *testing.T) {
|
||||
for _, said := range []string{"both", "end only"} {
|
||||
t.Run(said, func(t *testing.T) {
|
||||
h, _, _, c, _ := newHolder(t)
|
||||
ended := c.now().Add(10 * time.Minute) // by the controller's clock
|
||||
started := ended.Add(-20 * time.Second)
|
||||
body := `{"walk":"plan-k","phase":"ended","outcome":"delivered","touches":[{"machine":"novox","modules":["mesh-tools"]}],` +
|
||||
`"ended_at":"` + ended.Format(time.RFC3339Nano) + `"`
|
||||
if said == "both" {
|
||||
body += `,"started_at":"` + started.Format(time.RFC3339Nano) + `"`
|
||||
}
|
||||
end, _, err := DecodeWalk([]byte(body + `}`))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
h.Walked(end)
|
||||
c.pass(time.Second)
|
||||
walk(t, h, "plan-k", WalkStarted, "", started, 600, novox("mesh-tools"))
|
||||
if len(h.walks) != 0 {
|
||||
t.Fatalf("a start heard after its end opened a walk: %+v", h.walks)
|
||||
}
|
||||
walk(t, h, "plan-k", WalkStarted, "", ended.Add(time.Minute), 600, novox("mesh-tools"))
|
||||
if len(h.walks) != 1 {
|
||||
t.Fatalf("the next walk under its id, started after the end, did not open: %+v", h.walks)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -73,6 +73,7 @@
|
||||
"mesh-controller.condition-raised",
|
||||
"mesh-controller.condition-changed",
|
||||
"mesh-controller.condition-cleared",
|
||||
"mesh-controller.plan-moved",
|
||||
"intake.choice",
|
||||
"intake.link",
|
||||
"intake.failed",
|
||||
@@ -93,7 +94,8 @@
|
||||
"name": "asks",
|
||||
"ttl-seconds": 2592000
|
||||
},
|
||||
"identities"
|
||||
"identities",
|
||||
"walks"
|
||||
],
|
||||
"own-secrets": {
|
||||
"broker": "${dir:state}/broker"
|
||||
@@ -129,7 +131,7 @@
|
||||
"path": "${dir:state}/settings.json",
|
||||
"mode": "0600",
|
||||
"merge": "json",
|
||||
"content": "{\n \"time-zone\": \"\"\n}\n",
|
||||
"content": "{\n \"time-zone\": \"\",\n \"only-what-needs-me\": []\n}\n",
|
||||
"owner": "messenger"
|
||||
}
|
||||
],
|
||||
|
||||
Reference in New Issue
Block a user