Compare commits

..
Author SHA1 Message Date
mesh-tools eba4bab00a Merge pull request 'The image build context is the module's own directory, so a Dockerfile in a subdirectory builds (issue 1703)' (#512) from fix/1703-image-context into main 2026-10-26 20:36:08 +00:00
mesh-tools 216701fcba Merge pull request 'The build modules size their capacity by CPU share and keep build hours (issues 1890, 1806)' (#511) from fix/capacity-follows-cpu-share into main 2026-10-26 20:33:30 +00:00
jschoubben 8e116cb0e6 Merge main into fix/capacity-follows-cpu-share; pin the build modules at mesh-sdk 805b4d9
mesh/delivery merged: source.main moved to novox/mesh-catalog@216701fcbace
mesh/delivery-group group fix/capacity-follows-cpu-share merged: every member is merged
mesh/check checking 8e116cb0e6e4 on 7c5c29421f02 | novox/mesh-catalog#511 in group fix/capacity-follows-cpu-share
Main's #508 (the binary refusal) pinned mesh-sdk fe11db9; 805b4d9 is the capacity work merged with
sdk main, so it holds both.
2026-10-26 21:29:33 +01:00
jschoubben dcf32576a8 mesh-build-image: build with the module's directory as context, so a Dockerfile in a subdirectory finds its COPY paths (issue 1703)
mesh/check checking dcf32576a880 on eba4bab00aa7
mesh/delivery merged: source.main moved to novox/mesh-catalog@eba4bab00aa7
photos' admin-client, client and server Dockerfiles live in subdirectories and copy
paths from the module's root (admin-client/..., client-server/serve.mjs), as build-agent
built them and as the fingerprint reads the module's whole directory. Taking the
Dockerfile's own directory as context failed every such COPY with copier: stat
"/admin-client/package.json": no such file or directory, on shanks and on ace.

Quick fix (rejected): move photos' Dockerfiles to its repository root; it changes the
operator's photo application and leaves the trap for the next module.
Thorough fix (chosen): one context for every image, the module's directory; it never
leaves that directory, so the isolation of ADR 0487 section 2 holds.
2026-10-26 21:25:24 +01:00
jschoubben 3f8b0b4248 Build modules take asks only within their node's build hours
mesh/delivery-group group fix/capacity-follows-cpu-share rejected: composed together: the checker's verdict at the members' heads (63694fe9222d): unit: ..test
mesh/check checking 3f8b0b42480b on 7c5c29421f02 | novox/mesh-catalog#511 in group fix/capacity-follows-cpu-share
mesh/delivery closed: a newer head of the same pull request
mesh-build-go, -node and -image declare build-hours (node=HH:MM-HH:MM in the node's local time,
empty = any hour) and hand it to the SDK puller (the operator, 2026-10-26: g14 builds only from
18:00 to 06:00); their SDK pin moves to mesh-sdk d00dac1, which has it.
2026-10-26 21:22:09 +01:00
mesh-tools 7c5c29421f Merge pull request 'mesh-issues: a milestone or commit link moves no issue and keeps no worker current (issue 1781)' (#510) from fix/1781-moved-ignores-milestone-links into main 2026-10-26 20:17:09 +00:00
mesh-tools 3690c149c4 Merge pull request 'Forge rights read as the user sees them; write-only grant for mesh-tools; progress answers in minimal detail (issues 1945, 1947)' (#509) from fix/progress-and-unwritable into main 2026-10-26 20:08:05 +00:00
mesh-tools 5b75233406 Merge pull request 'The catalogue's check uses the shared binary refusal, commit by commit (issue 1577)' (#508) from fix/1577-checker-refuses-binaries into main 2026-10-26 19:32:31 +00:00
jschoubben 245143d948 Build modules run at most as many steps as their node's build CPU gives, and say a check's phases
Each of mesh-build-go, -node and -image declares cores-per-step (6, 2, 2) and hands it to the
SDK puller, which bounds the configured capacity by floor(slice quota cores / cores-per-step),
read live; thread limits and the status tools follow the effective capacity (issue 1806's
follow-up: three Go checks on shanks' 12-core slice starved past the 30-minute step bound).
mesh-build-go says its lint and test phases with the ask id (issue 1890).
2026-10-26 19:55:31 +01:00
jschoubben 8530a9730f The build modules take mesh-sdk's report of programs left in a branch's history (issue 1577)
mesh/delivery merged: source.main moved to novox/mesh-catalog@5b75233406fe
mesh/delivery-group group fix/1577-checker-refuses-binaries merged: every member is merged
mesh/check checking 8530a9730fbc on 5b75233406fe
2026-10-26 19:47:38 +01:00
jschoubben ee48e5a01e gitea_get_file refuses a bad path before its size listing reaches the forge; the guard test expects the listing (issue 1577) 2026-10-26 19:32:00 +01:00
jschoubben 854dbfb3f3 mesh-issues: a milestone or commit link moves no issue and keeps no worker current (issue 1781)
mesh/delivery merged: source.main moved to novox/mesh-catalog@7c5c29421f02
mesh/check being carried to 854dbfb3f3f0 on 7c5c29421f02 | from 3690c149c4ad
The seed linked 188 issues at once and every one read as moved at the seed's minute, hiding when each
really moved; on a worked issue the link also counted as its worker's activity. These links only place
an issue, so last-moved and the worker's activity leave them out.
2026-10-26 19:12:12 +01:00
jschoubben 69a3b3f7e0 The forge tools cut a request the forge never answers and refuse a file too big for an answer without fetching it: a 5 MB read stalled every read (issue 1577) 2026-10-26 19:03:18 +01:00
jschoubben 3cd826830b The build modules' fetch step says which changed files are compiled programs, so the checker can refuse them (issue 1577) 2026-10-26 19:00:54 +01:00
jschoubben c584dc57b9 gitea: mesh/check is the mesh's own status and never blocks; signed commits, an outdated branch and a rejected review are said as blocking the mesh's merge (issue 1945, review round 2)
mesh/delivery merged: source.main moved to novox/mesh-catalog@3690c149c4ad
mesh/check being carried to c584dc57b95e on 3690c149c4ad | a compiled program is not refused: mesh-build-node reported no programs
2026-10-26 18:37:02 +01:00
jschoubben e1e5a8ff0d gitea: mesh-tools writes a user's delivered repository and main's rule stays the operator's: only its merge whitelist gets mesh-tools, read back; what else blocks the mesh's merge is said (issue 1945, the operator's answer 2026-10-26 18:30) 2026-10-26 18:34:03 +01:00
jschoubben c85472285d gitea: main's rule read back must say what was written, since the forge silently drops a whitelist login that cannot write (issue 1945, review round 1) 2026-10-26 18:23:42 +01:00
jschoubben f0fbe193fa gitea: mesh-tools administers a user's delivered repository and puts main's rule right, permissions read as the account sees them; knowledge-base.progress: minimal answers leave items to the tree, failed calls warn before parse faults (issues 1945, 1947) 2026-10-26 17:59:34 +01:00
57 changed files with 600 additions and 849 deletions
-2
View File
@@ -35,8 +35,6 @@ modules/mesh-build-image/cmd/mesh-build-image/mesh-build-image
/checks/paths/paths /checks/paths/paths
/modules/mesh-build-go/mesh-build-go /modules/mesh-build-go/mesh-build-go
/modules/mesh-delivery-gate/mesh-delivery-gate /modules/mesh-delivery-gate/mesh-delivery-gate
modules/mesh-config/cmd/mesh-config/mesh-config
checks/placement/placement
# Built by their modules' build step (issue 1577), never committed. # Built by their modules' build step (issue 1577), never committed.
modules/forticlient/cmd/forticlient-tools/forticlient-tools modules/forticlient/cmd/forticlient-tools/forticlient-tools
-5
View File
@@ -1,5 +0,0 @@
module checks/placement
go 1.26.0
require git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026193323-03dce1237803
-14
View File
@@ -1,14 +0,0 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026193323-03dce1237803 h1:RInAboHBk5THYf6Agvy05iO9p2XV5zGb0ilYa+iSO7o=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026193323-03dce1237803/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/klauspost/compress v1.20.0 h1:a3C1ke2ohxFymNlb2HWAHjDeKCI90scRskErZkR0ezA=
github.com/klauspost/compress v1.20.0/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/nats-io/nats.go v1.54.0 h1:vsXoOxjHp/GmPUN+EcI7uOf/uB+iAP+kEsAFNQN0yzA=
github.com/nats-io/nats.go v1.54.0/go.mod h1:y+DZoD1oBOYfZTU681eTUiUjI0vbqYGixNVFHcjHJ0k=
github.com/nats-io/nkeys v0.4.16 h1:rd5oAuLOb8mnAycB0xleuEBNS1pVVnN0fv/FF34Eypg=
github.com/nats-io/nkeys v0.4.16/go.mod h1:llLgWoI0o4z/Q57q2R1kHfmocyhGV6VG/U18Glg1Afs=
github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw=
github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
-80
View File
@@ -1,80 +0,0 @@
// Command placement is the catalogue's check of its placement files (novox/hq ADR 0549 §1, ADR 0555; issue 1126): every
// `placement/<node>.json` reads as a placement of the node its name says, and the files together are refused, every
// finding named at once, for two modules of one node on one host port, a pin, binding or holding naming a provider not
// placed where it says, a seat held once for the mesh held twice, two nodes with one address on the private network, or
// more than one hub (mesh-sdk delivery.CheckPlacements).
//
// A module the catalogue does not hold and a node not enrolled are the composition check's to refuse: placement names
// modules of every repository the indexer reads and the nodes the mesh enrolled, neither of which this tree holds.
//
// go run . <catalogue root>
//
// Exits 1 on a finding.
package main
import (
"fmt"
"os"
"path/filepath"
"sort"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// Check reads every placement file under root and answers each finding; none is a catalogue whose placement holds. A
// catalogue with no placement directory has none to check.
func Check(root string) ([]string, error) {
dir := filepath.Join(root, delivery.PlacementDir)
entries, err := os.ReadDir(dir)
if os.IsNotExist(err) {
return nil, nil
}
if err != nil {
return nil, err
}
var found []string
placements := map[string]delivery.Placement{}
for _, e := range entries {
rel := delivery.PlacementDir + "/" + e.Name()
node, ok := delivery.PlacementFileNode(rel)
if e.IsDir() || !ok {
found = append(found, fmt.Sprintf("%s is not a placement file: the directory holds only <node>.json, one per node", rel))
continue
}
raw, err := os.ReadFile(filepath.Join(dir, e.Name()))
if err != nil {
return nil, err
}
p, err := delivery.DecodePlacement(raw)
if err != nil {
found = append(found, fmt.Sprintf("%s does not read as a placement: %v", rel, err))
continue
}
if p.Revision != "" {
found = append(found, fmt.Sprintf("%s names a revision, which only the indexer writes into the key", rel))
}
placements[node] = p
}
all := func(string) bool { return true }
found = append(found, delivery.CheckPlacements(placements, all, all)...)
sort.Strings(found)
return found, nil
}
func main() {
if len(os.Args) != 2 {
fmt.Fprintln(os.Stderr, "usage: placement <catalogue root>")
os.Exit(2)
}
found, err := Check(os.Args[1])
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(2)
}
for _, f := range found {
fmt.Println(f)
}
if len(found) > 0 {
os.Exit(1)
}
}
-55
View File
@@ -1,55 +0,0 @@
package main
import (
"os"
"path/filepath"
"strings"
"testing"
)
// The catalogue's placement files as the merge check reads them (ADR 0549 §1, ADR 0555): the real tree holds every
// rule, and a tree that breaks them is refused with every finding named at once.
func TestPlacementFiles(t *testing.T) {
found, err := Check("../..")
if err != nil {
t.Fatal(err)
}
for _, f := range found {
t.Errorf("%s (novox/hq ADR 0549 §1)", f)
}
root := t.TempDir()
write := func(name, body string) {
t.Helper()
if err := os.MkdirAll(filepath.Join(root, "placement"), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(root, "placement", name), []byte(body), 0o644); err != nil {
t.Fatal(err)
}
}
if found, err := Check(root); err != nil || len(found) != 0 {
t.Fatalf("a catalogue with no placement directory: %v, %v", found, err)
}
write("ace.json", `{"v":1,"node":"ace","decided":{"hub":true,"address":"10.77.0.1"},"modules":[{"module":"a","ports":{"80":8080}},{"module":"b","ports":{"81":8080}}]}`)
write("g14.json", `{"v":1,"node":"g14","decided":{"hub":true,"address":"10.77.0.1"},"modules":[{"module":"c","bindings":{"pg":{"node":"ace","module":"pg"}}}]}`)
write("novox.json", `{"v":1,"node":"shanks","modules":[]}`)
write("bad.json", `{"v":1,"node":"bad","modules":{}}`)
write("dated.json", `{"v":1,"node":"dated","revision":"abc","modules":[]}`)
write("notes.md", "x")
found, err = Check(root)
if err != nil {
t.Fatal(err)
}
for _, want := range []string{"a and b are both published on host port 8080", "bound to ace/pg", "both have the address 10.77.0.1",
"ace, g14 are each placed as it", "placement file of novox names the node", "bad.json does not read", "dated.json names a revision",
"notes.md is not a placement file"} {
hit := false
for _, f := range found {
hit = hit || strings.Contains(f, want)
}
if !hit {
t.Errorf("no finding says %q; found %v", want, found)
}
}
}
-1
View File
@@ -1 +0,0 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
+86 -28
View File
@@ -174,8 +174,27 @@ function ignoreSecretSetting(cfg: Record<string, unknown>, key: string): void {
console.error(`the setting ${key} is ignored: it comes from the vault`); console.error(`the setting ${key} is ignored: it comes from the vault`);
} }
/** How long the forge may take to answer one request before it is cut and said (novox/hq issue 1577: a request for a 5 MB
* file never ended, and every read through the module timed out behind it). */
export const FORGE_REQUEST_TIMEOUT_MS = 20_000;
/** The most a tool's answer carries of one file: the bus's message bound is 1 MiB and a file travels as text (issue 1577). */
export const MAX_FILE_BYTES = 512 * 1024;
/** One request to the forge, cut at the limit and failing loud naming the request. */
async function timedFetch(url: string, init: RequestInit, label: string, timeoutMs: number): Promise<Response> {
try {
return await fetch(url, { ...init, signal: init.signal ? AbortSignal.any([init.signal, AbortSignal.timeout(timeoutMs)]) : AbortSignal.timeout(timeoutMs) });
} catch (err) {
if (err instanceof Error && err.name === "TimeoutError") throw new Error(`Gitea API ${label}: no answer in ${timeoutMs / 1000} s, the request was cut`);
throw err;
}
}
export class GiteaClient { export class GiteaClient {
readonly baseUrl: string; readonly baseUrl: string;
/** The limit of one request; a test sets it short. */
requestTimeoutMs = FORGE_REQUEST_TIMEOUT_MS;
private readonly tokens: TokenSource; private readonly tokens: TokenSource;
/** A token given as a string is one somebody configured; a source decides for itself (token.ts). */ /** A token given as a string is one somebody configured; a source decides for itself (token.ts). */
@@ -230,14 +249,14 @@ export class GiteaClient {
private send(path: string, options: RequestInit, token: string): Promise<Response> { private send(path: string, options: RequestInit, token: string): Promise<Response> {
checkSentPath(path); checkSentPath(path);
return fetch(`${this.baseUrl}/api/v1${path}`, { return timedFetch(`${this.baseUrl}/api/v1${path}`, {
...options, ...options,
headers: { headers: {
"Content-Type": "application/json", "Content-Type": "application/json",
Authorization: `token ${token}`, Authorization: `token ${token}`,
...(options.headers as Record<string, string> | undefined), ...(options.headers as Record<string, string> | undefined),
}, },
}); }, `${options.method ?? "GET"} ${path}`, this.requestTimeoutMs);
} }
/** Generic authenticated API call — the escape hatch for endpoints without a dedicated method. /** Generic authenticated API call — the escape hatch for endpoints without a dedicated method.
@@ -540,7 +559,7 @@ export class GiteaClient {
* **A listing the forge cut is no listing** (it says `truncated` on a page with nothing more to read, or the pages * **A listing the forge cut is no listing** (it says `truncated` on a page with nothing more to read, or the pages
* run past a bound): the call fails naming the repository and the commit, never answering the directories it did * run past a bound): the call fails naming the repository and the commit, never answering the directories it did
* see, since a directory left out would be taken for one without a module. Reads only. */ * see, since a directory left out would be taken for one without a module. Reads only. */
async commitTrees(owner: string, repo: string, sha: string, paths?: Record<string, string[]>, listed?: string[]): Promise<{ root: string; trees: Record<string, string>; paths?: Record<string, Record<string, string>>; listed?: Record<string, Record<string, string>> }> { async commitTrees(owner: string, repo: string, sha: string, paths?: Record<string, string[]>): Promise<{ root: string; trees: Record<string, string>; paths?: Record<string, Record<string, string>> }> {
const base = `${repoPath(owner, repo)}/git/trees/${commitSegment(sha)}?recursive=true`; const base = `${repoPath(owner, repo)}/git/trees/${commitSegment(sha)}?recursive=true`;
const cut = (why: string) => new Error(`the forge's tree listing of ${owner}/${repo} at ${sha} was cut (${why}); no trees are answered from a part`); const cut = (why: string) => new Error(`the forge's tree listing of ${owner}/${repo} at ${sha} was cut (${why}); no trees are answered from a part`);
const entries: any[] = []; const entries: any[] = [];
@@ -567,23 +586,7 @@ export class GiteaClient {
if (!tree) throw new Error(`the forge's tree listing of ${owner}/${repo} at ${sha} holds ${e.path} but not the tree of ${dir}`); if (!tree) throw new Error(`the forge's tree listing of ${owner}/${repo} at ${sha} holds ${e.path} but not the tree of ${dir}`);
trees[dir] = tree; trees[dir] = tree;
} }
// **Listed directories** (novox/hq ADR 0549 §2, issue 1126): each asked directory's files directly under it, by name if (paths === undefined) return { root, trees };
// to blob id, from the same listing: how the forge bridge finds the catalogue's placement files, one per node, at a
// merge. A directory the commit lacks lists nothing (the catalogue before its first placement file).
let files: Record<string, Record<string, string>> | undefined;
if (listed !== undefined) {
files = {};
for (const dir of listed) {
files[dir] = {};
const prefix = dir === "" ? "" : `${dir}/`;
for (const e of entries) {
if (e?.type !== "blob" || typeof e.path !== "string" || !e.path.startsWith(prefix)) continue;
const name = e.path.slice(prefix.length);
if (name !== "" && !name.includes("/")) files[dir][name] = String(e.sha);
}
}
}
if (paths === undefined) return files === undefined ? { root, trees } : { root, trees, listed: files };
// **Named paths** (novox/hq ADR 0499 §3, issue 1758): each asked path's git object id at the commit, a file's or a // **Named paths** (novox/hq ADR 0499 §3, issue 1758): each asked path's git object id at the commit, a file's or a
// directory's, from the same listing, by directory and by path relative to it. A path the listing lacks fails // directory's, from the same listing, by directory and by path relative to it. A path the listing lacks fails
// naming both, and nothing is answered in part. // naming both, and nothing is answered in part.
@@ -598,7 +601,7 @@ export class GiteaClient {
named[dir][p] = id; named[dir][p] = id;
} }
} }
return files === undefined ? { root, trees, paths: named } : { root, trees, paths: named, listed: files }; return { root, trees, paths: named };
} }
/** Whether a repository holds a file at a commit: true, false for a 404, thrown when the forge cannot say. */ /** Whether a repository holds a file at a commit: true, false for a 404, thrown when the forge cannot say. */
@@ -787,6 +790,22 @@ export class GiteaClient {
/** One file's contents at a ref (default the repository's default branch), decoded. */ /** One file's contents at a ref (default the repository's default branch), decoded. */
async getFile(owner: string, repo: string, path: string, ref?: string): Promise<{ path: string; ref?: string; sha: string; size: number; content: string }> { async getFile(owner: string, repo: string, path: string, ref?: string): Promise<{ path: string; ref?: string; sha: string; size: number; content: string }> {
const qs = ref ? `?ref=${encodeURIComponent(ref)}` : ""; const qs = ref ? `?ref=${encodeURIComponent(ref)}` : "";
// The size first, from the file's directory listing, which carries no content: a file over what an answer can carry is
// refused without its body being fetched (issue 1577). A file the listing does not show is asked for as before.
const clean = path.replace(/^\/+|\/+$/g, "");
pathSegments("file path", clean); // a path that is another path or user is refused before anything reaches the forge
const slash = clean.lastIndexOf("/");
const dir = slash < 0 ? "" : clean.slice(0, slash);
let listing: any[] | null = null;
try {
listing = await this.request<any[]>(`${repoPath(owner, repo)}/contents${dir ? "/" + pathSegments("file path", dir) : ""}${qs}`);
} catch (err) {
if (!(err instanceof Error) || !/: 404 /.test(err.message)) throw err;
}
const entry = Array.isArray(listing) ? listing.find((e) => e?.path === clean) : undefined;
if (entry && Number(entry.size) > MAX_FILE_BYTES) {
throw new Error(`the file is ${Number(entry.size)} bytes, over the ${MAX_FILE_BYTES / 1024} KiB a tool answer can carry; read it from a clone`);
}
const f = await this.request<any>(`${repoPath(owner, repo)}/contents/${pathSegments("file path", path.replace(/^\/+|\/+$/g, ""))}${qs}`); const f = await this.request<any>(`${repoPath(owner, repo)}/contents/${pathSegments("file path", path.replace(/^\/+|\/+$/g, ""))}${qs}`);
if (!f || f.type !== "file") throw new Error(`Gitea API: ${path} is not a file`); if (!f || f.type !== "file") throw new Error(`Gitea API: ${path} is not a file`);
const content = f.encoding === "base64" ? Buffer.from(String(f.content ?? ""), "base64").toString("utf8") : String(f.content ?? ""); const content = f.encoding === "base64" ? Buffer.from(String(f.content ?? ""), "base64").toString("utf8") : String(f.content ?? "");
@@ -956,14 +975,14 @@ export class GiteaAdmin {
* throwing on it — the caller decides which non-2xx codes are idempotent successes. Only an * throwing on it — the caller decides which non-2xx codes are idempotent successes. Only an
* unexpected status becomes an error, and only where the caller says so. */ * unexpected status becomes an error, and only where the caller says so. */
private async request(path: string, options: RequestInit = {}): Promise<AdminResponse> { private async request(path: string, options: RequestInit = {}): Promise<AdminResponse> {
const res = await fetch(`${this.baseUrl}/api/v1${path}`, { const res = await timedFetch(`${this.baseUrl}/api/v1${path}`, {
...options, ...options,
headers: { headers: {
"Content-Type": "application/json", "Content-Type": "application/json",
Authorization: this.authorization, Authorization: this.authorization,
...(options.headers as Record<string, string> | undefined), ...(options.headers as Record<string, string> | undefined),
}, },
}); }, `${options.method ?? "GET"} ${path}`, FORGE_REQUEST_TIMEOUT_MS);
const text = await res.text(); const text = await res.text();
let body: any = null; let body: any = null;
if (text) { if (text) {
@@ -1063,12 +1082,19 @@ export class GiteaAdmin {
return true; return true;
} }
/** What the forge itself says an account may do on a repository: none, read, write, admin or owner. */ /** What the forge gives an account on a repository, effective: none, read, write, admin or owner. It is the repository as
* that account sees it (the site administrator acts as the account with `sudo`), so access through an organisation's team
* counts. The collaborator endpoint is not used: it names only a collaborator's own permission, and says "read" of an
* account that writes through a team (novox/hq issue 1945). */
async repositoryPermission(owner: string, repo: string, user: string): Promise<string> { async repositoryPermission(owner: string, repo: string, user: string): Promise<string> {
const path = `${repoPath(owner, repo)}/collaborators/${encodeURIComponent(user)}/permission`; const path = `${repoPath(owner, repo)}?sudo=${encodeURIComponent(user)}`;
const res = await this.request(path); const res = await this.request(path);
if (res.status === 404) return "none";
if (res.status !== 200) GiteaAdmin.fail(path, res); if (res.status !== 200) GiteaAdmin.fail(path, res);
return String(res.body?.permission ?? "none"); const p = res.body?.permissions;
if (p?.admin === true) return res.body?.owner?.login === user ? "owner" : "admin";
if (p?.push === true) return "write";
return p?.pull === true ? "read" : "none";
} }
/** Make an account a collaborator of a repository that reads only. The forge asks a repository's administrator for /** Make an account a collaborator of a repository that reads only. The forge asks a repository's administrator for
@@ -1083,6 +1109,38 @@ export class GiteaAdmin {
await this.addCollaborator(owner, repo, user, "write"); await this.addCollaborator(owner, repo, user, "write");
} }
/** The rule that applies to main in a repository, as its administrator reads it: a rule named main, else the first glob
* rule covering main; null when none does. With the administrator's login. */
async mainRule(owner: string, repo: string): Promise<BranchProtection | null> {
const rules = await this.branchProtections(owner, repo);
return rules.find((r) => (r.rule_name ?? r.branch_name) === MAIN_BRANCH)
?? rules.find((r) => ruleCovers(String(r.rule_name ?? r.branch_name ?? ""), MAIN_BRANCH)) ?? null;
}
/** Put an account on main's merge whitelist in a repository of a user, and nothing else (the operator, 2026-10-26:
* write, the rule untouched). Only an existing rule whose merge whitelist is on is changed, and only its whitelist: the
* account is added to the logins already there. Push, status checks, approvals and the administrator override stay as
* the operator set them; with no rule, or the whitelist off, nothing is written (whoever writes may merge). Read back:
* the forge drops a whitelist login that cannot write the repository and still answers success, so a whitelist read back
* without the account fails loud. Answers the rule as the forge holds it after. With the administrator's login. */
async ensureMergerListed(owner: string, repo: string, account: string): Promise<{ changed: boolean; rule: BranchProtection | null }> {
const rule = await this.mainRule(owner, repo);
const listed = rule?.merge_whitelist_usernames ?? [];
if (!rule || rule.enable_merge_whitelist !== true || listed.includes(account)) return { changed: false, rule };
const name = String(rule.rule_name ?? rule.branch_name);
const path = `${repoPath(owner, repo)}/branch_protections/${encodeURIComponent(name)}`;
const res = await this.request(path, { method: "PATCH", body: JSON.stringify({ merge_whitelist_usernames: [...listed, account] }) });
if (res.status !== 200) GiteaAdmin.fail(path, res);
const after = (await this.branchProtections(owner, repo)).find((r) => (r.rule_name ?? r.branch_name) === name);
if (!after) throw new Error(`main's rule ${name} of ${owner}/${repo} was written, but the forge does not list it`);
const held = after.merge_whitelist_usernames ?? [];
if (after.enable_merge_whitelist === true && !held.includes(account)) {
throw new Error(`${account} was added to the merge whitelist of main's rule ${name} of ${owner}/${repo}, but the forge ` +
`holds [${held.join(", ")}]: it drops a login that cannot write the repository`);
}
return { changed: true, rule: after };
}
private async addCollaborator(owner: string, repo: string, user: string, permission: "read" | "write"): Promise<void> { private async addCollaborator(owner: string, repo: string, user: string, permission: "read" | "write"): Promise<void> {
const path = `${repoPath(owner, repo)}/collaborators/${encodeURIComponent(user)}`; const path = `${repoPath(owner, repo)}/collaborators/${encodeURIComponent(user)}`;
const res = await this.request(path, { method: "PUT", body: JSON.stringify({ permission }) }); const res = await this.request(path, { method: "PUT", body: JSON.stringify({ permission }) });
@@ -1208,9 +1266,9 @@ export class GiteaAdmin {
* other answer rejects (novox/hq issue 120). * other answer rejects (novox/hq issue 120).
*/ */
async holdsTeamMember(org: string, team: string, username: string, password: string): Promise<boolean> { async holdsTeamMember(org: string, team: string, username: string, password: string): Promise<boolean> {
const me = await fetch(`${this.baseUrl}/api/v1/user`, { const me = await timedFetch(`${this.baseUrl}/api/v1/user`, {
headers: { Authorization: "Basic " + Buffer.from(`${username}:${password}`).toString("base64") }, headers: { Authorization: "Basic " + Buffer.from(`${username}:${password}`).toString("base64") },
}); }, "GET /user", FORGE_REQUEST_TIMEOUT_MS);
if (me.status === 401 || me.status === 403) return false; if (me.status === 401 || me.status === 403) return false;
if (me.status !== 200) throw new Error(`Gitea GET /user as ${username}: ${me.status}`); if (me.status !== 200) throw new Error(`Gitea GET /user as ${username}: ${me.status}`);
const teams = await this.request(`/orgs/${encodeURIComponent(org)}/teams?limit=50`); const teams = await this.request(`/orgs/${encodeURIComponent(org)}/teams?limit=50`);
+1 -15
View File
@@ -183,15 +183,6 @@ test("named paths: each asked path's git object id at the commit, a file's or a
}); });
assert.deepEqual(got.trees, { "": ROOT, "modules/a": h("4"), "modules/b/c": h("8") }); assert.deepEqual(got.trees, { "": ROOT, "modules/a": h("4"), "modules/b/c": h("8") });
assert.equal((await client.commitTrees("novox", "hq", COMMIT)).paths, undefined, "no paths asked, none answered"); assert.equal((await client.commitTrees("novox", "hq", COMMIT)).paths, undefined, "no paths asked, none answered");
// Listed directories (ADR 0549 §2, issue 1126): the files directly under each, not deeper; a missing one lists none.
const listed = await client.commitTrees("novox", "hq", COMMIT, undefined, ["modules/a", "modules/b", "", "placement"]);
assert.deepEqual(listed.listed, {
"modules/a": { "main.go": h("6"), "module.json": h("5") },
"modules/b": {},
"": { "README.md": h("1"), "module.json": h("2") },
placement: {},
});
assert.equal((await client.commitTrees("novox", "hq", COMMIT)).listed, undefined, "no directory asked, none listed");
} finally { } finally {
server.close(); server.close();
} }
@@ -218,12 +209,7 @@ test("gitea_commit_trees takes paths as {directory: [paths]} and refuses another
const { client, server, asked } = await serve([{ sha: COMMIT, url: "u", tree: LISTING, truncated: false, page: 1, total_count: LISTING.length }]); const { client, server, asked } = await serve([{ sha: COMMIT, url: "u", tree: LISTING, truncated: false, page: 1, total_count: LISTING.length }]);
try { try {
const tool = getGiteaTools(client).find((t) => t.name === "gitea_commit_trees")!; const tool = getGiteaTools(client).find((t) => t.name === "gitea_commit_trees")!;
assert.ok("paths" in tool.input && "listed" in tool.input, "the tool names the arguments"); assert.ok("paths" in tool.input, "the tool names the argument");
const listedAnswer: any = await tool.run({ owner: "novox", repo: "hq", sha: COMMIT, listed: ["modules/d"] } as any);
assert.deepEqual(listedAnswer.listed, { "modules/d": { "notes.md": h("b") } });
for (const bad of ["modules/d", ["../x"], ["/abs"], [1]]) {
await assert.rejects(tool.run({ owner: "novox", repo: "hq", sha: COMMIT, listed: bad } as any), /listed/, JSON.stringify(bad));
}
const got: any = await tool.run({ owner: "novox", repo: "hq", sha: COMMIT, paths: { "modules/a": ["main.go"] } } as any); const got: any = await tool.run({ owner: "novox", repo: "hq", sha: COMMIT, paths: { "modules/a": ["main.go"] } } as any);
assert.deepEqual(got.paths, { "modules/a": { "main.go": h("6") } }); assert.deepEqual(got.paths, { "modules/a": { "main.go": h("6") } });
const before = asked.length; const before = asked.length;
+1
View File
@@ -640,6 +640,7 @@ test("no tool's caller text becomes another path or another user: `..` and `?sud
await tool("gitea_delete_branch").run({ ...repo, branch: "feat/x" }); await tool("gitea_delete_branch").run({ ...repo, branch: "feat/x" });
await tool("gitea_get_file").run({ ...repo, path: "modules/gitea/module.json" }).catch(() => undefined); await tool("gitea_get_file").run({ ...repo, path: "modules/gitea/module.json" }).catch(() => undefined);
assert.deepEqual(state.asked, ["DELETE /repos/novox/mesh-catalog/branches/feat/x", assert.deepEqual(state.asked, ["DELETE /repos/novox/mesh-catalog/branches/feat/x",
"GET /repos/novox/mesh-catalog/contents/modules/gitea", // the file's size first (issue 1577)
"GET /repos/novox/mesh-catalog/contents/modules/gitea/module.json"]); "GET /repos/novox/mesh-catalog/contents/modules/gitea/module.json"]);
}); });
+45
View File
@@ -90,3 +90,48 @@ test("a pass pages until it reaches back past its floor, and no further", async
server.close(); server.close();
} }
}); });
// **One forge request never holds the others, and a file too big for an answer is not downloaded** (novox/hq issue 1577): a
// request for a 4.9 MB file never ended and every read through the module timed out behind it.
test("a request the forge never answers is cut at the limit and says which request", async () => {
const { GiteaClient } = await import("../client.ts");
const server = createServer((req) => { if (!req.url?.includes("fast")) return; });
await new Promise<void>((r) => server.listen(0, r));
const client = new GiteaClient(`http://127.0.0.1:${(server.address() as any).port}`, "t");
client.requestTimeoutMs = 150;
try {
const started = Date.now();
await assert.rejects(client.api("/repos/novox/hq/pulls"), /Gitea API GET \/repos\/novox\/hq\/pulls: no answer in 0\.15 s/);
assert.ok(Date.now() - started < 3000);
} finally {
server.closeAllConnections();
await new Promise<void>((r) => server.close(() => r()));
}
});
test("a file over 512 KiB is refused from its directory listing, and its body is never asked for", async () => {
const { GiteaClient } = await import("../client.ts");
const asked: string[] = [];
const server = createServer((req, res) => {
asked.push(req.url ?? "");
res.setHeader("Content-Type", "application/json");
if (req.url?.startsWith("/api/v1/repos/novox/hq/contents/bin?")) {
return res.end(JSON.stringify([{ path: "bin/big", type: "file", size: 5_000_000 }, { path: "bin/small", type: "file", size: 5 }]));
}
if (req.url?.startsWith("/api/v1/repos/novox/hq/contents/bin/small")) {
return res.end(JSON.stringify({ type: "file", encoding: "base64", sha: "s", size: 5, content: Buffer.from("hello").toString("base64") }));
}
res.statusCode = 404;
res.end("{}");
});
await new Promise<void>((r) => server.listen(0, r));
const client = new GiteaClient(`http://127.0.0.1:${(server.address() as any).port}`, "t");
try {
await assert.rejects(client.getFile("novox", "hq", "bin/big", "main"),
/the file is 5000000 bytes, over the 512 KiB a tool answer can carry; read it from a clone/);
assert.ok(!asked.some((u) => u.includes("/contents/bin/big")), asked.join("\n"));
assert.equal((await client.getFile("novox", "hq", "bin/small", "main")).content, "hello");
} finally {
await new Promise<void>((r) => server.close(() => r()));
}
});
+53 -4
View File
@@ -28,6 +28,7 @@ interface Team { id: number; name: string; units: any; members: Set<string> }
interface ForgeRepo { owner: string; name: string; full: string; private: boolean; archived?: boolean; ownerVisibility: string; collaborators: Map<string, string> } interface ForgeRepo { owner: string; name: string; full: string; private: boolean; archived?: boolean; ownerVisibility: string; collaborators: Map<string, string> }
const forgeRepos: ForgeRepo[] = []; const forgeRepos: ForgeRepo[] = [];
const userRules = new Map<string, any>();
const repoFaults = { refuse: new Set<string>(), permission: new Set<string>() }; const repoFaults = { refuse: new Set<string>(), permission: new Set<string>() };
function addRepo(full: string, isPrivate: boolean, ownerVisibility = "public"): ForgeRepo { function addRepo(full: string, isPrivate: boolean, ownerVisibility = "public"): ForgeRepo {
const [owner, name] = full.split("/"); const [owner, name] = full.split("/");
@@ -84,7 +85,7 @@ function fakeForge() {
const server = createServer(async (req, res) => { const server = createServer(async (req, res) => {
const url = new URL(req.url ?? "/", "http://fake"); const url = new URL(req.url ?? "/", "http://fake");
const p = url.pathname.replace(/^\/api\/v1/, ""); const p = url.pathname.replace(/^\/api\/v1/, "");
const me = who(req); const me = url.searchParams.has("sudo") && isAdmin(who(req)) ? url.searchParams.get("sudo") : who(req);
calls.push(`${req.method} ${p} as ${me}`); calls.push(`${req.method} ${p} as ${me}`);
if (me === null) return json(res, 401, { message: "unauthorized" }); if (me === null) return json(res, 401, { message: "unauthorized" });
@@ -168,6 +169,23 @@ function fakeForge() {
if (m[2] === "feat/unguarded") return json(res, 200, { name: m[2], protected: false }); if (m[2] === "feat/unguarded") return json(res, 200, { name: m[2], protected: false });
return json(res, 404, { message: "branch does not exist" }); return json(res, 404, { message: "branch does not exist" });
} }
// A repository of a user: its rules are its administrator's (a site administrator, or a collaborator with admin).
m = p.match(/^\/repos\/(?!novox\/)([^/]+)\/([^/]+)\/branch_protections(?:\/(.*))?$/);
if (m) {
const key = `${m[1]}/${m[2]}`;
const repo = forgeRepos.find((r) => r.full === key);
if (!repo || !["admin", "owner"].includes(permissionOf(me, repo))) return json(res, 403, { message: "user should be an owner or a collaborator with admin write of a repository" });
if (req.method === "GET" && m[3] === undefined) return json(res, 200, userRules.has(key) ? [userRules.get(key)] : []);
// As gitea does: a whitelist login that cannot write the repository is dropped, and the write still answered.
const keep = (b: any) => (Array.isArray(b.merge_whitelist_usernames)
? { ...b, merge_whitelist_usernames: b.merge_whitelist_usernames.filter((u: string) => ["write", "admin", "owner"].includes(permissionOf(u, repo))) } : b);
if (req.method === "PATCH" && m[3] === "main") {
if (!userRules.has(key)) return json(res, 404, { message: "branch protection does not exist" });
Object.assign(userRules.get(key), keep(await read(req)));
return json(res, 200, userRules.get(key));
}
if (req.method === "POST") { const b = keep(await read(req)); userRules.set(key, b); return json(res, 201, b); }
}
// gitea's reqAdmin on the whole group: reading the rules is administration too. // gitea's reqAdmin on the whole group: reading the rules is administration too.
m = p.match(/^\/repos\/novox\/([^/]+)\/branch_protections(?:\/(.*))?$/); m = p.match(/^\/repos\/novox\/([^/]+)\/branch_protections(?:\/(.*))?$/);
if (m) { if (m) {
@@ -205,6 +223,7 @@ function fakeForge() {
m = p.match(/^\/repos\/([^/]+)\/([^/]+)(?:\/collaborators\/([^/]+)(\/permission)?)?$/); m = p.match(/^\/repos\/([^/]+)\/([^/]+)(?:\/collaborators\/([^/]+)(\/permission)?)?$/);
if (m) { if (m) {
const repo = forgeRepos.find((r) => r.owner === decodeURIComponent(m![1]) && r.name === decodeURIComponent(m![2])); const repo = forgeRepos.find((r) => r.owner === decodeURIComponent(m![1]) && r.name === decodeURIComponent(m![2]));
if (repo && req.method === "GET" && !m[3] && url.searchParams.has("sudo") && repoFaults.permission.has(repo.full)) return json(res, 500, { message: "the forge is not well" });
if (!repo || permissionOf(me, repo) === "none") return json(res, 404, { message: "repository does not exist" }); if (!repo || permissionOf(me, repo) === "none") return json(res, 404, { message: "repository does not exist" });
if (req.method === "GET" && !m[3]) { if (req.method === "GET" && !m[3]) {
const mine = permissionOf(me, repo); const mine = permissionOf(me, repo);
@@ -214,7 +233,8 @@ function fakeForge() {
if (req.method === "GET" && m[4]) { if (req.method === "GET" && m[4]) {
if (repoFaults.permission.has(repo.full)) return json(res, 500, { message: "the forge is not well" }); if (repoFaults.permission.has(repo.full)) return json(res, 500, { message: "the forge is not well" });
if (!isAdmin(me) && me !== user) return json(res, 403, { message: "only the user or an administrator" }); if (!isAdmin(me) && me !== user) return json(res, 403, { message: "only the user or an administrator" });
return json(res, 200, { permission: permissionOf(user, repo) }); // As gitea answers: the collaborator's own permission, not what a team gives the account.
return json(res, 200, { permission: repo.collaborators.get(user) ?? "read" });
} }
if (req.method === "PUT" && !m[4]) { if (req.method === "PUT" && !m[4]) {
// gitea's reqAdmin: the repository's administration, which on a repository of another user the site administrator alone holds. // gitea's reqAdmin: the repository's administration, which on a repository of another user the site administrator alone holds.
@@ -608,7 +628,8 @@ test("the merging account is given write on each repository the indexer's module
await writes.take({ key: "car-hunter-2", op: "put", value: { v: 1, repository: "jschoubben/car-hunter" } }); await writes.take({ key: "car-hunter-2", op: "put", value: { v: 1, repository: "jschoubben/car-hunter" } });
assert.equal(forge.calls.length, from, "a repository already written asks nothing more"); assert.equal(forge.calls.length, from, "a repository already written asks nothing more");
const repo = (full: string) => forgeRepos.find((x) => x.full === full)!; const repo = (full: string) => forgeRepos.find((x) => x.full === full)!;
assert.equal(repo("jschoubben/car-hunter").collaborators.get(TOOLS_ACCOUNT), "write"); assert.equal(repo("jschoubben/car-hunter").collaborators.get(TOOLS_ACCOUNT), "write", "a user's repository is written, not administered");
assert.equal(userRules.has("jschoubben/car-hunter"), false, "a repository with no rule for main is given none");
assert.equal(repo("novox/hq").collaborators.size, 0, "the organisation's repository is written through its team, no collaborator"); assert.equal(repo("novox/hq").collaborators.size, 0, "the organisation's repository is written through its team, no collaborator");
assert.equal((await client.getRepo("jschoubben", "car-hunter")).writes, true); assert.equal((await client.getRepo("jschoubben", "car-hunter")).writes, true);
assert.deepEqual(said, [{ type: WRITABLE_EVENT, body: { owner: "jschoubben", repo: "car-hunter", account: TOOLS_ACCOUNT } }]); assert.deepEqual(said, [{ type: WRITABLE_EVENT, body: { owner: "jschoubben", repo: "car-hunter", account: TOOLS_ACCOUNT } }]);
@@ -636,9 +657,37 @@ test("the merging account is given write on each repository the indexer's module
assert.ok(!forge.calls.slice(puts).some((c) => c.startsWith("PUT ")), "no write is asked on a repository no module builds from"); assert.ok(!forge.calls.slice(puts).some((c) => c.startsWith("PUT ")), "no write is asked on a repository no module builds from");
assert.equal(repo("jschoubben/dotfiles").collaborators.get(TOOLS_ACCOUNT), "read"); assert.equal(repo("jschoubben/dotfiles").collaborators.get(TOOLS_ACCOUNT), "read");
repoFaults.refuse.clear(); repoFaults.refuse.clear();
// The operator's rule is left as set (the operator, 2026-10-26: "Write, rule untouched"): only mesh-tools is added to its
// merge whitelist; a status the mesh does not set and required approvals are said as blocking the mesh's merge, unchanged.
const operators = { rule_name: "main", enable_push: true, enable_merge_whitelist: true, merge_whitelist_usernames: ["jschoubben"],
enable_status_check: true, status_check_contexts: ["mesh/check", "mesh/merge-gate", "ci/external"], required_approvals: 2,
block_admin_merge_override: false, require_signed_commits: true, block_on_outdated_branch: true, block_on_rejected_reviews: true };
userRules.set("jschoubben/snake", structuredClone(operators));
const after = await grantTool.run({ repositories: ["jschoubben/snake"] }) as any; const after = await grantTool.run({ repositories: ["jschoubben/snake"] }) as any;
assert.deepEqual(after.granted, ["jschoubben/snake"]); assert.deepEqual(after.granted, ["jschoubben/snake"]);
assert.deepEqual(((await unwritableTool.run({})) as any).unwritable, []); assert.equal(repo("jschoubben/snake").collaborators.get(TOOLS_ACCOUNT), "write");
assert.deepEqual(userRules.get("jschoubben/snake"), { ...operators, merge_whitelist_usernames: ["jschoubben", TOOLS_ACCOUNT] }, "only the whitelist changed");
assert.deepEqual(after.merge_whitelist, [{ repository: "jschoubben/snake", changed: true, listed: ["jschoubben", TOOLS_ACCOUNT] }]);
const blockedWhy = "writable, but main's rule blocks the mesh's merge: it requires the status ci/external, which the mesh does not set; " +
"it requires 2 approving reviews on the forge, which the gate does not give; it requires signed commits, which the mesh does not make; " +
"it refuses a branch behind main, which the mesh does not bring up to date; " +
"it refuses a merge after a review asked for changes, which the gate does not clear";
assert.deepEqual(after.merge_blocked, [{ repository: "jschoubben/snake", why: blockedWhy }]);
const again = await unwritableTool.run({}) as any;
assert.deepEqual([again.unwritable, again.merge_blocked], [[], [{ repository: "jschoubben/snake", why: blockedWhy }]]);
const writesBefore = forge.calls.filter((c) => /^PATCH .*branch_protections/.test(c)).length;
assert.deepEqual(((await grantTool.run({ repositories: ["jschoubben/snake"] })) as any).merge_whitelist[0].changed, false);
assert.equal(forge.calls.filter((c) => /^PATCH .*branch_protections/.test(c)).length, writesBefore, "a listed account is not written again");
// The mesh's own statuses never block: a rule requiring only them, with mesh-tools listed, is not reported.
userRules.set("jschoubben/snake", { rule_name: "main", enable_merge_whitelist: true, merge_whitelist_usernames: [TOOLS_ACCOUNT],
enable_status_check: true, status_check_contexts: ["mesh/check", "mesh/merge-gate", "mesh/repo-check"] });
assert.deepEqual(((await unwritableTool.run({})) as any).merge_blocked, []);
assert.deepEqual(((await grantTool.run({ repositories: ["jschoubben/snake"] })) as any).merge_blocked, []);
// The forge drops a whitelist login that cannot write and still answers 200: the whitelist is read back and that fails loud.
userRules.set("jschoubben/dotfiles", { rule_name: "main", enable_merge_whitelist: true, merge_whitelist_usernames: ["jschoubben"] });
await assert.rejects(GiteaAdmin.fromEnv(env).ensureMergerListed("jschoubben", "dotfiles", TOOLS_ACCOUNT),
/mesh-tools was added to the merge whitelist of main's rule main of jschoubben\/dotfiles, but the forge holds \[jschoubben\]: it drops a login that cannot write/);
userRules.delete("jschoubben/dotfiles");
// A module gone from the indexer leaves its repository out of the delivered list. // A module gone from the indexer leaves its repository out of the delivered list.
await writes.take({ key: "snake", op: "delete" }); await writes.take({ key: "snake", op: "delete" });
assert.deepEqual(writes.delivered(), ["jschoubben/car-hunter", "novox/hq"]); assert.deepEqual(writes.delivered(), ["jschoubben/car-hunter", "novox/hq"]);
+3 -15
View File
@@ -76,17 +76,6 @@ function namedPaths(raw: unknown): Record<string, string[]> | undefined {
return out; return out;
} }
/** The `listed` argument of gitea_commit_trees: [directory], each plain (no `.` or `..` segment, not absolute, the root
* the empty name); anything else is refused before the forge is asked. Absent is absent. */
function listedDirs(raw: unknown): string[] | undefined {
if (raw === undefined || raw === null) return undefined;
if (!Array.isArray(raw) || raw.some((d) => typeof d !== "string")) throw new Error("listed is [directory, ...]");
for (const d of raw as string[]) {
if (d !== "" && d.split("/").some((seg) => seg === "" || seg === "." || seg === "..")) throw new Error(`listed: ${JSON.stringify(d)} is not a plain relative directory`);
}
return raw as string[];
}
export function getGiteaTools(gitea: GiteaClient, admin?: () => GiteaAdmin): ToolDefinition[] { export function getGiteaTools(gitea: GiteaClient, admin?: () => GiteaAdmin): ToolDefinition[] {
return namesChecked(allGiteaTools(gitea, admin)); return namesChecked(allGiteaTools(gitea, admin));
} }
@@ -385,7 +374,7 @@ function allGiteaTools(gitea: GiteaClient, admin?: () => GiteaAdmin): ToolDefini
// ---- Contents and branches ---- // ---- Contents and branches ----
{ {
name: "gitea_get_file", name: "gitea_get_file",
description: "One file's contents from a repository, decoded, at a branch, tag or commit (default the repository's default branch).", description: "One file's contents from a repository, decoded, at a branch, tag or commit (default the repository's default branch). Refuses a file over 512 KiB, which a tool answer cannot carry: read it from a clone.",
input: { input: {
owner: { type: "string", description: "the repository owner" }, owner: { type: "string", description: "the repository owner" },
repo: { type: "string", description: "the repository name" }, repo: { type: "string", description: "the repository name" },
@@ -561,20 +550,19 @@ function allGiteaTools(gitea: GiteaClient, admin?: () => GiteaAdmin): ToolDefini
}, },
{ {
name: "gitea_commit_trees", name: "gitea_commit_trees",
description: "A commit's trees from the forge's one recursive listing: the root tree, and the tree of every directory that holds a module.json (the root as an empty name), each by its git hash. What the forge bridge puts in a merge's message (novox/hq ADR 0437 §5). With paths, {directory: [path relative to it]}, the answer also carries paths, {directory: {path: git object id}} of each named file or directory at the commit (novox/hq ADR 0499 §3); a path the commit lacks fails naming the directory and the path. The commit is a full 40-hex sha. With listed, [directory], the answer also carries listed, {directory: {name: git blob id}} of the files directly under each (the catalogue's placement files, novox/hq ADR 0549 §2). A listing the forge cut is refused, never answered in part. Reads only.", description: "A commit's trees from the forge's one recursive listing: the root tree, and the tree of every directory that holds a module.json (the root as an empty name), each by its git hash. What the forge bridge puts in a merge's message (novox/hq ADR 0437 §5). With paths, {directory: [path relative to it]}, the answer also carries paths, {directory: {path: git object id}} of each named file or directory at the commit (novox/hq ADR 0499 §3); a path the commit lacks fails naming the directory and the path. The commit is a full 40-hex sha. A listing the forge cut is refused, never answered in part. Reads only.",
input: { input: {
owner: { type: "string", description: "the repository owner" }, owner: { type: "string", description: "the repository owner" },
repo: { type: "string", description: "the repository name" }, repo: { type: "string", description: "the repository name" },
sha: { type: "string", description: "the commit, its full 40-hex sha" }, sha: { type: "string", description: "the commit, its full 40-hex sha" },
paths: { type: "object", description: "optional, {directory: [path relative to it]}: the git object id of each named file or directory is answered under paths; the root directory is the empty name" }, paths: { type: "object", description: "optional, {directory: [path relative to it]}: the git object id of each named file or directory is answered under paths; the root directory is the empty name" },
listed: { type: "array", items: { type: "string" }, description: "optional, [directory]: the files directly under each, {directory: {name: git blob id}}, are answered under listed; a directory the commit lacks lists none" },
}, },
run: async (args) => { run: async (args) => {
const owner = String(args.owner ?? "").trim(), repo = String(args.repo ?? "").trim(); const owner = String(args.owner ?? "").trim(), repo = String(args.repo ?? "").trim();
const sha = String(args.sha ?? "").trim(); const sha = String(args.sha ?? "").trim();
if (!owner || !repo) throw new Error("name the repository: owner and repo"); if (!owner || !repo) throw new Error("name the repository: owner and repo");
if (!/^[0-9a-f]{40}$/i.test(sha)) throw new Error("sha is a commit's full 40-hex sha"); if (!/^[0-9a-f]{40}$/i.test(sha)) throw new Error("sha is a commit's full 40-hex sha");
return gitea.commitTrees(owner, repo, sha, namedPaths(args.paths), listedDirs(args.listed)); return gitea.commitTrees(owner, repo, sha, namedPaths(args.paths));
}, },
}, },
{ {
+100 -25
View File
@@ -18,10 +18,18 @@
// `repo.writable` {owner, repo, account}: mesh-delivery-gate, which does not ask the operator's Approve of a pull request // `repo.writable` {owner, repo, account}: mesh-delivery-gate, which does not ask the operator's Approve of a pull request
// whose merge the forge would refuse, reads that repository again on it. // whose merge the forge would refuse, reads that repository again on it.
// //
// **A repository of a user is written, and main's rule is left as the operator set it** (the operator, 2026-10-26, asked
// "Write, rule untouched"): the account is a write collaborator there, and only added to main's merge whitelist when that
// rule keeps one (client.ts ensureMergerListed). What else in main's rule stops the mesh's merge (a status the mesh does not
// set, approvals the gate does not give) is changed by nobody but said: both tools list such a repository as writable, but
// blocked, with why (merge_blocked).
//
// Not taken away: a repository whose last module goes from the indexer keeps its collaborator until a hand removes it; // Not taken away: a repository whose last module goes from the indexer keeps its collaborator until a hand removes it;
// gitea_unwritable_repos and gitea_grant_write say the delivered list as it is now. // gitea_unwritable_repos and gitea_grant_write say the delivered list as it is now.
import type { ForgeRepository } from "./client.js"; import type { ForgeRepository } from "./client.js";
import type { BranchProtection } from "./protection.js";
import { CHECK_CONTEXT, CHECK_STATUS, REPO_CHECK_CONTEXT } from "./pulls.js";
/** The indexer's state this module reads: one key per registered module, its value naming the repository. */ /** The indexer's state this module reads: one key per registered module, its value naming the repository. */
export const INDEXER_MODULES = "mesh-delivery-indexer.modules"; export const INDEXER_MODULES = "mesh-delivery-indexer.modules";
@@ -34,8 +42,35 @@ export interface WriteForge {
repository(owner: string, repo: string): Promise<ForgeRepository | null>; repository(owner: string, repo: string): Promise<ForgeRepository | null>;
repositoryPermission(owner: string, repo: string, user: string): Promise<string>; repositoryPermission(owner: string, repo: string, user: string): Promise<string>;
addWriteCollaborator(owner: string, repo: string, user: string): Promise<void>; addWriteCollaborator(owner: string, repo: string, user: string): Promise<void>;
mainRule(owner: string, repo: string): Promise<BranchProtection | null>;
ensureMergerListed(owner: string, repo: string, account: string): Promise<{ changed: boolean; rule: BranchProtection | null }>;
} }
/** The statuses every pull request the mesh delivers carries (mesh-delivery's checks); a rule requiring another blocks. */
const MESH_CONTEXTS = new Set([CHECK_STATUS, CHECK_CONTEXT, REPO_CHECK_CONTEXT]);
/** What in main's rule stops the mesh's merge, in words; empty when nothing does. Changed by nobody here: the operator's. */
export function mergeBlocks(rule: BranchProtection | null, account: string): string[] {
if (!rule) return [];
const blocks: string[] = [];
if (rule.enable_merge_whitelist === true && !(rule.merge_whitelist_usernames ?? []).includes(account)) {
blocks.push(`its merge whitelist [${(rule.merge_whitelist_usernames ?? []).join(", ")}] leaves out ${account} (gitea_grant_write adds it)`);
}
if (rule.enable_status_check === true) {
for (const c of rule.status_check_contexts ?? []) {
if (!MESH_CONTEXTS.has(c)) blocks.push(`it requires the status ${c}, which the mesh does not set`);
}
}
const approvals = Number(rule.required_approvals ?? 0);
if (approvals > 0) blocks.push(`it requires ${approvals} approving review${approvals === 1 ? "" : "s"} on the forge, which the gate does not give`);
if (rule.require_signed_commits === true) blocks.push("it requires signed commits, which the mesh does not make");
if (rule.block_on_outdated_branch === true) blocks.push("it refuses a branch behind main, which the mesh does not bring up to date");
if (rule.block_on_rejected_reviews === true) blocks.push("it refuses a merge after a review asked for changes, which the gate does not clear");
return blocks;
}
const blockedWhy = (blocks: string[]): string => `writable, but main's rule blocks the mesh's merge: ${blocks.join("; ")}`;
/** One change of the indexer's modules state, as the SDK's state watch hands it. */ /** One change of the indexer's modules state, as the SDK's state watch hands it. */
export interface ModulesChange { export interface ModulesChange {
key: string; key: string;
@@ -51,8 +86,13 @@ export interface Written {
delivered: string[]; delivered: string[];
/** Made writable now, each read back from the forge. */ /** Made writable now, each read back from the forge. */
granted: string[]; granted: string[];
/** Written already (the organisation's team, or a collaborator that writes). */ /** Written already (a collaborator of a user's repository), or written through the organisation's team. */
already: string[]; already: string[];
/** Each repository of a user: whether the account was added to main's merge whitelist now, and the whitelist read back
* (null: main's rule keeps no whitelist, or there is no rule, and whoever writes may merge). */
merge_whitelist: { repository: string; changed: boolean; listed: string[] | null }[];
/** Writable, but main's rule (the operator's, left as set) stops the mesh's merge, and why. */
merge_blocked: { repository: string; why: string }[];
/** The forge refused, in its own words; or the repository is the organisation's, whose team should write it. */ /** The forge refused, in its own words; or the repository is the organisation's, whose team should write it. */
refused: { repository: string; why: string }[]; refused: { repository: string; why: string }[];
/** Repositories the account cannot write that are left so, and why: they build no registered module. */ /** Repositories the account cannot write that are left so, and why: they build no registered module. */
@@ -118,7 +158,7 @@ export class DeliveredWrites {
"(the watch is not granted or not answered); nothing is granted"); "(the watch is not granted or not answered); nothing is granted");
} }
const delivered = this.delivered(); const delivered = this.delivered();
const result: Written = { account: this.account, delivered, granted: [], already: [], refused: [], left_read_only: [] }; const result: Written = { account: this.account, delivered, granted: [], already: [], merge_whitelist: [], merge_blocked: [], refused: [], left_read_only: [] };
const named = only && only.length ? only.map((x) => String(x).trim()) : null; const named = only && only.length ? only.map((x) => String(x).trim()) : null;
const targets: string[] = []; const targets: string[] = [];
for (const full of named ?? delivered) { for (const full of named ?? delivered) {
@@ -140,36 +180,55 @@ export class DeliveredWrites {
result.refused.push({ repository: full, why: `its permission could not be read — ${said(err)}` }); result.refused.push({ repository: full, why: `its permission could not be read — ${said(err)}` });
continue; continue;
} }
if (WRITES.has(permission)) {
this.writable.add(full);
result.already.push(full);
continue;
}
if (owner === this.org) { if (owner === this.org) {
if (WRITES.has(permission)) {
this.writable.add(full);
result.already.push(full);
continue;
}
// The organisation's repositories are written through its team (tools-account.ts), never a collaborator. // The organisation's repositories are written through its team (tools-account.ts), never a collaborator.
result.refused.push({ repository: full, why: `a repository of ${this.org}, which ${this.account} writes through the ` + result.refused.push({ repository: full, why: `a repository of ${this.org}, which ${this.account} writes through the ` +
`team ${this.org}/${this.account}; the forge gives it "${permission}", so the team is not as it should be (it is ` + `team ${this.org}/${this.account}; the forge gives it "${permission}", so the team is not as it should be (it is ` +
"set again at the next mint of the tools' token)" }); "set again at the next mint of the tools' token)" });
continue; continue;
} }
try { // A repository of a user: the account writes it, and is put on main's merge whitelist when the rule keeps one.
await this.forge.addWriteCollaborator(owner, name, this.account); let changed = false;
} catch (err) { if (!WRITES.has(permission)) {
result.refused.push({ repository: full, why: said(err) }); try {
continue; await this.forge.addWriteCollaborator(owner, name, this.account);
} catch (err) {
result.refused.push({ repository: full, why: said(err) });
continue;
}
let after: string;
try {
after = await this.forge.repositoryPermission(owner, name, this.account);
} catch (err) {
result.refused.push({ repository: full, why: `granted, but its permission could not be read back — ${said(err)}` });
continue;
}
if (!WRITES.has(after)) {
result.refused.push({ repository: full, why: `the grant was answered, but the forge still gives ${this.account} "${after}"` });
continue;
}
changed = true;
} }
let after: string;
try { try {
after = await this.forge.repositoryPermission(owner, name, this.account); const { changed: listedNow, rule } = await this.forge.ensureMergerListed(owner, name, this.account);
result.merge_whitelist.push({ repository: full, changed: listedNow,
listed: rule?.enable_merge_whitelist === true ? (rule.merge_whitelist_usernames ?? []) : null });
const blocks = mergeBlocks(rule, this.account);
if (blocks.length > 0) result.merge_blocked.push({ repository: full, why: blockedWhy(blocks) });
} catch (err) { } catch (err) {
result.refused.push({ repository: full, why: `granted, but its permission could not be read back — ${said(err)}` }); result.refused.push({ repository: full, why: `${this.account} writes it, but could not be put on main's merge whitelist — ${said(err)}` });
continue;
}
if (!WRITES.has(after)) {
result.refused.push({ repository: full, why: `the grant was answered, but the forge still gives ${this.account} "${after}"` });
continue; continue;
} }
this.writable.add(full); this.writable.add(full);
if (!changed) {
result.already.push(full);
continue;
}
result.granted.push(full); result.granted.push(full);
await this.emit(WRITABLE_EVENT, { owner, repo: name, account: this.account }).catch((err) => await this.emit(WRITABLE_EVENT, { owner, repo: name, account: this.account }).catch((err) =>
this.log(`${this.account} writes ${full} now, and that could not be said on the bus — ${said(err)}`)); this.log(`${this.account} writes ${full} now, and that could not be said on the bus — ${said(err)}`));
@@ -192,10 +251,12 @@ export class DeliveredWrites {
} }
/** Every delivered repository the account cannot write, as the forge says now, and why. */ /** Every delivered repository the account cannot write, as the forge says now, and why. */
async unwritable(): Promise<{ account: string; delivered: number; unwritable: { repository: string; permission: string; why: string }[] }> { async unwritable(): Promise<{ account: string; delivered: number; unwritable: { repository: string; permission: string; why: string }[];
merge_blocked: { repository: string; why: string }[] }> {
if (!this.heard) throw new Error(`which repositories build a registered module is not known yet: ${INDEXER_MODULES} has not been read`); if (!this.heard) throw new Error(`which repositories build a registered module is not known yet: ${INDEXER_MODULES} has not been read`);
const delivered = this.delivered(); const delivered = this.delivered();
const out: { repository: string; permission: string; why: string }[] = []; const out: { repository: string; permission: string; why: string }[] = [];
const blocked: { repository: string; why: string }[] = [];
for (const full of delivered) { for (const full of delivered) {
const [owner, name] = split(full); const [owner, name] = split(full);
let permission: string; let permission: string;
@@ -205,12 +266,21 @@ export class DeliveredWrites {
out.push({ repository: full, permission: "unknown", why: `the forge did not answer ${this.account}'s permission on it — ${said(err)}` }); out.push({ repository: full, permission: "unknown", why: `the forge did not answer ${this.account}'s permission on it — ${said(err)}` });
continue; continue;
} }
if (WRITES.has(permission)) continue; if (WRITES.has(permission)) {
if (owner === this.org) continue; // the organisation's main rules are the mesh's own (mainrules.ts)
try {
const blocks = mergeBlocks(await this.forge.mainRule(owner, name), this.account);
if (blocks.length > 0) blocked.push({ repository: full, why: blockedWhy(blocks) });
} catch (err) {
blocked.push({ repository: full, why: `writable, but main's rule could not be read — ${said(err)}` });
}
continue;
}
const modules = [...this.modules].filter(([, r]) => r === full).map(([m]) => m).sort(); const modules = [...this.modules].filter(([, r]) => r === full).map(([m]) => m).sort();
out.push({ repository: full, permission, why: `builds ${modules.join(", ")}, and the forge gives ${this.account} ` + out.push({ repository: full, permission, why: `builds ${modules.join(", ")}, and the forge gives ${this.account} ` +
`"${permission}" on it: its commit status and its merge are refused until gitea_grant_write grants it` }); `"${permission}" on it: its commit status and its merge are refused until gitea_grant_write grants it` });
} }
return { account: this.account, delivered: delivered.length, unwritable: out }; return { account: this.account, delivered: delivered.length, unwritable: out, merge_blocked: blocked };
} }
} }
@@ -224,14 +294,19 @@ export function writeTools(writes: () => DeliveredWrites): Tool[] {
description: "Every repository that builds a registered module (the indexer's modules, read over the bus) on which " + description: "Every repository that builds a registered module (the indexer's modules, read over the bus) on which " +
"the mesh's merging forge account (mesh-tools, which sets commit statuses and merges) cannot write, with the " + "the mesh's merging forge account (mesh-tools, which sets commit statuses and merges) cannot write, with the " +
"permission the forge gives it there and why: such a pull request can be checked but never merged, and the " + "permission the forge gives it there and why: such a pull request can be checked but never merged, and the " +
"delivery gate does not ask its Approve. Empty when it writes every one. Read with the site administrator's login.", "delivery gate does not ask its Approve. Empty when it writes every one. Its merge_blocked lists each repository of " +
"a user it writes whose main's rule (the operator's, left as set) still stops the mesh's merge: a merge whitelist " +
"without it, a required status the mesh does not set, required approving reviews. Read with the site administrator's login.",
input: {}, input: {},
run: async () => writes().unwritable(), run: async () => writes().unwritable(),
}, },
{ {
name: "gitea_grant_write", name: "gitea_grant_write",
description: "Give the mesh's merging forge account (mesh-tools) write, as a collaborator, on every repository that " + description: "Give the mesh's merging forge account (mesh-tools) write, as a collaborator, on every repository of a " +
"builds a registered module and that it cannot write now, or on the repositories named (each must build one). " + "user that builds a registered module, or on the repositories named (each must build one). Main's rule there is left " +
"as the operator set it; only when it keeps a merge whitelist is mesh-tools added to it, read back (merge_whitelist). " +
"What else in that rule stops the mesh's merge (a required status the mesh does not set, required approving reviews) " +
"is changed by nobody and said in merge_blocked. " +
"The module does this itself at its start and when a module of a new repository is registered; this is the same " + "The module does this itself at its start and when a module of a new repository is registered; this is the same " +
"act by hand, such as when a merge or a commit status was refused 403. A repository of the organisation novox " + "act by hand, such as when a merge or a commit status was refused 403. A repository of the organisation novox " +
"is written through its team and needs nothing. Answers what was granted (each read back from the forge), what " + "is written through its team and needs nothing. Answers what was granted (each read back from the forge), what " +
@@ -84,9 +84,17 @@ func (m *Module) Puller(steps []StepTool, source buildstep.AskSource) *buildstep
// The capacity's own change wakes the puller (issue 1887): the runtime's watch of the config value as mesh-config // The capacity's own change wakes the puller (issue 1887): the runtime's watch of the config value as mesh-config
// resolved it; the settings file is read only when that watch cannot start. A test's Capacity is its own. // resolved it; the settings file is read only when that watch cannot start. A test's Capacity is its own.
var watch func(string, func(stdio.ProvisionChange) error) error var watch func(string, func(stdio.ProvisionChange) error) error
// And the CPU bounds it (issue 1806's follow-up): the configured capacity is an upper bound, the node's build slice
// gives floor(its quota's cores / the module's cores-per-step) at once, read live; a test judges by its own Capacity.
// And the node's build hours (the operator, 2026-10-26: g14 builds only from 18:00 to 06:00): outside them it takes
// nothing new.
var cores func() float64
var hours func() (*buildstep.BuildWindow, string)
if capacity == nil { if capacity == nil {
hours = buildstep.NodeBuildHours(m.Settings, m.Node, m.Say)
capacity = buildstep.NodeCapacity(m.Settings, buildstep.DefaultCapacities, m.Node, m.Say) capacity = buildstep.NodeCapacity(m.Settings, buildstep.DefaultCapacities, m.Node, m.Say)
watch = buildstep.WatchConfigServed watch = buildstep.WatchConfigServed
cores = buildstep.NodeCoresPerStep(m.Settings, buildstep.DefaultCoresPerStep[delivery.AskGo], m.Say)
} }
ask := m.askSteps(steps) ask := m.askSteps(steps)
var load func(string) ([]byte, error) var load func(string) ([]byte, error)
@@ -97,7 +105,7 @@ func (m *Module) Puller(steps []StepTool, source buildstep.AskSource) *buildstep
} }
} }
return &buildstep.Puller{Source: source, Language: delivery.AskGo, Node: m.Node, Put: m.Steps, PausePut: m.Pauses, Load: load, return &buildstep.Puller{Source: source, Language: delivery.AskGo, Node: m.Node, Put: m.Steps, PausePut: m.Pauses, Load: load,
Capacity: capacity, WatchConfig: watch, Logf: m.Say, Has: m.Has, Capacity: capacity, WatchConfig: watch, CoresPerStep: cores, BuildHours: hours, Logf: m.Say, Has: m.Has,
Run: func(ctx context.Context, a delivery.StepAsk) (any, error) { Run: func(ctx context.Context, a delivery.StepAsk) (any, error) {
if err := m.Check(ctx); err != nil { if err := m.Check(ctx); err != nil {
return nil, err return nil, err
@@ -3,6 +3,7 @@ package main
import ( import (
"context" "context"
"encoding/json" "encoding/json"
"fmt"
"os" "os"
"slices" "slices"
"strings" "strings"
@@ -125,12 +126,28 @@ func TestManifestClaimsTheGoBuildSeatAndReadsItsCapacity(t *testing.T) {
raw, _ := os.ReadFile(moduleDir + "/module.json") raw, _ := os.ReadFile(moduleDir + "/module.json")
var doc struct { var doc struct {
Config map[string]struct { Config map[string]struct {
Default string `json:"default"` Default json.RawMessage `json:"default"`
} `json:"config"` } `json:"config"`
} }
_ = json.Unmarshal(raw, &doc) _ = json.Unmarshal(raw, &doc)
if doc.Config[buildstep.CapacitySetting].Default != buildstep.DefaultCapacities { // Issue 1806's follow-up: the cores a step needs is a setting of its own, in the settings file, with the SDK's default.
t.Errorf("the capacity's default = %q, want %q", doc.Config[buildstep.CapacitySetting].Default, buildstep.DefaultCapacities) if cores := string(doc.Config[buildstep.CoresPerStepSetting].Default); cores != fmt.Sprint(buildstep.DefaultCoresPerStep[delivery.AskGo]) {
t.Errorf("the cores-per-step default = %s, want %g", cores, buildstep.DefaultCoresPerStep[delivery.AskGo])
}
if got := buildstep.NodeCoresPerStep(moduleDir+"/files/settings.json", 0, nil)(); got != buildstep.DefaultCoresPerStep[delivery.AskGo] {
t.Errorf("the settings template holds cores-per-step %g", got)
}
// And the build hours (the operator, 2026-10-26): a setting of their own, empty by default (any hour), in the template.
if hours := string(doc.Config[buildstep.BuildHoursSetting].Default); hours != `""` {
t.Errorf("the build-hours default = %s, want empty", hours)
}
if w, fault := buildstep.NodeBuildHours(moduleDir+"/files/settings.json", "g14", t.Errorf)(); w != nil || fault != "" {
t.Errorf("the settings template holds build hours %v (%s)", w, fault)
}
var capacity string
_ = json.Unmarshal(doc.Config[buildstep.CapacitySetting].Default, &capacity)
if capacity != buildstep.DefaultCapacities {
t.Errorf("the capacity's default = %q, want %q", capacity, buildstep.DefaultCapacities)
} }
settings := resource(t, m, "settings") settings := resource(t, m, "settings")
if settings["path"] != "${dir:state}/settings.json" { if settings["path"] != "${dir:state}/settings.json" {
@@ -15,6 +15,8 @@ import (
func (m *Module) cpuLimits() buildcpu.Limits { func (m *Module) cpuLimits() buildcpu.Limits {
slots := 1 slots := 1
switch { switch {
case m.Held != nil:
slots = m.Held.CapacityNow() // the capacity the puller judges by, bounded by the node's build slice
case m.Capacity != nil: case m.Capacity != nil:
slots = m.Capacity() slots = m.Capacity()
case m.Settings != "": case m.Settings != "":
@@ -32,5 +34,9 @@ func (m *Module) threadEnv() []string {
// cpuStatus is what the status tool says of it: the limits a step is given, and what the kernel holds this runtime // cpuStatus is what the status tool says of it: the limits a step is given, and what the kernel holds this runtime
// to (its unit and its slice), read from its cgroup rather than assumed. // to (its unit and its slice), read from its cgroup rather than assumed.
func (m *Module) cpuStatus() map[string]any { func (m *Module) cpuStatus() map[string]any {
return map[string]any{"limits": m.cpuLimits(), "in-force": buildcpu.ReadInForce()} out := map[string]any{"limits": m.cpuLimits(), "in-force": buildcpu.ReadInForce()}
if m.Held != nil {
out["capacity"] = m.Held.CapacityDetail() // the capacity judged by and why (issue 1806's follow-up)
}
return out
} }
@@ -42,11 +42,13 @@ type FetchResult struct {
// head onto it, the paths the merge changes, and the tree of each module directory (a directory holding a // head onto it, the paths the merge changes, and the tree of each module directory (a directory holding a
// module.json) in the merge. A head that does not merge cleanly is an answer: Merge is empty, Conflicts names the // module.json) in the merge. A head that does not merge cleanly is an answer: Merge is empty, Conflicts names the
// files, and the workspace is left at the trunk commit. // files, and the workspace is left at the trunk commit.
Base string `json:"base,omitempty"` Base string `json:"base,omitempty"`
Merge string `json:"merge,omitempty"` Merge string `json:"merge,omitempty"`
Conflicts []string `json:"conflicts,omitempty"` Conflicts []string `json:"conflicts,omitempty"`
Changed []string `json:"changed,omitempty"` Changed []string `json:"changed,omitempty"`
Trees map[string]buildstep.ModuleTree `json:"trees,omitempty"` // Programs are the changed files that are compiled programs (issue 1577); a merge answers it, a pointer to an empty list when there are none, absent for a fetch that made no merge.
Programs *[]buildstep.Program `json:"programs,omitempty"`
Trees map[string]buildstep.ModuleTree `json:"trees,omitempty"`
// Beside are the sibling repositories placed as plain directories under the workspace's beside directory, which // Beside are the sibling repositories placed as plain directories under the workspace's beside directory, which
// build_go_test hands to the tests as MESH_CHECK_BESIDE (ADR 0484 clause 4). // build_go_test hands to the tests as MESH_CHECK_BESIDE (ADR 0484 clause 4).
Beside []Sibling `json:"beside,omitempty"` Beside []Sibling `json:"beside,omitempty"`
@@ -14,6 +14,7 @@ import (
"strings" "strings"
"sync" "sync"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/forgekey" "git.novox.be/novox/mesh-sdk/go/forgekey"
) )
@@ -132,6 +133,9 @@ func (m *Module) lint(ctx context.Context, run forgekey.Runner, tree, d string,
// lintAll lints each directory, up to m.parallel() at once like the level's tests; the failures join in directory order // lintAll lints each directory, up to m.parallel() at once like the level's tests; the failures join in directory order
// and the error is the first directory's, so the answer does not depend on which ended first (issue 1930). // and the error is the first directory's, so the answer does not depend on which ended first (issue 1930).
func (m *Module) lintAll(ctx context.Context, run forgekey.Runner, tree string, dirs []string, env func(d string) []string) ([]Failure, error) { func (m *Module) lintAll(ctx context.Context, run forgekey.Runner, tree string, dirs []string, env func(d string) []string) ([]Failure, error) {
// The ask's phases say it (issue 1890): lint of the directories, then their tests.
buildstep.Phase(ctx, fmt.Sprintf("unit: lint of %d directories", len(dirs)))
defer buildstep.Phase(ctx, fmt.Sprintf("unit: tests of %d directories", len(dirs)))
type answer struct { type answer struct {
failed []Failure failed []Failure
err error err error
@@ -41,6 +41,9 @@ func (m *Module) merge(ctx context.Context, res *FetchResult, tree, remote, base
} }
res.Base = mr.Base res.Base = mr.Base
res.Conflicts, res.Merge, res.Changed = mr.Conflicts, mr.Merge, mr.Changed res.Conflicts, res.Merge, res.Changed = mr.Conflicts, mr.Merge, mr.Changed
if mr.Merge != "" {
res.Programs = &mr.Programs
}
if res.Merge == "" { if res.Merge == "" {
return nil return nil
} }
@@ -77,7 +77,7 @@ func TestFetchWithABaseMergesAndAnswersThePreview(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
if res.Base != baseSHA || res.Merge != mergeSHA || res.Commit != commitSHA || len(res.Conflicts) != 0 || if res.Base != baseSHA || res.Merge != mergeSHA || res.Commit != commitSHA || len(res.Conflicts) != 0 ||
strings.Join(res.Changed, ",") != "a.go,modules/m/main.go" { strings.Join(res.Changed, ",") != "a.go,modules/m/main.go" || res.Programs == nil || len(*res.Programs) != 0 { // 1577: a merge says it found no program
t.Errorf("result = %s", st.Result) t.Errorf("result = %s", st.Result)
} }
mt, ok := res.Trees["modules/m"] mt, ok := res.Trees["modules/m"]
+3 -1
View File
@@ -1,3 +1,5 @@
{ {
"capacity": "shanks=3,ace=2,g14=0,novox=0" "capacity": "shanks=3,ace=2,g14=0,novox=0",
"cores-per-step": 6,
"build-hours": ""
} }
+1 -1
View File
@@ -3,7 +3,7 @@ module meshbuildgo
go 1.26.0 go 1.26.0
require ( require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026142629-ff81bdf8cc8f git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026202658-805b4d9a3536
github.com/distribution/distribution/v3 v3.1.2 github.com/distribution/distribution/v3 v3.1.2
github.com/nats-io/nats-server/v2 v2.11.17 github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0 github.com/nats-io/nats.go v1.54.0
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026142629-ff81bdf8cc8f h1:GYLCvz2x+JNUOTbXzyh2QwzX0sTzvRlXR7lGocWjk3A= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026202658-805b4d9a3536 h1:+SoDdDXF7Tgj38WP4kRpUSps6DJVl6P+w5u/KfdtwnQ=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026142629-ff81bdf8cc8f/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026202658-805b4d9a3536/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/alicebob/miniredis/v2 v2.35.0 h1:QwLphYqCEAo1eu1TqPRN2jgVMPBweeQcR21jeqDCONI= github.com/alicebob/miniredis/v2 v2.35.0 h1:QwLphYqCEAo1eu1TqPRN2jgVMPBweeQcR21jeqDCONI=
github.com/alicebob/miniredis/v2 v2.35.0/go.mod h1:TcL7YfarKPGDAthEtl5NBeHZfeUQj6OXMm/+iu5cLMM= github.com/alicebob/miniredis/v2 v2.35.0/go.mod h1:TcL7YfarKPGDAthEtl5NBeHZfeUQj6OXMm/+iu5cLMM=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
+13 -1
View File
@@ -29,6 +29,18 @@
"description": "How many asks of the Go build seat this module runs at once on each node, as node=asks pairs separated by commas, such as shanks=3,ace=2 (novox/hq ADR 0525 §5); a node set to 0 pulls nothing, a node not named runs one at a time. The control-node and the laptop build nothing.", "description": "How many asks of the Go build seat this module runs at once on each node, as node=asks pairs separated by commas, such as shanks=3,ace=2 (novox/hq ADR 0525 §5); a node set to 0 pulls nothing, a node not named runs one at a time. The control-node and the laptop build nothing.",
"pattern": "|[a-z0-9_-]+=[0-9]+(,[a-z0-9_-]+=[0-9]+)*", "pattern": "|[a-z0-9_-]+=[0-9]+(,[a-z0-9_-]+=[0-9]+)*",
"default": "shanks=3,ace=2,g14=0,novox=0" "default": "shanks=3,ace=2,g14=0,novox=0"
},
"build-hours": {
"type": "string",
"description": "When each node may take this module's build asks, as node=HH:MM-HH:MM pairs in the node's local time, separated by commas, such as g14=18:00-06:00 (an end before its start crosses midnight). Outside its window a node takes no new step and finishes the ones it runs; a node not named, or an empty value, builds at any hour. The operator, 2026-10-26: g14 builds only outside office hours.",
"pattern": "|[a-z0-9_-]+=([01][0-9]|2[0-3]):[0-5][0-9]-([01][0-9]|2[0-3]):[0-5][0-9](,[a-z0-9_-]+=([01][0-9]|2[0-3]):[0-5][0-9]-([01][0-9]|2[0-3]):[0-5][0-9])*",
"default": ""
},
"cores-per-step": {
"type": "number",
"description": "How many processors one Go step of this module needs (a full check of a large repository takes what it is given: three at about 4 cores each ran past the 30-minute step bound on 2026-10-26). The capacity above is an upper bound: on each node the module runs at most floor(the cores of its build slice's CPU quota / this) steps at once, the quota read live from its cgroup (issue 1806), so a lowered build CPU share never starves its steps. Above 0.",
"minimum": 0.1,
"default": 6
} }
}, },
"tools": [ "tools": [
@@ -107,7 +119,7 @@
"path": "${dir:state}/settings.json", "path": "${dir:state}/settings.json",
"mode": "0600", "mode": "0600",
"owner": "mesh-build-go", "owner": "mesh-build-go",
"takes-config": ["capacity"], "takes-config": ["capacity", "cores-per-step", "build-hours"],
"merge": "json", "merge": "json",
"template": "files/settings.json" "template": "files/settings.json"
}, },
+3 -2
View File
@@ -24,8 +24,9 @@ The Dockerfile's steps may use the network (ADR 0487 §6); nothing a step can re
- `build_image_fetch` fetches a commit over SSH into a workspace (a base commit makes the merge preview, as the other - `build_image_fetch` fetches a commit over SSH into a workspace (a base commit makes the merge preview, as the other
build modules' fetch). Long: answers a step id and ends on its key in `steps`. build modules' fetch). Long: answers a step id and ends on its key in `steps`.
- `build_image_build` builds one image artifact: `buildah bud` with the Dockerfile of the module's directory as file - `build_image_build` builds one image artifact: `buildah bud` with the Dockerfile `from` names (in the module's
and context, `--target`, and the images `build.on` pins as `--build-arg`. Answers the image's digest. With directory or a subdirectory of it) as file and the module's own directory as context, so a COPY names paths from the
module's root (issue 1703), `--target`, and the images `build.on` pins as `--build-arg`. Answers the image's digest. With
`publish: true` (a merged trunk commit only) the module's own process then pushes it to the registry under `publish: true` (a merged trunk commit only) the module's own process then pushes it to the registry under
`<module>/<artifact>:<tag of its own>` (the first 12 hex digits of the digest unless the call gives one), waiting out `<module>/<artifact>:<tag of its own>` (the first 12 hex digits of the digest unless the call gives one), waiting out
a registry held still, and the key carries the digest the registry serves and the reference pinned by it. Without a registry held still, and the key carries the digest the registry serves and the reference pinned by it. Without
@@ -40,9 +40,17 @@ func (m *Module) Puller(steps []StepTool, source buildstep.AskSource) *buildstep
// The capacity's own change wakes the puller (issue 1887): the runtime's watch of the config value as mesh-config // The capacity's own change wakes the puller (issue 1887): the runtime's watch of the config value as mesh-config
// resolved it; the settings file is read only when that watch cannot start. A test's Capacity is its own. // resolved it; the settings file is read only when that watch cannot start. A test's Capacity is its own.
var watch func(string, func(stdio.ProvisionChange) error) error var watch func(string, func(stdio.ProvisionChange) error) error
// And the CPU bounds it (issue 1806's follow-up): the configured capacity is an upper bound, the node's build slice
// gives floor(its quota's cores / the module's cores-per-step) at once, read live; a test judges by its own Capacity.
// And the node's build hours (the operator, 2026-10-26: g14 builds only from 18:00 to 06:00): outside them it takes
// nothing new.
var cores func() float64
var hours func() (*buildstep.BuildWindow, string)
if capacity == nil { if capacity == nil {
hours = buildstep.NodeBuildHours(m.Settings, m.Node, m.Say)
capacity = buildstep.NodeCapacity(m.Settings, buildstep.DefaultCapacities, m.Node, m.Say) capacity = buildstep.NodeCapacity(m.Settings, buildstep.DefaultCapacities, m.Node, m.Say)
watch = buildstep.WatchConfigServed watch = buildstep.WatchConfigServed
cores = buildstep.NodeCoresPerStep(m.Settings, buildstep.DefaultCoresPerStep[delivery.AskImage], m.Say)
} }
ask := m.askSteps(steps) ask := m.askSteps(steps)
var load func(string) ([]byte, error) var load func(string) ([]byte, error)
@@ -50,7 +58,7 @@ func (m *Module) Puller(steps []StepTool, source buildstep.AskSource) *buildstep
load = func(key string) ([]byte, error) { return m.ReadPauses.Get(key) } load = func(key string) ([]byte, error) { return m.ReadPauses.Get(key) }
} }
return &buildstep.Puller{Source: source, Language: delivery.AskImage, Node: m.Node, Put: m.Steps, PausePut: m.Pauses, Load: load, return &buildstep.Puller{Source: source, Language: delivery.AskImage, Node: m.Node, Put: m.Steps, PausePut: m.Pauses, Load: load,
Capacity: capacity, WatchConfig: watch, Logf: m.Say, Capacity: capacity, WatchConfig: watch, CoresPerStep: cores, BuildHours: hours, Logf: m.Say,
Run: func(ctx context.Context, a delivery.StepAsk) (any, error) { Run: func(ctx context.Context, a delivery.StepAsk) (any, error) {
if err := m.Check(ctx); err != nil { if err := m.Check(ctx); err != nil {
return nil, err return nil, err
@@ -55,18 +55,18 @@ func (m *Module) buildTool() StepTool {
} }
// unsupportedImage says why this module does not build the artifact, or "" when it does. It is the one place that // unsupportedImage says why this module does not build the artifact, or "" when it does. It is the one place that
// knows which image forms are out: ADR 0487 §1 takes the Dockerfile's directory as the build context and the base // knows which image forms are out: ADR 0487 §1 builds from the module's own directory (the build context) and the base
// from the Dockerfile or an image pinned in `build.on`, so a context that is another repository and a base that is // from the Dockerfile or an image pinned in `build.on`, so a context that is another repository and a base that is
// another module's artifact are not built (the lead's decision for R4c-9, which retires them with build-agent at R4g). // another module's artifact are not built (the lead's decision for R4c-9, which retires them with build-agent at R4g).
func unsupportedImage(manifest catalogue.Manifest, a catalogue.Artifact) string { func unsupportedImage(manifest catalogue.Manifest, a catalogue.Artifact) string {
if a.Context != nil { if a.Context != nil {
return fmt.Sprintf("the image %s of %s builds with the repository %s as its context; mesh-build-image takes the Dockerfile's directory as the build context (ADR 0487 §1) and does not build a whole repository", return fmt.Sprintf("the image %s of %s builds with the repository %s as its context; mesh-build-image takes the module's own directory as the build context (ADR 0487 §1) and does not build another repository",
a.Name, manifest.Module, a.Context.Repository) a.Name, manifest.Module, a.Context.Repository)
} }
if manifest.Build != nil { if manifest.Build != nil {
for _, on := range manifest.Build.On { for _, on := range manifest.Build.On {
if on.Module != "" { if on.Module != "" {
return fmt.Sprintf("the image %s of %s stands on the artifact %s of the module %s through the build argument %s; mesh-build-image builds only from the Dockerfile's directory and an image pinned in build.on (ADR 0487 §1)", return fmt.Sprintf("the image %s of %s stands on the artifact %s of the module %s through the build argument %s; mesh-build-image builds only from the module's own directory and an image pinned in build.on (ADR 0487 §1)",
a.Name, manifest.Module, on.Artifact, on.Module, on.Arg) a.Name, manifest.Module, on.Artifact, on.Module, on.Arg)
} }
} }
@@ -183,8 +183,14 @@ func (m *Module) build(ctx context.Context, args map[string]any) (any, error) {
return m.push(ctx, workspace, manifest.Module, local, res, tag) return m.push(ctx, workspace, manifest.Module, local, res, tag)
} }
// dockerfileOf is the Dockerfile an artifact's `from` names and the build context: the directory it lies in. It must // dockerfileOf is the Dockerfile an artifact's `from` names and the build context: the module's own directory, the
// be a regular file inside the module's own directory (checks/imageplace holds the catalogue to the same). // one its manifest lies in, whichever subdirectory the Dockerfile sits in. `from` is a path relative to that
// directory, and so is every COPY of the Dockerfile: a recipe in a subdirectory (photos' admin-client/Dockerfile
// copies admin-client/... and client-server/serve.mjs) builds as build-agent built it, and as the fingerprint reads
// the module's whole directory as its build source. Taking the Dockerfile's own directory instead failed every such
// COPY with `copier: stat ... no such file or directory` (issue 1703, 2026-10-26). The context never leaves the module's
// directory, so a module's build reads no other module's files. The Dockerfile must be a regular file inside that
// directory (checks/imageplace holds the catalogue to the same).
func dockerfileOf(moduleDir, from string) (dockerfile, contextDir string, err error) { func dockerfileOf(moduleDir, from string) (dockerfile, contextDir string, err error) {
clean := filepath.Clean(filepath.FromSlash(strings.TrimSpace(from))) clean := filepath.Clean(filepath.FromSlash(strings.TrimSpace(from)))
if from == "" || filepath.IsAbs(clean) || clean == ".." || strings.HasPrefix(clean, ".."+string(filepath.Separator)) { if from == "" || filepath.IsAbs(clean) || clean == ".." || strings.HasPrefix(clean, ".."+string(filepath.Separator)) {
@@ -195,7 +201,7 @@ func dockerfileOf(moduleDir, from string) (dockerfile, contextDir string, err er
if err != nil || !info.Mode().IsRegular() { if err != nil || !info.Mode().IsRegular() {
return "", "", fmt.Errorf("%q is not a file in the module's directory", from) return "", "", fmt.Errorf("%q is not a file in the module's directory", from)
} }
return dockerfile, filepath.Dir(dockerfile), nil return dockerfile, moduleDir, nil
} }
// localName names the image in buildah's store: one name for each artifact of a workspace. // localName names the image in buildah's store: one name for each artifact of a workspace.
@@ -94,7 +94,7 @@ func TestBuildRunsBuildahRootlessAndAnswersTheDigest(t *testing.T) {
} }
} }
if !strings.HasSuffix(bud[0].line, " "+dir) { if !strings.HasSuffix(bud[0].line, " "+dir) {
t.Errorf("the context is not the Dockerfile's directory:\n%s", bud[0].line) t.Errorf("the context is not the module's directory:\n%s", bud[0].line)
} }
for _, c := range f.calls { for _, c := range f.calls {
if c.name != "buildah" { if c.name != "buildah" {
@@ -122,6 +122,43 @@ func TestBuildRunsBuildahRootlessAndAnswersTheDigest(t *testing.T) {
} }
} }
// Issue 1703, 2026-10-26: a Dockerfile in a subdirectory of the module builds with the module's directory as its
// context, so its COPY of a path from the module's root (photos' admin-client/Dockerfile copies admin-client/... and
// client-server/serve.mjs) finds it; the Dockerfile's own directory as the context failed with `copier: stat
// "/admin-client/package.json": no such file or directory`.
func TestBuildContextIsTheModulesDirectoryForADockerfileInASubdirectory(t *testing.T) {
m, f, states := setup(t, func(s string) string {
return strings.Replace(s, `"from": "Dockerfile"`, `"from": "admin-client/Dockerfile"`, 1)
})
dir := filepath.Join(m.Work, "ws1", "tree", "modules", "demo")
for name, body := range map[string]string{
"admin-client/Dockerfile": "ARG DEMO_BASE\nFROM ${DEMO_BASE} AS final\nCOPY admin-client/package.json client-server/serve.mjs /\n",
"admin-client/package.json": "{}\n",
"client-server/serve.mjs": "\n",
} {
if err := os.MkdirAll(filepath.Dir(filepath.Join(dir, name)), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, name), []byte(body), 0o644); err != nil {
t.Fatal(err)
}
}
_, st := runBuild(t, m, states, nil)
if st.State != delivery.StepPassed {
t.Fatalf("step = %+v", st)
}
bud := f.called("bud")
if len(bud) != 1 {
t.Fatalf("buildah bud ran %d times", len(bud))
}
if !strings.Contains(bud[0].line, "--file "+dir+"/admin-client/Dockerfile ") {
t.Errorf("the Dockerfile is not the subdirectory's:\n%s", bud[0].line)
}
if !strings.HasSuffix(bud[0].line, " "+dir) {
t.Errorf("the context is not the module's directory, so admin-client/ and client-server/ are out of reach:\n%s", bud[0].line)
}
}
// 1703c, issue 1703: a failing Dockerfile step fails the step, its reason saying buildah's error from stderr, never the // 1703c, issue 1703: a failing Dockerfile step fails the step, its reason saying buildah's error from stderr, never the
// last STEP line buildah writes on stdout; nothing is pushed. // last STEP line buildah writes on stdout; nothing is pushed.
func TestBuildFailsWithBuildahsError(t *testing.T) { func TestBuildFailsWithBuildahsError(t *testing.T) {
@@ -178,7 +215,7 @@ func TestBuildRefusesAnImageWithARepositoryContextOrAModuleBase(t *testing.T) {
}{ }{
"context": {func(s string) string { "context": {func(s string) string {
return strings.Replace(s, `"target": "final"`, `"target": "final", "context": {"seat": "git", "repository": "novox/mesh-controller"}`, 1) return strings.Replace(s, `"target": "final"`, `"target": "final", "context": {"seat": "git", "repository": "novox/mesh-controller"}`, 1)
}, []string{"novox/mesh-controller", "Dockerfile's directory", "ADR 0487"}}, }, []string{"novox/mesh-controller", "module's own directory", "ADR 0487"}},
"module base": {func(s string) string { "module base": {func(s string) string {
return strings.Replace(s, `"image": "docker.io/library/busybox@sha256:0000000000000000000000000000000000000000000000000000000000000001"`, `"module": "mesh-sdk", "artifact": "lib"`, 1) return strings.Replace(s, `"image": "docker.io/library/busybox@sha256:0000000000000000000000000000000000000000000000000000000000000001"`, `"module": "mesh-sdk", "artifact": "lib"`, 1)
}, []string{"artifact lib of the module mesh-sdk", "DEMO_BASE"}}, }, []string{"artifact lib of the module mesh-sdk", "DEMO_BASE"}},
@@ -15,6 +15,8 @@ import (
func (m *Module) cpuLimits() buildcpu.Limits { func (m *Module) cpuLimits() buildcpu.Limits {
slots := 1 slots := 1
switch { switch {
case m.Held != nil:
slots = m.Held.CapacityNow() // the capacity the puller judges by, bounded by the node's build slice
case m.Capacity != nil: case m.Capacity != nil:
slots = m.Capacity() slots = m.Capacity()
case m.Settings != "": case m.Settings != "":
@@ -32,5 +34,9 @@ func (m *Module) threadEnv() []string {
// cpuStatus is what the status tool says of it: the limits a step is given, and what the kernel holds this runtime // cpuStatus is what the status tool says of it: the limits a step is given, and what the kernel holds this runtime
// to (its unit and its slice), read from its cgroup rather than assumed. // to (its unit and its slice), read from its cgroup rather than assumed.
func (m *Module) cpuStatus() map[string]any { func (m *Module) cpuStatus() map[string]any {
return map[string]any{"limits": m.cpuLimits(), "in-force": buildcpu.ReadInForce()} out := map[string]any{"limits": m.cpuLimits(), "in-force": buildcpu.ReadInForce()}
if m.Held != nil {
out["capacity"] = m.Held.CapacityDetail() // the capacity judged by and why (issue 1806's follow-up)
}
return out
} }
@@ -31,13 +31,15 @@ const ForgeAlias = "mesh-forge"
// files in conflict, the paths the merge changes and the tree of each module directory). A head that does not // files in conflict, the paths the merge changes and the tree of each module directory). A head that does not
// merge cleanly is an answer: Merge is empty, Conflicts names the files, and the workspace is left at the trunk. // merge cleanly is an answer: Merge is empty, Conflicts names the files, and the workspace is left at the trunk.
type FetchResult struct { type FetchResult struct {
Workspace string `json:"workspace"` Workspace string `json:"workspace"`
Commit string `json:"commit"` Commit string `json:"commit"`
Base string `json:"base,omitempty"` Base string `json:"base,omitempty"`
Merge string `json:"merge,omitempty"` Merge string `json:"merge,omitempty"`
Conflicts []string `json:"conflicts,omitempty"` Conflicts []string `json:"conflicts,omitempty"`
Changed []string `json:"changed,omitempty"` Changed []string `json:"changed,omitempty"`
Trees map[string]buildstep.ModuleTree `json:"trees,omitempty"` // Programs are the changed files that are compiled programs (issue 1577); a merge answers it, a pointer to an empty list when there are none, absent for a fetch that made no merge.
Programs *[]buildstep.Program `json:"programs,omitempty"`
Trees map[string]buildstep.ModuleTree `json:"trees,omitempty"`
} }
func (m *Module) fetchTool() StepTool { func (m *Module) fetchTool() StepTool {
@@ -127,6 +129,9 @@ func (m *Module) merge(ctx context.Context, res *FetchResult, tree, remote, base
} }
res.Base = mr.Base res.Base = mr.Base
res.Conflicts, res.Merge, res.Changed = mr.Conflicts, mr.Merge, mr.Changed res.Conflicts, res.Merge, res.Changed = mr.Conflicts, mr.Merge, mr.Changed
if mr.Merge != "" {
res.Programs = &mr.Programs
}
if res.Merge == "" { if res.Merge == "" {
return nil return nil
} }
+3 -1
View File
@@ -1,3 +1,5 @@
{ {
"capacity": "shanks=3,ace=2,g14=0,novox=0" "capacity": "shanks=3,ace=2,g14=0,novox=0",
"cores-per-step": 2,
"build-hours": ""
} }
+1 -1
View File
@@ -3,7 +3,7 @@ module meshbuildimage
go 1.26.0 go 1.26.0
require ( require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026142629-ff81bdf8cc8f git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026202658-805b4d9a3536
github.com/nats-io/nats-server/v2 v2.11.17 github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0 github.com/nats-io/nats.go v1.54.0
) )
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026142629-ff81bdf8cc8f h1:GYLCvz2x+JNUOTbXzyh2QwzX0sTzvRlXR7lGocWjk3A= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026202658-805b4d9a3536 h1:+SoDdDXF7Tgj38WP4kRpUSps6DJVl6P+w5u/KfdtwnQ=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026142629-ff81bdf8cc8f/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026202658-805b4d9a3536/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo= github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
+13 -1
View File
@@ -24,6 +24,18 @@
"description": "How many asks of the image build seat this module runs at once on each node, as node=asks pairs separated by commas, such as shanks=3,ace=2 (novox/hq ADR 0525 §5); a node set to 0 pulls nothing, a node not named runs one at a time. The control-node and the laptop build nothing.", "description": "How many asks of the image build seat this module runs at once on each node, as node=asks pairs separated by commas, such as shanks=3,ace=2 (novox/hq ADR 0525 §5); a node set to 0 pulls nothing, a node not named runs one at a time. The control-node and the laptop build nothing.",
"pattern": "|[a-z0-9_-]+=[0-9]+(,[a-z0-9_-]+=[0-9]+)*", "pattern": "|[a-z0-9_-]+=[0-9]+(,[a-z0-9_-]+=[0-9]+)*",
"default": "shanks=3,ace=2,g14=0,novox=0" "default": "shanks=3,ace=2,g14=0,novox=0"
},
"build-hours": {
"type": "string",
"description": "When each node may take this module's build asks, as node=HH:MM-HH:MM pairs in the node's local time, separated by commas, such as g14=18:00-06:00 (an end before its start crosses midnight). Outside its window a node takes no new step and finishes the ones it runs; a node not named, or an empty value, builds at any hour. The operator, 2026-10-26: g14 builds only outside office hours.",
"pattern": "|[a-z0-9_-]+=([01][0-9]|2[0-3]):[0-5][0-9]-([01][0-9]|2[0-3]):[0-5][0-9](,[a-z0-9_-]+=([01][0-9]|2[0-3]):[0-5][0-9]-([01][0-9]|2[0-3]):[0-5][0-9])*",
"default": ""
},
"cores-per-step": {
"type": "number",
"description": "How many processors one image step of this module needs (buildah, whose RUN lines are mostly fetches). The capacity above is an upper bound: on each node the module runs at most floor(the cores of its build slice's CPU quota / this) steps at once, the quota read live from its cgroup (issue 1806), so a lowered build CPU share never starves its steps. Above 0.",
"minimum": 0.1,
"default": 2
} }
}, },
"tools": [ "tools": [
@@ -78,7 +90,7 @@
"path": "${dir:state}/settings.json", "path": "${dir:state}/settings.json",
"mode": "0600", "mode": "0600",
"owner": "mesh-build-image", "owner": "mesh-build-image",
"takes-config": ["capacity"], "takes-config": ["capacity", "cores-per-step", "build-hours"],
"merge": "json", "merge": "json",
"template": "files/settings.json" "template": "files/settings.json"
}, },
@@ -48,9 +48,17 @@ func (m *Module) Puller(steps []StepTool, source buildstep.AskSource) *buildstep
// The capacity's own change wakes the puller (issue 1887): the runtime's watch of the config value as mesh-config // The capacity's own change wakes the puller (issue 1887): the runtime's watch of the config value as mesh-config
// resolved it; the settings file is read only when that watch cannot start. A test's Capacity is its own. // resolved it; the settings file is read only when that watch cannot start. A test's Capacity is its own.
var watch func(string, func(stdio.ProvisionChange) error) error var watch func(string, func(stdio.ProvisionChange) error) error
// And the CPU bounds it (issue 1806's follow-up): the configured capacity is an upper bound, the node's build slice
// gives floor(its quota's cores / the module's cores-per-step) at once, read live; a test judges by its own Capacity.
// And the node's build hours (the operator, 2026-10-26: g14 builds only from 18:00 to 06:00): outside them it takes
// nothing new.
var cores func() float64
var hours func() (*buildstep.BuildWindow, string)
if capacity == nil { if capacity == nil {
hours = buildstep.NodeBuildHours(m.Settings, m.Node, m.Say)
capacity = buildstep.NodeCapacity(m.Settings, buildstep.DefaultCapacities, m.Node, m.Say) capacity = buildstep.NodeCapacity(m.Settings, buildstep.DefaultCapacities, m.Node, m.Say)
watch = buildstep.WatchConfigServed watch = buildstep.WatchConfigServed
cores = buildstep.NodeCoresPerStep(m.Settings, buildstep.DefaultCoresPerStep[delivery.AskNode], m.Say)
} }
ask := m.askSteps(steps) ask := m.askSteps(steps)
var load func(string) ([]byte, error) var load func(string) ([]byte, error)
@@ -58,7 +66,7 @@ func (m *Module) Puller(steps []StepTool, source buildstep.AskSource) *buildstep
load = func(key string) ([]byte, error) { return m.ReadPauses.Get(key) } load = func(key string) ([]byte, error) { return m.ReadPauses.Get(key) }
} }
return &buildstep.Puller{Source: source, Language: delivery.AskNode, Node: m.Node, Put: m.Steps, PausePut: m.Pauses, Load: load, return &buildstep.Puller{Source: source, Language: delivery.AskNode, Node: m.Node, Put: m.Steps, PausePut: m.Pauses, Load: load,
Capacity: capacity, WatchConfig: watch, Logf: m.Say, Capacity: capacity, WatchConfig: watch, CoresPerStep: cores, BuildHours: hours, Logf: m.Say,
Run: func(ctx context.Context, a delivery.StepAsk) (any, error) { Run: func(ctx context.Context, a delivery.StepAsk) (any, error) {
if err := m.Prepare(ctx); err != nil { if err := m.Prepare(ctx); err != nil {
return nil, err return nil, err
@@ -3,6 +3,7 @@ package main
import ( import (
"context" "context"
"encoding/json" "encoding/json"
"fmt"
"os" "os"
"strings" "strings"
"testing" "testing"
@@ -83,12 +84,28 @@ func TestManifestTakesTheBuildCapacity(t *testing.T) {
raw, _ := os.ReadFile(moduleDir + "/module.json") raw, _ := os.ReadFile(moduleDir + "/module.json")
var doc struct { var doc struct {
Config map[string]struct { Config map[string]struct {
Default string `json:"default"` Default json.RawMessage `json:"default"`
} `json:"config"` } `json:"config"`
} }
_ = json.Unmarshal(raw, &doc) _ = json.Unmarshal(raw, &doc)
if doc.Config[buildstep.CapacitySetting].Default != buildstep.DefaultCapacities { // Issue 1806's follow-up: the cores a step needs is a setting of its own, in the settings file, with the SDK's default.
t.Errorf("the capacity's default = %q", doc.Config[buildstep.CapacitySetting].Default) if cores := string(doc.Config[buildstep.CoresPerStepSetting].Default); cores != fmt.Sprint(buildstep.DefaultCoresPerStep[delivery.AskNode]) {
t.Errorf("the cores-per-step default = %s, want %g", cores, buildstep.DefaultCoresPerStep[delivery.AskNode])
}
if got := buildstep.NodeCoresPerStep(moduleDir+"/files/settings.json", 0, nil)(); got != buildstep.DefaultCoresPerStep[delivery.AskNode] {
t.Errorf("the settings template holds cores-per-step %g", got)
}
// And the build hours (the operator, 2026-10-26): a setting of their own, empty by default (any hour), in the template.
if hours := string(doc.Config[buildstep.BuildHoursSetting].Default); hours != `""` {
t.Errorf("the build-hours default = %s, want empty", hours)
}
if w, fault := buildstep.NodeBuildHours(moduleDir+"/files/settings.json", "g14", t.Errorf)(); w != nil || fault != "" {
t.Errorf("the settings template holds build hours %v (%s)", w, fault)
}
var capacity string
_ = json.Unmarshal(doc.Config[buildstep.CapacitySetting].Default, &capacity)
if capacity != buildstep.DefaultCapacities {
t.Errorf("the capacity's default = %q", capacity)
} }
if resource(t, m, "settings")["path"] != "${dir:state}/settings.json" { if resource(t, m, "settings")["path"] != "${dir:state}/settings.json" {
t.Error("no settings file") t.Error("no settings file")
@@ -15,6 +15,8 @@ import (
func (m *Module) cpuLimits() buildcpu.Limits { func (m *Module) cpuLimits() buildcpu.Limits {
slots := 1 slots := 1
switch { switch {
case m.Held != nil:
slots = m.Held.CapacityNow() // the capacity the puller judges by, bounded by the node's build slice
case m.Capacity != nil: case m.Capacity != nil:
slots = m.Capacity() slots = m.Capacity()
case m.Settings != "": case m.Settings != "":
@@ -32,5 +34,9 @@ func (m *Module) threadEnv() []string {
// cpuStatus is what the status tool says of it: the limits a step is given, and what the kernel holds this runtime // cpuStatus is what the status tool says of it: the limits a step is given, and what the kernel holds this runtime
// to (its unit and its slice), read from its cgroup rather than assumed. // to (its unit and its slice), read from its cgroup rather than assumed.
func (m *Module) cpuStatus() map[string]any { func (m *Module) cpuStatus() map[string]any {
return map[string]any{"limits": m.cpuLimits(), "in-force": buildcpu.ReadInForce()} out := map[string]any{"limits": m.cpuLimits(), "in-force": buildcpu.ReadInForce()}
if m.Held != nil {
out["capacity"] = m.Held.CapacityDetail() // the capacity judged by and why (issue 1806's follow-up)
}
return out
} }
@@ -51,11 +51,13 @@ type FetchResult struct {
// commit, the merge commit of the head onto it, the paths the merge changes and the tree of each module directory. // commit, the merge commit of the head onto it, the paths the merge changes and the tree of each module directory.
// A head that does not merge cleanly is an answer: Merge is empty, Conflicts names the files, and the workspace is // A head that does not merge cleanly is an answer: Merge is empty, Conflicts names the files, and the workspace is
// left at the trunk commit with nothing installed. // left at the trunk commit with nothing installed.
Base string `json:"base,omitempty"` Base string `json:"base,omitempty"`
Merge string `json:"merge,omitempty"` Merge string `json:"merge,omitempty"`
Conflicts []string `json:"conflicts,omitempty"` Conflicts []string `json:"conflicts,omitempty"`
Changed []string `json:"changed,omitempty"` Changed []string `json:"changed,omitempty"`
Trees map[string]buildstep.ModuleTree `json:"trees,omitempty"` // Programs are the changed files that are compiled programs (issue 1577); a merge answers it, a pointer to an empty list when there are none, absent for a fetch that made no merge.
Programs *[]buildstep.Program `json:"programs,omitempty"`
Trees map[string]buildstep.ModuleTree `json:"trees,omitempty"`
// Group is the group's package the fetch packed and served to the TypeScript members (the call's `package`): // Group is the group's package the fetch packed and served to the TypeScript members (the call's `package`):
// "<name>@<version> <integrity>". // "<name>@<version> <integrity>".
Group string `json:"group,omitempty"` Group string `json:"group,omitempty"`
@@ -27,6 +27,9 @@ func (m *Module) merge(ctx context.Context, run forgekey.Runner, res *FetchResul
} }
res.Base = mr.Base res.Base = mr.Base
res.Conflicts, res.Merge, res.Changed = mr.Conflicts, mr.Merge, mr.Changed res.Conflicts, res.Merge, res.Changed = mr.Conflicts, mr.Merge, mr.Changed
if mr.Merge != "" {
res.Programs = &mr.Programs
}
if res.Merge == "" { if res.Merge == "" {
return nil return nil
} }
+3 -1
View File
@@ -1,3 +1,5 @@
{ {
"capacity": "shanks=3,ace=2,g14=0,novox=0" "capacity": "shanks=3,ace=2,g14=0,novox=0",
"cores-per-step": 2,
"build-hours": ""
} }
+1 -1
View File
@@ -3,7 +3,7 @@ module meshbuildnode
go 1.26.0 go 1.26.0
require ( require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026142629-ff81bdf8cc8f git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026202658-805b4d9a3536
github.com/distribution/distribution/v3 v3.1.2 github.com/distribution/distribution/v3 v3.1.2
github.com/nats-io/nats.go v1.54.0 github.com/nats-io/nats.go v1.54.0
) )
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026142629-ff81bdf8cc8f h1:GYLCvz2x+JNUOTbXzyh2QwzX0sTzvRlXR7lGocWjk3A= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026202658-805b4d9a3536 h1:+SoDdDXF7Tgj38WP4kRpUSps6DJVl6P+w5u/KfdtwnQ=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026142629-ff81bdf8cc8f/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026202658-805b4d9a3536/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/alicebob/miniredis/v2 v2.35.0 h1:QwLphYqCEAo1eu1TqPRN2jgVMPBweeQcR21jeqDCONI= github.com/alicebob/miniredis/v2 v2.35.0 h1:QwLphYqCEAo1eu1TqPRN2jgVMPBweeQcR21jeqDCONI=
github.com/alicebob/miniredis/v2 v2.35.0/go.mod h1:TcL7YfarKPGDAthEtl5NBeHZfeUQj6OXMm/+iu5cLMM= github.com/alicebob/miniredis/v2 v2.35.0/go.mod h1:TcL7YfarKPGDAthEtl5NBeHZfeUQj6OXMm/+iu5cLMM=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
+13 -1
View File
@@ -25,6 +25,18 @@
"description": "How many asks of the Node build seat this module runs at once on each node, as node=asks pairs separated by commas, such as shanks=3,ace=2 (novox/hq ADR 0525 §5); a node set to 0 pulls nothing, a node not named runs one at a time. The control-node and the laptop build nothing.", "description": "How many asks of the Node build seat this module runs at once on each node, as node=asks pairs separated by commas, such as shanks=3,ace=2 (novox/hq ADR 0525 §5); a node set to 0 pulls nothing, a node not named runs one at a time. The control-node and the laptop build nothing.",
"pattern": "|[a-z0-9_-]+=[0-9]+(,[a-z0-9_-]+=[0-9]+)*", "pattern": "|[a-z0-9_-]+=[0-9]+(,[a-z0-9_-]+=[0-9]+)*",
"default": "shanks=3,ace=2,g14=0,novox=0" "default": "shanks=3,ace=2,g14=0,novox=0"
},
"build-hours": {
"type": "string",
"description": "When each node may take this module's build asks, as node=HH:MM-HH:MM pairs in the node's local time, separated by commas, such as g14=18:00-06:00 (an end before its start crosses midnight). Outside its window a node takes no new step and finishes the ones it runs; a node not named, or an empty value, builds at any hour. The operator, 2026-10-26: g14 builds only outside office hours.",
"pattern": "|[a-z0-9_-]+=([01][0-9]|2[0-3]):[0-5][0-9]-([01][0-9]|2[0-3]):[0-5][0-9](,[a-z0-9_-]+=([01][0-9]|2[0-3]):[0-5][0-9]-([01][0-9]|2[0-3]):[0-5][0-9])*",
"default": ""
},
"cores-per-step": {
"type": "number",
"description": "How many processors one Node step of this module needs (tsc, esbuild and a test runner; a median step takes 15 s). The capacity above is an upper bound: on each node the module runs at most floor(the cores of its build slice's CPU quota / this) steps at once, the quota read live from its cgroup (issue 1806), so a lowered build CPU share never starves its steps. Above 0.",
"minimum": 0.1,
"default": 2
} }
}, },
"tools": [ "tools": [
@@ -100,7 +112,7 @@
"path": "${dir:state}/settings.json", "path": "${dir:state}/settings.json",
"mode": "0600", "mode": "0600",
"owner": "mesh-build-node", "owner": "mesh-build-node",
"takes-config": ["capacity"], "takes-config": ["capacity", "cores-per-step", "build-hours"],
"merge": "json", "merge": "json",
"template": "files/settings.json" "template": "files/settings.json"
}, },
+1 -1
View File
@@ -115,7 +115,7 @@ func resolving(c *Config) {
} }
return nil return nil
} }
for _, read := range []string{AssignmentsRead, PlacementRead, DeclaredConfigRead, PredefinedRead} { for _, read := range []string{AssignmentsRead, DeclaredConfigRead, PredefinedRead} {
go func(read string) { go func(read string) {
for wait := 2 * time.Second; ; wait = min(wait*2, time.Minute) { for wait := 2 * time.Second; ; wait = min(wait*2, time.Minute) {
time.Sleep(wait) time.Sleep(wait)
@@ -21,16 +21,4 @@ func TestAModulePulledInHasItsValuesResolvedOnItsNode(t *testing.T) {
if r, _ := c.Reconcile(); len(r.Moved) != 4 || !r.Moved[0].Gone { if r, _ := c.Reconcile(); len(r.Moved) != 4 || !r.Moved[0].Gone {
t.Errorf("a module no longer pulled in moved %+v", r.Moved) t.Errorf("a module no longer pulled in moved %+v", r.Moved)
} }
// 1126 (ADR 0549 §3, §5): a module the catalogue's placement names on a node resolves there as an assigned one does,
// while the two sources stand; it goes when neither names it.
m.mirror(t, PlacementRead, "placement.ace", `{"v":1,"node":"ace","modules":[{"module":"dunst"}]}`)
if r, err := c.Reconcile(); err != nil || len(r.Moved) != 4 {
t.Fatalf("a placed module's values moved %+v (%v)", r.Moved, err)
}
expect(t, m, "ace", "10", SourceDefault)
_ = m.Delete(PlacementRead, "placement.ace")
if r, _ := c.Reconcile(); len(r.Moved) != 4 || !r.Moved[0].Gone {
t.Errorf("a module no longer placed moved %+v", r.Moved)
}
} }
@@ -20,10 +20,7 @@ const (
PredefinedState = "predefined" PredefinedState = "predefined"
ProposalsState = "proposals" ProposalsState = "proposals"
AssignmentsRead = "mesh-controller.assignments" AssignmentsRead = "mesh-controller.assignments"
// PlacementRead is the indexer's placement, read from the catalogue at each merge (novox/hq ADR 0549 §2, §3): each
// node's key `placement.<node>` with the modules placed there.
PlacementRead = "mesh-delivery-indexer.placement"
DeclaredConfigRead = "mesh-controller.declared-config" DeclaredConfigRead = "mesh-controller.declared-config"
PredefinedRead = "mesh-controller.predefined" PredefinedRead = "mesh-controller.predefined"
@@ -27,38 +27,6 @@ func (c *Config) sources() (Sources, error) {
}); err != nil { }); err != nil {
return s, fmt.Errorf("the assignments cannot be read: %w", err) return s, fmt.Errorf("the assignments cannot be read: %w", err)
} }
// And every module the catalogue's placement names on a node (ADR 0549 §3): until ADR 0547's third milestone both
// sources stand (ADR 0549 §5), so a pair either names is resolved, and none loses its values while the two differ.
// Placement that cannot be read yet (the indexer's state not on the bus) leaves the assignments alone, said.
standing := map[string]bool{}
for _, a := range s.Assignments {
standing[a.Node+"."+a.Module] = true
}
if err := c.each(PlacementRead, func(key string, raw []byte) error {
node, ok := strings.CutPrefix(key, "placement.")
if !ok {
return nil
}
var p struct {
Node string `json:"node"`
Modules []struct {
Module string `json:"module"`
} `json:"modules"`
}
if err := json.Unmarshal(raw, &p); err != nil || p.Node != node {
problems = append(problems, fmt.Sprintf("the placement %s is not one", key))
return nil
}
for _, m := range p.Modules {
if m.Module != "" && !standing[node+"."+m.Module] {
standing[node+"."+m.Module] = true
s.Assignments = append(s.Assignments, Assignment{Node: node, Module: m.Module})
}
}
return nil
}); err != nil {
problems = append(problems, fmt.Sprintf("the catalogue's placement cannot be read, so the assignments alone are resolved: %v", err))
}
if err := c.each(DeclaredConfigRead, func(key string, raw []byte) error { if err := c.each(DeclaredConfigRead, func(key string, raw []byte) error {
m, err := ParseModuleDeclarations(raw) m, err := ParseModuleDeclarations(raw)
if err != nil { if err != nil {
-1
View File
@@ -29,7 +29,6 @@
], ],
"reads": [ "reads": [
"mesh-controller.assignments", "mesh-controller.assignments",
"mesh-delivery-indexer.placement",
"mesh-controller.declared-config", "mesh-controller.declared-config",
"mesh-controller.predefined" "mesh-controller.predefined"
], ],
@@ -2,6 +2,7 @@ package main
import ( import (
"reflect" "reflect"
"strings"
"testing" "testing"
"time" "time"
) )
@@ -26,6 +27,22 @@ func TestListSaysWhenEachIssueMovedAndKeepsTheLinkedOnes(t *testing.T) {
if _, err := h.Link(LinkAsk{Number: quiet, Kind: LinkDesign, Target: "03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md"}); err != nil { if _, err := h.Link(LinkAsk{Number: quiet, Kind: LinkDesign, Target: "03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md"}); err != nil {
t.Fatal(err) t.Fatal(err)
} }
// Issue 1781: a milestone or a commit link places the issue and moves nothing, nor keeps its worker current.
if _, err := h.milestoneSet("lead@shanks", MilestoneSetAsk{Design: 47, ID: "R6", Outcome: "x"}); err != nil {
t.Fatal(err)
}
h.fixing(t, noted)
worked, _ := h.Get(noted)
clock = clock.Add(time.Hour)
for _, l := range []LinkAsk{{Number: noted, Kind: LinkMilestone, Target: "47/R6"},
{Number: noted, Kind: LinkCommit, Target: "mesh-catalog@" + strings.Repeat("a", 40)}} {
if _, err := h.link("seed@shanks", l); err != nil {
t.Fatal(err)
}
}
if after, _ := h.Get(noted); !after.Worker.Active.Equal(worked.Worker.Active) {
t.Fatalf("a milestone link kept the worker current: %v, was %v", after.Worker.Active, worked.Worker.Active)
}
start := time.Date(2026, 10, 26, 12, 0, 0, 0, time.UTC) start := time.Date(2026, 10, 26, 12, 0, 0, 0, time.UTC)
for _, c := range []struct { for _, c := range []struct {
name string name string
@@ -35,7 +52,7 @@ func TestListSaysWhenEachIssueMovedAndKeepsTheLinkedOnes(t *testing.T) {
refuse bool refuse bool
}{ }{
{"numbers say moved", ListAsk{Numbers: []uint64{quiet, noted, linked}}, []uint64{quiet, noted, linked}, {"numbers say moved", ListAsk{Numbers: []uint64{quiet, noted, linked}}, []uint64{quiet, noted, linked},
[]time.Time{start.Add(2 * time.Hour), start.Add(time.Hour), start.Add(2 * time.Hour)}, false}, []time.Time{start.Add(2 * time.Hour), start.Add(2 * time.Hour), start.Add(2 * time.Hour)}, false},
{"linked to a decision", ListAsk{Linked: "decision:0493"}, []uint64{linked}, nil, false}, {"linked to a decision", ListAsk{Linked: "decision:0493"}, []uint64{linked}, nil, false},
{"linked to a design", ListAsk{Linked: " design:03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md"}, []uint64{quiet}, nil, false}, {"linked to a design", ListAsk{Linked: " design:03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md"}, []uint64{quiet}, nil, false},
{"linked and numbers together", ListAsk{Linked: "decision:0493", Numbers: []uint64{quiet, noted}}, nil, nil, false}, {"linked and numbers together", ListAsk{Linked: "decision:0493", Numbers: []uint64{quiet, noted}}, nil, nil, false},
+14 -4
View File
@@ -675,7 +675,9 @@ func (t *Tracker) link(by string, a LinkAsk) (*Linked, error) {
return fmt.Errorf("issue %d already has the %s link to %s", is.Number, kind, target) return fmt.Errorf("issue %d already has the %s link to %s", is.Number, kind, target)
} }
is.Links = append(is.Links[:i:i], is.Links[i+1:]...) is.Links = append(is.Links[:i:i], is.Links[i+1:]...)
touch(is, by, at) if !placesOnly(kind) {
touch(is, by, at)
}
return nil return nil
} }
} }
@@ -692,7 +694,9 @@ func (t *Tracker) link(by string, a LinkAsk) (*Linked, error) {
return fmt.Errorf("issue %d has %d links, the most one issue keeps", is.Number, MostLinks) return fmt.Errorf("issue %d has %d links, the most one issue keeps", is.Number, MostLinks)
} }
is.Links = append(is.Links, Link{Kind: kind, Target: target, By: by, At: at}) is.Links = append(is.Links, Link{Kind: kind, Target: target, By: by, At: at})
touch(is, by, at) if !placesOnly(kind) {
touch(is, by, at)
}
return nil return nil
}) })
if err != nil { if err != nil {
@@ -768,7 +772,7 @@ type Row struct {
} }
// lastMoved is when an issue last changed, as its own record says: the latest of its opening, its closing, its // lastMoved is when an issue last changed, as its own record says: the latest of its opening, its closing, its
// notes, its links, its worker's start and last act, its import and what the delivery last said of each fix. // notes, its links but those that only place it, its worker's start and last act, its import and what the delivery last said of each fix.
func lastMoved(is *Issue) time.Time { func lastMoved(is *Issue) time.Time {
at := is.Opened at := is.Opened
later := func(t time.Time) { later := func(t time.Time) {
@@ -783,7 +787,9 @@ func lastMoved(is *Issue) time.Time {
later(n.At) later(n.At)
} }
for _, l := range is.Links { for _, l := range is.Links {
later(l.At) if !placesOnly(l.Kind) {
later(l.At)
}
} }
if w := is.Worker; w != nil { if w := is.Worker; w != nil {
later(w.Since) later(w.Since)
@@ -798,6 +804,10 @@ func lastMoved(is *Issue) time.Time {
return at return at
} }
// placesOnly says a link of this kind only places the issue (its milestone) or names a commit that fixed it: no work
// on it, so it neither moves the issue nor keeps its worker current (issue 1781: the seed linked 188 issues at once).
func placesOnly(kind string) bool { return kind == LinkMilestone || kind == LinkCommit }
// checkLinked is a list's linked filter, kind:target, said the one way a link of that kind is kept. // checkLinked is a list's linked filter, kind:target, said the one way a link of that kind is kept.
func checkLinked(s string) (kind, target string, err error) { func checkLinked(s string) (kind, target string, err error) {
k, t, ok := strings.Cut(strings.TrimSpace(s), ":") k, t, ok := strings.Cut(strings.TrimSpace(s), ":")
@@ -168,6 +168,8 @@ func (k *KB) Progress(a ProgressAsk) (*Progress, error) {
if err != nil { if err != nil {
return nil, err return nil, err
} }
// A cut takes the warnings from the last: the calls that failed are said before the answers that did not parse.
callsFirst(p.Warnings)
fitProgress(p, boundOf(a.Bound)) fitProgress(p, boundOf(a.Bound))
return p, nil return p, nil
} }
@@ -835,8 +837,21 @@ func movedSay(t *time.Time) string {
return " · moved " + when(t) return " · moved " + when(t)
} }
// callsFirst puts the warnings of calls that failed before those of answers that did not parse, in place.
func callsFirst(w []string) {
sort.SliceStable(w, func(i, j int) bool {
return !strings.Contains(w[i], " answered something else") && strings.Contains(w[j], " answered something else")
})
}
// fitProgress keeps an answer within its bound: the items go first, then the tree's last lines, saying what was cut. // fitProgress keeps an answer within its bound: the items go first, then the tree's last lines, saying what was cut.
// A minimal answer never carries the items: its tree says the same, one line per milestone, so leaving them out is how
// the minimal form is made (the items of design 47 alone passed 8192 bytes) and no cut. A detailed answer's items, and
// everything else that goes, are said in cut.
func fitProgress(p *Progress, bound int) { func fitProgress(p *Progress, bound int) {
if p.Detail == DetailMinimal && p.Tree != "" {
p.Items = nil
}
if size(p) <= bound { if size(p) <= bound {
return return
} }
@@ -306,12 +306,23 @@ func TestProgressOfADesignMinimalAndDetailed(t *testing.T) {
if len(raw) > DefaultBound || p.Detail != DetailMinimal || !strings.Contains(p.Source, "the issue tracker's milestone records of design 47") { if len(raw) > DefaultBound || p.Detail != DetailMinimal || !strings.Contains(p.Source, "the issue tracker's milestone records of design 47") {
t.Fatalf("%d bytes, %s, %s", len(raw), p.Detail, p.Source) t.Fatalf("%d bytes, %s, %s", len(raw), p.Detail, p.Source)
} }
if p.Cut != "" || len(p.Items) != 0 {
t.Errorf("a minimal answer leaves the items to its tree and is not a cut: cut %q, %d items", p.Cut, len(p.Items))
}
warns := []string{"mesh-delivery.show a answered something else: x", "issue-tracker.get 1: refused", "the tracker did not name 2", "issue-tracker.get 3 answered something else: y"}
callsFirst(warns)
if strings.Join(warns, "|") != "issue-tracker.get 1: refused|the tracker did not name 2|mesh-delivery.show a answered something else: x|issue-tracker.get 3 answered something else: y" {
t.Errorf("call failures do not come first: %q", warns)
}
if seats.asked["seat:issue-tracker.list"] != 1 || seats.asked["seat:mesh-delivery.deliveries"] != 0 || seats.asked["seat:issue-tracker.get"] != 0 { if seats.asked["seat:issue-tracker.list"] != 1 || seats.asked["seat:mesh-delivery.deliveries"] != 0 || seats.asked["seat:issue-tracker.get"] != 0 {
t.Fatalf("minimal asked %v", seats.asked) t.Fatalf("minimal asked %v", seats.asked)
} }
if w := strings.Join(p.Warnings, "\n"); !strings.Contains(w, "the tracker neither answered nor named missing") || !strings.Contains(w, "973, 525") { if w := strings.Join(p.Warnings, "\n"); !strings.Contains(w, "the tracker neither answered nor named missing") || !strings.Contains(w, "973, 525") {
t.Errorf("a row the tracker did not answer is not said: %v", p.Warnings) t.Errorf("a row the tracker did not answer is not said: %v", p.Warnings)
} }
if big, _ := k.Progress(ProgressAsk{Of: "design 47", Bound: 8192}); big.Cut != "" || big.Tree != p.Tree {
t.Errorf("a minimal answer at bound 8192 is cut: %q", big.Cut)
}
lines := strings.Split(p.Tree, "\n") lines := strings.Split(p.Tree, "\n")
if !strings.HasPrefix(lines[0], "✗ design 47 · Delivery: from a commit to delivered — ") { if !strings.HasPrefix(lines[0], "✗ design 47 · Delivery: from a commit to delivered — ") {
t.Fatalf("first line %q", lines[0]) t.Fatalf("first line %q", lines[0])
@@ -5,8 +5,6 @@ package main
// key (or a first one) raises `secret-changed` for the machine's core secrets, the same key says nothing. // key (or a first one) raises `secret-changed` for the machine's core secrets, the same key says nothing.
import ( import (
"git.novox.be/novox/mesh-sdk/go/seal"
"encoding/json" "encoding/json"
"os" "os"
"path/filepath" "path/filepath"
@@ -102,17 +100,14 @@ authorization {
users = [ users = [
{ user: "novox.mesh-vault", password: "pw", permissions: { { user: "novox.mesh-vault", password: "pw", permissions: {
publish: { deny: [">"] } publish: { deny: [">"] }
subscribe: { allow: ["mesh.seat.mesh-vault.call.node-key.*", "mesh.seat.mesh-vault.call.own.*"] } subscribe: { allow: ["mesh.seat.mesh-vault.call.node-key.*"] }
allow_responses: true } } allow_responses: true } }
{ user: "node.ace", password: "pw", permissions: { { user: "node.ace", password: "pw", permissions: {
publish: { allow: ["mesh.seat.mesh-vault.call.node-key.node_ace", "mesh.seat.mesh-vault.call.own.node_ace"] } publish: { allow: ["mesh.seat.mesh-vault.call.node-key.node_ace"] }
subscribe: { allow: ["_INBOX.node.ace.>"] } } } subscribe: { allow: ["_INBOX.node.ace.>"] } } }
{ user: "node.g14", password: "pw", permissions: { { user: "node.g14", password: "pw", permissions: {
publish: { allow: ["mesh.seat.mesh-vault.call.node-key.node_g14", "mesh.seat.mesh-vault.call.own.node_g14"] } publish: { allow: ["mesh.seat.mesh-vault.call.node-key.node_g14"] }
subscribe: { allow: ["_INBOX.node.g14.>"] } } } subscribe: { allow: ["_INBOX.node.g14.>"] } } }
{ user: "controller", password: "pw", permissions: {
publish: { allow: ["mesh.seat.mesh-vault.call.own.controller"] }
subscribe: { allow: ["_INBOX.controller.>"] } } }
] ]
} }
` `
@@ -140,48 +135,39 @@ func TestOnARealBusANodeEngineStatesOnlyItsOwnKey(t *testing.T) {
t.Cleanup(s.Shutdown) t.Cleanup(s.Shutdown)
v := newVault(t, nil) v := newVault(t, nil)
v.Askers = Declared{Read: assignments{"ace.web": `{"node":"ace","module":"web"}`}.read, Store: v.Store} tool := toolNamed(t, definitions(&Env{Vault: v.Vault}), Seat+".node-key")
tools := map[string]toolDef{"node-key": toolNamed(t, definitions(&Env{Vault: v.Vault}), Seat+".node-key"),
"own": toolNamed(t, definitions(&Env{Vault: v.Vault}), Seat+".own")}
vaultConn := connectAs(t, s.ClientURL(), "novox.mesh-vault") vaultConn := connectAs(t, s.ClientURL(), "novox.mesh-vault")
serve := func(m *nats.Msg) { if _, err := vaultConn.Subscribe("mesh.seat.mesh-vault.call.node-key.*", func(m *nats.Msg) {
parts := strings.Split(m.Subject, ".") parts := strings.Split(m.Subject, ".")
var args map[string]any var args map[string]any
_ = json.Unmarshal(m.Data, &args) _ = json.Unmarshal(m.Data, &args)
answer := busAnswer{} answer := busAnswer{}
if got, err := tools[parts[len(parts)-2]].runAs(untoken(parts[len(parts)-1]), args); err != nil { if got, err := tool.runAs(untoken(parts[len(parts)-1]), args); err != nil {
answer.Error = err.Error() answer.Error = err.Error()
} else { } else {
answer.Result, _ = json.Marshal(got) answer.Result, _ = json.Marshal(got)
} }
raw, _ := json.Marshal(answer) raw, _ := json.Marshal(answer)
_ = m.Respond(raw) _ = m.Respond(raw)
} }); err != nil {
for _, verb := range []string{"node-key", "own"} { t.Fatal(err)
if _, err := vaultConn.Subscribe("mesh.seat.mesh-vault.call."+verb+".*", serve); err != nil {
t.Fatal(err)
}
} }
_ = vaultConn.Flush() _ = vaultConn.Flush()
call := func(user, subject string, args any) (json.RawMessage, error) { say := func(user, subject, key string) (string, error) {
raw, _ := json.Marshal(args) raw, _ := json.Marshal(map[string]string{"key": key})
msg, err := connectAs(t, s.ClientURL(), user).Request(subject, raw, 3*time.Second) msg, err := connectAs(t, s.ClientURL(), user).Request(subject, raw, 3*time.Second)
if err != nil { if err != nil {
return nil, err return "", err
} }
var a busAnswer var a busAnswer
_ = json.Unmarshal(msg.Data, &a) _ = json.Unmarshal(msg.Data, &a)
if a.Error != "" { if a.Error != "" {
return nil, errString(a.Error) return a.Error, errString(a.Error)
} }
return a.Result, nil return "", nil
} }
say := func(user, subject, key string) (string, error) { aceKey, _ := aSealingKey(t)
_, err := call(user, subject, map[string]string{"key": key})
return "", err
}
aceKey, acePriv := aSealingKey(t)
g14Key, _ := aSealingKey(t) g14Key, _ := aSealingKey(t)
if _, err := say("node.ace", "mesh.seat.mesh-vault.call.node-key.node_ace", aceKey); err != nil { if _, err := say("node.ace", "mesh.seat.mesh-vault.call.node-key.node_ace", aceKey); err != nil {
t.Fatal(err) t.Fatal(err)
@@ -202,32 +188,4 @@ func TestOnARealBusANodeEngineStatesOnlyItsOwnKey(t *testing.T) {
if got, held, _ := v.Store.NodeKey("g14"); !held || got != g14Key { if got, held, _ := v.Store.NodeKey("g14"); !held || got != g14Key {
t.Errorf("g14's key is %q, %v", got, held) t.Errorf("g14's key is %q, %v", got, held)
} }
// Issue 1126 (ADR 0547 §2, ADR 0550 §5): `own` on the same bus. node.ace asks its own and gets its module's secret
// sealed to the key it stated, opening with its key alone; g14 cannot ask in ace's name, and asking in its own gets
// none of ace's; the controller is refused, as no node-engine.
if _, err := moveIn(t, v, "node.ace", MoveOf{Node: "ace", Module: "web", Name: "password", Origin: "made", Generation: "g-1",
SealingKey: aceKey}, "ace-web-pw"); err != nil {
t.Fatal(err)
}
raw, err := call("node.ace", "mesh.seat.mesh-vault.call.own.node_ace", map[string]any{})
var own OwnAnswer
if err != nil || json.Unmarshal(raw, &own) != nil || own.Node != "ace" || len(own.Own) != 1 {
t.Fatalf("ace's own over the bus: %s, %v", raw, err)
}
if value, err := seal.Open(own.Own[0].Sealed, acePriv); err != nil || string(value) != "ace-web-pw" {
t.Errorf("ace's secret does not open with ace's key: %q, %v", value, err)
}
if strings.Contains(string(raw), "ace-web-pw") {
t.Error("ace's secret crossed the bus in the clear")
}
if _, err := call("node.g14", "mesh.seat.mesh-vault.call.own.node_ace", map[string]any{}); err == nil {
t.Error("node.g14 asked ace's own in ace's name")
}
if raw, err := call("node.g14", "mesh.seat.mesh-vault.call.own.node_g14", map[string]any{}); err != nil || strings.Contains(string(raw), "web") {
t.Errorf("g14 asking its own was answered %s, %v", raw, err)
}
if _, err := call("controller", "mesh.seat.mesh-vault.call.own.controller", map[string]any{}); err == nil || !strings.Contains(err.Error(), "not a node-engine") {
t.Errorf("the controller asking own: %v", err)
}
} }
@@ -1,288 +0,0 @@
package main
// A machine's node-engine composes its own modules (novox/hq ADR 0547 §2, issue 1126), and asks the vault for their own
// secrets sealed to its own key (ADR 0349 §11, ADR 0550 §5): the verb `own`, answered to the node-engine of a machine
// alone, about that machine alone (the machine is the caller's, as the bus proved it, never an argument). Each value is
// sealed to the sealing key the machine's engine stated (`node-key`), so what crosses the bus opens on that machine only,
// and the composition carries it sealed (no composition carries a secret in the clear). A module handed to the
// provisioner is left out: its client fills its secrets from `get` (ADR 0349 §17).
import (
"encoding/json"
"errors"
"fmt"
"sort"
"strings"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/seal"
)
// NodeOwnSecret is one own secret of a module on the asking machine, sealed to the machine's key.
type NodeOwnSecret struct {
Module string `json:"module"`
Name string `json:"name"`
Origin string `json:"origin"`
Generation int `json:"generation"`
Sealed seal.Box `json:"sealed"`
}
// OwnAnswer is what `own` answers: the machine, the sealing key every value is sealed to (raw X25519, base64, as its
// engine stated it), each own secret of its modules, and each pair credential it asked for that the vault holds.
type OwnAnswer struct {
Node string `json:"node"`
Key string `json:"key"`
Own []NodeOwnSecret `json:"own"`
Credentials []NodeCredential `json:"credentials,omitempty"`
Missing []CredentialAsked `json:"missing,omitempty"`
}
// CredentialAsked names one pair credential by its ends, as a node composing its own asks for it (mesh-sdk
// compose.CredentialKey): Role is `consumer` when the asking machine consumes it, `provider` when it grants it.
type CredentialAsked struct {
Role string `json:"role"`
Provision string `json:"provision"`
Consumer string `json:"consumer"`
ConsumerModule string `json:"consumer_module"`
Provider string `json:"provider"`
Local string `json:"local,omitempty"`
}
// NodeCredential is one pair credential answered, sealed to the asking machine's key, with the name the vault keeps it
// under (`pair/...` per pair, `provision/<provider>/<provision>/<node>/<module>` issued for one holder, or the provision's
// one shared value `provision/<provider>/<provision>`).
type NodeCredential struct {
CredentialAsked
Name string `json:"name"`
Generation int `json:"generation"`
Sealed seal.Box `json:"sealed"`
}
// credentialNames are the names a pair credential may be kept under, the most particular first.
func credentialNames(c CredentialAsked) []string {
pair := "pair/" + c.Provider + "/" + c.Provision + "/" + c.Consumer + "/" + c.ConsumerModule
if c.Local != "" {
pair += "/" + c.Local
}
return []string{pair, "provision/" + c.Provider + "/" + c.Provision + "/" + c.Consumer + "/" + c.ConsumerModule,
"provision/" + c.Provider + "/" + c.Provision}
}
// credentialRefusal says why node may not have c, or "": a consumer's credential to the machine its consuming module
// stands on, a provider's to the providing machine, and nothing to a machine that is neither.
func (v *Vault) credentialRefusal(node string, c CredentialAsked) string {
for _, part := range []string{c.Provision, c.Consumer, c.ConsumerModule, c.Provider} {
if !namePart.MatchString(part) {
return fmt.Sprintf("%q is not a name the mesh writes", part)
}
}
if c.Local != "" && !namePart.MatchString(c.Local) {
return fmt.Sprintf("%q is not a name the mesh writes", c.Local)
}
switch c.Role {
case "consumer":
if c.Consumer != node {
return fmt.Sprintf("%s asks as a consumer on %s, and is not that machine", node, c.Consumer)
}
// The consuming module stands on the machine: read, never assumed (an assignment that cannot be read refuses).
read, why := v.assignmentReader(node)
if why != "" {
return why
}
raw, err := read(node + "." + c.ConsumerModule)
if err != nil {
return fmt.Sprintf(notYetPrefix+"the assignment %s.%s could not be read: %v", node, c.ConsumerModule, err)
}
if raw == nil {
return fmt.Sprintf("%s is not on %s: neither assigned there nor required by a module there", c.ConsumerModule, node)
}
return ""
case "provider":
if c.Provider != node {
return fmt.Sprintf("%s asks as the provider %s, and is not that machine", node, c.Provider)
}
return ""
}
return fmt.Sprintf("the role %q is not consumer or provider", c.Role)
}
// assignmentReader is the vault's reader of the assignments, or why it has none: the rule `own` keeps reads them,
// and without them it refuses rather than hands (a rule that cannot be read is no rule).
func (v *Vault) assignmentReader(node string) (AssignmentReader, string) {
d, ok := v.Askers.(Declared)
if !ok || d.Read == nil {
return nil, "the vault cannot read the assignments, so it cannot tell what " + node + "'s modules are; nothing is handed"
}
return standing(d.Read), ""
}
// sharedRefusal says why node may not have the provision's one shared value `provision/<provider>/<provision>` for c,
// or "": the vault's record of who holds it names the consumer's module among its consumers, or the machine as its
// provider. Without that record nothing is handed: the shared value is not one any machine may have by naming a
// provider (ADR 0550 §5).
func (v *Vault) sharedRefusal(node string, c CredentialAsked) (string, error) {
name := "provision/" + c.Provider + "/" + c.Provision
sh, ok, err := v.Store.ReadShared(name)
if err != nil {
return "", err
}
if !ok {
return fmt.Sprintf("the vault keeps no record of who holds %s", name), nil
}
switch c.Role {
case "consumer":
want := consumerOf(c.Consumer, c.ConsumerModule, c.Local)
for _, held := range sh.Consumers {
if held == want {
return "", nil
}
}
return fmt.Sprintf("%s is held by %s, not by %s", name, strings.Join(sh.Consumers, ", "), want), nil
case "provider":
if sh.Provider == node {
return "", nil
}
return fmt.Sprintf("%s is provided from %s, not %s", name, sh.Provider, node), nil
}
return fmt.Sprintf("the role %q is not consumer or provider", c.Role), nil
}
// composeOrigin is the vault's origin of a value as the composition reads it: made by the mesh, or given (accepted at a
// terminal, typed at a desk, issued by a provider's software).
func composeOrigin(origin string) string {
if origin == "" || origin == OriginMade {
return OriginMade
}
return OriginAccepted
}
// OwnOfNode answers the node-engine of a machine the own secrets of its modules, each sealed to the machine's stated key.
func (v *Vault) OwnOfNode(caller string, asked ...CredentialAsked) (OwnAnswer, error) {
node, ok := nodeEngineOf(caller)
if !ok {
if caller == "" || caller == stdio.Unproven {
caller = "an unproven caller"
}
return OwnAnswer{}, fmt.Errorf("own answers a machine's node-engine alone, about its own machine; %s is not a node-engine", caller)
}
key, had, err := v.Store.NodeKey(node)
if err != nil {
return OwnAnswer{}, err
}
if !had {
return OwnAnswer{}, fmt.Errorf("the vault holds no secret named own/%s yet: %s's node-engine has stated no sealing key "+
"(node-key), so there is no key of its own to seal to", node, node)
}
to, err := publicPEM(key)
if err != nil {
return OwnAnswer{}, fmt.Errorf("%s's sealing key as recorded is not usable: %w", node, err)
}
all, err := v.Store.List()
if err != nil {
return OwnAnswer{}, err
}
out := OwnAnswer{Node: node, Key: key}
for _, r := range all {
parts := strings.Split(r.Name, "/")
if len(parts) != 4 || parts[0] != "own" || parts[1] != node {
continue
}
handed, err := v.handed(node, parts[2])
if err != nil {
return OwnAnswer{}, err
}
if handed {
continue
}
value, err := seal.Open(r.AtRest, v.Own.PrivateKey)
if err != nil {
return OwnAnswer{}, fmt.Errorf("%s does not open with the vault's key: %w", r.Name, err)
}
sealed, err := seal.Seal(value, to)
if err != nil {
return OwnAnswer{}, err
}
if err := v.recordHanding(Handing{To: caller, Name: r.Name, Generation: r.Generation, Outcome: "handed"}); err != nil {
return OwnAnswer{}, fmt.Errorf("%s is not handed out: who was handed it could not be recorded: %w", r.Name, err)
}
out.Own = append(out.Own, NodeOwnSecret{Module: parts[2], Name: parts[3], Origin: composeOrigin(r.Origin),
Generation: r.Generation, Sealed: sealed})
}
sort.Slice(out.Own, func(i, j int) bool {
if out.Own[i].Module != out.Own[j].Module {
return out.Own[i].Module < out.Own[j].Module
}
return out.Own[i].Name < out.Own[j].Name
})
for _, c := range asked {
if why := v.credentialRefusal(node, c); why != "" {
return OwnAnswer{}, fmt.Errorf("%s may not ask for the credential of %s from %s to %s on %s: %s", caller,
c.Provision, c.Provider, c.ConsumerModule, c.Consumer, why)
}
found := false
names := credentialNames(c)
for i, name := range names {
r, held, err := v.Store.Read(name)
if err != nil {
return OwnAnswer{}, err
}
if !held {
continue
}
if i == len(names)-1 {
// The provision's one shared value: handed only to a machine the record of its holders names.
why, err := v.sharedRefusal(node, c)
if err != nil {
return OwnAnswer{}, err
}
if why != "" {
v.log("%s is not handed %s: %s", node, name, why)
continue
}
}
value, err := seal.Open(r.AtRest, v.Own.PrivateKey)
if err != nil {
return OwnAnswer{}, fmt.Errorf("%s does not open with the vault's key: %w", name, err)
}
sealed, err := seal.Seal(value, to)
if err != nil {
return OwnAnswer{}, err
}
if err := v.recordHanding(Handing{To: caller, Name: name, Generation: r.Generation, Outcome: "handed"}); err != nil {
return OwnAnswer{}, fmt.Errorf("%s is not handed out: who was handed it could not be recorded: %w", name, err)
}
out.Credentials = append(out.Credentials, NodeCredential{CredentialAsked: c, Name: name, Generation: r.Generation, Sealed: sealed})
found = true
break
}
if !found {
out.Missing = append(out.Missing, c)
}
}
v.log("handed %s's node-engine the %d own secrets of its modules and %d credentials (%d not held), sealed to its key",
node, len(out.Own), len(out.Credentials), len(out.Missing))
return out, nil
}
// handed says the plan of module on node hands it to the provisioner, so its secrets are filled from `get`, not composed.
// A plan that cannot be read is an error, never "not handed": `own` refuses rather than hands a secret the provisioner
// holds.
func (v *Vault) handed(node, module string) (bool, error) {
read, why := v.assignmentReader(node)
if why != "" {
return false, errors.New(why)
}
raw, err := read(node + "." + module)
if err != nil {
return false, fmt.Errorf(notYetPrefix+"the assignment %s.%s could not be read, so whether it is handed to the "+
"provisioner is not known: %w", node, module, err)
}
if raw == nil {
return false, nil
}
var a plannedSecrets
if err := json.Unmarshal(raw, &a); err != nil {
return false, fmt.Errorf("the assignment %s.%s is not the controller's: %w", node, module, err)
}
return a.Provisioning != nil, nil
}
@@ -1,121 +0,0 @@
package main
import (
"errors"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/seal"
)
// Issue 1126 (ADR 0547 §2, ADR 0550 §5): `own` answers a machine's node-engine the own secrets of its modules, sealed to
// the key that machine's engine stated, about that machine alone; another caller, another machine's secrets and a
// machine with no stated key get nothing.
func TestOwnAnswersAMachineItsModulesSecretsSealedToItsKey(t *testing.T) {
plans := assignments{"anchor.web": `{"node":"anchor","module":"web"}`, "laptop.web": `{"node":"laptop","module":"web"}`}
v := newVault(t, nil)
v.Askers = Declared{Read: plans.read, Store: v.Store}
anchorPub, anchorPriv := aSealingKey(t)
laptopPub, _ := aSealingKey(t)
for _, m := range []struct {
caller string
of MoveOf
value string
}{
{"node.anchor", MoveOf{Node: "anchor", Module: "web", Name: "password", Origin: "made", Generation: "g-1", SealingKey: anchorPub}, "anchor-web-pw"},
{"node.anchor", MoveOf{Node: "anchor", Module: "web", Name: "api-key", Origin: "accepted", Generation: "g-1", SealingKey: anchorPub}, "anchor-web-key"},
{"node.laptop", MoveOf{Node: "laptop", Module: "web", Name: "password", Origin: "made", Generation: "g-1", SealingKey: laptopPub}, "laptop-web-pw"},
} {
if _, err := moveIn(t, v, m.caller, m.of, m.value); err != nil {
t.Fatal(err)
}
}
got, err := v.OwnOfNode("node.anchor")
if err != nil {
t.Fatal(err)
}
if got.Node != "anchor" || got.Key != anchorPub || len(got.Own) != 2 {
t.Fatalf("anchor was answered %+v", got)
}
want := map[string][2]string{"api-key": {"accepted", "anchor-web-key"}, "password": {"made", "anchor-web-pw"}}
for _, s := range got.Own {
w, ok := want[s.Name]
if !ok || s.Module != "web" || s.Origin != w[0] || s.Generation < 1 {
t.Errorf("anchor's %s: %+v", s.Name, s)
continue
}
value, err := seal.Open(s.Sealed, anchorPriv)
if err != nil || string(value) != w[1] {
t.Errorf("anchor's %s does not open to its value with anchor's key: %q, %v", s.Name, value, err)
}
}
for caller, says := range map[string]string{
"controller": "is not a node-engine",
"anchor.web": "is not a node-engine",
"": "an unproven caller",
"node.unknown": "no secret named own/unknown yet",
} {
if _, err := v.OwnOfNode(caller); err == nil || !strings.Contains(err.Error(), says) {
t.Errorf("%q: %v, want a refusal saying %q", caller, err, says)
}
}
if laptop, err := v.OwnOfNode("node.laptop"); err != nil || len(laptop.Own) != 1 || laptop.Own[0].Name != "password" {
t.Errorf("laptop was answered %+v, %v; want its one secret alone", laptop, err)
}
// A pair credential: laptop's web consumes anchor's postgres. Each end is answered the one value sealed to its own key;
// a credential the vault holds none of is named missing; a machine at neither end is refused.
if _, err := moveIn(t, v, "node.laptop", MoveOf{Provider: "anchor", Provision: "postgres", Consumer: "laptop",
ConsumerModule: "web", Origin: "made", Generation: "g-1", SealingKey: laptopPub}, "pg-pair"); err != nil {
t.Fatal(err)
}
pair := CredentialAsked{Provision: "postgres", Consumer: "laptop", ConsumerModule: "web", Provider: "anchor"}
asConsumer, asProvider := pair, pair
asConsumer.Role, asProvider.Role = "consumer", "provider"
none := CredentialAsked{Role: "consumer", Provision: "redis", Consumer: "laptop", ConsumerModule: "web", Provider: "anchor"}
laptop, err := v.OwnOfNode("node.laptop", asConsumer, none)
if err != nil || len(laptop.Credentials) != 1 || len(laptop.Missing) != 1 || laptop.Missing[0] != none {
t.Fatalf("laptop as consumer: %+v, %v", laptop, err)
}
if !strings.HasPrefix(laptop.Credentials[0].Name, "pair/anchor/postgres/laptop/web") {
t.Errorf("laptop's credential is kept as %s", laptop.Credentials[0].Name)
}
anchor, err := v.OwnOfNode("node.anchor", asProvider)
if err != nil || len(anchor.Credentials) != 1 {
t.Fatalf("anchor as provider: %+v, %v", anchor, err)
}
if value, err := seal.Open(anchor.Credentials[0].Sealed, anchorPriv); err != nil || string(value) != "pg-pair" {
t.Errorf("anchor's copy does not open to the value with its key: %q, %v", value, err)
}
for who, c := range map[string]CredentialAsked{"node.anchor": asConsumer, "node.laptop": asProvider} {
if _, err := v.OwnOfNode(who, c); err == nil || !strings.Contains(err.Error(), "is not that machine") {
t.Errorf("%s asking %+v: %v", who, c, err)
}
}
// Review of L2 (C1, C2): `own` fails closed. A provision's one shared value is handed only to a consumer the record
// of its holders names, never to a machine that names its provider; assignments that cannot be read, or a vault
// that reads none, refuse rather than hand.
if _, err := v.keep("provision/anchor/redis", "shared-redis", "test", "", OriginMade); err != nil {
t.Fatal(err)
}
if err := v.Store.WriteShared(Shared{Name: "provision/anchor/redis", Provider: "anchor", Provision: "redis",
Consumers: []string{"anchor/web"}}); err != nil {
t.Fatal(err)
}
if got, err := v.OwnOfNode("node.laptop", none); err != nil || len(got.Credentials) != 0 || len(got.Missing) != 1 {
t.Errorf("laptop, whose web the shared value's record does not name, got %+v, %v", got, err)
}
anchorRedis := CredentialAsked{Role: "consumer", Provision: "redis", Consumer: "anchor", ConsumerModule: "web", Provider: "anchor"}
if got, err := v.OwnOfNode("node.anchor", anchorRedis); err != nil || len(got.Credentials) != 1 {
t.Errorf("anchor's web, a holder the record names, got %+v, %v", got, err)
}
v.Askers = Declared{Read: func(string) ([]byte, error) { return nil, errors.New("the bucket did not answer") }, Store: v.Store}
if _, err := v.OwnOfNode("node.anchor"); err == nil || !strings.Contains(err.Error(), "could not be read") {
t.Errorf("own with the assignments unread: %v", err)
}
v.Askers = nil
if _, err := v.OwnOfNode("node.anchor", anchorRedis); err == nil || !strings.Contains(err.Error(), "cannot read the assignments") {
t.Errorf("own with no reader of the assignments: %v", err)
}
}
@@ -251,34 +251,6 @@ func definitions(env *Env) []toolDef {
// clause 9, adopt.go): a public half, never the private one. // clause 9, adopt.go): a public half, never the private one.
return map[string]any{"recorded": true, "vault_key": v.Own.PublicKey}, nil return map[string]any{"recorded": true, "vault_key": v.Own.PublicKey}, nil
}}, }},
{Name: Seat + ".own",
Description: "The own secrets of this machine's modules, each sealed to the machine's sealing key (novox/hq ADR 0547 §2, " +
"ADR 0550 §5, issue 1126): from the node-engine of the machine alone, about its own machine (the bus proves which), " +
"for the compositions it makes itself. Answers the machine, the key each value is sealed to, and per module and " +
"name the origin, the generation and the sealed value; a module handed to the provisioner is left out. With credentials, " +
"also each pair credential the machine consumes or grants, sealed the same way. Never a value in the clear.",
Input: map[string]any{"type": "object", "properties": map[string]any{
"credentials": map[string]any{"type": "array", "description": "the pair credentials the machine's composition " +
"reads, each {role: consumer|provider, provision, consumer, consumer_module, provider, local}: answered " +
"sealed to the machine's key when the machine is that consumer's or that provider, and named missing when " +
"the vault holds none", "items": map[string]any{"type": "object"}}}},
run: func(caller string, a map[string]any) (any, error) {
v, err := env.ready()
if err != nil {
return nil, err
}
var asked []CredentialAsked
if raw, ok := a["credentials"]; ok && raw != nil {
b, err := json.Marshal(raw)
if err == nil {
err = json.Unmarshal(b, &asked)
}
if err != nil {
return nil, fmt.Errorf("credentials is a list of {role, provision, consumer, consumer_module, provider, local}: %w", err)
}
}
return v.OwnOfNode(caller, asked...)
}},
{Name: Seat + ".move-asks", {Name: Seat + ".move-asks",
Description: "Copy rows of the controller's record of asks at a desk into the vault's, as they are and once " + Description: "Copy rows of the controller's record of asks at a desk into the vault's, as they are and once " +
"(design 57's V3): from the controller alone, at most 500 at a time. No value is in any of them.", "(design 57's V3): from the controller alone, at most 500 at a time. No value is in any of them.",
+1 -2
View File
@@ -177,8 +177,7 @@
"rotate", "rotate",
"move-in", "move-in",
"node-key", "node-key",
"move-asks", "move-asks"
"own"
] ]
} }
] ]