Merge pull request 'The gitea module makes the organisation's repositories from its config, protected before any team reaches them; no tool creates in the organisation (issue 1124, ADR 0428)' (#363) from fix/1124-repositories-config into main

This commit was merged in pull request #363.
This commit is contained in:
2026-10-18 15:05:04 +00:00
14 changed files with 1654 additions and 133 deletions
+37
View File
@@ -140,6 +140,43 @@ the value is emptied, never deleted. A team without that prefix is never touched
change of the value. It says each change in one line on stdout, and never a secret. `go test ./cmd/agent-teams`
runs it against a fake forge.
## The organisation's repositories (novox/hq ADR 0428, issue 1124)
Which repositories the organisation `novox` holds is one trusted config value, `repositories`:
`{"<name>": {"private": true, "description": "..."}}`, changed only at the controller's terminal or on an approval
that proves who answered, as `agent-read-teams` is. The module renders it into `repositories.json` and runs
`cmd/repositories sync` when it starts and whenever that file changes. With the setup login (`mesh-admin`), for each
listed repository:
- one that is not in the organisation and not under the setup account is made there, **empty and under the setup
account**, where neither the tools' team nor the agents' team (both on every repository of the organisation)
reaches it. Main gets a rule letting the setup login alone push; main's first commit is written with it: a README
naming the description, and a `merge-check.sh` declaring `# mesh-check-toolchain: go` that vets and runs the unit
tests, then the integration tests (`-tags integration`), of every Go module in the tree, and says so and passes on
a tree with none. The rule is then narrowed to main's rule of ADR 0398 §2 (no push, only `mesh-tools` merges),
requiring `mesh/repo-check` and `mesh/merge-gate`, and only then is the repository moved into the organisation.
There main's rule is read back and put right once more (the step `rule-after-move`): the forge drops from a merge
list a user without access to the repository, and under the setup account `mesh-tools` has none;
- one found still under the setup account is finished: the steps it lacks are done, main is never written twice;
- one in the organisation is only put right: main's rule narrowed, a missing one of the two statuses added and every
status already there kept.
A step the forge refuses stops that repository, and the others go on. A repository taken out of the value is never
asked about: nothing is deleted, renamed, moved out or unprotected. The pass keeps what it did per repository in
`repositories-outcome.json`.
**How a refusal is seen.** `repositories sync` is a run-once process, declared as the module's last resource, so
the node-engine's gate on a step that did not complete (ADR 0136) stops nothing of the module after it. When any
step was refused it exits 1, and its last stderr lines are one per refused repository,
`repositories: <name>: <step> refused: <the forge's answer>`, then, when there were several, a summary line naming
them. The node-engine reports the module's step failed and raises apply-failing, and runs it again on every apply
until it passes: that is the next pass. The condition says that the `gitea-repositories` step did not complete;
the lines naming the repository, the step and the forge's answer are in that unit's journal
(`node-service-manager.journal`).
No tool creates in the organisation: `gitea_create_repo` creates under the tools' own user only, the tools' team
keeps `can_create_org_repo` false and its token has no `write:organization` (ADR 0428 §8).
`go test ./cmd/repositories` runs it against a fake forge.
## The package registry's retention (novox/hq ADR 0251 §4, to-be 51)
Every publish adds a version to the registry, and until this nothing removed one. The Go bundle
+5 -12
View File
@@ -244,23 +244,16 @@ export class GiteaClient {
return all;
}
/** A repository, created under the tools' own user, or with `owner` in that organisation (novox/hq issue 1124).
* The forge's refusal to create in an organisation (403 not allowed, 404 no such organisation, 409 exists) is
* thrown with its status and its reason, naming the organisation. */
/** A repository, created under the tools' own user and nowhere else (novox/hq ADR 0428 §8): the forge makes whoever
* creates an organisation's repository its administrator, so the organisation's repositories are made by the
* module's setup login from its config value `repositories` (cmd/repositories), never through a tool. */
async createRepo(data: {
name: string;
description?: string;
private?: boolean;
auto_init?: boolean;
}, owner?: string): Promise<GiteaRepo> {
const options = { method: "POST", body: JSON.stringify(data) };
if (owner === undefined) return GiteaClient.mapRepo(await this.request<any>("/user/repos", options));
const path = `/orgs/${pathSegment("organisation", owner)}/repos`;
try {
return GiteaClient.mapRepo(await this.request<any>(path, options));
} catch (e) {
throw new Error(`the forge did not create ${owner}/${data.name} in the organisation ${owner}: ${(e as Error).message}`);
}
}): Promise<GiteaRepo> {
return GiteaClient.mapRepo(await this.request<any>("/user/repos", { method: "POST", body: JSON.stringify(data) }));
}
async deleteRepo(owner: string, repo: string): Promise<void> {
+323
View File
@@ -0,0 +1,323 @@
package main
import (
"context"
"encoding/base64"
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"regexp"
"sort"
"strings"
"sync"
"testing"
"gitea/internal/forgehttp"
)
// The fake forge of issue 1124 / ADR 0428: it keeps repositories by owner/name with their branch protection rules,
// their branches and their files, records every call in order, and refuses a call a test names. Like the forge, it
// refuses a commit onto a branch whose rule does not let the caller push, and it answers only the setup login.
const (
admin = "mesh-admin"
tools = "mesh-tools"
org = "novox"
)
type fRepo struct {
Private bool
Description string
Rules map[string]map[string]any // by rule_name
Branches map[string]bool
Files map[string]string
Commits int
}
type fakeForge struct {
mu sync.Mutex
repos map[string]*fRepo // "owner/name"
calls []string // "METHOD /path"
// refuse answers a call with a status and a message when it returns a status above zero.
refuse func(method, path string) (int, string)
}
func newFake() *fakeForge { return &fakeForge{repos: map[string]*fRepo{}} }
func (f *fakeForge) add(full string, rules map[string]map[string]any, branches ...string) *fRepo {
r := &fRepo{Rules: map[string]map[string]any{}, Branches: map[string]bool{}, Files: map[string]string{}}
for k, v := range rules {
r.Rules[k] = v
}
for _, b := range branches {
r.Branches[b] = true
}
f.repos[full] = r
return r
}
// writes are the calls that change something, in order.
func (f *fakeForge) writes() []string {
f.mu.Lock()
defer f.mu.Unlock()
var out []string
for _, c := range f.calls {
if !strings.HasPrefix(c, "GET ") {
out = append(out, c)
}
}
return out
}
func (f *fakeForge) all() []string {
f.mu.Lock()
defer f.mu.Unlock()
return append([]string(nil), f.calls...)
}
var (
repoPath = regexp.MustCompile(`^/api/v1/repos/([^/]+)/([^/]+)$`)
rulesPath = regexp.MustCompile(`^/api/v1/repos/([^/]+)/([^/]+)/branch_protections(?:/([^/]+))?$`)
branchPath = regexp.MustCompile(`^/api/v1/repos/([^/]+)/([^/]+)/branches/([^/]+)$`)
filesPath = regexp.MustCompile(`^/api/v1/repos/([^/]+)/([^/]+)/contents$`)
movePath = regexp.MustCompile(`^/api/v1/repos/([^/]+)/([^/]+)/transfer$`)
)
func strs(v any) []string {
var out []string
if l, ok := v.([]any); ok {
for _, s := range l {
out = append(out, fmt.Sprint(s))
}
}
if l, ok := v.([]string); ok {
out = append(out, l...)
}
return out
}
// lets says whether a rule lets the login push.
func lets(rule map[string]any, login string) bool {
if rule == nil {
return true
}
if rule["enable_push"] != true {
return false
}
if rule["enable_push_whitelist"] != true {
return true
}
for _, u := range strs(rule["push_whitelist_usernames"]) {
if u == login {
return true
}
}
return false
}
// reaches says whether a login has access to a repository, as the forge judges a merge whitelist: its owner does,
// and in the organisation the tools' team does (it is on every repository there); nobody else here.
func reaches(full, login string) bool {
owner := strings.SplitN(full, "/", 2)[0]
return owner == login || (owner == org && login == tools)
}
// keepReaching drops from a rule's merge whitelist every user without access to the repository, as gitea's
// updateUserWhitelist does on creating or editing a branch protection.
func keepReaching(full string, rule map[string]any) {
if _, ok := rule["merge_whitelist_usernames"]; !ok {
return
}
kept := []any{}
for _, u := range strs(rule["merge_whitelist_usernames"]) {
if reaches(full, u) {
kept = append(kept, u)
}
}
rule["merge_whitelist_usernames"] = kept
}
func (f *fakeForge) ServeHTTP(w http.ResponseWriter, r *http.Request) {
f.mu.Lock()
defer f.mu.Unlock()
if u, p, ok := r.BasicAuth(); !ok || u != admin || p != "pw" {
http.Error(w, "no", http.StatusUnauthorized)
return
}
reply := func(status int, v any) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(status)
if v != nil {
_ = json.NewEncoder(w).Encode(v)
}
}
p := r.URL.Path
f.calls = append(f.calls, r.Method+" "+p)
if f.refuse != nil {
if status, said := f.refuse(r.Method, p); status > 0 {
reply(status, map[string]string{"message": said})
return
}
}
var in map[string]any
_ = json.NewDecoder(r.Body).Decode(&in)
switch {
case p == "/api/v1/version":
reply(200, map[string]string{"version": "1.27"})
case p == "/api/v1/user/repos" && r.Method == http.MethodPost:
name := fmt.Sprint(in["name"])
if f.repos[admin+"/"+name] != nil {
reply(409, map[string]string{"message": "The repository with the same name already exists."})
return
}
if in["auto_init"] == true {
reply(400, map[string]string{"message": "this fake takes only an empty repository"})
return
}
repo := f.add(admin+"/"+name, nil)
repo.Private, _ = in["private"].(bool)
repo.Description, _ = in["description"].(string)
reply(201, map[string]any{"full_name": admin + "/" + name, "empty": true})
case repoPath.MatchString(p) && r.Method == http.MethodGet:
m := repoPath.FindStringSubmatch(p)
repo := f.repos[m[1]+"/"+m[2]]
if repo == nil {
reply(404, map[string]string{"message": "not found"})
return
}
reply(200, map[string]any{"full_name": m[1] + "/" + m[2], "empty": len(repo.Branches) == 0})
case rulesPath.MatchString(p):
m := rulesPath.FindStringSubmatch(p)
repo := f.repos[m[1]+"/"+m[2]]
if repo == nil {
reply(404, nil)
return
}
switch {
case r.Method == http.MethodGet && m[3] == "":
names := make([]string, 0, len(repo.Rules))
for n := range repo.Rules {
names = append(names, n)
}
sort.Strings(names)
out := []map[string]any{}
for _, n := range names {
rule := map[string]any{"rule_name": n}
for k, v := range repo.Rules[n] {
rule[k] = v
}
out = append(out, rule)
}
reply(200, out)
case r.Method == http.MethodPost && m[3] == "":
name := fmt.Sprint(in["rule_name"])
if repo.Rules[name] != nil {
reply(403, map[string]string{"message": "branch protection already exist"})
return
}
delete(in, "rule_name")
keepReaching(m[1]+"/"+m[2], in)
repo.Rules[name] = in
reply(201, in)
case r.Method == http.MethodPatch && m[3] != "":
rule := repo.Rules[m[3]]
if rule == nil {
reply(404, nil)
return
}
for k, v := range in {
rule[k] = v
}
keepReaching(m[1]+"/"+m[2], rule)
reply(200, rule)
default:
reply(405, nil)
}
case branchPath.MatchString(p) && r.Method == http.MethodGet:
m := branchPath.FindStringSubmatch(p)
repo := f.repos[m[1]+"/"+m[2]]
if repo == nil || !repo.Branches[m[3]] {
reply(404, nil)
return
}
reply(200, map[string]any{"name": m[3]})
case filesPath.MatchString(p) && r.Method == http.MethodPost:
m := filesPath.FindStringSubmatch(p)
repo := f.repos[m[1]+"/"+m[2]]
branch := fmt.Sprint(in["branch"])
if repo == nil {
reply(404, nil)
return
}
if !lets(repo.Rules[branch], admin) {
reply(403, map[string]string{"message": "branch " + branch + " is protected"})
return
}
files, _ := in["files"].([]any)
for _, x := range files {
op, _ := x.(map[string]any)
raw, _ := base64.StdEncoding.DecodeString(fmt.Sprint(op["content"]))
repo.Files[fmt.Sprint(op["path"])] = string(raw)
}
repo.Branches[branch] = true
repo.Commits++
reply(201, map[string]any{"commit": map[string]any{"sha": fmt.Sprintf("%040d", repo.Commits)}})
case movePath.MatchString(p) && r.Method == http.MethodPost:
m := movePath.FindStringSubmatch(p)
from := m[1] + "/" + m[2]
repo := f.repos[from]
to := fmt.Sprint(in["new_owner"]) + "/" + m[2]
if repo == nil || f.repos[to] != nil {
reply(422, map[string]string{"message": "cannot transfer"})
return
}
delete(f.repos, from)
f.repos[to] = repo
reply(202, map[string]any{"full_name": to})
default:
reply(404, map[string]string{"message": "not found"})
}
}
func syncer(t *testing.T, f *fakeForge) (*Syncer, *[]string) {
t.Helper()
srv := httptest.NewServer(f)
t.Cleanup(srv.Close)
pw := filepath.Join(t.TempDir(), "pw")
if err := os.WriteFile(pw, []byte("pw\n"), 0o600); err != nil {
t.Fatal(err)
}
var said []string
return &Syncer{Forge: forgehttp.Client{URL: srv.URL, User: admin, PasswordFile: pw}, Org: org, Admin: admin, Tools: tools,
Log: func(format string, a ...any) { said = append(said, fmt.Sprintf(format, a...)) }}, &said
}
func run(t *testing.T, s *Syncer, v Value) Outcome {
t.Helper()
o, err := s.Sync(context.Background(), v)
if err != nil {
t.Fatal(err)
}
return o
}
func result(o Outcome, name string) Result {
for _, r := range o.Repositories {
if r.Name == name {
return r
}
}
return Result{}
}
func has(list []string, want string) bool {
for _, s := range list {
if s == want {
return true
}
}
return false
}
+43
View File
@@ -0,0 +1,43 @@
package main
import "strings"
// MergeCheck is the first commit's merge-check.sh (ADR 0428 §4): it declares the Go toolchain (ADR 0238 §4, read
// among a script's first lines by the build seat), and runs the unit tests, then with -tags integration the
// integration tests, of every Go module in the tree; on a tree with none it says so and passes.
const MergeCheck = `#!/bin/sh
# mesh-check-toolchain: go
#
# This repository's own check, mesh/repo-check (novox/hq ADR 0238 §4), written with its first commit (ADR 0428 §4):
# every Go module in the tree is vetted, its unit tests run, and then its integration tests (-tags integration).
# A tree with no Go module has nothing to test, and the check says so and passes.
set -eu
cd "$(dirname "$0")"
modules=$(find . -name go.mod -not -path '*/vendor/*' -not -path './.git/*' | sort)
if [ -z "$modules" ]; then
echo "This tree holds no Go module, so there are no tests to run; the check passes."
exit 0
fi
for mod in $modules; do
dir=$(dirname "$mod")
echo "== $dir: go vet and the unit tests"
(cd "$dir" && go vet ./... && go test ./...)
echo "== $dir: the integration tests (-tags integration)"
(cd "$dir" && go vet -tags integration ./... && go test -tags integration ./...)
done
`
// FirstCommit is main's first commit of a listed repository: its README, naming the repository's purpose, and
// its own check.
func FirstCommit(name string, r Repo) map[string]string {
purpose := strings.TrimSpace(r.Description)
if purpose == "" {
purpose = "(its purpose is not described yet: the gitea module's config value `repositories` describes it)"
}
readme := "# " + name + "\n\n" + purpose + "\n\n" +
"Made by the mesh from the gitea module's config value `repositories` (novox/hq ADR 0428). Main takes no push: " +
"a change reaches it by a pull request whose `mesh/repo-check` (this repository's `merge-check.sh`) and " +
"`mesh/merge-gate` passed.\n"
return map[string]string{"README.md": readme, "merge-check.sh": MergeCheck}
}
+69
View File
@@ -0,0 +1,69 @@
// repositories: the organisation's repositories from the gitea module's config value `repositories` (novox/hq
// ADR 0428, issue 1124). `repositories sync` is a run-once process the node-engine runs when the module starts and
// again whenever the value changes: it reads the file the mesh renders from that value and, with the setup login,
// makes every listed repository that does not exist under that login's own account, protects its main, writes
// main's first commit, narrows main's rule and only then moves it into the organisation; one that exists only has
// its main rule put right. A repository the value does not list is never touched. It keeps what it did, per
// repository, in the outcome file, says each change on stdout in one line and never a secret, and exits 1 when a
// step was refused, its last stderr lines naming each (outcome.go Report): the node-engine's gate on a run-once
// step (ADR 0136) reports it and runs it again on every apply until it passes.
package main
import (
"context"
"fmt"
"os"
"time"
"gitea/internal/forgehttp"
)
func main() {
if len(os.Args) != 2 || os.Args[1] != "sync" {
fmt.Fprintln(os.Stderr, "usage: repositories sync")
os.Exit(2)
}
os.Exit(runSync())
}
func runSync() int {
forge, missing := forgehttp.FromEnv()
file, outcome := os.Getenv("MESH_REPOSITORIES_FILE"), os.Getenv("MESH_REPOSITORIES_OUTCOME")
for k, v := range map[string]string{"MESH_REPOSITORIES_FILE": file, "MESH_REPOSITORIES_OUTCOME": outcome} {
if v == "" {
missing = append(missing, k)
}
}
if forge.TokenFile != "" {
// The tools' account cannot make or protect a repository, and must not (ADR 0428 §8).
missing = append(missing, "the setup login (MESH_GITEA_TOKEN_FILE is set instead)")
}
if len(missing) > 0 {
fmt.Fprintf(os.Stderr, "repositories: %v not set by the manifest; nothing was changed\n", missing)
return 1
}
toolsAccount := os.Getenv("MESH_GITEA_TOOLS_USER")
if toolsAccount == "" {
toolsAccount = forgehttp.ToolsAccount
}
s := &Syncer{Forge: forge, Org: "novox", Admin: forge.User, Tools: toolsAccount,
Log: func(format string, a ...any) { fmt.Printf(format+"\n", a...) },
WaitFor: 3 * time.Minute, Poll: 5 * time.Second}
value, err := ReadValue(file)
if err != nil {
o := Outcome{At: time.Now().UTC(), Org: s.Org, Failed: err.Error()}
_ = WriteOutcome(outcome, o)
return Report(o, os.Stderr)
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute)
defer cancel()
o, _ := s.Sync(ctx, value) // a forge not reachable is in o.Failed
werr := WriteOutcome(outcome, o)
if werr != nil {
fmt.Fprintln(os.Stderr, "repositories: the outcome could not be kept:", werr)
}
if code := Report(o, os.Stderr); code != 0 || werr == nil {
return code
}
return 1
}
@@ -0,0 +1,87 @@
package main
import (
"encoding/json"
"os"
"strings"
"testing"
)
// 1124/AC1: the manifest declares `repositories` under config as a trusted object value with an empty default, as
// it declares agent-read-teams (ADR 0365 §3, ADR 0348), renders it into a file of its own, and runs this program
// at the module's start and whenever that file changes, as a run-once process (1124/AC6: its failed exit is the
// node-engine's gate, ADR 0136) declared last, so that gate stops nothing of the module after it.
func TestManifestDeclaresTheValueAndItsTrigger(t *testing.T) {
raw, err := os.ReadFile("../../module.json")
if err != nil {
t.Fatal(err)
}
var m struct {
Config map[string]map[string]any `json:"config"`
Resources []map[string]any `json:"resources"`
Build struct {
Artifacts []map[string]any `json:"artifacts"`
} `json:"build"`
}
if err := json.Unmarshal(raw, &m); err != nil {
t.Fatal(err)
}
v := m.Config[ConfigKey]
if v == nil || v["type"] != "object" || v["trusted"] != true {
t.Fatalf("config %s = %v, want a trusted object", ConfigKey, v)
}
if d, ok := v["default"].(map[string]any); !ok || len(d) != 0 {
t.Fatalf("its default is %v, want {}", v["default"])
}
var file, process map[string]any
for _, r := range m.Resources {
for _, k := range strs(r["takes-config"]) {
if k == ConfigKey {
file = r
}
}
}
if file == nil || file["type"] != "file" {
t.Fatalf("no file takes the value %s", ConfigKey)
}
for _, r := range m.Resources {
if r["type"] == "process" && has(strs(r["restart-on"]), file["id"].(string)) {
process = r
}
}
if process == nil {
t.Fatalf("no process runs again when %s changes", file["id"])
}
if process["run-once"] != true {
t.Errorf("the process is not run-once: %v", process)
}
if last := m.Resources[len(m.Resources)-1]; last["id"] != process["id"] {
t.Errorf("the last resource is %v, want %v: a refused step gates whatever of the module comes after it", last["id"], process["id"])
}
if process["health"] != nil {
t.Errorf("the process declares a health check: %v", process["health"])
}
if strings.Join(strs(process["run"]), " ") != "./repositories sync" || process["artifact"] != "repositories" {
t.Errorf("the process runs %v from %v", process["run"], process["artifact"])
}
env, _ := process["env"].(map[string]any)
for _, k := range []string{"MESH_GITEA_URL", "MESH_GITEA_ADMIN_USER", "MESH_GITEA_ADMIN_PASSWORD_FILE", "MESH_REPOSITORIES_FILE", "MESH_REPOSITORIES_OUTCOME"} {
if env[k] == nil {
t.Errorf("the process is not given %s", k)
}
}
found := false
for _, a := range m.Build.Artifacts {
if a["name"] == "repositories" && a["from"] == "cmd/repositories" && a["binary"] == "repositories" {
found = true
}
}
if !found {
t.Error("no bundle builds cmd/repositories")
}
// Like agent-read-teams: the same kind of value, the same way of applying it.
teams := m.Config["agent-read-teams"]
if teams["trusted"] != v["trusted"] || teams["type"] != v["type"] {
t.Errorf("repositories is declared unlike agent-read-teams: %v vs %v", v, teams)
}
}
+79
View File
@@ -0,0 +1,79 @@
package main
// What a pass left (ADR 0428 §6): the outcome file, and the pass's exit. A refused step is not judged by a health
// check of its own: the sync is a run-once process, so its non-zero exit is the node-engine's gate (ADR 0136). The
// module's step is reported failed, apply-failing is raised, and every apply runs it again until it passes, which is
// the "next pass" of §6. Its last stderr lines name each refused repository, its step and the forge's answer.
import (
"encoding/json"
"fmt"
"io"
"os"
"path/filepath"
"strings"
)
// WriteOutcome keeps what the pass did per repository, whole or not at all.
func WriteOutcome(file string, o Outcome) error {
raw, err := json.MarshalIndent(o, "", " ")
if err != nil {
return err
}
tmp, err := os.CreateTemp(filepath.Dir(file), ".repositories-*")
if err != nil {
return err
}
defer os.Remove(tmp.Name())
if _, err := tmp.Write(append(raw, '\n')); err != nil {
tmp.Close()
return err
}
if err := tmp.Chmod(0o644); err != nil {
tmp.Close()
return err
}
if err := tmp.Close(); err != nil {
return err
}
return os.Rename(tmp.Name(), file)
}
// ReadOutcome reads what the last pass left.
func ReadOutcome(file string) (*Outcome, error) {
raw, err := os.ReadFile(file)
if err != nil {
return nil, err
}
var o Outcome
if err := json.Unmarshal(raw, &o); err != nil {
return nil, fmt.Errorf("the outcome %s is not readable: %w", file, err)
}
return &o, nil
}
// Report says on stderr what the pass refused and is the pass's exit: 0 when every listed repository was made, put
// right or already right; 1 otherwise. One line per refused repository, "repositories: <name>: <step> refused:
// <forge answer>", then a summary line, so the line the node-engine keeps last names what to look at.
func Report(o Outcome, stderr io.Writer) int {
if o.Failed != "" {
fmt.Fprintf(stderr, "repositories: the pass did not run: %s; nothing was made\n", o.Failed)
return 1
}
refused := o.Refused()
if len(refused) == 0 {
return 0
}
for _, r := range refused {
fmt.Fprintf(stderr, "repositories: %s: %s refused: %s\n", r.Name, r.Step, r.Said)
}
if len(refused) > 1 {
names := make([]string, 0, len(refused))
for _, r := range refused {
names = append(names, r.Name)
}
fmt.Fprintf(stderr, "repositories: %d of %d repositories had a step refused (%s); nothing further of them was done, the rest were\n",
len(refused), len(o.Repositories), strings.Join(names, ", "))
}
return 1
}
@@ -0,0 +1,421 @@
package main
import (
"context"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"net/http"
"net/url"
"os"
"path"
"regexp"
"sort"
"strings"
"time"
"gitea/internal/forgehttp"
)
const (
// ConfigKey is the module's config value this program applies (ADR 0428 §1).
ConfigKey = "repositories"
// MainBranch is the branch whose rule takes no push and lets only the tools' account merge (ADR 0398 §2,
// protection.ts MAIN_BRANCH).
MainBranch = "main"
)
// RequiredStatuses are what main requires in every listed repository from its first commit (ADR 0428 §5): the
// repository's own check and the module graph's gate. A status already on the rule is kept beside them.
var RequiredStatuses = []string{"mesh/merge-gate", "mesh/repo-check"}
// The steps of one repository, as an outcome names a refused one (ADR 0428 §6).
const (
StepRead = "read"
StepCreate = "create"
StepAdminRule = "admin-rule"
StepFirstCommit = "first-commit"
StepNarrow = "narrow"
StepMove = "move"
StepPutRight = "put-right"
// StepRuleAfterMove reads main's rule back once the repository is in the organisation and puts it right: the
// forge drops from a merge whitelist a user without access, and under the setup account the tools' have none.
StepRuleAfterMove = "rule-after-move"
)
// What a pass did with one repository.
const (
DidMade = "made"
DidPutRight = "put right"
DidUnchanged = "unchanged"
DidRefused = "refused"
)
// repoName is a repository name the forge takes and that is never another path.
var repoName = regexp.MustCompile(`^[A-Za-z0-9_][A-Za-z0-9_.-]{0,99}$`)
// Repo is one listed repository: whether it is private, and its description, which its README names as its purpose.
type Repo struct {
Private bool `json:"private"`
Description string `json:"description"`
}
// Value is the config value: each repository of the organisation by name.
type Value map[string]Repo
// ReadValue reads the file the mesh renders from the config value. A file without the key is the empty value,
// which makes nothing and touches nothing.
func ReadValue(file string) (Value, error) {
raw, err := os.ReadFile(file)
if err != nil {
return nil, fmt.Errorf("the repositories file cannot be read (%s): %w", file, err)
}
var whole map[string]json.RawMessage
if err := json.Unmarshal(raw, &whole); err != nil {
return nil, fmt.Errorf("the repositories file %s is not JSON: %w", file, err)
}
v := Value{}
if r, ok := whole[ConfigKey]; ok {
dec := json.NewDecoder(strings.NewReader(string(r)))
dec.DisallowUnknownFields()
if err := dec.Decode(&v); err != nil {
return nil, fmt.Errorf("%s in %s is not {name: {private, description}}: %w", ConfigKey, file, err)
}
}
for name := range v {
if !repoName.MatchString(name) || strings.Contains(name, "..") || strings.HasSuffix(name, ".git") {
return nil, fmt.Errorf("%s in %s lists %q, which is not a repository name", ConfigKey, file, name)
}
}
return v, nil
}
// Result is what a pass did with one repository; a refusal names its step and the forge's answer.
type Result struct {
Name string `json:"name"`
Did string `json:"did"`
Step string `json:"step,omitempty"`
Said string `json:"said,omitempty"`
}
// Outcome is what a pass left, kept in the outcome file and said on stderr (outcome.go).
type Outcome struct {
At time.Time `json:"at"`
Org string `json:"org"`
Failed string `json:"failed,omitempty"` // the pass did not run: why
Repositories []Result `json:"repositories"`
}
// Refused names the repositories a step of was refused.
func (o Outcome) Refused() []Result {
var out []Result
for _, r := range o.Repositories {
if r.Did == DidRefused {
out = append(out, r)
}
}
return out
}
// Syncer makes every listed repository of the organisation, with the setup login, and puts main's rule right in
// those that exist. A repository the value does not list is never asked about (ADR 0428 §7).
type Syncer struct {
Forge forgehttp.Client // the setup login's
Org string // the organisation, novox
Admin string // the setup login, under whose own account a repository is made
Tools string // the tools' account, the only one that merges into main
Log func(format string, a ...any)
// WaitFor is how long to wait for the forge to answer; none when zero.
WaitFor, Poll time.Duration
}
func (s *Syncer) say(format string, a ...any) {
if s.Log != nil {
s.Log(format, a...)
}
}
// Sync applies the value. The error is only the forge not being reachable; a refused step is in the outcome.
func (s *Syncer) Sync(ctx context.Context, v Value) (Outcome, error) {
out := Outcome{At: time.Now().UTC(), Org: s.Org, Repositories: []Result{}}
if err := s.waitForForge(ctx); err != nil {
out.Failed = err.Error()
return out, err
}
names := make([]string, 0, len(v))
for n := range v {
names = append(names, n)
}
sort.Strings(names)
for _, n := range names {
r := s.one(ctx, n, v[n])
if r.Did == DidRefused {
s.say("refused: %s/%s at the step %s: %s", s.Org, n, r.Step, r.Said)
} else if r.Did != DidUnchanged {
s.say("%s: %s/%s", r.Did, s.Org, n)
}
out.Repositories = append(out.Repositories, r)
}
return out, nil
}
// waitForForge is cmd/agent-teams' (teams.go waitForForge): the forge starting with the module answers late.
func (s *Syncer) waitForForge(ctx context.Context) error {
deadline := time.Now().Add(s.WaitFor)
for {
err := s.Forge.GetJSON(ctx, "/api/v1/version", nil)
if err == nil {
return nil
}
var se *forgehttp.StatusError
if errors.As(err, &se) || time.Now().After(deadline) || ctx.Err() != nil {
return fmt.Errorf("the forge is not reachable: %w", err)
}
time.Sleep(s.Poll)
}
}
func refused(name, step string, err error) Result {
return Result{Name: name, Did: DidRefused, Step: step, Said: err.Error()}
}
func repoAPI(owner, name string) string {
return "/api/v1/repos/" + url.PathEscape(owner) + "/" + url.PathEscape(name)
}
// exists says whether owner/name is on the forge.
func (s *Syncer) exists(ctx context.Context, owner, name string) (bool, error) {
err := s.Forge.GetJSON(ctx, repoAPI(owner, name), nil)
if errors.Is(err, forgehttp.ErrNotFound) {
return false, nil
}
return err == nil, err
}
// one is one listed repository (ADR 0428 §2, §3): put right when it is in the organisation; else made, or finished
// when found under the setup account, and moved in only once main's rule is narrowed. A refused step stops it.
func (s *Syncer) one(ctx context.Context, name string, r Repo) Result {
inOrg, err := s.exists(ctx, s.Org, name)
if err != nil {
return refused(name, StepRead, err)
}
if inOrg {
changed, err := s.putRight(ctx, s.Org, name)
if err != nil {
return refused(name, StepPutRight, err)
}
if changed {
return Result{Name: name, Did: DidPutRight}
}
return Result{Name: name, Did: DidUnchanged}
}
held, err := s.exists(ctx, s.Admin, name)
if err != nil {
return refused(name, StepRead, err)
}
if !held {
// Empty, so main exists only once the rule below is on it (ADR 0428 §3).
if err := s.Forge.SendJSON(ctx, http.MethodPost, "/api/v1/user/repos", map[string]any{
"name": name, "private": r.Private, "description": r.Description, "auto_init": false, "default_branch": MainBranch,
}, nil); err != nil {
return refused(name, StepCreate, err)
}
}
rules, err := s.rules(ctx, s.Admin, name)
if err != nil {
return refused(name, StepRead, err)
}
rule := ruleNamed(rules, MainBranch)
if rule == nil {
body := map[string]any{"rule_name": MainBranch, "enable_push": true, "enable_push_whitelist": true,
"push_whitelist_usernames": []string{s.Admin}}
if err := s.Forge.SendJSON(ctx, http.MethodPost, repoAPI(s.Admin, name)+"/branch_protections", body, nil); err != nil {
return refused(name, StepAdminRule, err)
}
}
hasMain, err := s.branchExists(ctx, s.Admin, name, MainBranch)
if err != nil {
return refused(name, StepRead, err)
}
if !hasMain {
if err := s.firstCommit(ctx, name, r); err != nil {
return refused(name, StepFirstCommit, err)
}
}
if rule == nil || !isRight(rule, s.Tools) {
if err := s.Forge.SendJSON(ctx, http.MethodPatch, repoAPI(s.Admin, name)+"/branch_protections/"+MainBranch,
narrowed(rule, s.Tools), nil); err != nil {
return refused(name, StepNarrow, err)
}
}
if err := s.Forge.SendJSON(ctx, http.MethodPost, repoAPI(s.Admin, name)+"/transfer",
map[string]any{"new_owner": s.Org}, nil); err != nil {
return refused(name, StepMove, err)
}
if _, err := s.putRight(ctx, s.Org, name); err != nil {
return refused(name, StepRuleAfterMove, err)
}
return Result{Name: name, Did: DidMade}
}
// firstCommit writes main's first commit: the README and the repository's own check (ADR 0428 §4).
func (s *Syncer) firstCommit(ctx context.Context, name string, r Repo) error {
files := FirstCommit(name, r)
paths := make([]string, 0, len(files))
for p := range files {
paths = append(paths, p)
}
sort.Strings(paths)
var ops []map[string]any
for _, p := range paths {
ops = append(ops, map[string]any{"operation": "create", "path": p,
"content": base64.StdEncoding.EncodeToString([]byte(files[p]))})
}
return s.Forge.SendJSON(ctx, http.MethodPost, repoAPI(s.Admin, name)+"/contents", map[string]any{
"branch": MainBranch, "message": "Begin " + name + " with its README and its own check (ADR 0428)", "files": ops,
}, nil)
}
// rule is a branch protection rule as the forge answers it; only the fields read here are typed.
type rule map[string]any
func ruleNamed(rules []rule, name string) rule {
for _, r := range rules {
if ruleName(r) == name {
return r
}
}
return nil
}
func ruleName(r rule) string {
if n, _ := r["rule_name"].(string); n != "" {
return n
}
n, _ := r["branch_name"].(string)
return n
}
func strList(v any) []string {
var out []string
if l, ok := v.([]any); ok {
for _, s := range l {
if t, ok := s.(string); ok && strings.TrimSpace(t) != "" {
out = append(out, t)
}
}
}
return out
}
// statuses is what a rule requires, with RequiredStatuses added and none dropped.
func statuses(rules ...rule) []string {
seen := map[string]bool{}
var out []string
add := func(s string) {
if !seen[s] {
seen[s] = true
out = append(out, s)
}
}
for _, r := range rules {
if r != nil && r["enable_status_check"] == true {
for _, s := range strList(r["status_check_contexts"]) {
add(s)
}
}
}
for _, s := range RequiredStatuses {
add(s)
}
return out
}
// narrowed is main's rule of ADR 0398 §2 (protection.ts mainRuleBody): no push, only the tools' account merges,
// requiring the statuses (ADR 0428 §5). A field the forge keeps beside these is left as it is.
func narrowed(r rule, toolsAccount string, also ...rule) map[string]any {
return map[string]any{
"enable_push": false, "enable_push_whitelist": false, "push_whitelist_usernames": []string{},
"enable_merge_whitelist": true, "merge_whitelist_usernames": []string{toolsAccount},
"enable_status_check": true, "status_check_contexts": statuses(append([]rule{r}, also...)...),
}
}
// isRight says whether a rule already is narrowed and requires both statuses.
func isRight(r rule, toolsAccount string) bool {
if r["enable_push"] != false || r["enable_push_whitelist"] == true || r["enable_merge_whitelist"] != true {
return false
}
if m := strList(r["merge_whitelist_usernames"]); len(m) != 1 || m[0] != toolsAccount {
return false
}
return r["enable_status_check"] == true && hasAll(strList(r["status_check_contexts"]))
}
func hasAll(have []string) bool {
for _, want := range RequiredStatuses {
found := false
for _, h := range have {
found = found || h == want
}
if !found {
return false
}
}
return true
}
// putRight makes main's rule in an organisation's repository the narrowed one, keeping every status it requires.
// A repository with no rule named main gets one, carrying what a glob rule covering main asks (client.ts
// ensureMainMerges, issue 948). It never takes a status away and never unprotects (ADR 0428 §5, §7).
func (s *Syncer) putRight(ctx context.Context, owner, name string) (bool, error) {
rules, err := s.rules(ctx, owner, name)
if err != nil {
return false, err
}
if own := ruleNamed(rules, MainBranch); own != nil {
if isRight(own, s.Tools) {
return false, nil
}
return true, s.Forge.SendJSON(ctx, http.MethodPatch, repoAPI(owner, name)+"/branch_protections/"+MainBranch,
narrowed(own, s.Tools), nil)
}
var covering []rule
for _, r := range rules {
if ok, _ := path.Match(ruleName(r), MainBranch); ok {
covering = append(covering, r)
}
}
body := narrowed(nil, s.Tools, covering...)
body["rule_name"] = MainBranch
approvals := 0.0
for _, r := range covering {
if n, _ := r["required_approvals"].(float64); n > approvals {
approvals = n
}
if r["block_admin_merge_override"] == true {
body["block_admin_merge_override"] = true
}
}
if approvals > 0 {
body["required_approvals"] = approvals
}
return true, s.Forge.SendJSON(ctx, http.MethodPost, repoAPI(owner, name)+"/branch_protections", body, nil)
}
func (s *Syncer) rules(ctx context.Context, owner, name string) ([]rule, error) {
var out []rule
if err := s.Forge.GetJSON(ctx, repoAPI(owner, name)+"/branch_protections", &out); err != nil {
return nil, err
}
return out, nil
}
func (s *Syncer) branchExists(ctx context.Context, owner, name, branch string) (bool, error) {
err := s.Forge.GetJSON(ctx, repoAPI(owner, name)+"/branches/"+url.PathEscape(branch), nil)
if errors.Is(err, forgehttp.ErrNotFound) {
return false, nil
}
return err == nil, err
}
@@ -0,0 +1,480 @@
package main
import (
"bytes"
"encoding/json"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
"time"
)
// The tests of issue 1124 / ADR 0428: the organisation's repositories are the gitea module's config value
// `repositories`, made and protected by the setup login, never by a tool.
const mainRule = "main"
// 1124/AC2: a listed repository that does not exist is made under the setup account, given main's rule letting that
// login alone push, given its first commit, narrowed to no push and the tools' merge requiring mesh/repo-check (and
// mesh/merge-gate, 1124/AC5), and only then moved into the organisation. Nothing reaches the organisation before the
// narrowed rule, and nothing else is pushed to main.
func TestNewRepositoryIsMadeInOrder(t *testing.T) {
f := newFake()
s, _ := syncer(t, f)
o := run(t, s, Value{"mesh-delivery": {Private: false, Description: "the mesh's delivery"}})
want := []string{
"POST /api/v1/user/repos",
"POST /api/v1/repos/mesh-admin/mesh-delivery/branch_protections",
"POST /api/v1/repos/mesh-admin/mesh-delivery/contents",
"PATCH /api/v1/repos/mesh-admin/mesh-delivery/branch_protections/main",
"POST /api/v1/repos/mesh-admin/mesh-delivery/transfer",
"PATCH /api/v1/repos/novox/mesh-delivery/branch_protections/main",
}
if got := f.writes(); strings.Join(got, "\n") != strings.Join(want, "\n") {
t.Fatalf("writes, in order:\n%s\nwant:\n%s", strings.Join(got, "\n"), strings.Join(want, "\n"))
}
// Nothing names the organisation's copy before the narrowed rule but the read that found it missing.
narrowed := false
for _, c := range f.all() {
if strings.HasPrefix(c, "PATCH ") {
narrowed = true
}
if strings.Contains(c, "/repos/novox/mesh-delivery") && !narrowed && c != "GET /api/v1/repos/novox/mesh-delivery" {
t.Errorf("%s before the narrowed rule", c)
}
}
repo := f.repos["novox/mesh-delivery"]
if repo == nil || f.repos["mesh-admin/mesh-delivery"] != nil {
t.Fatalf("not moved into the organisation: %v", f.repos)
}
if repo.Private || repo.Description != "the mesh's delivery" {
t.Errorf("made as %+v, want public with its description", repo)
}
if repo.Commits != 1 || len(repo.Files) != 2 || repo.Files["README.md"] == "" || repo.Files["merge-check.sh"] == "" {
t.Errorf("first commit: %d commits, files %v", repo.Commits, keysOf(repo.Files))
}
if !strings.Contains(repo.Files["README.md"], "the mesh's delivery") {
t.Errorf("the README does not name its purpose: %q", repo.Files["README.md"])
}
rule := repo.Rules[mainRule]
if rule["enable_push"] != false || rule["enable_push_whitelist"] == true || len(strs(rule["push_whitelist_usernames"])) != 0 {
t.Errorf("main still takes a push: %v", rule)
}
if rule["enable_merge_whitelist"] != true || strings.Join(strs(rule["merge_whitelist_usernames"]), ",") != tools {
t.Errorf("main's merge whitelist is not the tools' account alone: %v", rule)
}
checks := strs(rule["status_check_contexts"])
if rule["enable_status_check"] != true || !has(checks, "mesh/repo-check") || !has(checks, "mesh/merge-gate") {
t.Errorf("main does not require mesh/repo-check and mesh/merge-gate (1124/AC5): %v", rule)
}
if r := result(o, "mesh-delivery"); r.Did != DidMade || r.Step != "" {
t.Errorf("outcome %+v, want made", r)
}
// A second pass changes nothing.
before := len(f.writes())
o = run(t, s, Value{"mesh-delivery": {Private: false, Description: "the mesh's delivery"}})
if got := f.writes(); len(got) != before {
t.Errorf("a second pass wrote %v", got[before:])
}
if r := result(o, "mesh-delivery"); r.Did != DidUnchanged {
t.Errorf("second outcome %+v, want unchanged", r)
}
}
// 1124/AC2: the forge drops from a merge whitelist a user without access to the repository, and under the setup
// account the tools' account has none; so once the repository is in the organisation its main rule is read back and
// put right: the tools' account the only one that merges, both statuses required.
func TestOrganisationRuleListsTheToolsAfterTheMove(t *testing.T) {
f := newFake()
s, _ := syncer(t, f)
o := run(t, s, Value{"mesh-delivery": {}})
repo := f.repos["novox/mesh-delivery"]
if repo == nil {
t.Fatal("not moved into the organisation")
}
rule := repo.Rules[mainRule]
if rule["enable_merge_whitelist"] != true || strings.Join(strs(rule["merge_whitelist_usernames"]), ",") != tools {
t.Errorf("after the move main's merge whitelist is %v, want %s alone", rule["merge_whitelist_usernames"], tools)
}
if rule["enable_push"] != false {
t.Errorf("after the move main takes a push: %v", rule)
}
checks := strs(rule["status_check_contexts"])
if rule["enable_status_check"] != true || !has(checks, "mesh/repo-check") || !has(checks, "mesh/merge-gate") {
t.Errorf("after the move main does not require both statuses: %v", rule)
}
if r := result(o, "mesh-delivery"); r.Did != DidMade {
t.Errorf("outcome %+v, want made", r)
}
}
// 1124/AC2: the setup login's rule lets that login alone push, and it is what the first commit lands through.
func TestAdminRuleLetsTheSetupLoginAlonePush(t *testing.T) {
f := newFake()
var seen map[string]any
f.refuse = func(method, path string) (int, string) {
if method == "POST" && strings.HasSuffix(path, "/contents") {
seen = map[string]any{}
for k, v := range f.repos["mesh-admin/x"].Rules[mainRule] {
seen[k] = v
}
}
return 0, ""
}
s, _ := syncer(t, f)
run(t, s, Value{"x": {Private: true}})
if seen["enable_push"] != true || seen["enable_push_whitelist"] != true || strings.Join(strs(seen["push_whitelist_usernames"]), ",") != admin {
t.Fatalf("the rule at the first commit was %v, want a push by %s alone", seen, admin)
}
if !f.repos["novox/x"].Private {
t.Error("made public, want private")
}
}
// 1124/AC3: a listed repository that exists in the organisation is not made again, only its main rule put right;
// a rule still letting the setup login push is narrowed, its statuses kept (1124/AC5).
func TestExistingOrganisationRepositoryIsOnlyPutRight(t *testing.T) {
f := newFake()
hq := f.add("novox/hq", map[string]map[string]any{mainRule: {
"enable_push": true, "enable_push_whitelist": true, "push_whitelist_usernames": []any{admin},
"enable_status_check": true, "status_check_contexts": []any{"other/x", "mesh/merge-gate"},
}}, "main")
bare := f.add("novox/bare", nil, "main")
s, _ := syncer(t, f)
o := run(t, s, Value{"hq": {}, "bare": {}})
for _, c := range f.writes() {
if strings.HasSuffix(c, "/user/repos") || strings.HasSuffix(c, "/contents") || strings.HasSuffix(c, "/transfer") {
t.Errorf("an existing repository got %s", c)
}
}
rule := hq.Rules[mainRule]
if rule["enable_push"] != false || strings.Join(strs(rule["merge_whitelist_usernames"]), ",") != tools {
t.Errorf("hq's rule not narrowed: %v", rule)
}
checks := strs(rule["status_check_contexts"])
for _, c := range []string{"other/x", "mesh/merge-gate", "mesh/repo-check"} {
if !has(checks, c) {
t.Errorf("hq's rule lacks %s: %v", c, checks)
}
}
made := bare.Rules[mainRule]
if made == nil || made["enable_push"] != false || !has(strs(made["status_check_contexts"]), "mesh/repo-check") ||
!has(strs(made["status_check_contexts"]), "mesh/merge-gate") {
t.Errorf("bare's rule: %v", made)
}
if hq.Commits != 0 || bare.Commits != 0 {
t.Error("something was pushed to an existing main")
}
if r := result(o, "hq"); r.Did != DidPutRight {
t.Errorf("hq outcome %+v", r)
}
}
// 1124/AC3: a repository found still under the setup account is finished: with main already written it is not
// written again; without a rule it gets the steps it lacks.
func TestRepositoryUnderTheSetupAccountIsFinished(t *testing.T) {
f := newFake()
f.add("mesh-admin/half", map[string]map[string]any{mainRule: {
"enable_push": true, "enable_push_whitelist": true, "push_whitelist_usernames": []any{admin},
}}, "main")
f.add("mesh-admin/empty", nil)
s, _ := syncer(t, f)
o := run(t, s, Value{"half": {}, "empty": {}})
got := strings.Join(f.writes(), "\n")
want := strings.Join([]string{
"POST /api/v1/repos/mesh-admin/empty/branch_protections",
"POST /api/v1/repos/mesh-admin/empty/contents",
"PATCH /api/v1/repos/mesh-admin/empty/branch_protections/main",
"POST /api/v1/repos/mesh-admin/empty/transfer",
"PATCH /api/v1/repos/novox/empty/branch_protections/main",
"PATCH /api/v1/repos/mesh-admin/half/branch_protections/main",
"POST /api/v1/repos/mesh-admin/half/transfer",
"PATCH /api/v1/repos/novox/half/branch_protections/main",
}, "\n")
if got != want {
t.Fatalf("writes:\n%s\nwant:\n%s", got, want)
}
for _, n := range []string{"half", "empty"} {
repo := f.repos["novox/"+n]
if repo == nil || repo.Rules[mainRule]["enable_push"] != false {
t.Errorf("%s not finished: %+v", n, repo)
}
if r := result(o, n); r.Did != DidMade {
t.Errorf("%s outcome %+v", n, r)
}
}
if f.repos["novox/half"].Commits != 0 {
t.Error("main of half was written again")
}
}
// 1124/AC5: putting a rule right adds a missing one of mesh/repo-check and mesh/merge-gate and keeps every status
// already there; no pass drops one.
func TestStatusesAreAddedAndNeverDropped(t *testing.T) {
f := newFake()
only := f.add("novox/only", map[string]map[string]any{mainRule: {
"enable_push": false, "enable_merge_whitelist": true, "merge_whitelist_usernames": []any{tools},
"enable_status_check": true, "status_check_contexts": []any{"mesh/repo-check"},
}}, "main")
more := f.add("novox/more", map[string]map[string]any{mainRule: {
"enable_push": false, "enable_merge_whitelist": true, "merge_whitelist_usernames": []any{tools},
"enable_status_check": true, "status_check_contexts": []any{"mesh/merge-gate", "mesh/repo-check", "mesh/delivery"},
}}, "main")
s, _ := syncer(t, f)
o := run(t, s, Value{"only": {}, "more": {}})
if c := strs(only.Rules[mainRule]["status_check_contexts"]); !has(c, "mesh/merge-gate") || !has(c, "mesh/repo-check") {
t.Errorf("only: %v", c)
}
if c := strs(more.Rules[mainRule]["status_check_contexts"]); len(c) != 3 || !has(c, "mesh/delivery") {
t.Errorf("more lost a status: %v", c)
}
for _, w := range f.writes() {
if strings.Contains(w, "/more/") {
t.Errorf("a right rule was written: %s", w)
}
}
if result(o, "more").Did != DidUnchanged || result(o, "only").Did != DidPutRight {
t.Errorf("outcome %+v", o)
}
}
// 1124/AC6: a refused step is said with the repository, the step and the forge's answer, nothing further of that
// repository is sent, and the next listed repository is still made; the pass exits 1, its stderr naming them.
func TestRefusedStepStopsThatRepositoryOnly(t *testing.T) {
steps := map[string]func(method, path string) bool{
StepCreate: func(m, p string) bool { return m == "POST" && p == "/api/v1/user/repos" },
StepAdminRule: func(m, p string) bool { return m == "POST" && strings.HasSuffix(p, "/a/branch_protections") },
StepFirstCommit: func(m, p string) bool { return m == "POST" && strings.HasSuffix(p, "/a/contents") },
StepNarrow: func(m, p string) bool { return m == "PATCH" && strings.Contains(p, "/a/branch_protections/") },
StepMove: func(m, p string) bool { return m == "POST" && strings.HasSuffix(p, "/a/transfer") },
StepRuleAfterMove: func(m, p string) bool {
return m == "PATCH" && strings.HasPrefix(p, "/api/v1/repos/novox/a/branch_protections/")
},
}
for step, match := range steps {
t.Run(step, func(t *testing.T) {
f := newFake()
refused := false
f.refuse = func(method, path string) (int, string) {
if refused {
return 0, ""
}
// The create of a is refused, not b's.
if step == StepCreate && match(method, path) && !strings.Contains(strings.Join(f.calls, "\n"), "/b") {
refused = true
return 403, "the forge says no to " + step
}
if step != StepCreate && match(method, path) {
refused = true
return 403, "the forge says no to " + step
}
return 0, ""
}
s, _ := syncer(t, f)
o := run(t, s, Value{"a": {}, "b": {}})
r := result(o, "a")
if r.Did != DidRefused || r.Step != step || !strings.Contains(r.Said, "the forge says no to "+step) {
t.Fatalf("a: %+v, want refused at %s with the forge's answer", r, step)
}
// Nothing of a after the refused call.
calls := f.all()
at := -1
for i, c := range calls {
m := strings.SplitN(c, " ", 2)
if match(m[0], m[1]) {
at = i
break
}
}
for _, c := range calls[at+1:] {
if strings.Contains(c, "/a/") || strings.HasSuffix(c, "/a") {
t.Errorf("after the refused %s: %s", step, c)
}
}
if f.repos["novox/b"] == nil || result(o, "b").Did != DidMade {
t.Errorf("b was not made after a's refusal: %+v", result(o, "b"))
}
var stderr bytes.Buffer
if code := Report(o, &stderr); code == 0 {
t.Errorf("a refused %s exits 0", step)
}
lines := strings.Split(strings.TrimRight(stderr.String(), "\n"), "\n")
if want := "repositories: a: " + step + " refused: "; len(lines) != 1 || !strings.HasPrefix(lines[0], want) ||
!strings.Contains(lines[0], "the forge says no to "+step) {
t.Errorf("stderr %q, want one line %q<the forge's answer>", stderr.String(), want)
}
})
}
}
// 1124/AC7: a repository taken out of the value gets no call at all.
func TestRepositoryOutOfTheValueIsNotTouched(t *testing.T) {
f := newFake()
f.add("novox/old", map[string]map[string]any{mainRule: {"enable_push": true}}, "main")
f.add("mesh-admin/older", nil)
s, _ := syncer(t, f)
run(t, s, Value{})
run(t, s, Value{"new": {}})
for _, c := range f.all() {
if strings.Contains(c, "old") {
t.Errorf("a repository out of the value was asked: %s", c)
}
}
if f.repos["novox/old"].Rules[mainRule]["enable_push"] != true || f.repos["mesh-admin/older"] == nil {
t.Error("a repository out of the value was changed")
}
}
// 1124/AC6: the pass's exit is the node-engine's gate on a run-once step (ADR 0136): 0 when every repository was
// made, put right or unchanged and nothing on stderr; 1 with one line per refused repository, then a summary line
// naming them all, when several were; 1 naming why when the pass did not run.
func TestReportIsTheExitAndTheStderrLines(t *testing.T) {
now := time.Now()
var stderr bytes.Buffer
ok := Outcome{At: now, Repositories: []Result{{Name: "hq", Did: DidUnchanged}, {Name: "x", Did: DidMade}, {Name: "y", Did: DidPutRight}}}
if code := Report(ok, &stderr); code != 0 || stderr.Len() != 0 {
t.Errorf("all made: exit %d, stderr %q", code, stderr.String())
}
stderr.Reset()
two := Outcome{At: now, Repositories: []Result{
{Name: "a", Did: DidRefused, Step: StepNarrow, Said: "403 Forbidden: user does not have permission"},
{Name: "b", Did: DidMade},
{Name: "c", Did: DidRefused, Step: StepMove, Said: "422: cannot transfer"},
}}
if code := Report(two, &stderr); code != 1 {
t.Errorf("two refused: exit %d", code)
}
lines := strings.Split(strings.TrimRight(stderr.String(), "\n"), "\n")
want := []string{
"repositories: a: narrow refused: 403 Forbidden: user does not have permission",
"repositories: c: move refused: 422: cannot transfer",
}
if len(lines) != 3 || lines[0] != want[0] || lines[1] != want[1] {
t.Fatalf("stderr:\n%s\nwant the lines:\n%s\nthen a summary", stderr.String(), strings.Join(want, "\n"))
}
if !strings.Contains(lines[2], "2 of 3") || !strings.Contains(lines[2], "a, c") {
t.Errorf("summary %q, want the count and the names", lines[2])
}
stderr.Reset()
if code := Report(Outcome{At: now, Failed: "the forge is not reachable"}, &stderr); code != 1 ||
!strings.Contains(stderr.String(), "the forge is not reachable") {
t.Errorf("a pass that did not run: exit %d, stderr %q", code, stderr.String())
}
}
// 1124/AC6: the outcome is written whole, and read back whole.
func TestOutcomeRoundTrip(t *testing.T) {
p := filepath.Join(t.TempDir(), "outcome.json")
o := Outcome{At: time.Now().UTC().Truncate(time.Second), Repositories: []Result{{Name: "a", Did: DidRefused, Step: StepMove, Said: "no"}}}
if err := WriteOutcome(p, o); err != nil {
t.Fatal(err)
}
got, err := ReadOutcome(p)
if err != nil || got.Repositories[0] != o.Repositories[0] || !got.At.Equal(o.At) {
t.Fatalf("%+v %v", got, err)
}
}
// 1124/AC1: the value is read from the file the mesh renders; a missing key is the empty value; an unknown field
// or a name that is no repository name is refused.
func TestReadValue(t *testing.T) {
p := filepath.Join(t.TempDir(), "r.json")
_ = os.WriteFile(p, []byte(`{"repositories":{"mesh-delivery":{"private":true,"description":"d"}}}`), 0o600)
v, err := ReadValue(p)
if err != nil || !v["mesh-delivery"].Private || v["mesh-delivery"].Description != "d" {
t.Fatalf("%v %v", v, err)
}
_ = os.WriteFile(p, []byte(`{}`), 0o600)
if v, err := ReadValue(p); err != nil || len(v) != 0 {
t.Fatalf("%v %v", v, err)
}
_ = os.WriteFile(p, []byte(`{"repositories":{"a":{"owner":"novox"}}}`), 0o600)
if _, err := ReadValue(p); err == nil {
t.Fatal("an unknown field was accepted")
}
for _, bad := range []string{"..", "a/b", "", "a b", "-x"} {
raw, _ := json.Marshal(map[string]any{"repositories": map[string]any{bad: map[string]any{}}})
_ = os.WriteFile(p, raw, 0o600)
if _, err := ReadValue(p); err == nil {
t.Errorf("%q was accepted as a repository name", bad)
}
}
}
// 1124/AC4: the first commit's merge-check.sh declares its toolchain in its first lines, runs the tests of a Go
// module in the tree, and on a tree with none says so and passes.
func TestFirstCommitMergeCheck(t *testing.T) {
files := FirstCommit("mesh-delivery", Repo{Description: "the mesh's delivery"})
script := files["merge-check.sh"]
lines := strings.SplitN(script, "\n", 4)
if lines[0] != "#!/bin/sh" || lines[1] != "# mesh-check-toolchain: go" {
t.Fatalf("first lines %q", lines[:2])
}
if !strings.Contains(files["README.md"], "mesh-delivery") || !strings.Contains(files["README.md"], "the mesh's delivery") {
t.Errorf("README %q", files["README.md"])
}
goBin, err := exec.LookPath("go")
if err != nil {
t.Skip("no go on PATH to run the script with")
}
tree := t.TempDir()
mod := filepath.Join(tree, "sub")
_ = os.MkdirAll(mod, 0o755)
_ = os.WriteFile(filepath.Join(tree, "merge-check.sh"), []byte(script), 0o755)
_ = os.WriteFile(filepath.Join(mod, "go.mod"), []byte("module tiny\n\ngo 1.21\n"), 0o644)
_ = os.WriteFile(filepath.Join(mod, "tiny_test.go"), []byte(`package tiny
import (
"os"
"testing"
)
func TestRan(t *testing.T) { _ = os.WriteFile("ran", []byte("yes"), 0o644) }
`), 0o644)
cmd := exec.Command("sh", "merge-check.sh")
cmd.Dir = tree
cmd.Env = append(os.Environ(), "GOFLAGS=", "GOTOOLCHAIN=local", "PATH="+filepath.Dir(goBin)+":"+os.Getenv("PATH"))
out, err := cmd.CombinedOutput()
if err != nil {
t.Fatalf("on a tree with a Go module: %v\n%s", err, out)
}
if _, err := os.Stat(filepath.Join(mod, "ran")); err != nil || !strings.Contains(string(out), "integration") {
t.Fatalf("its tests did not run, or not with -tags integration:\n%s", out)
}
// A failing test fails the check.
_ = os.WriteFile(filepath.Join(mod, "fail_test.go"), []byte("package tiny\n\nimport \"testing\"\n\nfunc TestFails(t *testing.T) { t.Fatal(\"no\") }\n"), 0o644)
if out, err := cmd2(tree, cmd.Env).CombinedOutput(); err == nil {
t.Fatalf("a failing test passed the check:\n%s", out)
}
empty := t.TempDir()
_ = os.WriteFile(filepath.Join(empty, "merge-check.sh"), []byte(script), 0o755)
out, err = cmd2(empty, cmd.Env).CombinedOutput()
if err != nil || !strings.Contains(string(out), "no Go module") {
t.Fatalf("on an empty tree: %v\n%s", err, out)
}
}
func cmd2(dir string, env []string) *exec.Cmd {
c := exec.Command("sh", "merge-check.sh")
c.Dir, c.Env = dir, env
return c
}
func keysOf(m map[string]string) []string {
out := []string{}
for k := range m {
out = append(out, k)
}
return out
}
+1
View File
@@ -0,0 +1 @@
{}
+50
View File
@@ -169,6 +169,12 @@
"description": "The read teams of the agents' forge accounts (ADR 0365 clause 3): each team by name, {\"<team>\": {\"repos\": [\"<owner>/<repo>\", ...], \"members\": [\"<node>\", ...]}}. The forge gets a team agent-read-<team> that reads only, with exactly those repositories and the accounts agent-<node>; a team taken out of the value is emptied. Nothing else on the forge is touched; an agent account is in no team until it is listed here.",
"trusted": true,
"default": {}
},
"repositories": {
"type": "object",
"description": "The repositories of the organisation novox the mesh holds (ADR 0428): each by name, {\"<name>\": {\"private\": true, \"description\": \"...\"}}. One that does not exist is made by the module's setup login under its own account, its main protected and given a first commit (a README naming the description and a merge-check.sh), its rule narrowed to no push and the tools' merge requiring mesh/repo-check and mesh/merge-gate, and only then moved into the organisation; one that exists only has main's rule put right, its statuses kept. A repository taken out of the value is never touched.",
"trusted": true,
"default": {}
}
},
"facts": {
@@ -311,6 +317,17 @@
"agent-read-teams"
]
},
{
"id": "repositories-file",
"type": "file",
"path": "${dir:mesh-state}/repositories.json",
"mode": "0600",
"template": "files/repositories.json",
"merge": "json",
"takes-config": [
"repositories"
]
},
{
"id": "consumers",
"type": "file",
@@ -336,6 +353,28 @@
"restart-on": [
"agent-teams-file"
]
},
{
"id": "repositories",
"type": "process",
"name": "gitea-repositories",
"artifact": "repositories",
"run": [
"./repositories",
"sync"
],
"run-once": true,
"env": {
"MESH_GITEA_URL": "http://127.0.0.1:${port:3000}",
"MESH_GITEA_ADMIN_USER": "mesh-admin",
"MESH_GITEA_ADMIN_PASSWORD_FILE": "${dir:state}/admin.secret",
"MESH_GITEA_TOOLS_USER": "mesh-tools",
"MESH_REPOSITORIES_FILE": "${dir:mesh-state}/repositories.json",
"MESH_REPOSITORIES_OUTCOME": "${dir:runtime-state}/repositories-outcome.json"
},
"restart-on": [
"repositories-file"
]
}
],
"provides": [
@@ -460,6 +499,17 @@
"agent-teams"
]
},
{
"name": "repositories",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/repositories",
"binary": "repositories",
"loads": [
"repositories"
]
},
{
"name": "forge-account-provider",
"kind": "bundle",
-117
View File
@@ -1,117 +0,0 @@
// A repository is created where it is asked for (novox/hq issue 1124): gitea_create_repo created under the tools'
// own user alone, so a repository of the novox organisation could not be made through it. With `owner` it is created
// in that organisation (POST /orgs/{org}/repos), the forge's refusal said with its status and its reason; without
// it, under the tools' own user as before.
import { test, after, beforeEach } from "node:test";
import assert from "node:assert/strict";
import { createServer, type IncomingMessage, type ServerResponse } from "node:http";
import { GiteaClient } from "../dist/client.js";
import { ConfiguredToken } from "../dist/token.js";
import { getGiteaTools } from "../dist/tools/index.js";
interface Sent { method: string; path: string; body: any }
const state = {
sent: [] as Sent[],
// When set, what the forge answers a create in an organisation: its status and its message.
refuse: null as { status: number; message: string } | null,
};
function json(res: ServerResponse, status: number, body: unknown) {
res.statusCode = status;
res.setHeader("Content-Type", "application/json");
res.end(JSON.stringify(body));
}
const server = createServer(async (req: IncomingMessage, res: ServerResponse) => {
const chunks: Buffer[] = [];
for await (const c of req) chunks.push(c as Buffer);
const raw = Buffer.concat(chunks).toString();
const path = new URL(req.url ?? "", "http://x").pathname.replace(/^\/api\/v1/, "");
const method = req.method ?? "GET";
state.sent.push({ method, path, body: raw ? JSON.parse(raw) : undefined });
const created = (owner: string) => {
const body = raw ? JSON.parse(raw) : {};
return json(res, 201, { full_name: `${owner}/${body.name}`, name: body.name, owner: { login: owner },
private: body.private, description: body.description, html_url: `http://forge/${owner}/${body.name}`,
clone_url: `http://forge/${owner}/${body.name}.git`, default_branch: "main" });
};
if (method === "POST" && path === "/user/repos") return created("agent");
const org = /^\/orgs\/([^/]+)\/repos$/.exec(path);
if (method === "POST" && org) {
if (state.refuse) return json(res, state.refuse.status, { message: state.refuse.message, url: "http://forge/api/swagger" });
return created(decodeURIComponent(org[1]));
}
return json(res, 404, { message: "not found" });
});
await new Promise<void>((r) => server.listen(0, "127.0.0.1", () => r()));
after(() => new Promise<void>((r) => server.close(() => r())));
const { port } = server.address() as { port: number };
const client = new GiteaClient(`http://127.0.0.1:${port}`, new ConfiguredToken("t"));
const tool = (name: string) => getGiteaTools(client).find((t) => t.name === name)!;
const body = { name: "mesh-delivery", description: "the delivery's own repository", private: true, auto_init: true };
beforeEach(() => {
state.sent = [];
state.refuse = null;
});
test("with an organisation the repository is created there, with the same body, and answered (1124/AC1)", async () => {
// Fails before the fix: createRepo takes no owner and posts to /user/repos, so the repository is the tools' user's.
const repo = await client.createRepo({ ...body }, "novox");
assert.deepEqual(state.sent, [{ method: "POST", path: "/orgs/novox/repos", body }]);
assert.equal(repo.full_name, "novox/mesh-delivery");
assert.equal(repo.owner, "novox");
// The tool passes its owner through; before the fix it has no owner argument and posts to /user/repos.
state.sent = [];
const answered = await tool("gitea_create_repo").run({ owner: "novox", ...body }) as { repo: { full_name: string } };
assert.deepEqual(state.sent, [{ method: "POST", path: "/orgs/novox/repos", body }]);
assert.equal(answered.repo.full_name, "novox/mesh-delivery");
assert.match(tool("gitea_create_repo").description, /organisation/, "the description says where it creates");
});
test("without an owner the repository is created under the tools' own user, as before (1124/AC2)", async () => {
// Holds before the fix: it is the behaviour the fix must leave unchanged.
const repo = await client.createRepo({ ...body });
assert.deepEqual(state.sent, [{ method: "POST", path: "/user/repos", body }]);
assert.equal(repo.full_name, "agent/mesh-delivery");
state.sent = [];
await tool("gitea_create_repo").run({ name: "mesh-delivery" });
assert.deepEqual(state.sent, [{ method: "POST", path: "/user/repos", body: { name: "mesh-delivery", private: true, auto_init: true } }]);
});
test("the forge's refusal to create in an organisation is said with its status, its reason and the organisation (1124/AC3)", async () => {
// Fails before the fix: the create goes to /user/repos, which the fake forge answers 201, so nothing is refused.
for (const [status, message] of [
[403, "user should be an owner or a member of the organisation"],
[404, "GetOrgByName: org does not exist [name: novox]"],
[409, "The repository with the same name already exists."],
] as const) {
state.refuse = { status, message };
for (const run of [
() => client.createRepo({ ...body }, "novox"),
() => tool("gitea_create_repo").run({ owner: "novox", ...body }),
]) {
await assert.rejects(run(), (e: Error) => {
assert.match(e.message, new RegExp(`\\b${status}\\b`), `the status ${status} is said`);
assert.ok(e.message.includes(message), `the forge's reason is said: ${e.message}`);
assert.match(e.message, /organisation novox/, "the organisation is named");
return true;
});
}
}
});
test("an owner that is not a name is refused before anything is sent (1124/AC4)", async () => {
// Holds before the fix too: namesChecked (tools/index.ts) checks args.owner for every tool. It stays so the guard
// keeps covering this tool's new argument.
for (const owner of ["..", "novox/../agent", "", "no vox", "%2e%2e", "-x"]) {
await assert.rejects(tool("gitea_create_repo").run({ owner, name: "mesh-delivery" }), /not a repository owner/,
JSON.stringify(owner));
}
assert.deepEqual(state.sent, [], "nothing reached the forge");
});
+56
View File
@@ -0,0 +1,56 @@
// No tool and no agent creates a repository in an organisation (novox/hq issue 1124, ADR 0428 §8): the forge makes
// whoever creates an organisation's repository its administrator, and an administrator can take its protection off.
// So gitea_create_repo creates only under the tools' own user, with no owner argument; the tools' team keeps
// can_create_org_repo false; and the tools' token has no write:organization. The organisation's repositories are
// made by the module's setup login from its config value `repositories` (cmd/repositories).
import { test, after } from "node:test";
import assert from "node:assert/strict";
import { createServer, type IncomingMessage, type ServerResponse } from "node:http";
import { GiteaClient } from "../dist/client.js";
import { ConfiguredToken, TOKEN_SCOPES } from "../dist/token.js";
import { getGiteaTools } from "../dist/tools/index.js";
import { TOOLS_TEAM_UNITS } from "../dist/tools-account.js";
const sent: { method: string; path: string; body: any }[] = [];
const server = createServer(async (req: IncomingMessage, res: ServerResponse) => {
const chunks: Buffer[] = [];
for await (const c of req) chunks.push(c as Buffer);
const raw = Buffer.concat(chunks).toString();
const path = new URL(req.url ?? "", "http://x").pathname.replace(/^\/api\/v1/, "");
const body = raw ? JSON.parse(raw) : undefined;
sent.push({ method: req.method ?? "GET", path, body });
res.setHeader("Content-Type", "application/json");
if (req.method === "POST" && path === "/user/repos") {
res.statusCode = 201;
res.end(JSON.stringify({ full_name: `mesh-tools/${body.name}`, name: body.name, owner: { login: "mesh-tools" },
private: body.private, html_url: "", clone_url: "", default_branch: "main" }));
return;
}
res.statusCode = 404;
res.end(JSON.stringify({ message: "not found" }));
});
await new Promise<void>((r) => server.listen(0, "127.0.0.1", () => r()));
after(() => new Promise<void>((r) => server.close(() => r())));
const { port } = server.address() as { port: number };
const client = new GiteaClient(`http://127.0.0.1:${port}`, new ConfiguredToken("t"));
const tool = getGiteaTools(client).find((t) => t.name === "gitea_create_repo")!;
test("gitea_create_repo has no owner argument and posts only to the tools' own user's repositories (1124/AC8)", async () => {
assert.equal("owner" in tool.input, false, "the tool takes no owner");
assert.equal(GiteaClient.prototype.createRepo.length, 1, "createRepo takes the repository alone, no owner");
// An owner given anyway is not an argument of the tool: the repository is the tools' user's all the same.
await tool.run({ owner: "novox", name: "mesh-delivery" });
await client.createRepo({ name: "x" });
assert.deepEqual(sent.map((s) => `${s.method} ${s.path}`), ["POST /user/repos", "POST /user/repos"]);
assert.ok(sent.every((s) => !s.path.startsWith("/orgs/")), "nothing is sent to an organisation");
assert.doesNotMatch(tool.description, /organisation/i, "the description offers no organisation");
});
test("the tools' team cannot create in the organisation and its token holds no write:organization (1124/AC8)", () => {
assert.equal(TOOLS_TEAM_UNITS.can_create_org_repo, false);
assert.ok(!TOKEN_SCOPES.includes("write:organization"), `scopes ${TOKEN_SCOPES.join(", ")}`);
assert.ok(TOKEN_SCOPES.every((s) => !/organization|admin/.test(s)), `scopes ${TOKEN_SCOPES.join(", ")}`);
});
+3 -4
View File
@@ -82,10 +82,9 @@ function allGiteaTools(gitea: GiteaClient, admin?: () => GiteaAdmin): ToolDefini
},
{
name: "gitea_create_repo",
description: "Create a repository: with owner, in that organisation (the forge's refusal said with its status " +
"and reason); without owner, under the tools' own user.",
description: "Create a repository under the tools' own user. A repository of the mesh's own is made by the gitea " +
"module from its config value repositories, never by a tool (novox/hq ADR 0428 §8).",
input: {
owner: { type: "string", description: "an organisation to create the repository in (default: the tools' own user)" },
name: { type: "string", description: "the repository name" },
description: { type: "string", description: "an optional description" },
private: { type: "boolean", description: "private repo (default true)" },
@@ -97,7 +96,7 @@ function allGiteaTools(gitea: GiteaClient, admin?: () => GiteaAdmin): ToolDefini
description: args.description ? String(args.description) : undefined,
private: args.private === undefined ? true : Boolean(args.private),
auto_init: args.auto_init === undefined ? true : Boolean(args.auto_init),
}, args.owner === undefined ? undefined : String(args.owner));
});
return { repo };
},
},