Join: merge A3 (wip/registry-a3)
Conflicts, both sides kept: artifact-store-tools' invokes (the assigner's and indexer's read tools, followed through views, beside the registry seats' hold, release and remove), its main (the views reader and the seat asker), its manifest row; gitea's invokes (the indexer's read tool, the hand act and the builder's versions) and its consumes of the indexer's views. gitea's account test takes the packages account's names; repoCheckFacts, removed on main, is not imported. No catalogue module reads the indexer's state. Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
This commit is contained in:
+33
-10
@@ -1,8 +1,8 @@
|
||||
# gitea
|
||||
|
||||
The forge, and the mesh's npm package registry: it claims the `git` and `npm-package-registry` seats.
|
||||
Its tools, its events and the provisioner of registry accounts are the TypeScript bundle (`index.ts`,
|
||||
`tools/`, `provisioner/`). This README covers the two Go bundles beside it.
|
||||
Its tools and its events are the TypeScript bundle (`index.ts`,
|
||||
`tools/`). This README covers the two Go bundles beside it.
|
||||
|
||||
**What a registry consumer is told** (`serves.npm-package-registry`): where the registry answers on the
|
||||
private network (`scheme`, `port`, `npm-path`), and its `public-root`, the address the forge writes every
|
||||
@@ -11,6 +11,11 @@ tarball at: `https://` and the forge's route's composed public name, under the n
|
||||
locked tarball only from the registry it is configured with, so a consumer that installs from lock files
|
||||
trusts that root. On a machine with no public domain the key is left out, and the plan says so.
|
||||
|
||||
**Both package provisions carry no credential** (`credential.form` none, design 51 *Publishing*): reads of the
|
||||
organisation's packages are anonymous, and every write goes through the seats' verbs, so no consumer is given an
|
||||
account of the forge. The registry accounts the TypeScript pair provisioner made before stay on the forge until
|
||||
removed by hand; nothing hands their passwords out any more.
|
||||
|
||||
## Announcing pull requests to the merge check (novox/hq to-be 45 §9, issue 347)
|
||||
|
||||
Every new head of an open pull request is announced as `pull.updated`, once, and the controller has it checked.
|
||||
@@ -91,7 +96,7 @@ is read once and searched in memory; a binary file is passed over, a file over 4
|
||||
`skipped`, and at most `max` matches (default 100) are answered, with `truncated` when more were found. A
|
||||
search that reaches the call's deadline stops there and answers what it found, with `stopped_at_deadline`.
|
||||
|
||||
It reaches the forge through the same HTTP client as the npm-registry bundle (`internal/forgehttp`), as the
|
||||
It reaches the forge through the same HTTP client as the package-registry bundle (`internal/forgehttp`), as the
|
||||
tools' own account (novox/hq ADR 0365 §2, issue 469): the token the TypeScript bundle mints for `mesh-tools` and
|
||||
keeps (`MESH_GITEA_TOKEN_FILE`), sent only when its scopes line names that account, never the admin login. One tool
|
||||
call takes at most 50 seconds.
|
||||
@@ -181,16 +186,28 @@ No tool creates in the organisation: `gitea_create_repo` creates under the tools
|
||||
keeps `can_create_org_repo` false and its token has no `write:organization` (ADR 0428 §8).
|
||||
`go test ./cmd/repositories` runs it against a fake forge.
|
||||
|
||||
## The two package seats (novox/hq design 51, ADR 0605, ADR 0611)
|
||||
|
||||
The forge module holds `npm-package-registry` and `go-package-registry`, and the Go bundle `cmd/package-registry`
|
||||
serves both seats' verbs as tools `<seat>.<verb>`: `publish` and `publish-part` (a package's one file in chunks into
|
||||
the module's own object store `incoming`, its transfers in the state `transfers`; mesh-sdk's seat-neutral receiving
|
||||
side), `versions` and `remove` (a tagged version, by the builder). Before anything is uploaded the holder asks the
|
||||
builder whether it numbered the version with that digest; a package goes up as npm's publish sends it, a Go module's
|
||||
zip to the Go upload. Every outcome is said on the seat's events (`published`, `refused`, `removed`), a Go package's
|
||||
with its module path. Who may call each verb is the trusted config value `callers` (empty: design 51's list).
|
||||
|
||||
## The package registry's retention (novox/hq ADR 0251 §4, to-be 51)
|
||||
|
||||
Every publish adds a version to the registry, and until this nothing removed one. The Go bundle
|
||||
`cmd/npm-registry` says which versions each package holds, what keeps each one, and — asked with a why —
|
||||
`cmd/package-registry` says which versions each package holds, what keeps each one, and — asked with a why —
|
||||
deletes the versions nothing keeps.
|
||||
|
||||
| tool | | what |
|
||||
|---|---|---|
|
||||
| `npm_packages` | r | every package of the registry's owner, each version newest first with its publish time, its size and why it is kept |
|
||||
| `npm_retention` | a | the versions retention would delete, and the bytes. **A dry run unless `dry_run` is false**; a real run needs `why` |
|
||||
| `go_packages` | r | every Go module of the registry's owner (the Go seat's), each version newest first with its publish time, its size and why it is kept: a `go.mod` on a default branch requires it, or it is among the newest `keep` |
|
||||
| `go_retention` | a | the Go module versions retention would delete, and the bytes; as `npm_retention`, and a `go.mod` that could not be read stops a real run. Each deletion says `removed` on the Go seat |
|
||||
|
||||
**A version is kept when:**
|
||||
|
||||
@@ -223,11 +240,18 @@ version already gone counts as deleted.
|
||||
|
||||
### How it reaches the forge
|
||||
|
||||
Over the forge's HTTP interface on the machine (`MESH_GITEA_URL`), as the admin account the vault
|
||||
delivered (`MESH_GITEA_ADMIN_USER`, the password in `MESH_GITEA_ADMIN_PASSWORD_FILE` — the same file
|
||||
the TypeScript bundle mints its token with), by basic authentication: this bundle mints no token and
|
||||
keeps nothing. The password is read per call and never logged, answered or put in an error. The
|
||||
registry's owner is `MESH_NPM_OWNER`, the owner in the seat's `npm-path`.
|
||||
Over the forge's HTTP interface on the machine (`MESH_GITEA_URL`), never as the admin account (design 51: each
|
||||
holder keeps its write side as an own secret beside a read-only tool token). It **reads** as the tools' account,
|
||||
with the token the TypeScript bundle keeps for it (`MESH_GITEA_TOKEN_FILE`, scopes including `read:package`), and
|
||||
**writes** (a publish, a remove, retention's deletion) as the packages account `mesh-packages`
|
||||
(`MESH_GITEA_PACKAGES_USER`), with its own token (`MESH_GITEA_PACKAGES_TOKEN_FILE`, the one scope `write:package`).
|
||||
That account is plain, a member of the team `mesh-packages` that writes the organisation's packages and nothing else;
|
||||
the watcher makes it with the administrator's login and mints and keeps its token at its start (`packages-account.ts`).
|
||||
No timer looks at it after that: when the forge refuses a write (401) this bundle raises the finding
|
||||
`packages-token-refused` (key `seat.gitea.packages-token-refused_write` in the module's `findings` state), the watcher,
|
||||
watching that key, mints the token afresh and clears it; a write the forge takes clears it too. This bundle mints nothing; a token is
|
||||
read per call and never logged, answered or put in an error. The registry's owner is `MESH_NPM_OWNER`, the owner in
|
||||
the seat's `npm-path`.
|
||||
|
||||
Calls to the forge run eight at a time, and one tool call reads for at most 50 seconds, inside a
|
||||
module's 60-second ask.
|
||||
@@ -347,5 +371,4 @@ Each directory or archive this module declares says how it treats a file it did
|
||||
- `${dir:mesh-state}/state`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
|
||||
- `/var/lib/mesh/forge-patches`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
|
||||
- `${dir:state}`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
|
||||
- `${dir:grants}`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
|
||||
- `${dir:data}`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
|
||||
|
||||
@@ -207,7 +207,8 @@ export class GiteaClient {
|
||||
/**
|
||||
* Build from the module's resolved environment. The URL comes from MESH_GITEA_URL (the mesh's own
|
||||
* name), falling back to the bare GITEA_URL and to the forge's loopback port. The token, in order:
|
||||
* one configured in the environment (MESH_GITEA_TOKEN / GITEA_TOKEN), which wins — never a setting, which is
|
||||
* one configured in the environment for a hand-run instance (GITEA_TOKEN; nothing on the mesh sets it, so no
|
||||
* MESH_* word names it: checks/environment), which wins — never a setting, which is
|
||||
* ignored and said (hq issue 342); else
|
||||
* one the module mints for itself with the admin account the vault delivered and keeps in its own
|
||||
* state (token.ts; hq issue 100). Throws only when neither is possible, naming what is missing,
|
||||
@@ -217,7 +218,7 @@ export class GiteaClient {
|
||||
const cfg = meshConfig(env.MESH_GITEA_CONFIG_FILE);
|
||||
const url = cfg.url ?? env.MESH_GITEA_URL ?? env.GITEA_URL ?? `http://127.0.0.1:${env.GITEA_PORT ?? "3000"}`;
|
||||
ignoreSecretSetting(cfg, "token");
|
||||
const configured = env.MESH_GITEA_TOKEN ?? env.GITEA_TOKEN;
|
||||
const configured = env.GITEA_TOKEN;
|
||||
if (configured) return new GiteaClient(url, new ConfiguredToken(configured));
|
||||
return new GiteaClient(url, MintedToken.fromEnv(url, env));
|
||||
}
|
||||
|
||||
@@ -17,7 +17,7 @@ import (
|
||||
stdio "git.novox.be/novox/mesh-sdk/go"
|
||||
)
|
||||
|
||||
// ToolNames are the tools this bundle serves. The module's manifest lists no `tools` (see the npm-registry bundle
|
||||
// ToolNames are the tools this bundle serves. The module's manifest lists no `tools` (see the package-registry bundle
|
||||
// for why); the README names this one, and a test holds the three together.
|
||||
var ToolNames = []string{ToolName}
|
||||
|
||||
|
||||
@@ -12,7 +12,7 @@ import (
|
||||
)
|
||||
|
||||
// The forge as this bundle reaches it: its own HTTP interface on the machine, as the tools' own account (novox/hq
|
||||
// ADR 0365 §2; internal/forgehttp, shared with the npm-registry bundle beside it).
|
||||
// ADR 0365 §2; internal/forgehttp, shared with the package-registry bundle beside it).
|
||||
|
||||
// Forge is the forge's HTTP interface.
|
||||
type Forge struct {
|
||||
|
||||
@@ -648,7 +648,13 @@ func TestToolsAreTheOnesTheReadmeNamesAndTheManifestBuilds(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var m struct {
|
||||
Tools []string `json:"tools"`
|
||||
Tools []string `json:"tools"`
|
||||
Serves map[string]map[string]any `json:"serves"`
|
||||
Provides []struct {
|
||||
Name string `json:"name"`
|
||||
Scope string `json:"scope"`
|
||||
Credential map[string]string `json:"credential"`
|
||||
} `json:"provides"`
|
||||
Build struct {
|
||||
Artifacts []struct {
|
||||
From string `json:"from"`
|
||||
@@ -664,28 +670,52 @@ func TestToolsAreTheOnesTheReadmeNamesAndTheManifestBuilds(t *testing.T) {
|
||||
if len(m.Tools) != 0 {
|
||||
t.Error("the manifest lists tools: its claims would offer them as both seats' verbs")
|
||||
}
|
||||
found := false
|
||||
// CAT-G2b (ADR 0346 §6, the operator 2026-11-04): the forge keeps serving the npm registry's public root, so a lock
|
||||
// file naming it installs as today. CAT-G2 (design 51, E2): the Go seat's read side is the provision
|
||||
// go-package-registry, mesh scope, read with no credential, offered only by this build.
|
||||
if root, _ := m.Serves["npm-package-registry"]["public-root"].(string); !strings.HasPrefix(root, "https://") || !strings.HasSuffix(root, "/api/packages/novox/npm/") {
|
||||
t.Errorf("npm-package-registry serves public-root %q: the public root must stay", root)
|
||||
}
|
||||
if g := m.Serves["go-package-registry"]; g["go-path"] != "/api/packages/novox/go" {
|
||||
t.Errorf("go-package-registry serves %v, want the forge's Go proxy path", g)
|
||||
}
|
||||
goProvided := false
|
||||
for _, p := range m.Provides {
|
||||
if p.Name == "go-package-registry" {
|
||||
goProvided = p.Scope == "mesh" && p.Credential["form"] == "none"
|
||||
}
|
||||
}
|
||||
if !goProvided {
|
||||
t.Error("go-package-registry is not provided at mesh scope with no credential")
|
||||
}
|
||||
// The tools' own account, never the admin login (novox/hq ADR 0365 §2); the package registry's bundle also writes
|
||||
// as the packages account (design 51: the write side an own secret beside a read-only tool token).
|
||||
wants := map[string][]string{
|
||||
"cmd/forge": {"MESH_GITEA_URL", "MESH_GITEA_TOOLS_USER", "MESH_GITEA_TOKEN_FILE"},
|
||||
"cmd/package-registry": {"MESH_GITEA_URL", "MESH_GITEA_TOOLS_USER", "MESH_GITEA_TOKEN_FILE", "MESH_GITEA_PACKAGES_USER", "MESH_GITEA_PACKAGES_TOKEN_FILE"},
|
||||
}
|
||||
for _, a := range m.Build.Artifacts {
|
||||
if a.From == "cmd/forge" {
|
||||
found = true
|
||||
if len(a.Loads) != 1 || a.Loads[0] != a.Binary {
|
||||
t.Errorf("the bundle loads %v, not its binary %s", a.Loads, a.Binary)
|
||||
want, ok := wants[a.From]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
delete(wants, a.From)
|
||||
if len(a.Loads) != 1 || a.Loads[0] != a.Binary {
|
||||
t.Errorf("%s loads %v, not its binary %s", a.From, a.Loads, a.Binary)
|
||||
}
|
||||
for _, k := range want {
|
||||
if a.Env[k] == "" {
|
||||
t.Errorf("%s is not given %s", a.From, k)
|
||||
}
|
||||
// The tools' own account, never the admin login (novox/hq ADR 0365 §2).
|
||||
for _, k := range []string{"MESH_GITEA_URL", "MESH_GITEA_TOOLS_USER", "MESH_GITEA_TOKEN_FILE"} {
|
||||
if a.Env[k] == "" {
|
||||
t.Errorf("the bundle is not given %s", k)
|
||||
}
|
||||
}
|
||||
for _, k := range []string{"MESH_GITEA_ADMIN_USER", "MESH_GITEA_ADMIN_PASSWORD_FILE"} {
|
||||
if a.Env[k] != "" {
|
||||
t.Errorf("the bundle is given %s: it acts as the tools' account, not the administrator", k)
|
||||
}
|
||||
}
|
||||
for _, k := range []string{"MESH_GITEA_ADMIN_USER", "MESH_GITEA_ADMIN_PASSWORD_FILE"} {
|
||||
if a.Env[k] != "" {
|
||||
t.Errorf("%s is given %s: it acts as an account of its own, not the administrator", a.From, k)
|
||||
}
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Error("the manifest builds no cmd/forge bundle")
|
||||
for from := range wants {
|
||||
t.Errorf("the manifest builds no %s bundle", from)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
// forge: the forge module's Go bundle for an agent's commit and the reads around it (novox/hq issue 375).
|
||||
// A process the node's runtime launches and speaks MCP over stdio to, beside the module's TypeScript bundle
|
||||
// and its npm-registry bundle. gitea_commit puts one commit on a branch of its own — never the default branch,
|
||||
// and its package-registry bundle. gitea_commit puts one commit on a branch of its own — never the default branch,
|
||||
// never one a protection rule names (novox/hq ADR 0266) — and the reads answer a commit history, a compare, a
|
||||
// pull request's reviews with their line comments, a directory's entries and a code search, summarised.
|
||||
// stdout is the protocol; it says what it says on stderr, and never the forge's credential.
|
||||
@@ -16,7 +16,7 @@ import (
|
||||
stdio "git.novox.be/novox/mesh-sdk/go"
|
||||
)
|
||||
|
||||
// ToolNames are the tools this bundle serves. The module's manifest lists no `tools` (see the npm-registry
|
||||
// ToolNames are the tools this bundle serves. The module's manifest lists no `tools` (see the package-registry
|
||||
// bundle for why); the README names these, and a test holds the three together.
|
||||
var ToolNames = []string{"gitea_commit", "gitea_commits", "gitea_compare", "gitea_pull_reviews", "gitea_list_dir", "gitea_search_code"}
|
||||
|
||||
|
||||
+85
-12
@@ -13,20 +13,81 @@ import (
|
||||
"gitea/internal/forgehttp"
|
||||
)
|
||||
|
||||
// The forge as this bundle reaches it: its own HTTP interface on the machine, as the forge's admin account
|
||||
// (internal/forgehttp, shared with the forge bundle beside it). Basic authentication, so this bundle mints
|
||||
// nothing and keeps nothing: the TypeScript bundle's token is its own.
|
||||
// The forge as this bundle reaches it: its own HTTP interface on the machine (internal/forgehttp, shared with the
|
||||
// forge bundle beside it), never as the admin account. It reads as the tools' account, whose token carries
|
||||
// read:package, and writes (a publish, a remove, retention's deletion) as the packages account, whose token carries
|
||||
// write:package (design 51: each holder keeps its write side as an own secret beside a read-only tool token). The
|
||||
// TypeScript bundle mints and keeps both tokens (token.ts, packages-account.ts); this bundle mints nothing.
|
||||
|
||||
// Forge is the forge's HTTP interface.
|
||||
type Forge struct {
|
||||
forgehttp.Client
|
||||
Owner string // the package registry's owner, as the seat serves it
|
||||
forgehttp.Client // reads, as the tools' account
|
||||
// Writer writes into the package registry, as the packages account; nil writes nothing.
|
||||
Writer *forgehttp.Client
|
||||
Owner string // the package registry's owner, as the seat serves it
|
||||
// Type is the registry's package type retention reads: TypeNpm when empty, or TypeGo.
|
||||
Type string
|
||||
// Refused says whether the forge refused the write side's token (401), as the finding PackagesTokenRefused in the
|
||||
// module's findings state: the TypeScript bundle mints a new token on that key's event and clears it (no timer
|
||||
// looks at the token). Nil says nothing.
|
||||
Refused func(raised bool)
|
||||
}
|
||||
|
||||
// PackagesTokenRefused is the finding this bundle raises when the forge refuses the packages account's token, and its
|
||||
// key in the module's findings state (packages-account.ts PACKAGES_TOKEN_REFUSED watches that key).
|
||||
const (
|
||||
PackagesTokenRefused = "packages-token-refused"
|
||||
PackagesTokenRefusedKey = "seat.gitea.packages-token-refused_write"
|
||||
)
|
||||
|
||||
// wrote is a write's answer, said on Refused: a 401 raises the finding, a write the forge took clears it.
|
||||
func (f *Forge) wrote(err error) error {
|
||||
if f.Refused == nil {
|
||||
return err
|
||||
}
|
||||
var status *forgehttp.StatusError
|
||||
switch {
|
||||
case errors.As(err, &status) && status.Status == http.StatusUnauthorized:
|
||||
f.Refused(true)
|
||||
case err == nil:
|
||||
f.Refused(false)
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
// The package types the retention reads: npm packages (the npm seat) and Go modules (the Go seat).
|
||||
const (
|
||||
TypeNpm = "npm"
|
||||
TypeGo = "go"
|
||||
)
|
||||
|
||||
// typ is the package type this forge's retention reads.
|
||||
func (f *Forge) typ() string {
|
||||
if f.Type == "" {
|
||||
return TypeNpm
|
||||
}
|
||||
return f.Type
|
||||
}
|
||||
|
||||
// As is the same forge, its retention reading the package type typ.
|
||||
func (f *Forge) As(typ string) *Forge {
|
||||
out := *f
|
||||
out.Type = typ
|
||||
return &out
|
||||
}
|
||||
|
||||
// ForgeFromEnv is the forge as the manifest hands it to this bundle.
|
||||
func ForgeFromEnv() (*Forge, error) {
|
||||
c, missing := forgehttp.FromEnv()
|
||||
f := &Forge{Client: c, Owner: os.Getenv("MESH_NPM_OWNER")}
|
||||
w, writeMissing := forgehttp.WriterFromEnv()
|
||||
for _, m := range writeMissing {
|
||||
if m != "MESH_GITEA_URL" {
|
||||
missing = append(missing, m)
|
||||
}
|
||||
}
|
||||
f := &Forge{Client: c, Writer: &w, Owner: os.Getenv("MESH_NPM_OWNER"), Refused: func(raised bool) {
|
||||
finding(PackagesTokenRefused, "write", "the forge refused the package seats' write token; the module mints it afresh", raised)
|
||||
}}
|
||||
if f.Owner == "" {
|
||||
missing = append(missing, "MESH_NPM_OWNER")
|
||||
}
|
||||
@@ -56,12 +117,12 @@ type PackageVersion struct {
|
||||
|
||||
const pageSize = 50
|
||||
|
||||
// Packages is every npm package version of the owner, all pages.
|
||||
// Packages is every package version of the owner of this forge's type, all pages.
|
||||
func (f *Forge) Packages(ctx context.Context) ([]PackageVersion, error) {
|
||||
var all []PackageVersion
|
||||
for page := 1; page <= 1000; page++ {
|
||||
var batch []PackageVersion
|
||||
path := fmt.Sprintf("/api/v1/packages/%s?type=npm&page=%d&limit=%d", url.PathEscape(f.Owner), page, pageSize)
|
||||
path := fmt.Sprintf("/api/v1/packages/%s?type=%s&page=%d&limit=%d", url.PathEscape(f.Owner), f.typ(), page, pageSize)
|
||||
if err := f.GetJSON(ctx, path, &batch); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -74,7 +135,7 @@ func (f *Forge) Packages(ctx context.Context) ([]PackageVersion, error) {
|
||||
}
|
||||
|
||||
func (f *Forge) versionPath(name, version string) string {
|
||||
return fmt.Sprintf("/api/v1/packages/%s/npm/%s/%s", url.PathEscape(f.Owner), url.PathEscape(name), url.PathEscape(version))
|
||||
return f.packagePath(f.typ(), name, version)
|
||||
}
|
||||
|
||||
// Size is the bytes of one version's files.
|
||||
@@ -104,10 +165,22 @@ func (f *Forge) DistTags(ctx context.Context, name string) (map[string]string, e
|
||||
return meta.DistTags, nil
|
||||
}
|
||||
|
||||
// Delete removes one version from the registry.
|
||||
// write is the registry's write side, refused when the manifest handed none.
|
||||
func (f *Forge) write() (*forgehttp.Client, error) {
|
||||
if f.Writer == nil {
|
||||
return nil, errors.New("the package registry's write side is not handed to this bundle (MESH_GITEA_PACKAGES_TOKEN_FILE); nothing was written")
|
||||
}
|
||||
return f.Writer, nil
|
||||
}
|
||||
|
||||
// Delete removes one version from the registry, as the packages account.
|
||||
func (f *Forge) Delete(ctx context.Context, name, version string) error {
|
||||
_, err := f.do(ctx, http.MethodDelete, f.versionPath(name, version))
|
||||
return err
|
||||
w, err := f.write()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = w.Do(ctx, http.MethodDelete, f.versionPath(name, version), nil, maxBody)
|
||||
return f.wrote(err)
|
||||
}
|
||||
|
||||
// Repository is one repository on the forge, as far as reading its manifests needs.
|
||||
+45
-16
@@ -1,4 +1,5 @@
|
||||
// npm-registry: the forge module's Go bundle for its package registry (novox/hq ADR 0251 §4, to-be 51).
|
||||
// package-registry: the forge module's Go bundle for its package registry (novox/hq ADR 0251 §4, design 51): its
|
||||
// retention tools, and the verbs of the two package seats it holds (seats.go).
|
||||
// A process the node's runtime launches and speaks MCP over stdio to, beside the module's TypeScript
|
||||
// bundle. It says which versions of each npm package the registry holds, what keeps each one, and —
|
||||
// asked with a why — deletes the versions nothing keeps. stdout is the protocol; it says what it says on
|
||||
@@ -13,15 +14,19 @@ import (
|
||||
"strings"
|
||||
|
||||
stdio "git.novox.be/novox/mesh-sdk/go"
|
||||
|
||||
"gitea/internal/packageseats"
|
||||
)
|
||||
|
||||
// ToolNames are the tools this bundle serves. The module's manifest lists no `tools`: its TypeScript
|
||||
// bundle's tools are served without one, and a claim without `serves` would offer a `tools` list to both
|
||||
// of the module's seats as their verbs. The README names these two, and a test holds the two together.
|
||||
var ToolNames = []string{"npm_packages", "npm_retention"}
|
||||
var ToolNames = []string{"go_packages", "go_retention", "npm_packages", "npm_retention"}
|
||||
|
||||
func main() {
|
||||
if err := stdio.Serve("", Tools(func() (*Forge, error) { return ForgeFromEnv() })); err != nil {
|
||||
forge := func() (*Forge, error) { return ForgeFromEnv() }
|
||||
tools := append(Tools(forge), SeatTools(func() (*packageseats.Holder, error) { return busHolder(forge) })...)
|
||||
if err := stdio.Serve("", tools); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
@@ -30,17 +35,45 @@ func main() {
|
||||
// Tools are the bundle's tools over the forge the function gives; asked per call, so a manifest that
|
||||
// does not set the environment is a refusal naming what is missing, not a bundle that will not start.
|
||||
func Tools(forge func() (*Forge, error)) []stdio.Tool {
|
||||
return append(typeTools(TypeGo, forge), typeTools(TypeNpm, forge)...)
|
||||
}
|
||||
|
||||
// typeTools are one package type's two tools, <type>_packages and <type>_retention, over the forge reading that type.
|
||||
func typeTools(typ string, forgeOf func() (*Forge, error)) []stdio.Tool {
|
||||
forge := func() (*Forge, error) {
|
||||
f, err := forgeOf()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return f.As(typ), nil
|
||||
}
|
||||
pkg := map[string]any{"type": "string", "description": "one package, by its name (default every package)"}
|
||||
keep := map[string]any{"type": "integer", "description": fmt.Sprintf("how many of each package's newest versions are kept whatever names them (default %d, at least 1)", DefaultKeep)}
|
||||
listed := "Every npm package in the forge's package registry, each version newest first with when it was published, " +
|
||||
"its size and what keeps it: a lockfile on a repository's default branch that names it, the highest version " +
|
||||
"satisfying a range a package.json there names, a dist-tag, or being among the newest `keep` of its package " +
|
||||
"(novox/hq ADR 0251). Repositories and files that could not be read are named. Changes nothing. Replaces " +
|
||||
"npm view and npm dist-tag ls. (r)"
|
||||
deletes := "What retention would delete from the forge's package registry: every version nothing keeps — no lockfile " +
|
||||
"on a repository's default branch names it, it is not the highest version satisfying any range a package.json " +
|
||||
"there names, no dist-tag names it, and it is not among the newest `keep` of its package — with the bytes. " +
|
||||
"A dry run unless dry_run is false; a real run needs why, and deletes nothing at all when any repository or " +
|
||||
"file could not be read (novox/hq ADR 0251). Each deletion says removed on the npm seat. Replaces npm unpublish. (a)"
|
||||
if typ == TypeGo {
|
||||
listed = "Every Go module in the forge's package registry (the Go seat's), each version newest first with when it " +
|
||||
"was published, its size and what keeps it: a go.mod on a repository's default branch that requires it, or " +
|
||||
"being among the newest `keep` of its module (design 51 *Retention*). Repositories and go.mod files that " +
|
||||
"could not be read are named. Changes nothing. Replaces go list -m -versions. (r)"
|
||||
deletes = "What retention would delete from the forge's Go registry: every module version no go.mod on a " +
|
||||
"repository's default branch requires and not among the newest `keep` of its module, with the bytes. A dry " +
|
||||
"run unless dry_run is false; a real run needs why, and deletes nothing at all when any repository or go.mod " +
|
||||
"could not be read. Each deletion says removed on the Go seat. (a)"
|
||||
}
|
||||
return []stdio.Tool{
|
||||
{
|
||||
Name: "npm_packages",
|
||||
Description: "Every npm package in the forge's package registry, each version newest first with when it was published, " +
|
||||
"its size and what keeps it: a lockfile on a repository's default branch that names it, the highest version " +
|
||||
"satisfying a range a package.json there names, a dist-tag, or being among the newest `keep` of its package " +
|
||||
"(novox/hq ADR 0251). Repositories and files that could not be read are named. Changes nothing. Replaces " +
|
||||
"npm view and npm dist-tag ls. (r)",
|
||||
Input: map[string]any{"package": pkg, "keep": keep},
|
||||
Name: typ + "_packages",
|
||||
Description: listed,
|
||||
Input: map[string]any{"package": pkg, "keep": keep},
|
||||
Run: func(args map[string]any) (any, error) {
|
||||
f, err := forge()
|
||||
if err != nil {
|
||||
@@ -70,12 +103,8 @@ func Tools(forge func() (*Forge, error)) []stdio.Tool {
|
||||
},
|
||||
},
|
||||
{
|
||||
Name: "npm_retention",
|
||||
Description: "What retention would delete from the forge's package registry: every version nothing keeps — no lockfile " +
|
||||
"on a repository's default branch names it, it is not the highest version satisfying any range a package.json " +
|
||||
"there names, no dist-tag names it, and it is not among the newest `keep` of its package — with the bytes. " +
|
||||
"A dry run unless dry_run is false; a real run needs why, and deletes nothing at all when any repository or " +
|
||||
"file could not be read (novox/hq ADR 0251). Replaces npm unpublish. (a)",
|
||||
Name: typ + "_retention",
|
||||
Description: deletes,
|
||||
Input: map[string]any{
|
||||
"package": pkg,
|
||||
"keep": keep,
|
||||
+64
-4
@@ -7,6 +7,7 @@ import (
|
||||
"fmt"
|
||||
"path"
|
||||
"regexp"
|
||||
"slices"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
@@ -36,18 +37,77 @@ var (
|
||||
"pnpm-lock.yaml": true}
|
||||
)
|
||||
|
||||
// interesting is whether a path is a file this reads; nothing under node_modules, which is what a
|
||||
// lockfile already says and is not the repository's own.
|
||||
func interesting(p string) bool {
|
||||
// goModName is the file a Go module's requirements are read from.
|
||||
const goModName = "go.mod"
|
||||
|
||||
// interesting is whether a path is a file this reads for packages of the type typ: an npm manifest or lockfile, or
|
||||
// a go.mod; nothing under node_modules or vendor, which is what a lockfile or go.mod already says and is not the
|
||||
// repository's own.
|
||||
func interesting(p, typ string) bool {
|
||||
for _, part := range strings.Split(p, "/") {
|
||||
if part == "node_modules" {
|
||||
if part == "node_modules" || part == "vendor" {
|
||||
return false
|
||||
}
|
||||
}
|
||||
base := path.Base(p)
|
||||
if typ == TypeGo {
|
||||
return base == goModName
|
||||
}
|
||||
return manifestNames[base] || lockNames[base]
|
||||
}
|
||||
|
||||
// ReadGoMod is every registry module version a go.mod requires, or names on the right of a replace (design 51
|
||||
// *Retention*: a Go version is referenced when a go.mod on a repository's default branch requires it). A line it
|
||||
// cannot read, or a block left open, is an error beside what was read: the file is then not read whole, and a real
|
||||
// run stops before anything is deleted.
|
||||
func ReadGoMod(content []byte, where string, known map[string]bool) ([]Pinned, error) {
|
||||
var out []Pinned
|
||||
pin := func(module, version string) {
|
||||
if known[module] {
|
||||
out = append(out, Pinned{Package: module, Version: version, Where: where})
|
||||
}
|
||||
}
|
||||
block := ""
|
||||
for n, line := range strings.Split(string(content), "\n") {
|
||||
if i := strings.Index(line, "//"); i >= 0 {
|
||||
line = line[:i]
|
||||
}
|
||||
f := strings.Fields(line)
|
||||
if len(f) == 0 {
|
||||
continue
|
||||
}
|
||||
if block != "" {
|
||||
if f[0] == ")" {
|
||||
block = ""
|
||||
continue
|
||||
}
|
||||
f = append([]string{block}, f...)
|
||||
} else if len(f) == 2 && f[1] == "(" {
|
||||
block = f[0]
|
||||
continue
|
||||
}
|
||||
switch f[0] {
|
||||
case "require":
|
||||
if len(f) != 3 {
|
||||
return out, fmt.Errorf("line %d: a require is <module> <version>, not %q", n+1, strings.Join(f[1:], " "))
|
||||
}
|
||||
pin(f[1], f[2])
|
||||
case "replace":
|
||||
arrow := slices.Index(f, "=>")
|
||||
if arrow < 0 {
|
||||
return out, fmt.Errorf("line %d: a replace without =>", n+1)
|
||||
}
|
||||
if right := f[arrow+1:]; len(right) == 2 {
|
||||
pin(right[0], right[1])
|
||||
}
|
||||
}
|
||||
}
|
||||
if block != "" {
|
||||
return out, fmt.Errorf("its %s block is never closed", block)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
var dependencyFields = []string{"dependencies", "devDependencies", "peerDependencies", "optionalDependencies"}
|
||||
|
||||
// ReadManifest is every range a package.json names for a registry package. An `npm:` alias names the
|
||||
+1
-1
@@ -85,7 +85,7 @@ func TestAPnpmLockNamesWhatItPinsInEachFormat(t *testing.T) {
|
||||
func TestOnlyTheRepositorysOwnManifestsAreRead(t *testing.T) {
|
||||
for p, want := range map[string]bool{"package.json": true, "modules/x/package-lock.json": true, "yarn.lock": true,
|
||||
"pnpm-lock.yaml": true, "node_modules/a/package.json": false, "README.md": false, "a/package.json.bak": false} {
|
||||
if interesting(p) != want {
|
||||
if interesting(p, TypeNpm) != want {
|
||||
t.Errorf("%s: %v", p, !want)
|
||||
}
|
||||
}
|
||||
+35
-4
@@ -9,6 +9,8 @@ import (
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea/internal/packageseats"
|
||||
)
|
||||
|
||||
// Retention for the package registry (novox/hq ADR 0251 §4, to-be 51): a version is kept when a
|
||||
@@ -48,6 +50,7 @@ type Package struct {
|
||||
// Survey is the registry and what the forge's repositories say of it.
|
||||
type Survey struct {
|
||||
Owner string `json:"owner"`
|
||||
Type string `json:"type"`
|
||||
Keep int `json:"keep"`
|
||||
Repositories int `json:"repositories_read"`
|
||||
Unread []string `json:"repositories_unread,omitempty"`
|
||||
@@ -71,7 +74,7 @@ func survey(ctx context.Context, f *Forge, only string, keep int, sizes func(Ite
|
||||
}
|
||||
if only != "" {
|
||||
if _, has := byName[only]; !has {
|
||||
return nil, fmt.Errorf("the registry of %s holds no npm package %q", f.Owner, only)
|
||||
return nil, fmt.Errorf("the registry of %s holds no %s package %q", f.Owner, f.typ(), only)
|
||||
}
|
||||
}
|
||||
known := map[string]bool{}
|
||||
@@ -79,7 +82,7 @@ func survey(ctx context.Context, f *Forge, only string, keep int, sizes func(Ite
|
||||
known[name] = true
|
||||
}
|
||||
|
||||
s := &Survey{Owner: f.Owner, Keep: keep, Said: []string{}}
|
||||
s := &Survey{Owner: f.Owner, Type: f.typ(), Keep: keep, Said: []string{}}
|
||||
wanted, pinned, err := readRepositories(ctx, f, known, s)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
@@ -92,7 +95,10 @@ func survey(ctx context.Context, f *Forge, only string, keep int, sizes func(Ite
|
||||
}
|
||||
}
|
||||
sort.Strings(names)
|
||||
tags := distTags(ctx, f, names)
|
||||
tags := map[string]tagAnswer{} // a Go module has no dist-tags
|
||||
if f.typ() == TypeNpm {
|
||||
tags = distTags(ctx, f, names)
|
||||
}
|
||||
for _, name := range names {
|
||||
s.Packages = append(s.Packages, decide(name, byName[name], tags[name], wanted[name], pinned[name], keep))
|
||||
}
|
||||
@@ -167,7 +173,7 @@ func readRepositories(ctx context.Context, f *Forge, known map[string]bool, s *S
|
||||
var ps []Pinned
|
||||
var notRead, unread []string
|
||||
for _, p := range files {
|
||||
if !interesting(p) {
|
||||
if !interesting(p, f.typ()) {
|
||||
continue
|
||||
}
|
||||
where := r.FullName + ":" + p
|
||||
@@ -177,6 +183,14 @@ func readRepositories(ctx context.Context, f *Forge, known map[string]bool, s *S
|
||||
continue
|
||||
}
|
||||
base := path.Base(p)
|
||||
if base == goModName {
|
||||
got, err := ReadGoMod(content, where, known)
|
||||
ps = append(ps, got...)
|
||||
if err != nil {
|
||||
notRead = append(notRead, fmt.Sprintf("%s: not readable (%v)", where, err))
|
||||
}
|
||||
continue
|
||||
}
|
||||
if manifestNames[base] {
|
||||
got, err := ReadManifest(content, where, known)
|
||||
if err != nil {
|
||||
@@ -418,12 +432,29 @@ func retention(ctx context.Context, f *Forge, only string, keep int, dryRun bool
|
||||
}
|
||||
del, cancel := context.WithTimeout(context.Background(), budget)
|
||||
defer cancel()
|
||||
store := seatStore{forge: func() (*Forge, error) { return f, nil }}
|
||||
seat := packageseats.NpmSeat
|
||||
if f.typ() == TypeGo {
|
||||
seat = packageseats.GoSeat
|
||||
}
|
||||
for _, c := range p.Candidates {
|
||||
digest, _, _ := store.Held(seat, c.Package, c.Version)
|
||||
if err := f.Delete(del, c.Package, c.Version); err != nil && !errors.Is(err, errNotFound) {
|
||||
p.Refused = append(p.Refused, fmt.Sprintf("%s@%s: %v", c.Package, c.Version, err))
|
||||
continue
|
||||
}
|
||||
p.Deleted = append(p.Deleted, c.Package+"@"+c.Version)
|
||||
// The seat says every removal (design 51 *Retention*), so the builder's entry turns removed.
|
||||
// The seat's version is the builder's, without Go's leading v (forgeVersion adds it for the forge).
|
||||
o := packageseats.Outcome{Name: c.Package, Version: strings.TrimPrefix(c.Version, "v"), Kind: packageseats.KindPackage, Digest: digest, Reason: packageseats.ReasonRetention}
|
||||
if seat == packageseats.NpmSeat {
|
||||
o.Version = c.Version
|
||||
} else {
|
||||
o.Module = c.Package
|
||||
}
|
||||
if err := say(seat, packageseats.EventRemoved, o); err != nil {
|
||||
s.Said = append(s.Said, fmt.Sprintf("%s@%s is deleted but its removed could not be said: %v", c.Package, c.Version, err))
|
||||
}
|
||||
}
|
||||
s.Said = append(s.Said, fmt.Sprintf("deleted %d, refused %d, because: %s", len(p.Deleted), len(p.Refused), p.Why))
|
||||
return p, nil
|
||||
+102
-11
@@ -17,12 +17,14 @@ import (
|
||||
"time"
|
||||
|
||||
"gitea/internal/forgehttp"
|
||||
"gitea/internal/packageseats"
|
||||
)
|
||||
|
||||
// fakeForge answers the forge's interface from maps, and records every delete.
|
||||
type fakeForge struct {
|
||||
t *testing.T
|
||||
versions map[string][]string // package -> versions
|
||||
modules map[string][]string // Go module -> versions, listed as type go
|
||||
tags map[string]map[string]string // package -> dist-tags
|
||||
repos map[string]map[string]string // repository -> path -> content
|
||||
broken map[string]bool // repositories whose tree answers 500
|
||||
@@ -40,11 +42,12 @@ func (f *fakeForge) serve() *Forge {
|
||||
if err := os.WriteFile(file, []byte(f.password+"\n"), 0o600); err != nil {
|
||||
f.t.Fatal(err)
|
||||
}
|
||||
return &Forge{Client: forgehttp.Client{URL: srv.URL, User: "admin", PasswordFile: file}, Owner: "acme"}
|
||||
c := forgehttp.Client{URL: srv.URL, User: "admin", PasswordFile: file}
|
||||
return &Forge{Client: c, Writer: &c, Owner: "acme"}
|
||||
}
|
||||
|
||||
var (
|
||||
versionFiles = regexp.MustCompile(`^/api/v1/packages/acme/npm/([^/]+)/([^/]+)(/files)?$`)
|
||||
versionFiles = regexp.MustCompile(`^/api/v1/packages/acme/(?:npm|go)/([^/]+)/([^/]+)(/files)?$`)
|
||||
treePath = regexp.MustCompile(`^/api/v1/repos/([^/]+)/([^/]+)/git/trees/`)
|
||||
rawPath = regexp.MustCompile(`^/api/v1/repos/([^/]+)/([^/]+)/raw/(.+)$`)
|
||||
)
|
||||
@@ -62,10 +65,14 @@ func (f *fakeForge) handle(w http.ResponseWriter, r *http.Request) {
|
||||
switch {
|
||||
case p == "/api/v1/packages/acme" && r.Method == http.MethodGet:
|
||||
var all []map[string]any
|
||||
names := keys(f.versions)
|
||||
listed := f.versions
|
||||
if q.Get("type") == TypeGo {
|
||||
listed = f.modules
|
||||
}
|
||||
names := keys(listed)
|
||||
for _, n := range names {
|
||||
for i, v := range f.versions[n] {
|
||||
all = append(all, map[string]any{"name": n, "version": v, "type": "npm",
|
||||
for i, v := range listed[n] {
|
||||
all = append(all, map[string]any{"name": n, "version": v, "type": q.Get("type"),
|
||||
"created_at": time.Date(2026, 1, 1+i, 0, 0, 0, 0, time.UTC)})
|
||||
}
|
||||
}
|
||||
@@ -276,10 +283,19 @@ func TestARealRunWithoutWhyIsRefusedBeforeAnythingIsRead(t *testing.T) {
|
||||
|
||||
func TestARealRunDeletesOnlyWhatNothingKeeps(t *testing.T) {
|
||||
f := aRegistry(t)
|
||||
var removed []string
|
||||
say = func(seat, event string, o packageseats.Outcome) error {
|
||||
removed = append(removed, seat+" "+event+" "+o.Name+"@"+o.Version+" "+o.Reason)
|
||||
return nil
|
||||
}
|
||||
t.Cleanup(func() { say = sayOn })
|
||||
p, err := retention(context.Background(), f.serve(), "", 3, false, "the registry keeps what is named")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if sort.Strings(removed); len(removed) != 4 || removed[0] != "npm-package-registry removed @acme/sdk@0.1.1 retention" {
|
||||
t.Errorf("said %v", removed)
|
||||
}
|
||||
sort.Strings(f.deleted)
|
||||
if strings.Join(f.deleted, " ") != "@acme/sdk@0.1.1 @acme/sdk@0.1.2 @acme/sdk@0.2.0 @acme/sdk@0.2.1" {
|
||||
t.Fatalf("deleted %v", f.deleted)
|
||||
@@ -289,6 +305,48 @@ func TestARealRunDeletesOnlyWhatNothingKeeps(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestAVersionAGoModRequiresSurvivesGoRetention: the Go seat keeps what the npm seat keeps (design 51 *Retention*): a
|
||||
// version a go.mod on a default branch requires, or names on the right of a replace, and the newest keep; the others are
|
||||
// deleted and said removed on the Go seat, without Go's leading v; a go.mod that cannot be read stops a real run.
|
||||
func TestAVersionAGoModRequiresSurvivesGoRetention(t *testing.T) {
|
||||
f := aRegistry(t)
|
||||
f.modules = map[string][]string{"git.acme/sdk": {"v0.1.0", "v0.1.1", "v0.1.2", "v0.1.3", "v0.1.4"}}
|
||||
f.repos["acme/svc"] = map[string]string{
|
||||
"go.mod": "module git.acme/svc\n\ngo 1.26\n\nrequire (\n\tgit.acme/sdk v0.1.0 // indirect\n\tgolang.org/x/mod v0.20.0\n)\n",
|
||||
"tools/go.mod": "module git.acme/svc/tools\nrequire git.acme/other v1.0.0\nreplace git.acme/x => git.acme/sdk v0.1.2\n",
|
||||
"vendor/go.mod": "module broken\nrequire (\n",
|
||||
}
|
||||
var removed []string
|
||||
say = func(seat, event string, o packageseats.Outcome) error {
|
||||
removed = append(removed, seat+" "+event+" "+o.Module+"@"+o.Version+" "+o.Reason)
|
||||
return nil
|
||||
}
|
||||
t.Cleanup(func() { say = sayOn })
|
||||
p, err := retention(context.Background(), f.serve().As(TypeGo), "", 1, false, "tidy the Go registry")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
sort.Strings(f.deleted)
|
||||
if strings.Join(f.deleted, " ") != "git.acme/sdk@v0.1.1 git.acme/sdk@v0.1.3" {
|
||||
t.Errorf("deleted %v, want all but the required v0.1.0, the replaced-to v0.1.2 and the newest v0.1.4", f.deleted)
|
||||
}
|
||||
if sort.Strings(removed); strings.Join(removed, ",") != "go-package-registry removed git.acme/sdk@0.1.1 retention,go-package-registry removed git.acme/sdk@0.1.3 retention" {
|
||||
t.Errorf("said %v", removed)
|
||||
}
|
||||
if p.Type != TypeGo || len(p.Packages) != 1 {
|
||||
t.Errorf("surveyed %s %v, want the Go registry alone", p.Type, p.Packages)
|
||||
}
|
||||
// A go.mod outside vendor that cannot be read stops the real run before anything is deleted.
|
||||
f.deleted = nil
|
||||
f.repos["acme/svc"]["cmd/go.mod"] = "module git.acme/svc/cmd\nrequire (\n\tgit.acme/sdk v0.1.1\n"
|
||||
if _, err := retention(context.Background(), f.serve().As(TypeGo), "", 1, false, "tidy"); err == nil || !strings.Contains(err.Error(), "not read whole") {
|
||||
t.Errorf("an unreadable go.mod = %v, want the real run refused", err)
|
||||
}
|
||||
if len(f.deleted) != 0 {
|
||||
t.Errorf("deleted %v while a go.mod was unread", f.deleted)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnUnreadRepositoryStopsARealRunBeforeAnythingIsDeleted(t *testing.T) {
|
||||
f := aRegistry(t)
|
||||
f.broken = map[string]bool{"acme/site": true}
|
||||
@@ -349,7 +407,7 @@ func TestToolsAreTheOnesTheReadmeNamesAndSayWhatTheyDo(t *testing.T) {
|
||||
for _, tl := range tools {
|
||||
served = append(served, tl.Name)
|
||||
suffix := "(r)"
|
||||
if tl.Name == "npm_retention" {
|
||||
if strings.HasSuffix(tl.Name, "_retention") {
|
||||
suffix = "(a)"
|
||||
}
|
||||
if !strings.HasSuffix(tl.Description, suffix) {
|
||||
@@ -373,8 +431,14 @@ func TestToolsAreTheOnesTheReadmeNamesAndSayWhatTheyDo(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var m struct {
|
||||
Tools []string `json:"tools"`
|
||||
Build struct {
|
||||
Tools []string `json:"tools"`
|
||||
Claims []struct {
|
||||
Name string `json:"name"`
|
||||
Serves []string `json:"serves"`
|
||||
} `json:"claims"`
|
||||
Requires []any `json:"requires"`
|
||||
Invokes []string `json:"invokes"`
|
||||
Build struct {
|
||||
Artifacts []struct {
|
||||
From string `json:"from"`
|
||||
Loads []string `json:"loads"`
|
||||
@@ -385,14 +449,41 @@ func TestToolsAreTheOnesTheReadmeNamesAndSayWhatTheyDo(t *testing.T) {
|
||||
if err := json.Unmarshal(raw, &m); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Both package seats are claimed serving the four verbs this bundle answers as <seat>.<verb>, over its own incoming
|
||||
// and transfers, asking the builder's numbering (design 51 *Publishing*).
|
||||
claimed := map[string]string{}
|
||||
for _, c := range m.Claims {
|
||||
claimed[c.Name] = strings.Join(c.Serves, ",")
|
||||
}
|
||||
for _, seat := range []string{packageseats.NpmSeat, packageseats.GoSeat} {
|
||||
if claimed[seat] != strings.Join(packageseats.Verbs, ",") {
|
||||
t.Errorf("%s is claimed serving %q, want %v", seat, claimed[seat], packageseats.Verbs)
|
||||
}
|
||||
}
|
||||
names := map[string]bool{}
|
||||
for _, tool := range SeatTools(nil) {
|
||||
names[tool.Name] = true
|
||||
}
|
||||
if len(names) != 8 || !names[packageseats.GoSeat+".publish-part"] {
|
||||
t.Errorf("the seat tools are %v", names)
|
||||
}
|
||||
req, _ := json.Marshal(m.Requires)
|
||||
for _, want := range []string{`"name":"incoming"`, `"name":"transfers"`, `"name":"findings"`} {
|
||||
if !strings.Contains(string(req), want) {
|
||||
t.Errorf("the manifest does not require %s", want)
|
||||
}
|
||||
}
|
||||
if strings.Join(m.Invokes, ",") != "mesh-delivery-builder.builder_versions" {
|
||||
t.Errorf("the manifest invokes %v", m.Invokes)
|
||||
}
|
||||
if len(m.Tools) != 0 {
|
||||
t.Error("the manifest lists tools: its claims would offer them as both seats' verbs")
|
||||
}
|
||||
found := false
|
||||
for _, a := range m.Build.Artifacts {
|
||||
if a.From == "cmd/npm-registry" {
|
||||
if a.From == "cmd/package-registry" {
|
||||
found = true
|
||||
for _, k := range []string{"MESH_GITEA_URL", "MESH_GITEA_ADMIN_USER", "MESH_GITEA_ADMIN_PASSWORD_FILE", "MESH_NPM_OWNER"} {
|
||||
for _, k := range []string{"MESH_GITEA_URL", "MESH_GITEA_TOKEN_FILE", "MESH_GITEA_PACKAGES_TOKEN_FILE", "MESH_NPM_OWNER"} {
|
||||
if a.Env[k] == "" {
|
||||
t.Errorf("the bundle is not given %s", k)
|
||||
}
|
||||
@@ -400,7 +491,7 @@ func TestToolsAreTheOnesTheReadmeNamesAndSayWhatTheyDo(t *testing.T) {
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Error("the manifest builds no cmd/npm-registry bundle")
|
||||
t.Error("the manifest builds no cmd/package-registry bundle")
|
||||
}
|
||||
// Refused cleanly when the environment is missing, not a crash.
|
||||
if _, err := tools[0].Run(map[string]any{}); err == nil {
|
||||
@@ -0,0 +1,372 @@
|
||||
package main
|
||||
|
||||
// The two package seats' verbs (novox/hq design 51 *The four registry seats*, *One contract*): publish, publish-part,
|
||||
// versions and remove on `npm-package-registry` and `go-package-registry`, served as tools named <seat>.<verb> and
|
||||
// answered in each seat's own contract types (mesh-contracts npmpackageregistry, gopackageregistry; ADR 0620). The
|
||||
// holding itself is internal/packageseats; this file binds it to the forge's package registry, the runtime's bus state
|
||||
// and the seats' events.
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"archive/zip"
|
||||
"bytes"
|
||||
"compress/gzip"
|
||||
"context"
|
||||
"crypto/sha1"
|
||||
"crypto/sha512"
|
||||
"encoding/base64"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"path"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
goreg "git.novox.be/novox/mesh-contracts/go/seats/gopackageregistry"
|
||||
npmreg "git.novox.be/novox/mesh-contracts/go/seats/npmpackageregistry"
|
||||
stdio "git.novox.be/novox/mesh-sdk/go"
|
||||
"git.novox.be/novox/mesh-sdk/go/delivery"
|
||||
"git.novox.be/novox/mesh-sdk/go/health"
|
||||
"git.novox.be/novox/mesh-sdk/go/parts"
|
||||
|
||||
"gitea/internal/packageseats"
|
||||
)
|
||||
|
||||
// seatStore is the forge's package registry as both seats store into it.
|
||||
type seatStore struct{ forge func() (*Forge, error) }
|
||||
|
||||
func forgeType(seat string) string { return packageseats.Seats[seat] }
|
||||
|
||||
// forgeVersion is the version as the forge keeps it: a Go module's carries Go's leading v.
|
||||
func forgeVersion(seat, version string) string {
|
||||
if seat == packageseats.GoSeat && !strings.HasPrefix(version, "v") {
|
||||
return "v" + version
|
||||
}
|
||||
return version
|
||||
}
|
||||
|
||||
func (f *Forge) packagePath(typ, name, version string) string {
|
||||
return fmt.Sprintf("/api/v1/packages/%s/%s/%s/%s", url.PathEscape(f.Owner), typ, url.PathEscape(name), url.PathEscape(version))
|
||||
}
|
||||
|
||||
func (s seatStore) Held(seat, name, version string) (string, bool, error) {
|
||||
f, err := s.forge()
|
||||
if err != nil {
|
||||
return "", false, err
|
||||
}
|
||||
var files []struct {
|
||||
SHA256 string `json:"sha256"`
|
||||
}
|
||||
err = f.GetJSON(context.Background(), f.packagePath(forgeType(seat), name, forgeVersion(seat, version))+"/files", &files)
|
||||
switch {
|
||||
case errors.Is(err, errNotFound):
|
||||
return "", false, nil
|
||||
case err != nil:
|
||||
return "", false, err
|
||||
case len(files) == 0:
|
||||
return "", false, fmt.Errorf("%s@%s is listed with no file", name, version)
|
||||
}
|
||||
return "sha256:" + files[0].SHA256, true, nil
|
||||
}
|
||||
|
||||
func (s seatStore) Put(seat string, p packageseats.Publish, file []byte) (string, error) {
|
||||
f, err := s.forge()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
w, err := f.write()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
ctx := context.Background()
|
||||
if seat == packageseats.GoSeat {
|
||||
module, err := goModuleOf(file, forgeVersion(seat, p.Version))
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
_, err = w.Send(ctx, http.MethodPut, fmt.Sprintf("/api/packages/%s/go/upload", url.PathEscape(f.Owner)), file, "application/zip", maxBody)
|
||||
return module, f.wrote(err)
|
||||
}
|
||||
doc, err := npmDocument(f.URL, f.Owner, p, file)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
_, err = w.Do(ctx, http.MethodPut, fmt.Sprintf("/api/packages/%s/npm/%s", url.PathEscape(f.Owner), url.PathEscape(p.Name)), doc, maxBody)
|
||||
return "", f.wrote(err)
|
||||
}
|
||||
|
||||
func (s seatStore) Versions(seat, name string) ([]packageseats.Version, error) {
|
||||
f, err := s.forge()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var out []packageseats.Version
|
||||
for page := 1; page <= 1000; page++ {
|
||||
var batch []PackageVersion
|
||||
p := fmt.Sprintf("/api/v1/packages/%s?type=%s&q=%s&page=%d&limit=%d", url.PathEscape(f.Owner), forgeType(seat), url.QueryEscape(name), page, pageSize)
|
||||
if err := f.GetJSON(context.Background(), p, &batch); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, v := range batch {
|
||||
if v.Name != name {
|
||||
continue
|
||||
}
|
||||
d, _, err := s.Held(seat, name, v.Version)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, packageseats.Version{Version: strings.TrimPrefix(v.Version, "v"), Digest: d, Published: v.CreatedAt})
|
||||
}
|
||||
if len(batch) < pageSize {
|
||||
break
|
||||
}
|
||||
}
|
||||
// Newest first, by when each was published.
|
||||
for i := 1; i < len(out); i++ {
|
||||
for j := i; j > 0 && out[j].Published.After(out[j-1].Published); j-- {
|
||||
out[j], out[j-1] = out[j-1], out[j]
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (s seatStore) Remove(seat, name, version string) error {
|
||||
f, err := s.forge()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return f.As(forgeType(seat)).Delete(context.Background(), name, forgeVersion(seat, version))
|
||||
}
|
||||
|
||||
// goModuleOf is the Go module path of a module zip whose every file sits under <module>@<version>/, as Go's module
|
||||
// proxy protocol has it; a zip of another version is refused before the forge sees it.
|
||||
func goModuleOf(file []byte, version string) (string, error) {
|
||||
z, err := zip.NewReader(bytes.NewReader(file), int64(len(file)))
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("the Go package is not a module zip: %w", err)
|
||||
}
|
||||
if len(z.File) == 0 {
|
||||
return "", errors.New("the Go package's zip holds no file")
|
||||
}
|
||||
module, rest, at := strings.Cut(z.File[0].Name, "@")
|
||||
v, _, ok := strings.Cut(rest, "/")
|
||||
if !ok || !at || module == "" {
|
||||
return "", fmt.Errorf("the Go package's zip does not hold <module>@<version>/: %s", z.File[0].Name)
|
||||
}
|
||||
if v != version {
|
||||
return "", fmt.Errorf("the Go package's zip is %s, not %s", v, version)
|
||||
}
|
||||
return module, nil
|
||||
}
|
||||
|
||||
// npmDocument is the body npm's publish sends: the package.json of the tarball's package/, its dist, the tarball as
|
||||
// its one attachment, and the dist-tag: the version's tag for a tagged version, so that it never becomes latest.
|
||||
func npmDocument(base, owner string, p packageseats.Publish, tgz []byte) (map[string]any, error) {
|
||||
manifest, err := npmManifest(tgz)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if manifest["name"] != p.Name || manifest["version"] != p.Version {
|
||||
return nil, fmt.Errorf("the tarball is %v@%v, not %s@%s", manifest["name"], manifest["version"], p.Name, p.Version)
|
||||
}
|
||||
file := path.Base(p.Name) + "-" + p.Version + ".tgz"
|
||||
s1, s512 := sha1.Sum(tgz), sha512.Sum512(tgz)
|
||||
manifest["_id"] = p.Name + "@" + p.Version
|
||||
manifest["dist"] = map[string]any{"shasum": hex.EncodeToString(s1[:]), "integrity": "sha512-" + base64.StdEncoding.EncodeToString(s512[:]),
|
||||
"tarball": fmt.Sprintf("%s/api/packages/%s/npm/%s/-/%s/%s", base, owner, p.Name, p.Version, file)}
|
||||
tag := "latest"
|
||||
if p.Tag != "" {
|
||||
tag = p.Tag
|
||||
}
|
||||
return map[string]any{"_id": p.Name, "name": p.Name, "dist-tags": map[string]string{tag: p.Version},
|
||||
"versions": map[string]any{p.Version: manifest},
|
||||
"_attachments": map[string]any{file: map[string]any{"content_type": "application/octet-stream", "data": base64.StdEncoding.EncodeToString(tgz), "length": len(tgz)}}}, nil
|
||||
}
|
||||
|
||||
func npmManifest(tgz []byte) (map[string]any, error) {
|
||||
gz, err := gzip.NewReader(bytes.NewReader(tgz))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("the npm package is not a gzipped tarball: %w", err)
|
||||
}
|
||||
tr := tar.NewReader(gz)
|
||||
for {
|
||||
h, err := tr.Next()
|
||||
if err == io.EOF {
|
||||
return nil, errors.New("the npm package holds no package/package.json")
|
||||
}
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("the npm package's tarball cannot be read: %w", err)
|
||||
}
|
||||
if h.Name != "package/package.json" {
|
||||
continue
|
||||
}
|
||||
var m map[string]any
|
||||
if err := json.NewDecoder(io.LimitReader(tr, 1<<20)).Decode(&m); err != nil {
|
||||
return nil, fmt.Errorf("the npm package's package.json is not JSON: %w", err)
|
||||
}
|
||||
return m, nil
|
||||
}
|
||||
}
|
||||
|
||||
// as is v in T's shape: what crosses the bus is the seat's contract type, read or written through its JSON.
|
||||
func as[T any](v any) (T, error) {
|
||||
var out T
|
||||
raw, err := json.Marshal(v)
|
||||
if err == nil {
|
||||
err = json.Unmarshal(raw, &out)
|
||||
}
|
||||
return out, err
|
||||
}
|
||||
|
||||
// outcomeBody is an outcome as the seat's contract's Outcome names it.
|
||||
func outcomeBody(o packageseats.Outcome) map[string]any {
|
||||
return map[string]any{"name": o.Name, "version": o.Version, "kind": o.Kind, "tag": o.Tag, "digest": o.Digest,
|
||||
"fingerprint": o.Fingerprint, "commit": o.Commit, "holder": packageseats.Self, "transfer": o.Transfer,
|
||||
"reference": o.Reference, "reason": o.Reason, "module": o.Module}
|
||||
}
|
||||
|
||||
// say says a seat's event; a test hears it instead.
|
||||
var say = sayOn
|
||||
|
||||
// sayOn says an event on the seat's family, in the seat's contract's Outcome.
|
||||
func sayOn(seat, event string, o packageseats.Outcome) error {
|
||||
var body any
|
||||
var err error
|
||||
if seat == packageseats.GoSeat {
|
||||
body, err = as[goreg.Outcome](outcomeBody(o))
|
||||
} else {
|
||||
body, err = as[npmreg.Outcome](outcomeBody(o))
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return stdio.SeatPublish("mesh.seat."+seat+".event."+event, body, "")
|
||||
}
|
||||
|
||||
func finding(kind, subject, words string, raised bool) {
|
||||
key := health.FindingKey(health.ScopeSeat, packageseats.Self, kind+"."+strings.NewReplacer(".", "_", "@", "_", "/", "_").Replace(subject))
|
||||
if !raised {
|
||||
_ = stdio.State(health.FindingsState).Delete(key)
|
||||
return
|
||||
}
|
||||
f := health.Finding{V: health.FindingVersion, Key: key, Kind: kind, Ref: key[strings.LastIndex(key, ".")+1:],
|
||||
Subject: health.Subject{Scope: health.ScopeSeat, ID: packageseats.Self}, Severity: health.Warning,
|
||||
Summary: words, Headline: words, Explanation: words, Resolved: "cleared once it no longer holds",
|
||||
Source: kind, Observer: packageseats.Self, At: time.Now().UTC()}
|
||||
if _, err := stdio.State(health.FindingsState).Put(key, f); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "the finding %s could not be written: %v\n", key, err)
|
||||
}
|
||||
}
|
||||
|
||||
// busHolder is the holder over the runtime's bus state, the builder's numbering and the forge.
|
||||
func busHolder(forge func() (*Forge, error)) (*packageseats.Holder, error) {
|
||||
callers, err := packageseats.ConfiguredCallers()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &packageseats.Holder{Callers: callers, Objects: parts.BusObjects(packageseats.IncomingObjects),
|
||||
Transfers: parts.BusTransfers[packageseats.Opening](packageseats.TransfersState),
|
||||
Builder: delivery.BuilderVersions{Ask: stdio.Ask}, Store: seatStore{forge}, Say: say, Finding: finding}, nil
|
||||
}
|
||||
|
||||
// SeatTools are both seats' verbs as tools <seat>.<verb>.
|
||||
func SeatTools(h func() (*packageseats.Holder, error)) []stdio.Tool {
|
||||
return append(verbTools[npmreg.PublishInput, npmreg.PublishAnswer, npmreg.PublishPartInput, npmreg.PublishPartAnswer,
|
||||
npmreg.VersionsInput, npmreg.VersionsAnswer, npmreg.RemoveInput, npmreg.RemoveAnswer](packageseats.NpmSeat, h),
|
||||
verbTools[goreg.PublishInput, goreg.PublishAnswer, goreg.PublishPartInput, goreg.PublishPartAnswer,
|
||||
goreg.VersionsInput, goreg.VersionsAnswer, goreg.RemoveInput, goreg.RemoveAnswer](packageseats.GoSeat, h)...)
|
||||
}
|
||||
|
||||
// chunk is publish-part's input as the holder reads it.
|
||||
type chunk struct {
|
||||
Data, Part, Transfer string
|
||||
Last bool
|
||||
Sequence int
|
||||
}
|
||||
|
||||
// verbTools are one seat's four verbs, each reading its input as the seat's contract type and answering in it.
|
||||
func verbTools[PI, PA, PPI, PPA, VI, VA, RI, RA any](seat string, holder func() (*packageseats.Holder, error)) []stdio.Tool {
|
||||
obj := map[string]any{"type": "object"}
|
||||
run := func(do func(h *packageseats.Holder, args map[string]any) (any, error)) func(map[string]any) (any, error) {
|
||||
return func(args map[string]any) (any, error) {
|
||||
h, err := holder()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return do(h, args)
|
||||
}
|
||||
}
|
||||
return []stdio.Tool{
|
||||
{Name: seat + "." + packageseats.VerbPublish, Input: obj,
|
||||
Description: "Open a transfer of a package's one file (ADR 0611): its name, the builder's version, kind package, its sha256 digest and the file as its one part; answers the part the forge lacks. A version held with that digest answers held and says published.",
|
||||
Run: run(func(h *packageseats.Holder, args map[string]any) (any, error) {
|
||||
in, err := as[PI](args)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("publish: %w", err)
|
||||
}
|
||||
p, err := as[packageseats.Publish](in)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
a, err := h.Publish(seat, stdio.CallerOf(args), p)
|
||||
out, _ := as[PA](map[string]any{"transfer": a.Transfer, "lacks": a.Lacks, "held": a.Ended && len(a.Lacks) == 0})
|
||||
return out, err
|
||||
})},
|
||||
{Name: seat + "." + packageseats.VerbPublishPart, Input: obj,
|
||||
Description: "Send one chunk (at most 760 KiB before base64) of the package's file, from the caller that opened the transfer; the last chunk ends it and the forge says published or refused.",
|
||||
Run: run(func(h *packageseats.Holder, args map[string]any) (any, error) {
|
||||
in, err := as[PPI](args)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("publish-part: %w", err)
|
||||
}
|
||||
c, err := as[chunk](in)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
data, err := base64.StdEncoding.DecodeString(c.Data)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("publish-part: data is not base64: %w", err)
|
||||
}
|
||||
a, err := h.PublishPart(seat, stdio.CallerOf(args), c.Transfer, c.Part, c.Sequence, c.Last, data)
|
||||
out, _ := as[PPA](map[string]any{"transfer": c.Transfer, "part": c.Part, "received": c.Sequence + 1, "complete": a.Ended})
|
||||
return out, err
|
||||
})},
|
||||
{Name: seat + "." + packageseats.VerbVersions, Input: obj,
|
||||
Description: "The versions the seat holds of one package, newest first, each with its digest.",
|
||||
Run: run(func(h *packageseats.Holder, args map[string]any) (any, error) {
|
||||
in, err := as[VI](args)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("versions: %w", err)
|
||||
}
|
||||
q, _ := as[struct{ Name string }](in)
|
||||
got, err := h.Versions(seat, stdio.CallerOf(args), q.Name)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
list := []map[string]any{}
|
||||
for _, v := range got {
|
||||
list = append(list, map[string]any{"name": q.Name, "version": v.Version, "digest": v.Digest, "kind": packageseats.KindPackage})
|
||||
}
|
||||
return as[VA](map[string]any{"name": q.Name, "versions": list})
|
||||
})},
|
||||
{Name: seat + "." + packageseats.VerbRemove, Input: obj,
|
||||
Description: "Remove a tagged version (the builder, when its branch ended); says removed. An untagged version goes by the forge's retention alone.",
|
||||
Run: run(func(h *packageseats.Holder, args map[string]any) (any, error) {
|
||||
in, err := as[RI](args)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("remove: %w", err)
|
||||
}
|
||||
r, _ := as[struct{ Name, Version, Reason string }](in)
|
||||
o, err := h.Remove(seat, stdio.CallerOf(args), r.Name, r.Version, r.Reason)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return as[RA](map[string]any{"removed": true, "artifact": map[string]any{"name": o.Name, "version": o.Version, "digest": o.Digest, "kind": o.Kind}})
|
||||
})},
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,197 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"archive/zip"
|
||||
"bytes"
|
||||
"compress/gzip"
|
||||
"crypto/sha256"
|
||||
"encoding/base64"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/health"
|
||||
"gitea/internal/forgehttp"
|
||||
"gitea/internal/packageseats"
|
||||
)
|
||||
|
||||
func tarball(t *testing.T, manifest string) []byte {
|
||||
var b bytes.Buffer
|
||||
gz := gzip.NewWriter(&b)
|
||||
tw := tar.NewWriter(gz)
|
||||
_ = tw.WriteHeader(&tar.Header{Name: "package/package.json", Mode: 0o644, Size: int64(len(manifest))})
|
||||
_, _ = tw.Write([]byte(manifest))
|
||||
_ = tw.Close()
|
||||
_ = gz.Close()
|
||||
return b.Bytes()
|
||||
}
|
||||
|
||||
func moduleZip(t *testing.T, prefix string) []byte {
|
||||
var b bytes.Buffer
|
||||
z := zip.NewWriter(&b)
|
||||
w, _ := z.Create(prefix + "/go.mod")
|
||||
_, _ = w.Write([]byte("module x\n"))
|
||||
_ = z.Close()
|
||||
return b.Bytes()
|
||||
}
|
||||
|
||||
func sha(b []byte) string { s := sha256.Sum256(b); return hex.EncodeToString(s[:]) }
|
||||
|
||||
// registry is the forge's package registry: what was uploaded, by path, and what the files API answers.
|
||||
type registry struct {
|
||||
mu sync.Mutex
|
||||
uploaded map[string][]byte // "npm <name>" or "go"
|
||||
held map[string][]byte // "<type>/<name>/<version>" -> file
|
||||
deleted []string
|
||||
}
|
||||
|
||||
func (g *registry) handle(w http.ResponseWriter, r *http.Request) {
|
||||
g.mu.Lock()
|
||||
defer g.mu.Unlock()
|
||||
body, _ := io.ReadAll(r.Body)
|
||||
p := r.URL.EscapedPath()
|
||||
// Reads come as the tools' account (read:package), writes as the packages account (write:package), never the admin.
|
||||
want := "token reader"
|
||||
if r.Method != http.MethodGet {
|
||||
want = "token writer"
|
||||
}
|
||||
if got := r.Header.Get("Authorization"); got == "token stale" {
|
||||
http.Error(w, "unauthorized", http.StatusUnauthorized)
|
||||
return
|
||||
} else if got != want {
|
||||
http.Error(w, r.Method+" "+p+" came as "+got+", not "+want, http.StatusForbidden)
|
||||
return
|
||||
}
|
||||
switch {
|
||||
case r.Method == http.MethodPut && p == "/api/packages/acme/go/upload":
|
||||
if r.Header.Get("Content-Type") != "application/zip" {
|
||||
http.Error(w, "not a zip", 400)
|
||||
return
|
||||
}
|
||||
g.uploaded["go"] = body
|
||||
w.WriteHeader(http.StatusCreated)
|
||||
case r.Method == http.MethodPut && strings.HasPrefix(p, "/api/packages/acme/npm/"):
|
||||
g.uploaded["npm "+strings.TrimPrefix(p, "/api/packages/acme/npm/")] = body
|
||||
w.WriteHeader(http.StatusCreated)
|
||||
case strings.HasPrefix(p, "/api/v1/packages/acme/") && strings.HasSuffix(p, "/files"):
|
||||
f, ok := g.held[strings.TrimSuffix(strings.TrimPrefix(r.URL.Path, "/api/v1/packages/acme/"), "/files")]
|
||||
if !ok {
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
_ = json.NewEncoder(w).Encode([]map[string]any{{"sha256": sha(f), "size": len(f)}})
|
||||
case r.Method == http.MethodGet && p == "/api/v1/packages/acme":
|
||||
var out []map[string]any
|
||||
for k := range g.held {
|
||||
parts := strings.Split(k, "/")
|
||||
if parts[0] == r.URL.Query().Get("type") {
|
||||
out = append(out, map[string]any{"name": strings.Join(parts[1:len(parts)-1], "/"), "version": parts[len(parts)-1], "created_at": time.Unix(int64(len(parts[len(parts)-1])), 0)})
|
||||
}
|
||||
}
|
||||
_ = json.NewEncoder(w).Encode(out)
|
||||
case r.Method == http.MethodDelete:
|
||||
g.deleted = append(g.deleted, strings.TrimPrefix(p, "/api/v1/packages/acme/"))
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
default:
|
||||
http.Error(w, "unexpected "+r.Method+" "+p, 500)
|
||||
}
|
||||
}
|
||||
|
||||
// **The package seats store into the forge's package registry** (design 51 *Publishing*): an npm package goes up as
|
||||
// npm's publish sends it, its package.json from the tarball and its tag as its dist-tag; a Go module's zip goes up to
|
||||
// the Go upload, its module path read from the zip; held, versions and remove read and delete through the forge's
|
||||
// package API, a Go version with Go's leading v.
|
||||
func TestThePackageSeatsStoreIntoTheForge(t *testing.T) {
|
||||
g := ®istry{uploaded: map[string][]byte{}, held: map[string][]byte{}}
|
||||
srv := httptest.NewServer(http.HandlerFunc(g.handle))
|
||||
defer srv.Close()
|
||||
dir := t.TempDir()
|
||||
kept := func(name, token, line string) string {
|
||||
file := filepath.Join(dir, name)
|
||||
_ = os.WriteFile(file, []byte(token+"\n"), 0o600)
|
||||
_ = os.WriteFile(file+".scopes", []byte(line+"\n"), 0o600)
|
||||
return file
|
||||
}
|
||||
reader := forgehttp.Client{URL: srv.URL, TokenFile: kept("token", "reader", "read:package,read:user for mesh-tools"), Account: "mesh-tools"}
|
||||
writer := forgehttp.Client{URL: srv.URL, TokenFile: kept("packages-token", "writer", "write:package for mesh-packages"), Account: "mesh-packages"}
|
||||
forge := &Forge{Client: reader, Writer: &writer, Owner: "acme"}
|
||||
if _, err := (seatStore{forge: func() (*Forge, error) { return &Forge{Client: reader, Owner: "acme"}, nil }}).Put(packageseats.GoSeat, packageseats.Publish{}, nil); err == nil ||
|
||||
!strings.Contains(err.Error(), "write side") {
|
||||
t.Errorf("a forge handed no write side wrote: %v", err)
|
||||
}
|
||||
s := seatStore{forge: func() (*Forge, error) { return forge, nil }}
|
||||
// The forge refuses the write side's token: the finding the TypeScript bundle renews on is raised under its key,
|
||||
// and the next write the forge takes clears it (no timer looks at the token).
|
||||
var refused []bool
|
||||
forge.Refused = func(raised bool) { refused = append(refused, raised) }
|
||||
if k := health.FindingKey(health.ScopeSeat, packageseats.Self, PackagesTokenRefused+".write"); k != PackagesTokenRefusedKey {
|
||||
t.Errorf("the refusal's key is %s, the watcher watches %s", k, PackagesTokenRefusedKey)
|
||||
}
|
||||
_ = os.WriteFile(writer.TokenFile, []byte("stale\n"), 0o600)
|
||||
if _, err := s.Put(packageseats.GoSeat, packageseats.Publish{Version: "0.1.15"}, moduleZip(t, "git.example/acme/sdk@v0.1.15")); err == nil ||
|
||||
len(refused) != 1 || !refused[0] {
|
||||
t.Errorf("a write the forge refused with 401 raised no finding: %v %v", err, refused)
|
||||
}
|
||||
_ = os.WriteFile(writer.TokenFile, []byte("writer\n"), 0o600)
|
||||
|
||||
tgz := tarball(t, `{"name":"@acme/contracts","version":"0.1.15-feat-x.2","main":"index.js"}`)
|
||||
p := packageseats.Publish{Name: "@acme/contracts", Version: "0.1.15-feat-x.2", Tag: "feat-x", Kind: packageseats.KindPackage, Digest: "sha256:" + sha(tgz)}
|
||||
if module, err := s.Put(packageseats.NpmSeat, p, tgz); err != nil || module != "" {
|
||||
t.Fatalf("npm put: %q, %v", module, err)
|
||||
}
|
||||
var doc struct {
|
||||
DistTags map[string]string `json:"dist-tags"`
|
||||
Versions map[string]map[string]any
|
||||
Attachments map[string]struct{ Data string } `json:"_attachments"`
|
||||
}
|
||||
if err := json.Unmarshal(g.uploaded["npm @acme%2Fcontracts"], &doc); err != nil {
|
||||
t.Fatalf("npm's publish body %v: %s", err, g.uploaded)
|
||||
}
|
||||
att := doc.Attachments["contracts-0.1.15-feat-x.2.tgz"].Data
|
||||
if doc.DistTags["feat-x"] != "0.1.15-feat-x.2" || doc.DistTags["latest"] != "" || doc.Versions["0.1.15-feat-x.2"]["main"] != "index.js" ||
|
||||
att != base64.StdEncoding.EncodeToString(tgz) {
|
||||
t.Errorf("npm's publish body: %+v", doc)
|
||||
}
|
||||
p.Name = "@acme/other"
|
||||
if _, err := s.Put(packageseats.NpmSeat, p, tgz); err == nil || !strings.Contains(err.Error(), "not @acme/other") {
|
||||
t.Errorf("a tarball of another package was uploaded: %v", err)
|
||||
}
|
||||
|
||||
zipped := moduleZip(t, "git.example/acme/sdk@v0.1.15")
|
||||
gp := packageseats.Publish{Name: "git.example/acme/sdk", Version: "0.1.15", Kind: packageseats.KindPackage, Digest: "sha256:" + sha(zipped)}
|
||||
if module, err := s.Put(packageseats.GoSeat, gp, zipped); err != nil || module != "git.example/acme/sdk" || !bytes.Equal(g.uploaded["go"], zipped) {
|
||||
t.Fatalf("go put: %q, %v", module, err)
|
||||
}
|
||||
gp.Version = "0.1.16"
|
||||
if _, err := s.Put(packageseats.GoSeat, gp, zipped); err == nil {
|
||||
t.Error("a zip of another version was uploaded")
|
||||
}
|
||||
|
||||
g.held["go/git.example/acme/sdk/v0.1.15"] = zipped
|
||||
if d, ok, err := s.Held(packageseats.GoSeat, "git.example/acme/sdk", "0.1.15"); err != nil || !ok || d != "sha256:"+sha(zipped) {
|
||||
t.Errorf("held: %s %v %v", d, ok, err)
|
||||
}
|
||||
if _, ok, err := s.Held(packageseats.NpmSeat, "x", "0.1.0"); ok || err != nil {
|
||||
t.Errorf("a version the forge does not hold: %v %v", ok, err)
|
||||
}
|
||||
g.held["go/git.example/acme/sdk/v0.1.9"] = zipped
|
||||
if v, err := s.Versions(packageseats.GoSeat, "git.example/acme/sdk"); err != nil || len(v) != 2 || v[0].Version != "0.1.15" {
|
||||
t.Errorf("versions: %+v %v", v, err)
|
||||
}
|
||||
if err := s.Remove(packageseats.GoSeat, "git.example/acme/sdk", "0.1.15-feat.1"); err != nil || len(g.deleted) != 1 ||
|
||||
g.deleted[0] != "go/git.example%2Facme%2Fsdk/v0.1.15-feat.1" {
|
||||
t.Errorf("remove: %v %v", err, g.deleted)
|
||||
}
|
||||
if len(refused) < 2 || refused[len(refused)-1] {
|
||||
t.Errorf("a write the forge took after the refusal did not clear it: %v", refused)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,408 @@
|
||||
//go:build integration
|
||||
|
||||
package main
|
||||
|
||||
// The forge module's package seats whole against a real bus (CAT-G3; design 51 *Acceptance tests*, the holder's cases):
|
||||
// the built package-registry bundle, played over its real stdin and stdout as the node's tool runtime plays it, its
|
||||
// `incoming` an object store and its `transfers` and `findings` key-value buckets of a nats-server of this test alone
|
||||
// (so the publish in parts runs through mesh-sdk's parts.BusObjects and BusTransfers, ruling 4 of the lead's 14:0x), the
|
||||
// builder's builder_versions answered on mesh/ask, the config value `callers` on mesh/provision.get, and the forge an
|
||||
// httptest server that takes reads only with the tools' token and writes only with the packages account's.
|
||||
//
|
||||
// A Go package the builder numbered goes up in two chunks, is uploaded whole as the packages account and says
|
||||
// published on the Go seat with its module path; one the builder did not number is refused before the forge sees it,
|
||||
// says refused and writes the seat's finding; versions answers what the forge holds.
|
||||
|
||||
import (
|
||||
"archive/zip"
|
||||
"bufio"
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/delivery"
|
||||
"git.novox.be/novox/mesh-sdk/go/testbus"
|
||||
|
||||
"gitea/internal/packageseats"
|
||||
)
|
||||
|
||||
type wire struct {
|
||||
ID json.RawMessage `json:"id,omitempty"`
|
||||
Method string `json:"method,omitempty"`
|
||||
Params json.RawMessage `json:"params,omitempty"`
|
||||
Result json.RawMessage `json:"result,omitempty"`
|
||||
}
|
||||
|
||||
// player is the node's tool runtime, as far as the package seats need it, over a real bus.
|
||||
type player struct {
|
||||
ctx context.Context
|
||||
t *testing.T
|
||||
nc *nats.Conn
|
||||
js jetstream.JetStream
|
||||
numbered map[string]string // name@version -> the digest the builder numbered
|
||||
mu sync.Mutex
|
||||
in io.Writer
|
||||
next int
|
||||
answers map[string]chan wire
|
||||
}
|
||||
|
||||
func (p *player) send(m map[string]any) {
|
||||
m["jsonrpc"] = "2.0"
|
||||
raw, _ := json.Marshal(m)
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
_, _ = p.in.Write(append(raw, '\n'))
|
||||
}
|
||||
|
||||
func (p *player) answer(id json.RawMessage, result any, err error) {
|
||||
if err != nil {
|
||||
p.send(map[string]any{"id": id, "error": map[string]any{"code": -32000, "message": err.Error()}})
|
||||
return
|
||||
}
|
||||
p.send(map[string]any{"id": id, "result": result})
|
||||
}
|
||||
|
||||
func (p *player) serve(out io.Reader) {
|
||||
scanner := bufio.NewScanner(out)
|
||||
scanner.Buffer(make([]byte, 1<<20), 1<<26)
|
||||
for scanner.Scan() {
|
||||
var m wire
|
||||
if json.Unmarshal(scanner.Bytes(), &m) != nil {
|
||||
continue
|
||||
}
|
||||
if m.Method == "" {
|
||||
var id string
|
||||
_ = json.Unmarshal(m.ID, &id)
|
||||
p.mu.Lock()
|
||||
ch := p.answers[id]
|
||||
p.mu.Unlock()
|
||||
if ch != nil {
|
||||
ch <- m
|
||||
}
|
||||
continue
|
||||
}
|
||||
go p.handle(m)
|
||||
}
|
||||
}
|
||||
|
||||
// handle answers one call of the bundle on the runtime.
|
||||
func (p *player) handle(m wire) {
|
||||
var a struct {
|
||||
State string `json:"state"`
|
||||
Key string `json:"key"`
|
||||
Value json.RawMessage `json:"value"`
|
||||
Objects string `json:"objects"`
|
||||
Name string `json:"name"`
|
||||
Data []byte `json:"data"`
|
||||
Subject string `json:"subject"`
|
||||
Body json.RawMessage `json:"body"`
|
||||
Kind string `json:"kind"`
|
||||
}
|
||||
_ = json.Unmarshal(m.Params, &a)
|
||||
switch {
|
||||
case m.Method == "mesh/ask" && a.Key == delivery.BuilderVersionsAddress:
|
||||
var q struct{ Name, Version, Digest string }
|
||||
_ = json.Unmarshal(a.Body, &q)
|
||||
d, ok := p.numbered[q.Name+"@"+q.Version]
|
||||
p.answer(m.ID, map[string]any{"recorded": ok && d == q.Digest, "why": "not numbered with that digest"}, nil)
|
||||
case m.Method == "mesh/provision.get" && a.Kind == "config":
|
||||
p.answer(m.ID, nil, nil) // `callers` unset: design 51's list
|
||||
case m.Method == "mesh/seat.publish":
|
||||
p.answer(m.ID, map[string]any{}, p.nc.Publish(a.Subject, a.Body))
|
||||
case m.Method == "mesh/publish":
|
||||
p.answer(m.ID, map[string]any{}, nil)
|
||||
case strings.HasPrefix(m.Method, "mesh/objects."):
|
||||
p.objects(m, a.Objects, a.Name, a.Data)
|
||||
case strings.HasPrefix(m.Method, "mesh/state."):
|
||||
p.state(m, a.State, a.Key, a.Value)
|
||||
default:
|
||||
p.answer(m.ID, nil, errors.New("not played here: "+m.Method))
|
||||
}
|
||||
}
|
||||
|
||||
func (p *player) objects(m wire, local, name string, data []byte) {
|
||||
os, err := p.js.ObjectStore(p.ctx, "gitea_"+local)
|
||||
if err != nil {
|
||||
p.answer(m.ID, nil, err)
|
||||
return
|
||||
}
|
||||
switch m.Method {
|
||||
case "mesh/objects.put":
|
||||
info, err := os.PutBytes(p.ctx, name, data)
|
||||
if err != nil {
|
||||
p.answer(m.ID, nil, err)
|
||||
return
|
||||
}
|
||||
p.answer(m.ID, map[string]any{"name": info.Name, "size": info.Size, "digest": info.Digest}, nil)
|
||||
case "mesh/objects.get":
|
||||
got, err := os.GetBytes(p.ctx, name)
|
||||
p.answer(m.ID, map[string]any{"info": map[string]any{"name": name, "size": len(got)}, "data": got}, err)
|
||||
case "mesh/objects.delete":
|
||||
err := os.Delete(p.ctx, name)
|
||||
if errors.Is(err, jetstream.ErrObjectNotFound) {
|
||||
err = nil
|
||||
}
|
||||
p.answer(m.ID, map[string]any{}, err)
|
||||
case "mesh/objects.info":
|
||||
st, err := os.Status(p.ctx)
|
||||
if err != nil {
|
||||
p.answer(m.ID, nil, err)
|
||||
return
|
||||
}
|
||||
p.answer(m.ID, map[string]any{"bytes": st.Size(), "max_bytes": 2048 << 20}, nil)
|
||||
default:
|
||||
p.answer(m.ID, nil, errors.New("not played here: "+m.Method))
|
||||
}
|
||||
}
|
||||
|
||||
func (p *player) state(m wire, state, key string, value json.RawMessage) {
|
||||
kv, err := p.js.KeyValue(p.ctx, "gitea_"+state)
|
||||
if err != nil {
|
||||
p.answer(m.ID, nil, err)
|
||||
return
|
||||
}
|
||||
switch m.Method {
|
||||
case "mesh/state.get":
|
||||
e, err := kv.Get(p.ctx, key)
|
||||
if errors.Is(err, jetstream.ErrKeyNotFound) {
|
||||
p.answer(m.ID, nil, nil)
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
p.answer(m.ID, nil, err)
|
||||
return
|
||||
}
|
||||
p.answer(m.ID, map[string]any{"key": e.Key(), "value": json.RawMessage(e.Value()), "revision": e.Revision()}, nil)
|
||||
case "mesh/state.put":
|
||||
rev, err := kv.Put(p.ctx, key, value)
|
||||
p.answer(m.ID, map[string]any{"revision": rev}, err)
|
||||
case "mesh/state.delete":
|
||||
p.answer(m.ID, map[string]any{}, kv.Delete(p.ctx, key))
|
||||
case "mesh/state.keys":
|
||||
keys, err := kv.Keys(p.ctx)
|
||||
if errors.Is(err, jetstream.ErrNoKeysFound) {
|
||||
keys, err = []string{}, nil
|
||||
}
|
||||
p.answer(m.ID, keys, err)
|
||||
default:
|
||||
p.answer(m.ID, nil, errors.New("not played here: "+m.Method))
|
||||
}
|
||||
}
|
||||
|
||||
// call is one tools/call as the runtime makes it, the caller named in _meta as the runtime proves it.
|
||||
func (p *player) call(caller, tool string, args map[string]any) (json.RawMessage, error) {
|
||||
p.mu.Lock()
|
||||
p.next++
|
||||
id := fmt.Sprintf("call-%d", p.next)
|
||||
ch := make(chan wire, 1)
|
||||
p.answers[id] = ch
|
||||
p.mu.Unlock()
|
||||
p.send(map[string]any{"id": id, "method": "tools/call", "params": map[string]any{"name": tool, "arguments": args,
|
||||
"_meta": map[string]any{"mesh/caller": caller}}})
|
||||
select {
|
||||
case m := <-ch:
|
||||
var r struct {
|
||||
Content []struct{ Text string } `json:"content"`
|
||||
IsError bool `json:"isError"`
|
||||
}
|
||||
if err := json.Unmarshal(m.Result, &r); err != nil || len(r.Content) == 0 {
|
||||
return nil, fmt.Errorf("%s answered %s", tool, m.Result)
|
||||
}
|
||||
if r.IsError {
|
||||
return nil, errors.New(r.Content[0].Text)
|
||||
}
|
||||
return json.RawMessage(r.Content[0].Text), nil
|
||||
case <-time.After(30 * time.Second):
|
||||
return nil, fmt.Errorf("%s did not answer", tool)
|
||||
}
|
||||
}
|
||||
|
||||
// bigZip is a Go module zip of about 1.2 MiB that does not compress, so its file goes in two chunks.
|
||||
func bigZip(t *testing.T, prefix string) []byte {
|
||||
noise := make([]byte, 1200<<10)
|
||||
_, _ = rand.Read(noise)
|
||||
var b bytes.Buffer
|
||||
z := zip.NewWriter(&b)
|
||||
w, _ := z.Create(prefix + "/go.mod")
|
||||
_, _ = w.Write([]byte("module " + strings.Split(prefix, "@")[0] + "\n"))
|
||||
w, _ = z.CreateHeader(&zip.FileHeader{Name: prefix + "/noise.bin", Method: zip.Store})
|
||||
_, _ = w.Write(noise)
|
||||
_ = z.Close()
|
||||
return b.Bytes()
|
||||
}
|
||||
|
||||
func TestThePackageSeatsWhole(t *testing.T) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
|
||||
t.Cleanup(cancel)
|
||||
bin := filepath.Join(t.TempDir(), "package-registry")
|
||||
build := exec.Command("go", "build", "-o", bin, ".")
|
||||
build.Stderr = os.Stderr
|
||||
if err := build.Run(); err != nil {
|
||||
t.Fatalf("package-registry does not build: %v", err)
|
||||
}
|
||||
nc, err := nats.Connect(testbus.URL(t))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(nc.Close)
|
||||
js, _ := jetstream.New(nc)
|
||||
for _, b := range []string{"gitea_" + packageseats.TransfersState, "gitea_findings"} {
|
||||
if _, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: b}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if _, err := js.CreateObjectStore(ctx, jetstream.ObjectStoreConfig{Bucket: "gitea_" + packageseats.IncomingObjects}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
heard := make(chan *nats.Msg, 8)
|
||||
if _, err := nc.ChanSubscribe("mesh.seat.go-package-registry.event.>", heard); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// The forge: reads only as the tools' account, writes only as the packages account.
|
||||
g := ®istry{uploaded: map[string][]byte{}, held: map[string][]byte{}}
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
g.handle(w, r)
|
||||
if r.Method == http.MethodPut && r.URL.Path == "/api/packages/acme/go/upload" {
|
||||
g.mu.Lock()
|
||||
g.held["go/git.example/acme/sdk/v0.1.15"] = g.uploaded["go"]
|
||||
g.mu.Unlock()
|
||||
}
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
dir := t.TempDir()
|
||||
kept := func(name, token, line string) string {
|
||||
file := filepath.Join(dir, name)
|
||||
_ = os.WriteFile(file, []byte(token+"\n"), 0o600)
|
||||
_ = os.WriteFile(file+".scopes", []byte(line+"\n"), 0o600)
|
||||
return file
|
||||
}
|
||||
cmd := exec.CommandContext(ctx, bin)
|
||||
cmd.Env = append(os.Environ(), "MESH_GITEA_URL="+srv.URL, "MESH_NPM_OWNER=acme", "MESH_GITEA_TOOLS_USER=mesh-tools",
|
||||
"MESH_GITEA_TOKEN_FILE="+kept("token", "reader", "read:package,read:user,write:issue,write:repository for mesh-tools"),
|
||||
"MESH_GITEA_PACKAGES_TOKEN_FILE="+kept("packages-token", "writer", "write:package for mesh-packages"))
|
||||
cmd.Stderr = os.Stderr
|
||||
in, _ := cmd.StdinPipe()
|
||||
out, _ := cmd.StdoutPipe()
|
||||
if err := cmd.Start(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { _ = in.Close(); _ = cmd.Wait() })
|
||||
|
||||
zipped := bigZip(t, "git.example/acme/sdk@v0.1.15")
|
||||
digest := "sha256:" + sha(zipped)
|
||||
p := &player{ctx: ctx, t: t, nc: nc, js: js, in: in, answers: map[string]chan wire{},
|
||||
numbered: map[string]string{"git.example/acme/sdk@0.1.15": digest}}
|
||||
go p.serve(out)
|
||||
p.send(map[string]any{"id": "init", "method": "initialize", "params": map[string]any{}})
|
||||
|
||||
builder := "ace.mesh-build-go"
|
||||
publish := func(version, digest string, size int) (string, []string) {
|
||||
raw, err := p.call(builder, "go-package-registry.publish", map[string]any{"name": "git.example/acme/sdk", "version": version,
|
||||
"kind": "package", "digest": digest, "transfer": "sdk-" + strings.ReplaceAll(version, ".", "-"), "parts": []map[string]any{{"digest": digest, "size": size}}})
|
||||
if err != nil {
|
||||
t.Fatalf("publish %s: %v", version, err)
|
||||
}
|
||||
var a struct {
|
||||
Transfer string `json:"transfer"`
|
||||
Lacks []string `json:"lacks"`
|
||||
}
|
||||
_ = json.Unmarshal(raw, &a)
|
||||
return a.Transfer, a.Lacks
|
||||
}
|
||||
send := func(transfer, digest string, file []byte) error {
|
||||
const chunk = 700 << 10
|
||||
for seq, at := 0, 0; at < len(file); seq, at = seq+1, at+chunk {
|
||||
end := min(at+chunk, len(file))
|
||||
if _, err := p.call(builder, "go-package-registry.publish-part", map[string]any{"transfer": transfer, "part": digest,
|
||||
"sequence": seq, "last": end == len(file), "data": base64.StdEncoding.EncodeToString(file[at:end])}); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
event := func(want string) map[string]any {
|
||||
select {
|
||||
case m := <-heard:
|
||||
if !strings.HasSuffix(m.Subject, "."+want) {
|
||||
t.Fatalf("the Go seat said %s, want %s: %s", m.Subject, want, m.Data)
|
||||
}
|
||||
var o map[string]any
|
||||
_ = json.Unmarshal(m.Data, &o)
|
||||
return o
|
||||
case <-time.After(30 * time.Second):
|
||||
t.Fatalf("the Go seat said no %s", want)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// A publish naming no transfer is refused: the transfer's name is the caller's own.
|
||||
if _, err := p.call(builder, "go-package-registry.publish", map[string]any{"name": "git.example/acme/sdk", "version": "0.1.15",
|
||||
"kind": "package", "digest": digest, "parts": []map[string]any{{"digest": digest, "size": len(zipped)}}}); err == nil ||
|
||||
!strings.Contains(err.Error(), "no transfer named") {
|
||||
t.Errorf("a publish naming no transfer: %v", err)
|
||||
}
|
||||
// A version the builder numbered: two chunks through incoming and transfers, uploaded whole as the packages account.
|
||||
transfer, lacks := publish("0.1.15", digest, len(zipped))
|
||||
if transfer == "" || len(lacks) != 1 || lacks[0] != digest {
|
||||
t.Fatalf("publish answered transfer %q lacks %v", transfer, lacks)
|
||||
}
|
||||
if err := send(transfer, digest, zipped); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if o := event("published"); o["module"] != "git.example/acme/sdk" || o["reference"] != "git.example/acme/sdk@0.1.15" || o["holder"] != "gitea" {
|
||||
t.Errorf("published says %v", o)
|
||||
}
|
||||
g.mu.Lock()
|
||||
up := g.uploaded["go"]
|
||||
g.mu.Unlock()
|
||||
if !bytes.Equal(up, zipped) {
|
||||
t.Errorf("the forge took %d bytes, not the %d published", len(up), len(zipped))
|
||||
}
|
||||
transfers, _ := js.KeyValue(ctx, "gitea_"+packageseats.TransfersState)
|
||||
if keys, _ := transfers.Keys(ctx); len(keys) != 0 {
|
||||
t.Errorf("an ended transfer is still kept: %v", keys)
|
||||
}
|
||||
|
||||
// One the builder did not number: refused before the forge sees it, said, and the seat's finding written.
|
||||
other := bigZip(t, "git.example/acme/sdk@v0.1.16")
|
||||
od := "sha256:" + sha(other)
|
||||
transfer, _ = publish("0.1.16", od, len(other))
|
||||
_ = send(transfer, od, other)
|
||||
if o := event("refused"); !strings.Contains(fmt.Sprint(o["reason"]), "did not number") {
|
||||
t.Errorf("refused says %v", o)
|
||||
}
|
||||
findings, _ := js.KeyValue(ctx, "gitea_findings")
|
||||
if keys, _ := findings.Keys(ctx); len(keys) == 0 {
|
||||
t.Error("a refused publish wrote no finding")
|
||||
}
|
||||
g.mu.Lock()
|
||||
_, uploadedOther := g.held["go/git.example/acme/sdk/v0.1.16"]
|
||||
g.mu.Unlock()
|
||||
if uploadedOther {
|
||||
t.Error("a version the builder did not number reached the forge")
|
||||
}
|
||||
|
||||
// Anyone reads versions; the forge was read as the tools' account.
|
||||
raw, err := p.call("g14.someone", "go-package-registry.versions", map[string]any{"name": "git.example/acme/sdk"})
|
||||
if err != nil || !strings.Contains(string(raw), `"version":"0.1.15"`) || !strings.Contains(string(raw), digest) {
|
||||
t.Errorf("versions answered %s, %v", raw, err)
|
||||
}
|
||||
}
|
||||
@@ -3,6 +3,8 @@ module gitea
|
||||
go 1.26.0
|
||||
|
||||
require (
|
||||
git.novox.be/novox/mesh-contracts/go/seats/gopackageregistry v0.0.0-00010101000000-000000000000
|
||||
git.novox.be/novox/mesh-contracts/go/seats/npmpackageregistry v0.0.0-00010101000000-000000000000
|
||||
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261102191058-f02f142af128
|
||||
github.com/nats-io/nats-server/v2 v2.11.17
|
||||
github.com/nats-io/nats.go v1.54.0
|
||||
|
||||
@@ -62,6 +62,7 @@ import { announcementOf, weighTrunks } from "./trunk.js";
|
||||
import { ReadAll, readAllTools } from "./readall.js";
|
||||
import { DeliveredWrites, INDEXER_MODULES_EVENTS, INDEXER_MODULES_TOOL, INDEXER_OWNER, writeTools, type ModulesChange } from "./writes.js";
|
||||
import { TOOLS_ACCOUNT, TOOLS_ORG, meshAccounts } from "./tools-account.js";
|
||||
import { PACKAGES_TOKEN_REFUSED, keepPackagesToken, packagesToken, renewOnRefusal } from "./packages-account.js";
|
||||
|
||||
// Without a way to a token — configured, or mintable with the admin account (token.ts) — there is
|
||||
// nothing to watch; log and stay quiet rather than crash the runtime. With one, the first poll mints
|
||||
@@ -538,6 +539,18 @@ if (gitea) {
|
||||
// Main takes no push and only the tools' account merges, in every repository, kept and not only when a token is
|
||||
// minted: at the start and every ten minutes after, so a repository made later gets it too (issue 948, ADR 0398 §2).
|
||||
// Only while the setting keep-main-rules is true (mainrules.ts); false, the default, leaves every repository's rule as set.
|
||||
// The package seats' write token (packages-account.ts): minted at the start; after that renewed only when the Go
|
||||
// bundle that writes with it says the forge refused it (the finding PACKAGES_TOKEN_REFUSED), never on a timer.
|
||||
try {
|
||||
const url = process.env.MESH_GITEA_URL ?? "http://127.0.0.1:3000";
|
||||
const source = packagesToken(url);
|
||||
const say = (line: string) => console.log(`[gitea] ${line}`);
|
||||
void keepPackagesToken(source, url).then(say, (err) => console.error(`[gitea] the package seats' write token could not be minted at the start — ${err instanceof Error ? err.message : err}`));
|
||||
const findings = state<unknown>("findings");
|
||||
void bindUntilHeld("findings", () => findings.watch(renewOnRefusal(source, url, findings, say), { key: PACKAGES_TOKEN_REFUSED }));
|
||||
} catch (err) {
|
||||
console.error(`[gitea] the package seats cannot write — ${err instanceof Error ? err.message : err}`);
|
||||
}
|
||||
tick(mainRulesStep(keepMainRules(), () => GiteaAdmin.fromEnv().ensureMainMerges("novox"), (line) => console.log(line)), 600_000);
|
||||
tick(() => pollRepos(client), 60_000);
|
||||
// **No timer asks the forge for pull requests or merges** (issue 1422): the forge's events announce them. One pass over
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
// Package forgehttp is the forge's HTTP interface as this module's Go bundles reach it (cmd/forge,
|
||||
// cmd/npm-registry), on the forge's own machine. One copy for both bundles (novox/hq issue 539).
|
||||
// cmd/package-registry), on the forge's own machine. One copy for both bundles (novox/hq issue 539).
|
||||
//
|
||||
// Two ways in, by what the manifest hands the bundle:
|
||||
//
|
||||
@@ -46,6 +46,29 @@ type Client struct {
|
||||
// ToolsAccount is the account the forge's tools act as when the manifest names none (token.ts, tools-account.ts).
|
||||
const ToolsAccount = "mesh-tools"
|
||||
|
||||
// PackagesAccount is the account the package seats write as when the manifest names none (packages-account.ts):
|
||||
// its token carries write:package and nothing else (design 51: each holder keeps its write side as an own secret
|
||||
// beside a read-only tool token).
|
||||
const PackagesAccount = "mesh-packages"
|
||||
|
||||
// WriterFromEnv is the package registry's write side as the manifest hands it to a bundle: the packages account's
|
||||
// token (MESH_GITEA_PACKAGES_TOKEN_FILE, kept by the module beside the tools' token) and the names of the variables
|
||||
// it left unset. Never the admin login: a bundle handed no writer's token writes nothing.
|
||||
func WriterFromEnv() (Client, []string) {
|
||||
c := Client{URL: strings.TrimRight(os.Getenv("MESH_GITEA_URL"), "/"), TokenFile: os.Getenv("MESH_GITEA_PACKAGES_TOKEN_FILE"),
|
||||
Account: os.Getenv("MESH_GITEA_PACKAGES_USER")}
|
||||
if c.Account == "" {
|
||||
c.Account = PackagesAccount
|
||||
}
|
||||
var missing []string
|
||||
for _, kv := range [][2]string{{"MESH_GITEA_URL", c.URL}, {"MESH_GITEA_PACKAGES_TOKEN_FILE", c.TokenFile}} {
|
||||
if kv[1] == "" {
|
||||
missing = append(missing, kv[0])
|
||||
}
|
||||
}
|
||||
return c, missing
|
||||
}
|
||||
|
||||
// FromEnv is the forge as the manifest hands it to a bundle, and the names of the variables it left unset. A bundle
|
||||
// handed MESH_GITEA_TOKEN_FILE acts as the tools' account and needs no admin login.
|
||||
func FromEnv() (Client, []string) {
|
||||
@@ -113,7 +136,7 @@ func (c *Client) password() (string, error) {
|
||||
func (c *Client) token() (string, error) {
|
||||
raw, err := os.ReadFile(c.TokenFile)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("the tools' token is not kept yet (the module mints it for %s at its first call): %w", c.Account, err)
|
||||
return "", fmt.Errorf("the token for %s is not kept yet (the module mints it): %w", c.Account, err)
|
||||
}
|
||||
line, err := os.ReadFile(c.TokenFile + ".scopes")
|
||||
if err != nil || !strings.HasSuffix(strings.TrimSpace(string(line)), " for "+c.Account) {
|
||||
@@ -148,12 +171,21 @@ func (c *Client) authorize(req *http.Request) error {
|
||||
// 404 is ErrNotFound. One byte past the limit is read, so a body the limit cut is told from one that fits it
|
||||
// exactly: a cut body is refused, never read as if it were whole.
|
||||
func (c *Client) Do(ctx context.Context, method, path string, payload any, limit int64) ([]byte, error) {
|
||||
if payload == nil {
|
||||
return c.Send(ctx, method, path, nil, "", limit)
|
||||
}
|
||||
raw, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return c.Send(ctx, method, path, raw, "application/json", limit)
|
||||
}
|
||||
|
||||
// Send is Do with a body of its own content type, sent as it is (a package's file uploaded into the registry);
|
||||
// a nil body sends none.
|
||||
func (c *Client) Send(ctx context.Context, method, path string, raw []byte, contentType string, limit int64) ([]byte, error) {
|
||||
var body io.Reader
|
||||
if payload != nil {
|
||||
raw, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if raw != nil {
|
||||
body = bytes.NewReader(raw)
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, method, c.URL+path, body)
|
||||
@@ -164,8 +196,8 @@ func (c *Client) Do(ctx context.Context, method, path string, payload any, limit
|
||||
return nil, err
|
||||
}
|
||||
req.Header.Set("Accept", "application/json")
|
||||
if payload != nil {
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
if raw != nil {
|
||||
req.Header.Set("Content-Type", contentType)
|
||||
}
|
||||
res, err := c.client().Do(req)
|
||||
if err != nil {
|
||||
@@ -181,8 +213,8 @@ func (c *Client) Do(ctx context.Context, method, path string, payload any, limit
|
||||
}
|
||||
switch {
|
||||
case res.StatusCode == http.StatusUnauthorized && c.TokenFile != "":
|
||||
return nil, fmt.Errorf("the forge rejected the tools' token for %s (401): the module mints it afresh at its next call; nothing was done: %w",
|
||||
method+" "+path, &StatusError{method, path, res.StatusCode, "unauthorized"})
|
||||
return nil, fmt.Errorf("the forge rejected %s's token for %s (401): the module mints it afresh; nothing was done: %w",
|
||||
c.Account, method+" "+path, &StatusError{method, path, res.StatusCode, "unauthorized"})
|
||||
case res.StatusCode == http.StatusNotFound:
|
||||
return nil, fmt.Errorf("%s %s: %w", method, path, ErrNotFound)
|
||||
case res.StatusCode >= 300:
|
||||
|
||||
@@ -0,0 +1,348 @@
|
||||
// Package packageseats is the forge module's holding of the two package seats, `npm-package-registry` and
|
||||
// `go-package-registry` (novox/hq design 51 *The four registry seats*, ADR 0605, ADR 0611). A build module publishes
|
||||
// a package through `publish` and `publish-part`: the package's one file lands in the forge module's own object store
|
||||
// `incoming`, is hashed, checked against the builder's numbering and then uploaded into the forge's package registry
|
||||
// with the module's own write side. The receiving side is mesh-sdk's seat-neutral parts.Receiver and the caller list
|
||||
// mesh-sdk's VerbCallers (ADR 0627 §2): this package binds both to the two seats and to the forge.
|
||||
package packageseats
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
stdio "git.novox.be/novox/mesh-sdk/go"
|
||||
"git.novox.be/novox/mesh-sdk/go/parts"
|
||||
)
|
||||
|
||||
// The seats the forge module holds.
|
||||
const (
|
||||
NpmSeat = "npm-package-registry"
|
||||
GoSeat = "go-package-registry"
|
||||
)
|
||||
|
||||
// Seats are the package seats, each with the forge's own package type behind it.
|
||||
var Seats = map[string]string{NpmSeat: "npm", GoSeat: "go"}
|
||||
|
||||
// The verbs both seats serve (design 51 *One contract*).
|
||||
const (
|
||||
VerbPublish = "publish"
|
||||
VerbPublishPart = "publish-part"
|
||||
VerbVersions = "versions"
|
||||
VerbRemove = "remove"
|
||||
)
|
||||
|
||||
// Verbs are the verbs of both seats, in order.
|
||||
var Verbs = []string{VerbPublish, VerbPublishPart, VerbRemove, VerbVersions}
|
||||
|
||||
// The events of both seats.
|
||||
const (
|
||||
EventPublished = "published"
|
||||
EventRefused = "refused"
|
||||
EventRemoved = "removed"
|
||||
)
|
||||
|
||||
// The bounds of incoming (design 51 *Publishing*, ADR 0611): a chunk at most 1 MiB, a transfer with no chunk for 24
|
||||
// hours dropped, a finding past 80 % of the cap. A package is one file, far smaller than an image, so the cap is the
|
||||
// manifest's 2 GiB.
|
||||
const (
|
||||
MaxChunk = 1 << 20
|
||||
IncomingCap = 2 << 30
|
||||
TransferIdle = 24 * time.Hour
|
||||
IncomingFullPct = 80
|
||||
IncomingObjects = "incoming"
|
||||
TransfersState = "transfers"
|
||||
// KindPackage is the one kind a package seat holds.
|
||||
KindPackage = "package"
|
||||
// Self is the forge module, the holder every outcome names.
|
||||
Self = "gitea"
|
||||
// ReasonRetention is the reason of the forge's own retention removing a version nothing keeps.
|
||||
ReasonRetention = "retention"
|
||||
)
|
||||
|
||||
// Callers is the trusted config value `callers`, read as mesh-sdk's list per verb.
|
||||
type Callers = stdio.VerbCallers
|
||||
|
||||
// DefaultCallers is design 51's list for the package seats: publish and publish-part the two build modules that
|
||||
// publish packages; remove the builder (a tagged version whose branch ended); versions anyone on the mesh, as the
|
||||
// registry's reads are anonymous.
|
||||
func DefaultCallers() Callers {
|
||||
builds := []string{"mesh-build-go", "mesh-build-node"}
|
||||
return Callers{VerbPublish: builds, VerbPublishPart: builds, VerbRemove: {"mesh-delivery-builder"},
|
||||
VerbVersions: {stdio.AnyCaller}}
|
||||
}
|
||||
|
||||
// ConfiguredCallers is the config value `callers` as mesh-config resolved it; unset is DefaultCallers.
|
||||
func ConfiguredCallers() (Callers, error) {
|
||||
return stdio.ConfiguredVerbCallers("callers", Verbs, DefaultCallers())
|
||||
}
|
||||
|
||||
// Publish is a publish as the seats' contract names it (mesh-contracts npmpackageregistry and gopackageregistry share
|
||||
// one shape): what the tools bundle reads from either seat's PublishInput.
|
||||
type Publish struct {
|
||||
Transfer string `json:"transfer"`
|
||||
Name string `json:"name"`
|
||||
Version string `json:"version,omitempty"`
|
||||
Tag string `json:"tag,omitempty"`
|
||||
Kind string `json:"kind"`
|
||||
Fingerprint string `json:"fingerprint,omitempty"`
|
||||
Commit string `json:"commit,omitempty"`
|
||||
Digest string `json:"digest"`
|
||||
Parts []parts.Want `json:"parts,omitempty"`
|
||||
}
|
||||
|
||||
// Opening is what the holder keeps of a publish beside the receiving side's own fields.
|
||||
type Opening struct {
|
||||
Publish
|
||||
Seat string `json:"seat"`
|
||||
}
|
||||
|
||||
// Transfer is one open publish in the state transfers.
|
||||
type Transfer = parts.Transfer[Opening]
|
||||
|
||||
// Outcome is the body of published, refused and removed, as the contract's Outcome names it.
|
||||
type Outcome struct {
|
||||
Name, Version, Kind, Tag, Digest, Fingerprint, Commit, Transfer, Reference, Reason string
|
||||
// Module is the Go module path a Go package is fetched under; empty on the npm seat.
|
||||
Module string
|
||||
}
|
||||
|
||||
// Answer is what publish and publish-part answer: the transfer, the parts lacked, and whether it ended.
|
||||
type Answer struct {
|
||||
Transfer string
|
||||
Lacks []string
|
||||
Ended bool
|
||||
}
|
||||
|
||||
// Version is one version a seat holds of a name.
|
||||
type Version struct {
|
||||
Version string
|
||||
Digest string
|
||||
Published time.Time
|
||||
}
|
||||
|
||||
// Builder answers whether the builder numbered a version with a digest (mesh-sdk delivery.BuilderVersions).
|
||||
type Builder interface {
|
||||
Recorded(name, version, digest string) (ok bool, why string, err error)
|
||||
}
|
||||
|
||||
// Store is the forge's package registry, written with the module's own write side alone.
|
||||
type Store interface {
|
||||
// Held is the sha256 digest of name@version's file when the registry holds it.
|
||||
Held(seat, name, version string) (digest string, ok bool, err error)
|
||||
// Put uploads the package's file and answers the Go module path (empty for npm).
|
||||
Put(seat string, p Publish, file []byte) (module string, err error)
|
||||
// Versions is every version of name, newest first.
|
||||
Versions(seat, name string) ([]Version, error)
|
||||
// Remove deletes name@version.
|
||||
Remove(seat, name, version string) error
|
||||
}
|
||||
|
||||
// Holder answers the verbs of both seats.
|
||||
type Holder struct {
|
||||
Callers Callers
|
||||
Objects parts.Objects
|
||||
Transfers parts.Transfers[Opening]
|
||||
Builder Builder
|
||||
Store Store
|
||||
// Say says an event on a seat's family.
|
||||
Say func(seat, event string, o Outcome) error
|
||||
// Finding writes (raised) or clears the module's finding of a kind for a subject.
|
||||
Finding func(kind, subject, words string, raised bool)
|
||||
Now func() time.Time
|
||||
Cap uint64
|
||||
}
|
||||
|
||||
func (h *Holder) now() time.Time {
|
||||
if h.Now != nil {
|
||||
return h.Now().UTC()
|
||||
}
|
||||
return time.Now().UTC()
|
||||
}
|
||||
|
||||
// Reference is the seat's answer for a package: <name>@<version> (ADR 0605 §9).
|
||||
func Reference(name, version string) string { return name + "@" + version }
|
||||
|
||||
// key is a transfer's key: the caller's own name under its seat, so the two seats' transfers never meet.
|
||||
func key(seat, transfer string) string { return seat + "_" + transfer }
|
||||
|
||||
func (h *Holder) receiver() *parts.Receiver[Opening] {
|
||||
cap := h.Cap
|
||||
if cap == 0 {
|
||||
cap = IncomingCap
|
||||
}
|
||||
return &parts.Receiver[Opening]{Objects: h.Objects, Transfers: h.Transfers, Cap: cap, MaxChunk: MaxChunk,
|
||||
Idle: TransferIdle, Full: IncomingFullPct, Now: h.now,
|
||||
// The builder's numbering is asked before anything is uploaded (ADR 0605 §4).
|
||||
Check: func(t Transfer) error {
|
||||
ok, why, err := h.Builder.Recorded(t.Shape.Name, t.Shape.Version, t.Shape.Digest)
|
||||
if err != nil {
|
||||
return errors.New("builder_versions did not answer, so nothing is stored: " + err.Error())
|
||||
}
|
||||
if !ok {
|
||||
return errors.New("the builder did not number it: " + why)
|
||||
}
|
||||
return nil
|
||||
},
|
||||
Store: func(t Transfer, held map[string][]byte) error {
|
||||
file := held[t.Shape.Digest]
|
||||
module, err := h.Store.Put(t.Shape.Seat, t.Shape.Publish, file)
|
||||
if err != nil {
|
||||
reason := "the forge did not take it: " + err.Error()
|
||||
return errors.Join(errors.New("refused: "+reason), h.refused(t, reason))
|
||||
}
|
||||
h.Finding("publish-refused", t.Shape.Name+"@"+t.Shape.Version, "", false)
|
||||
h.Finding("publish-refused", IncomingObjects, "", false)
|
||||
o := outcomeOf(t.Shape.Publish, "")
|
||||
o.Reference, o.Module = Reference(t.Shape.Name, t.Shape.Version), module
|
||||
return h.Say(t.Shape.Seat, EventPublished, o)
|
||||
},
|
||||
Refuse: h.refused,
|
||||
Filled: func(used, cap uint64, full bool) {
|
||||
h.Finding("incoming-full", IncomingObjects, fmt.Sprintf("incoming holds %d of its %d bytes (%d %%)", used, cap, used*100/cap), full)
|
||||
}}
|
||||
}
|
||||
|
||||
func (h *Holder) refused(t Transfer, reason string) error {
|
||||
subject := t.Shape.Name + "@" + t.Shape.Version
|
||||
if parts.RefusedAtCap(reason) {
|
||||
subject = IncomingObjects
|
||||
}
|
||||
h.Finding("publish-refused", subject, reason, true)
|
||||
return h.Say(t.Shape.Seat, EventRefused, outcomeOf(t.Shape.Publish, reason))
|
||||
}
|
||||
|
||||
func outcomeOf(p Publish, reason string) Outcome {
|
||||
return Outcome{Name: p.Name, Version: p.Version, Kind: p.Kind, Tag: p.Tag, Digest: p.Digest, Fingerprint: p.Fingerprint,
|
||||
Commit: p.Commit, Transfer: p.Transfer, Reason: reason}
|
||||
}
|
||||
|
||||
func (h *Holder) answer(transfer string, r parts.Received) Answer {
|
||||
return Answer{Transfer: transfer, Lacks: r.Lacks, Ended: r.Ended}
|
||||
}
|
||||
|
||||
// Publish is `publish` on seat for the proved caller: a package is one file, its digest the file's sha256.
|
||||
func (h *Holder) Publish(seat, caller string, p Publish) (Answer, error) {
|
||||
if err := h.Callers.Allowed(VerbPublish, caller); err != nil {
|
||||
return Answer{}, err
|
||||
}
|
||||
if err := checkPublish(seat, p); err != nil {
|
||||
return Answer{}, err
|
||||
}
|
||||
_, _ = h.receiver().DropIdle()
|
||||
// A rebuild of a published version answers it held and uploads nothing; the same version with other content is
|
||||
// refused, as the registry would refuse it.
|
||||
if d, ok, err := h.Store.Held(seat, p.Name, p.Version); err != nil {
|
||||
return Answer{}, fmt.Errorf("publish: the forge cannot say whether it holds %s@%s: %w", p.Name, p.Version, err)
|
||||
} else if ok {
|
||||
if d != p.Digest {
|
||||
reason := fmt.Sprintf("%s@%s is published with %s, not %s", p.Name, p.Version, d, p.Digest)
|
||||
return Answer{Transfer: p.Transfer, Ended: true}, h.refused(Transfer{Shape: Opening{Publish: p, Seat: seat}}, reason)
|
||||
}
|
||||
o := outcomeOf(p, "")
|
||||
o.Reference = Reference(p.Name, p.Version)
|
||||
return Answer{Transfer: p.Transfer, Ended: true}, h.Say(seat, EventPublished, o)
|
||||
}
|
||||
// The transfer's name is the caller's own (the seat's contract): none would open the seat's one nameless transfer,
|
||||
// which key() still makes a valid id.
|
||||
if strings.TrimSpace(p.Transfer) == "" {
|
||||
return Answer{}, errors.New("publish: no transfer named; the caller names its own")
|
||||
}
|
||||
got, err := h.receiver().Open(key(seat, p.Transfer), caller, p.Parts, Opening{Publish: p, Seat: seat})
|
||||
if err != nil {
|
||||
return h.answer(p.Transfer, got), fmt.Errorf("publish: %w", err)
|
||||
}
|
||||
return h.answer(p.Transfer, got), nil
|
||||
}
|
||||
|
||||
// PublishPart is `publish-part`: one chunk, from the caller that opened the transfer alone.
|
||||
func (h *Holder) PublishPart(seat, caller, transfer, part string, sequence int, last bool, data []byte) (Answer, error) {
|
||||
if err := h.Callers.Allowed(VerbPublishPart, caller); err != nil {
|
||||
return Answer{}, err
|
||||
}
|
||||
got, err := h.receiver().Take(key(seat, transfer), caller, part, sequence, last, data, func(t Transfer) error {
|
||||
if t.Shape.Seat != seat {
|
||||
return fmt.Errorf("transfer %s is %s's, not %s's", transfer, t.Shape.Seat, seat)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
return h.answer(transfer, got), fmt.Errorf("publish-part: %w", err)
|
||||
}
|
||||
return h.answer(transfer, got), nil
|
||||
}
|
||||
|
||||
// Versions is `versions`: every version the seat holds of name, newest first.
|
||||
func (h *Holder) Versions(seat, caller, name string) ([]Version, error) {
|
||||
if err := h.Callers.Allowed(VerbVersions, caller); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if _, ok := Seats[seat]; !ok {
|
||||
return nil, fmt.Errorf("versions: the forge holds no seat %s", seat)
|
||||
}
|
||||
if strings.TrimSpace(name) == "" {
|
||||
return nil, errors.New("versions: no name")
|
||||
}
|
||||
v, err := h.Store.Versions(seat, name)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("versions: the forge cannot list %s: %w", name, err)
|
||||
}
|
||||
return v, nil
|
||||
}
|
||||
|
||||
// Remove is `remove`: the builder removes a tagged version (its branch ended); an untagged version goes only by the
|
||||
// forge's own retention, never by a call. A version already gone is said removed all the same, so a retry ends.
|
||||
func (h *Holder) Remove(seat, caller, name, version, reason string) (Outcome, error) {
|
||||
if err := h.Callers.Allowed(VerbRemove, caller); err != nil {
|
||||
return Outcome{}, err
|
||||
}
|
||||
if _, ok := Seats[seat]; !ok {
|
||||
return Outcome{}, fmt.Errorf("remove: the forge holds no seat %s", seat)
|
||||
}
|
||||
if strings.TrimSpace(name) == "" || strings.TrimSpace(reason) == "" {
|
||||
return Outcome{}, errors.New("remove: a name and a reason are required")
|
||||
}
|
||||
if !strings.Contains(version, "-") {
|
||||
return Outcome{}, fmt.Errorf("remove: refused: %s@%s has no tag; only a tagged version is removed, an untagged one by retention alone", name, version)
|
||||
}
|
||||
d, held, err := h.Store.Held(seat, name, version)
|
||||
if err != nil {
|
||||
return Outcome{}, fmt.Errorf("remove: the forge cannot read %s@%s: %w", name, version, err)
|
||||
}
|
||||
if held {
|
||||
if err := h.Store.Remove(seat, name, version); err != nil {
|
||||
return Outcome{}, fmt.Errorf("remove: the forge did not remove %s@%s: %w", name, version, err)
|
||||
}
|
||||
}
|
||||
o := Outcome{Name: name, Version: version, Kind: KindPackage, Digest: d, Reason: reason}
|
||||
return o, h.Say(seat, EventRemoved, o)
|
||||
}
|
||||
|
||||
// Retained says `removed` for a version the forge's own retention deleted (design 51 *Retention*).
|
||||
func (h *Holder) Retained(seat, name, version, digest string) error {
|
||||
return h.Say(seat, EventRemoved, Outcome{Name: name, Version: version, Kind: KindPackage, Digest: digest, Reason: ReasonRetention})
|
||||
}
|
||||
|
||||
// Drop drops a transfer by hand; DropIdle every transfer idle for TransferIdle.
|
||||
func (h *Holder) Drop(seat, transfer string) (Transfer, error) {
|
||||
return h.receiver().Drop(key(seat, transfer))
|
||||
}
|
||||
|
||||
func checkPublish(seat string, p Publish) error {
|
||||
switch {
|
||||
case Seats[seat] == "":
|
||||
return fmt.Errorf("publish: the forge holds no seat %s", seat)
|
||||
case strings.TrimSpace(p.Name) == "":
|
||||
return errors.New("publish: no name")
|
||||
case strings.TrimSpace(p.Version) == "":
|
||||
return errors.New("publish: a package is published under the version the builder gave it")
|
||||
case p.Kind != KindPackage:
|
||||
return fmt.Errorf("publish: %s holds packages, not a %s", seat, p.Kind)
|
||||
case !strings.HasPrefix(p.Digest, "sha256:"):
|
||||
return fmt.Errorf("publish: digest %q is no sha256 digest", p.Digest)
|
||||
case len(p.Parts) != 1 || p.Parts[0].Digest != p.Digest:
|
||||
return errors.New("publish: a package is one file, its one part the package's digest")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,249 @@
|
||||
package packageseats
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
stdio "git.novox.be/novox/mesh-sdk/go"
|
||||
"git.novox.be/novox/mesh-sdk/go/parts"
|
||||
)
|
||||
|
||||
type memObjects map[string][]byte
|
||||
|
||||
func (m memObjects) Put(n string, d []byte) error { m[n] = append([]byte(nil), d...); return nil }
|
||||
func (m memObjects) Get(n string) ([]byte, error) {
|
||||
if d, ok := m[n]; ok {
|
||||
return d, nil
|
||||
}
|
||||
return nil, errors.New("no object " + n)
|
||||
}
|
||||
func (m memObjects) Delete(n string) error { delete(m, n); return nil }
|
||||
func (m memObjects) Used() (uint64, error) {
|
||||
var u uint64
|
||||
for _, d := range m {
|
||||
u += uint64(len(d))
|
||||
}
|
||||
return u, nil
|
||||
}
|
||||
|
||||
type memTransfers map[string]Transfer
|
||||
|
||||
func (m memTransfers) Get(id string) (Transfer, bool, error) { t, ok := m[id]; return t, ok, nil }
|
||||
func (m memTransfers) Put(t Transfer) error { m[t.ID] = t; return nil }
|
||||
func (m memTransfers) Delete(id string) error { delete(m, id); return nil }
|
||||
func (m memTransfers) List() ([]Transfer, error) {
|
||||
var out []Transfer
|
||||
for _, t := range m {
|
||||
out = append(out, t)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
type builder struct{ ok bool }
|
||||
|
||||
func (b builder) Recorded(_, _, _ string) (bool, string, error) { return b.ok, "not numbered", nil }
|
||||
|
||||
// forge is the forge's package registry: seat -> name@version -> file.
|
||||
type forge struct {
|
||||
files map[string]map[string][]byte
|
||||
removed []string
|
||||
}
|
||||
|
||||
func (f *forge) Held(seat, name, version string) (string, bool, error) {
|
||||
b, ok := f.files[seat][name+"@"+version]
|
||||
if !ok {
|
||||
return "", false, nil
|
||||
}
|
||||
return digest(b), true, nil
|
||||
}
|
||||
func (f *forge) Put(seat string, p Publish, file []byte) (string, error) {
|
||||
if f.files[seat] == nil {
|
||||
f.files[seat] = map[string][]byte{}
|
||||
}
|
||||
f.files[seat][p.Name+"@"+p.Version] = file
|
||||
if seat == GoSeat {
|
||||
return "git.example/" + p.Name, nil
|
||||
}
|
||||
return "", nil
|
||||
}
|
||||
func (f *forge) Versions(seat, name string) ([]Version, error) {
|
||||
var out []Version
|
||||
for k, b := range f.files[seat] {
|
||||
if n, v, _ := strings.Cut(k, "@"); n == name {
|
||||
out = append(out, Version{Version: v, Digest: digest(b)})
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
func (f *forge) Remove(seat, name, version string) error {
|
||||
delete(f.files[seat], name+"@"+version)
|
||||
f.removed = append(f.removed, seat+" "+name+"@"+version)
|
||||
return nil
|
||||
}
|
||||
|
||||
func digest(b []byte) string { s := sha256.Sum256(b); return "sha256:" + hex.EncodeToString(s[:]) }
|
||||
|
||||
type said struct {
|
||||
seat, event string
|
||||
o Outcome
|
||||
}
|
||||
|
||||
type bed struct {
|
||||
h *Holder
|
||||
forge *forge
|
||||
said []said
|
||||
findings map[string]bool
|
||||
}
|
||||
|
||||
func newBed(numbered bool) *bed {
|
||||
b := &bed{forge: &forge{files: map[string]map[string][]byte{}}, findings: map[string]bool{}}
|
||||
b.h = &Holder{Callers: DefaultCallers(), Objects: memObjects{}, Transfers: memTransfers{}, Builder: builder{ok: numbered},
|
||||
Store: b.forge, Now: func() time.Time { return time.Unix(1e9, 0) },
|
||||
Say: func(seat, event string, o Outcome) error { b.said = append(b.said, said{seat, event, o}); return nil },
|
||||
Finding: func(kind, subject, _ string, raised bool) { b.findings[kind+" "+subject] = raised }}
|
||||
return b
|
||||
}
|
||||
|
||||
func pub(name, version string, file []byte) Publish {
|
||||
return Publish{Transfer: "t-" + name, Name: name, Version: version, Kind: KindPackage, Digest: digest(file),
|
||||
Parts: []parts.Want{{Digest: digest(file), Size: int64(len(file))}}}
|
||||
}
|
||||
|
||||
const goBuild, nodeBuild = "ace.mesh-build-go", "ace.mesh-build-node"
|
||||
|
||||
// **The forge module holds both package seats** (design 51 *Publishing*, *Testing* for the holder): a package's one
|
||||
// file comes in parts, is checked against the builder's numbering and its digest, and is uploaded with the forge's own
|
||||
// write side; the outcome is said on the seat's events.
|
||||
func TestTheForgeHoldsBothPackageSeats(t *testing.T) {
|
||||
file := []byte("a package's tarball")
|
||||
t.Run("a publish in parts is uploaded and said published", func(t *testing.T) {
|
||||
for _, seat := range []string{NpmSeat, GoSeat} {
|
||||
b := newBed(true)
|
||||
a, err := b.h.Publish(seat, goBuild, pub("mesh-sdk", "0.1.15", file))
|
||||
if err != nil || len(a.Lacks) != 1 || a.Transfer != "t-mesh-sdk" {
|
||||
t.Fatalf("%s: publish answered %+v, %v", seat, a, err)
|
||||
}
|
||||
a, err = b.h.PublishPart(seat, goBuild, "t-mesh-sdk", digest(file), 0, true, file)
|
||||
if err != nil || !a.Ended {
|
||||
t.Fatalf("%s: the last chunk answered %+v, %v", seat, a, err)
|
||||
}
|
||||
if len(b.said) != 1 || b.said[0].event != EventPublished || b.said[0].o.Reference != "mesh-sdk@0.1.15" {
|
||||
t.Fatalf("%s: said %+v", seat, b.said)
|
||||
}
|
||||
if (seat == GoSeat) != (b.said[0].o.Module != "") {
|
||||
t.Errorf("%s: the Go module path said is %q", seat, b.said[0].o.Module)
|
||||
}
|
||||
if len(b.h.Objects.(memObjects)) != 0 || len(b.h.Transfers.(memTransfers)) != 0 {
|
||||
t.Errorf("%s: the transfer's chunks or key stayed after its end", seat)
|
||||
}
|
||||
if v, _ := b.h.Versions(seat, "g14.anyone", "mesh-sdk"); len(v) != 1 || v[0].Digest != digest(file) {
|
||||
t.Errorf("%s: versions answered %+v", seat, v)
|
||||
}
|
||||
// A rebuild of the published version is held: nothing uploaded, published said again.
|
||||
if a, err := b.h.Publish(seat, goBuild, pub("mesh-sdk", "0.1.15", file)); err != nil || !a.Ended || len(a.Lacks) != 0 {
|
||||
t.Errorf("%s: a held version answered %+v, %v", seat, a, err)
|
||||
}
|
||||
}
|
||||
})
|
||||
t.Run("refusals", func(t *testing.T) {
|
||||
for _, c := range []struct {
|
||||
name string
|
||||
do func(b *bed) error
|
||||
want string
|
||||
}{
|
||||
{"a caller not on publish's list", func(b *bed) error {
|
||||
_, err := b.h.Publish(NpmSeat, "ace.claude-code", pub("x", "0.1.0", file))
|
||||
return err
|
||||
}, "not on the holder's caller list"},
|
||||
{"an unproven caller", func(b *bed) error {
|
||||
_, err := b.h.Publish(NpmSeat, stdio.Unproven, pub("x", "0.1.0", file))
|
||||
return err
|
||||
}, "no caller the tool runner proved"},
|
||||
{"a seat the forge does not hold", func(b *bed) error {
|
||||
_, err := b.h.Publish("cargo-package-registry", goBuild, pub("x", "0.1.0", file))
|
||||
return err
|
||||
}, "holds no seat"},
|
||||
{"a package in two files", func(b *bed) error {
|
||||
p := pub("x", "0.1.0", file)
|
||||
p.Parts = append(p.Parts, parts.Want{Digest: digest([]byte("b")), Size: 1})
|
||||
_, err := b.h.Publish(NpmSeat, goBuild, p)
|
||||
return err
|
||||
}, "one file"},
|
||||
{"a part from another module than its opener", func(b *bed) error {
|
||||
if _, err := b.h.Publish(NpmSeat, nodeBuild, pub("x", "0.1.0", file)); err != nil {
|
||||
return err
|
||||
}
|
||||
_, err := b.h.PublishPart(NpmSeat, goBuild, "t-x", digest(file), 0, true, file)
|
||||
return err
|
||||
}, "opened"},
|
||||
{"a transfer of the other seat", func(b *bed) error {
|
||||
if _, err := b.h.Publish(NpmSeat, nodeBuild, pub("x", "0.1.0", file)); err != nil {
|
||||
return err
|
||||
}
|
||||
_, err := b.h.PublishPart(GoSeat, nodeBuild, "t-x", digest(file), 0, true, file)
|
||||
return err
|
||||
}, "not open"},
|
||||
{"remove by a build module", func(b *bed) error { _, err := b.h.Remove(GoSeat, goBuild, "x", "0.1.1-feat.1", "tag-gone"); return err }, "not on the holder's caller list"},
|
||||
{"remove of a version with no tag", func(b *bed) error {
|
||||
_, err := b.h.Remove(GoSeat, "novox.mesh-delivery-builder", "x", "0.1.1", "tag-gone")
|
||||
return err
|
||||
}, "has no tag"},
|
||||
} {
|
||||
b := newBed(true)
|
||||
if err := c.do(b); err == nil || !strings.Contains(err.Error(), c.want) {
|
||||
t.Errorf("%s: %v, want %q", c.name, err, c.want)
|
||||
}
|
||||
}
|
||||
})
|
||||
t.Run("a refused publish is said refused and raises the finding; nothing is uploaded", func(t *testing.T) {
|
||||
for _, c := range []struct {
|
||||
name string
|
||||
numbered bool
|
||||
data []byte
|
||||
want string
|
||||
}{{"the builder did not number it", false, file, "did not number"}} {
|
||||
b := newBed(c.numbered)
|
||||
if _, err := b.h.Publish(NpmSeat, nodeBuild, pub("x", "0.1.0", file)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_, _ = b.h.PublishPart(NpmSeat, nodeBuild, "t-x", digest(file), 0, true, c.data)
|
||||
if len(b.said) != 1 || b.said[0].event != EventRefused || !strings.Contains(b.said[0].o.Reason, c.want) {
|
||||
t.Errorf("%s: said %+v", c.name, b.said)
|
||||
}
|
||||
if !b.findings["publish-refused x@0.1.0"] || len(b.forge.files[NpmSeat]) != 0 {
|
||||
t.Errorf("%s: finding %v, uploaded %v", c.name, b.findings, b.forge.files)
|
||||
}
|
||||
}
|
||||
// A part whose bytes do not match its digest is refused to its sender, to send again; nothing is said yet.
|
||||
b := newBed(true)
|
||||
if _, err := b.h.Publish(NpmSeat, nodeBuild, pub("x", "0.1.0", file)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := b.h.PublishPart(NpmSeat, nodeBuild, "t-x", digest(file), 0, true, []byte("other bytes")); err == nil ||
|
||||
!strings.Contains(err.Error(), "send the part again") || len(b.said) != 0 || len(b.forge.files[NpmSeat]) != 0 {
|
||||
t.Errorf("a part with other bytes: %v, said %+v", err, b.said)
|
||||
}
|
||||
// The same version published with other content is refused, never uploaded over.
|
||||
b = newBed(true)
|
||||
b.forge.files[NpmSeat] = map[string][]byte{"x@0.1.0": []byte("first")}
|
||||
if _, err := b.h.Publish(NpmSeat, nodeBuild, pub("x", "0.1.0", file)); err != nil || len(b.said) != 1 || b.said[0].event != EventRefused {
|
||||
t.Errorf("a published version with other content: %v, said %+v", err, b.said)
|
||||
}
|
||||
})
|
||||
t.Run("the builder removes a tagged version; one already gone is said removed all the same", func(t *testing.T) {
|
||||
b := newBed(true)
|
||||
b.forge.files[GoSeat] = map[string][]byte{"x@0.1.1-feat.1": file}
|
||||
for range 2 {
|
||||
if _, err := b.h.Remove(GoSeat, "novox.mesh-delivery-builder", "x", "0.1.1-feat.1", "tag-gone"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if len(b.forge.removed) != 1 || len(b.said) != 2 || b.said[0].event != EventRemoved || b.said[0].o.Digest != digest(file) {
|
||||
t.Errorf("removed %v, said %+v", b.forge.removed, b.said)
|
||||
}
|
||||
})
|
||||
}
|
||||
+87
-47
@@ -5,7 +5,28 @@
|
||||
"requires": [
|
||||
"postgres-database",
|
||||
"route",
|
||||
"secret"
|
||||
"secret",
|
||||
{
|
||||
"name": "incoming",
|
||||
"storage": "object-store",
|
||||
"template": "objects",
|
||||
"options": {
|
||||
"max-mib": 2048
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "transfers",
|
||||
"storage": "key-value",
|
||||
"template": "state",
|
||||
"options": {
|
||||
"ttl-seconds": 86400
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "findings",
|
||||
"storage": "key-value",
|
||||
"template": "state"
|
||||
}
|
||||
],
|
||||
"contributes": {
|
||||
"postgres-database": {
|
||||
@@ -99,7 +120,7 @@
|
||||
"port": 22,
|
||||
"protocol": "tcp",
|
||||
"from": "mesh",
|
||||
"why": "git over ssh, gitea's own unmodified sshd. Published on the machine's own side at 222, the mesh's fixed public convention — not 22, which the machine's own daemon holds and a module does not take"
|
||||
"why": "git over ssh, gitea's own unmodified sshd. Published on the machine's own side at 222, the mesh's fixed public convention \u2014 not 22, which the machine's own daemon holds and a module does not take"
|
||||
},
|
||||
{
|
||||
"name": "forge-events",
|
||||
@@ -119,18 +140,33 @@
|
||||
"git": {
|
||||
"scheme": "http",
|
||||
"port": 3000
|
||||
},
|
||||
"go-package-registry": {
|
||||
"scheme": "http",
|
||||
"port": 3000,
|
||||
"go-path": "/api/packages/novox/go"
|
||||
}
|
||||
},
|
||||
"receives": {
|
||||
"npm-package-registry": "${dir:grants}/npm.json"
|
||||
},
|
||||
"grants": {
|
||||
"npm-package-registry": "${dir:grants}"
|
||||
},
|
||||
"claims": [
|
||||
{
|
||||
"name": "npm-package-registry",
|
||||
"scope": "mesh"
|
||||
"scope": "mesh",
|
||||
"serves": [
|
||||
"publish",
|
||||
"publish-part",
|
||||
"remove",
|
||||
"versions"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "go-package-registry",
|
||||
"scope": "mesh",
|
||||
"serves": [
|
||||
"publish",
|
||||
"publish-part",
|
||||
"remove",
|
||||
"versions"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "git",
|
||||
@@ -145,14 +181,7 @@
|
||||
"class": "valuable",
|
||||
"why": "every repository, its issues and attachments, and the package registry"
|
||||
}
|
||||
],
|
||||
"consumers": {
|
||||
"npm-package-registry": {
|
||||
"class": "rebuildable",
|
||||
"in": "data",
|
||||
"why": "a consumer's packages are published again from its source"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"config": {
|
||||
"url": {
|
||||
@@ -169,7 +198,7 @@
|
||||
},
|
||||
"keep-main-rules": {
|
||||
"type": "boolean",
|
||||
"description": "Whether the module keeps main's branch protection as the mesh wants it. True: every ten minutes, and at the start, main's rule in every repository of the organisation novox is put to no push and the tools' account alone merging, and each apply of the `repositories` value narrows the main rule of the repositories it lists, requiring no status. False (the default): the module changes no existing repository's branch protection, whoever set it at the forge, but takes away a required status it finds on a rule covering main, once (ADR 0534 §1); a repository listed in `repositories` that does not exist is still made, with its first protection.",
|
||||
"description": "Whether the module keeps main's branch protection as the mesh wants it. True: every ten minutes, and at the start, main's rule in every repository of the organisation novox is put to no push and the tools' account alone merging, and each apply of the `repositories` value narrows the main rule of the repositories it lists, requiring no status. False (the default): the module changes no existing repository's branch protection, whoever set it at the forge, but takes away a required status it finds on a rule covering main, once (ADR 0534 \u00a71); a repository listed in `repositories` that does not exist is still made, with its first protection.",
|
||||
"trusted": true,
|
||||
"default": false
|
||||
},
|
||||
@@ -181,15 +210,23 @@
|
||||
},
|
||||
"check-gates-merge": {
|
||||
"type": "boolean",
|
||||
"description": "Whether gitea_merge_pull_request judges mesh/check before it merges. True (the default): a head merges only on a passing mesh/check judged against the trunk's head (ADR 0447, ADR 0534 §4). False (the operator, 2026-10-25 22:56: \"run the check locally only, but not as a gate, it loses us too much time\"): a head the operator's Approve names in approved_heads merges whatever its mesh/check says (pending, failed, missing or judged on an older trunk); a head that moved, a branch no rule guards and a merge git refuses are still refused. Set together with mesh-delivery-gate's check-gates-merge.",
|
||||
"description": "Whether gitea_merge_pull_request judges mesh/check before it merges. True (the default): a head merges only on a passing mesh/check judged against the trunk's head (ADR 0447, ADR 0534 \u00a74). False (the operator, 2026-10-25 22:56: \"run the check locally only, but not as a gate, it loses us too much time\"): a head the operator's Approve names in approved_heads merges whatever its mesh/check says (pending, failed, missing or judged on an older trunk); a head that moved, a branch no rule guards and a merge git refuses are still refused. Set together with mesh-delivery-gate's check-gates-merge.",
|
||||
"trusted": true,
|
||||
"default": true
|
||||
},
|
||||
"main-mergers": {
|
||||
"type": "list",
|
||||
"description": "The forge logins that may merge into main beside the tools' account, on every repository of the organisation (ADR 0534 §2): the operator's own, so the operator can always merge by hand at the forge. The tools' account is always on main's merge whitelist; no agent's account belongs here.",
|
||||
"description": "The forge logins that may merge into main beside the tools' account, on every repository of the organisation (ADR 0534 \u00a72): the operator's own, so the operator can always merge by hand at the forge. The tools' account is always on main's merge whitelist; no agent's account belongs here.",
|
||||
"trusted": true,
|
||||
"default": ["jschoubben"]
|
||||
"default": [
|
||||
"jschoubben"
|
||||
]
|
||||
},
|
||||
"callers": {
|
||||
"type": "string",
|
||||
"trusted": true,
|
||||
"default": "",
|
||||
"description": "per verb of the two package seats, the callers that may call it (JSON object of verb to callers; * every proved caller, operator the operator's session); empty is design 51's list: publish and publish-part mesh-build-go and mesh-build-node, remove mesh-delivery-builder, versions anyone"
|
||||
}
|
||||
},
|
||||
"facts": {
|
||||
@@ -228,12 +265,6 @@
|
||||
"place": ".",
|
||||
"unknown": "program"
|
||||
},
|
||||
{
|
||||
"id": "grants",
|
||||
"type": "directory",
|
||||
"mode": "0700",
|
||||
"unknown": "program"
|
||||
},
|
||||
{
|
||||
"id": "server-env",
|
||||
"type": "file",
|
||||
@@ -397,16 +428,23 @@
|
||||
"provides": [
|
||||
{
|
||||
"name": "npm-package-registry",
|
||||
"credential": {"form": "pair"},
|
||||
"scope": "mesh",
|
||||
"identity": {
|
||||
"max": 40,
|
||||
"in": "a Gitea user name"
|
||||
}
|
||||
"credential": {
|
||||
"form": "none"
|
||||
},
|
||||
"scope": "mesh"
|
||||
},
|
||||
{
|
||||
"name": "go-package-registry",
|
||||
"credential": {
|
||||
"form": "none"
|
||||
},
|
||||
"scope": "mesh"
|
||||
},
|
||||
{
|
||||
"name": "git",
|
||||
"credential": {"form": "none"},
|
||||
"credential": {
|
||||
"form": "none"
|
||||
},
|
||||
"scope": "mesh"
|
||||
},
|
||||
{
|
||||
@@ -419,7 +457,9 @@
|
||||
},
|
||||
{
|
||||
"name": "forge-account",
|
||||
"credential": {"form": "none"},
|
||||
"credential": {
|
||||
"form": "none"
|
||||
},
|
||||
"scope": "mesh",
|
||||
"answered-by": "helper"
|
||||
}
|
||||
@@ -432,13 +472,11 @@
|
||||
"language": "typescript",
|
||||
"entrypoints": [
|
||||
"index.js",
|
||||
"tools/index.js",
|
||||
"provisioner/index.js"
|
||||
"tools/index.js"
|
||||
],
|
||||
"loads": [
|
||||
"index.js",
|
||||
"tools/index.js",
|
||||
"provisioner/index.js"
|
||||
"tools/index.js"
|
||||
],
|
||||
"env": {
|
||||
"MESH_GITEA_URL": "http://127.0.0.1:${port:3000}",
|
||||
@@ -447,26 +485,27 @@
|
||||
"MESH_GITEA_ADMIN_PASSWORD_FILE": "${dir:state}/admin.secret",
|
||||
"MESH_GITEA_TOOLS_USER": "mesh-tools",
|
||||
"MESH_GITEA_STATE_DIR": "${dir:runtime-state}",
|
||||
"MESH_RECEIVES": "${dir:grants}/npm.json",
|
||||
"MESH_GITEA_HOOK_URL": "http://172.17.0.1:3047",
|
||||
"MESH_GITEA_HOOK_PORT": "3047",
|
||||
"MESH_GITEA_HOOK_HOST": "172.17.0.1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "npm-registry",
|
||||
"name": "package-registry",
|
||||
"kind": "bundle",
|
||||
"language": "go",
|
||||
"system": "arch",
|
||||
"from": "cmd/npm-registry",
|
||||
"binary": "npm-registry",
|
||||
"from": "cmd/package-registry",
|
||||
"binary": "package-registry",
|
||||
"loads": [
|
||||
"npm-registry"
|
||||
"package-registry"
|
||||
],
|
||||
"env": {
|
||||
"MESH_GITEA_URL": "http://127.0.0.1:${port:3000}",
|
||||
"MESH_GITEA_ADMIN_USER": "mesh-admin",
|
||||
"MESH_GITEA_ADMIN_PASSWORD_FILE": "${dir:state}/admin.secret",
|
||||
"MESH_GITEA_TOOLS_USER": "mesh-tools",
|
||||
"MESH_GITEA_TOKEN_FILE": "${dir:runtime-state}/token",
|
||||
"MESH_GITEA_PACKAGES_USER": "mesh-packages",
|
||||
"MESH_GITEA_PACKAGES_TOKEN_FILE": "${dir:runtime-state}/packages-token",
|
||||
"MESH_NPM_OWNER": "novox"
|
||||
}
|
||||
},
|
||||
@@ -577,7 +616,8 @@
|
||||
],
|
||||
"invokes": [
|
||||
"mesh-delivery-indexer.indexer_read_modules",
|
||||
"seat:mesh-hand-acts.hand-act"
|
||||
"seat:mesh-hand-acts.hand-act",
|
||||
"mesh-delivery-builder.builder_versions"
|
||||
],
|
||||
"consumes": [
|
||||
"mesh-delivery-indexer.views.modules"
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
"type": "module",
|
||||
"private": true,
|
||||
"scripts": {
|
||||
"build": "tsc client.ts token.ts index.ts provisioner/index.ts provisioner/teams.ts tools/index.ts --module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist",
|
||||
"build": "tsc client.ts token.ts index.ts tools/index.ts --module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist",
|
||||
"test": "npm run build && tsc --module NodeNext --moduleResolution NodeNext --target ES2022 --types node --rootDir . --outDir . $(find . -name '*.ts' ! -name '*.d.ts' ! -path './test/*' ! -path './node_modules/*' ! -path './dist/*') && node --test --experimental-strip-types 'test/*.test.ts'"
|
||||
},
|
||||
"dependencies": {
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
// The package seats write as an account of their own, never the administrator (design 51: each holder keeps its write
|
||||
// side as an own secret beside a read-only tool token; the forge module's is `write:package`, its tool token
|
||||
// `read:package`).
|
||||
//
|
||||
// The Go bundle (cmd/package-registry) serves both package seats' verbs: it reads the registry with the tools' token
|
||||
// (token.ts, which carries `read:package`) and writes (a publish, a remove, retention's deletion) with this one:
|
||||
//
|
||||
// - **the account**: `mesh-packages`, plain, made by the module with the administrator's login as tools-account.ts
|
||||
// makes `mesh-tools`; its password random, sent once and kept nowhere.
|
||||
// - **the team**: `mesh-packages` in the organisation that owns the registry, writing packages and nothing else:
|
||||
// no code, no issues, no pull requests, no administration.
|
||||
// - **the token**: minted for the account by the administrator's login with the one scope `write:package`, kept at
|
||||
// 0600 in the module's runtime state as `packages-token`, its scopes line beside it saying whose it is
|
||||
// (MintedToken: one mint at a time across the bundle's processes, re-minted when the kept one is gone or other).
|
||||
//
|
||||
// The watcher mints it at its start (keepPackagesToken). No timer looks at it after that: the Go bundle mints nothing,
|
||||
// and when the forge refuses its write it raises the finding PACKAGES_TOKEN_REFUSED in the module's findings state; the
|
||||
// watcher, watching that key, looks again (a refused token is minted afresh) and clears the finding (renewOnRefusal).
|
||||
|
||||
import { randomBytes } from "node:crypto";
|
||||
import { join } from "node:path";
|
||||
import { MintedToken } from "./token.js";
|
||||
import { TOOLS_ORG, type ToolsForge } from "./tools-account.js";
|
||||
|
||||
/** The account the package seats write as (internal/forgehttp PackagesAccount). */
|
||||
export const PACKAGES_ACCOUNT = "mesh-packages";
|
||||
/** Its team in the registry's organisation, and the token's name in the account's list. */
|
||||
export const PACKAGES_TEAM = "mesh-packages";
|
||||
export const PACKAGES_TOKEN_NAME = "mesh-packages";
|
||||
/** The one scope the write side needs; reads go with the tools' token. */
|
||||
export const PACKAGES_TOKEN_SCOPES: readonly string[] = ["write:package"];
|
||||
/** Where the token is kept in the module's runtime state (the manifest's MESH_GITEA_PACKAGES_TOKEN_FILE). */
|
||||
export const PACKAGES_TOKEN_FILE = "packages-token";
|
||||
|
||||
/** What the team grants: write on the organisation's packages, and no more. */
|
||||
export const PACKAGES_TEAM_UNITS = {
|
||||
units_map: { "repo.packages": "write" },
|
||||
includes_all_repositories: false,
|
||||
can_create_org_repo: false,
|
||||
} as const;
|
||||
|
||||
/** Make the packages account, plain, a member of its team. Its password is set afresh each time and kept nowhere. */
|
||||
export async function setUpPackagesAccount(forge: ToolsForge, account = PACKAGES_ACCOUNT, password: () => string): Promise<void> {
|
||||
await forge.ensureUser(account, password(), `${account}@noreply.localhost`, true);
|
||||
const team = await forge.ensureTeamWith(TOOLS_ORG, PACKAGES_TEAM, { name: PACKAGES_TEAM, ...PACKAGES_TEAM_UNITS });
|
||||
await forge.addUserToTeam(team, account);
|
||||
}
|
||||
|
||||
/** The packages account's token, minted by the administrator's login from what the manifest hands the watcher. */
|
||||
export function packagesToken(url: string, env: NodeJS.ProcessEnv = process.env, log?: (line: string) => void): MintedToken {
|
||||
const admin = env.MESH_GITEA_ADMIN_USER;
|
||||
const passwordFile = env.MESH_GITEA_ADMIN_PASSWORD_FILE;
|
||||
const stateDir = env.MESH_GITEA_STATE_DIR;
|
||||
const missing = [
|
||||
admin ? null : "MESH_GITEA_ADMIN_USER",
|
||||
passwordFile ? null : "MESH_GITEA_ADMIN_PASSWORD_FILE",
|
||||
stateDir ? null : "MESH_GITEA_STATE_DIR",
|
||||
].filter((v): v is string => v !== null);
|
||||
if (missing.length) throw new Error(`the package seats' write token cannot be minted: ${missing.join(", ")} not set by the manifest`);
|
||||
const account = env.MESH_GITEA_PACKAGES_USER || PACKAGES_ACCOUNT;
|
||||
if (account === admin) throw new Error(`MESH_GITEA_PACKAGES_USER names the admin account "${admin}"; the package seats write as an account of their own`);
|
||||
return new MintedToken({
|
||||
url, admin: admin!, passwordFile: passwordFile!, file: join(stateDir!, PACKAGES_TOKEN_FILE), account,
|
||||
name: PACKAGES_TOKEN_NAME, scopes: PACKAGES_TOKEN_SCOPES, log,
|
||||
setUp: async (password) => {
|
||||
const { GiteaAdmin } = await import("./client.js");
|
||||
await setUpPackagesAccount(new GiteaAdmin(url, admin!, password), account, () => randomBytes(24).toString("base64url"));
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* One look at the packages token: mint it when none is kept, then ask the forge one read the scope covers (the
|
||||
* organisation's package list); a 401 renews it. Answers what was done, for the watcher's line.
|
||||
*/
|
||||
export async function keepPackagesToken(source: MintedToken, url: string, owner = TOOLS_ORG, fetchImpl: typeof fetch = fetch): Promise<string> {
|
||||
const token = await source.current();
|
||||
const look = (t: string) => fetchImpl(`${url.replace(/\/+$/, "")}/api/v1/packages/${encodeURIComponent(owner)}?limit=1`,
|
||||
{ headers: { Authorization: `token ${t}`, Accept: "application/json" } });
|
||||
const res = await look(token);
|
||||
if (res.status === 401 || MintedToken.lacksScope(res.status, await res.text())) {
|
||||
const fresh = await source.renew(token);
|
||||
const again = await look(fresh);
|
||||
if (!again.ok) throw new Error(`the forge answered ${again.status} to the renewed package token's read`);
|
||||
return "the forge rejected the package seats' write token; minted afresh";
|
||||
}
|
||||
if (!res.ok) throw new Error(`the forge answered ${res.status} to the package token's read`);
|
||||
return "the package seats' write token holds";
|
||||
}
|
||||
|
||||
/** The finding the Go bundle raises in the module's findings state when the forge refuses the write token (cmd/package-registry PackagesTokenRefusedKey). */
|
||||
export const PACKAGES_TOKEN_REFUSED = "seat.gitea.packages-token-refused_write";
|
||||
|
||||
/** One change of the findings state, as its watch hands it. */
|
||||
export interface FindingChange { readonly key: string; readonly op: "put" | "delete" }
|
||||
|
||||
/**
|
||||
* The watcher's answer to the Go bundle's refusal: on the finding's put, look at the token (a refused one is minted
|
||||
* afresh) and clear the finding; a look that fails leaves the finding standing, said. Other keys are not its own.
|
||||
*/
|
||||
export function renewOnRefusal(source: MintedToken, url: string, findings: { delete(key: string): Promise<void> },
|
||||
log: (line: string) => void, fetchImpl: typeof fetch = fetch): (change: FindingChange) => Promise<void> {
|
||||
return async (change) => {
|
||||
if (change.op !== "put" || change.key !== PACKAGES_TOKEN_REFUSED) return;
|
||||
try {
|
||||
log(await keepPackagesToken(source, url, TOOLS_ORG, fetchImpl));
|
||||
} catch (err) {
|
||||
log(`the package seats' write token was refused and could not be minted afresh; the finding stands — ${err instanceof Error ? err.message : err}`);
|
||||
return;
|
||||
}
|
||||
await findings.delete(change.key);
|
||||
};
|
||||
}
|
||||
@@ -1,54 +0,0 @@
|
||||
// gitea's provisioner — the adapter that makes gitea a provider of the mesh
|
||||
// `npm-package-registry` interface. The reconcile loop, the contributions file, and reading the
|
||||
// mesh's minted password are the sdk harness's; this writes only the per-service half: how gitea
|
||||
// creates and removes a consumer's npm credential (novox/hq ADR 0048/0076).
|
||||
//
|
||||
// **A package registry seat is one per ecosystem (novox/hq ADR 0109).** gitea holds the npm seat
|
||||
// (ADR 0110). Adding cargo or PyPI is adding a provision — another `provides` entry, another
|
||||
// `receives` path and another registration below — not widening this one. `git`, which gitea also
|
||||
// provides, mints nothing and so registers nothing here: the mesh's own repositories are public,
|
||||
// and a clone credential is not yet decided (ADR 0111).
|
||||
//
|
||||
// The `npm-package-registry` interface: a consumer authenticates to the npm registry at
|
||||
// `/api/packages/novox/npm/` with basic auth, as `as` with the password the mesh minted, and can
|
||||
// read packages under the `@novox` scope, and write them unless it asked for read only (teams.ts). The
|
||||
// registry's npm owner is the gitea org `novox`; a consumer is a gitea *user* on one of that org's teams.
|
||||
//
|
||||
// **The user name and password are the mesh's, not the provisioner's (ADR 0048).** The mesh derives
|
||||
// the login and hands it to both ends, and mints the password. gitea creates a user under exactly
|
||||
// that login and sets exactly that password every run — so a rotation takes — and seals nothing: the
|
||||
// consumer already has its copy through the mesh's own channel.
|
||||
//
|
||||
// The admin calls run through GiteaAdmin (basic auth as the mesh's gitea admin), which is the
|
||||
// module's one boundary to the forge's admin API (see client.ts).
|
||||
|
||||
import { runProvisioner, type Provision } from "@novox/mesh-sdk/provisioner";
|
||||
import { GiteaAdmin } from "../client.js";
|
||||
import { grant, holds } from "./teams.js";
|
||||
|
||||
// Which team a consumer joins — `packages` to publish, `readers` to read only — is teams.ts's.
|
||||
|
||||
const gitea = GiteaAdmin.fromEnv();
|
||||
|
||||
// Where this registration's contributions land comes from $MESH_RECEIVES, never a path written
|
||||
// here: the mesh writes the file where the manifest's `receives` says, and a second copy of that
|
||||
// path in code would drift from it. One variable carries one path, so a second registration in this
|
||||
// module needs the mesh to say where each provision's file is — not yet possible, and not faked.
|
||||
runProvisioner("npm-package-registry", {
|
||||
async create(p: Provision): Promise<void> {
|
||||
// The org and its teams are ensured per create: idempotent, and no separate bootstrap step. The
|
||||
// user carries the consumer's login and the mesh's minted password, set every run so a rotation
|
||||
// takes; membership of its team is what grants it what it may do.
|
||||
await grant(gitea, p);
|
||||
},
|
||||
|
||||
async remove(p: { as: string }): Promise<void> {
|
||||
// Login prohibited, never deleted (novox/hq issue 241: a withdrawal never destroys a consumer's data — on 2026-10-04 a misread grants file withdrew every consumer at once): deleting purges every repository the user owns.
|
||||
await gitea.prohibitLogin(p.as);
|
||||
},
|
||||
// Asked every minute by the harness: whether the backend still holds this consumer exactly as
|
||||
// the mesh gave it, so a login lost behind the provisioner's back is made again (novox/hq issue 120).
|
||||
async holds(p: Provision): Promise<boolean> {
|
||||
return holds(gitea, p);
|
||||
},
|
||||
});
|
||||
@@ -1,64 +0,0 @@
|
||||
// Which team of the forge's npm owner a consumer of `npm-package-registry` joins, and the per-consumer
|
||||
// half of the provisioner around it — pure of the forge's address, so it is tested with a stand-in.
|
||||
//
|
||||
// **Two teams, by what the consumer said it needs** (novox/hq issue 624). A consumer that publishes —
|
||||
// the build modules, whose builds publish the SDK and every TypeScript module's packages — joins `packages`:
|
||||
// code read on the org's repositories and write on its packages. A consumer that contributes
|
||||
// `{"access": "read"}` joins `readers`: code read and package read, and no write. The lab is the first: it
|
||||
// clones branches that are not yet reviewed and runs their code beside its credential, and code that is
|
||||
// not reviewed must not be able to publish the mesh's packages with it.
|
||||
|
||||
/** The npm registry's owner: the gitea org every consumer's packages live under (ADR 0076). */
|
||||
export const ORG = "novox";
|
||||
/** The team whose members read and write the org's packages. */
|
||||
export const PACKAGE_TEAM = "packages";
|
||||
/** The team whose members read the org's code and packages, and write nothing. */
|
||||
export const READ_TEAM = "readers";
|
||||
|
||||
/** The team a consumer joins, from what it contributed; an access it may not ask for is refused. */
|
||||
export function teamFor(values: Readonly<Record<string, unknown>> | undefined): { team: string; packageWrite: boolean } {
|
||||
const access = values?.access;
|
||||
if (access === undefined || access === "write") return { team: PACKAGE_TEAM, packageWrite: true };
|
||||
if (access === "read") return { team: READ_TEAM, packageWrite: false };
|
||||
throw new Error(`npm-package-registry: access ${JSON.stringify(access)} is not "read" or "write"`);
|
||||
}
|
||||
|
||||
/** What the adapter asks of the forge's admin API (client.ts GiteaAdmin). */
|
||||
export interface Forge {
|
||||
ensureOrg(org: string): Promise<void>;
|
||||
ensureTeam(org: string, team: string, packageWrite: boolean): Promise<number>;
|
||||
ensureUser(username: string, password: string, email: string): Promise<void>;
|
||||
addUserToTeam(teamId: number, username: string): Promise<void>;
|
||||
removeUserFromTeam(teamId: number, username: string): Promise<void>;
|
||||
holdsTeamMember(org: string, team: string, username: string, password: string): Promise<boolean>;
|
||||
}
|
||||
|
||||
export interface Consumer {
|
||||
readonly as: string;
|
||||
readonly password: string;
|
||||
readonly values: Readonly<Record<string, unknown>>;
|
||||
}
|
||||
|
||||
/**
|
||||
* grant makes the consumer's user, with exactly the mesh's password, a member of its one team. A consumer
|
||||
* that asks for read only is also taken out of the package team, so one that asked for write before and
|
||||
* read now loses the write at once.
|
||||
*/
|
||||
export async function grant(forge: Forge, p: Consumer): Promise<void> {
|
||||
const { team, packageWrite } = teamFor(p.values);
|
||||
await forge.ensureOrg(ORG);
|
||||
const teamId = await forge.ensureTeam(ORG, team, packageWrite);
|
||||
// The address is gitea's own convention for one that is not real: its email validation requires a
|
||||
// dotted domain, so `@localhost` was refused at create, while `@noreply.localhost` is the shape gitea
|
||||
// itself gives hidden addresses.
|
||||
await forge.ensureUser(p.as, p.password, `${p.as}@noreply.localhost`);
|
||||
await forge.addUserToTeam(teamId, p.as);
|
||||
if (!packageWrite) {
|
||||
await forge.removeUserFromTeam(await forge.ensureTeam(ORG, PACKAGE_TEAM, true), p.as);
|
||||
}
|
||||
}
|
||||
|
||||
/** holds is whether the forge still holds the consumer as the mesh gave it: its login, in its team. */
|
||||
export async function holds(forge: Forge, p: Consumer): Promise<boolean> {
|
||||
return forge.holdsTeamMember(ORG, teamFor(p.values).team, p.as, p.password);
|
||||
}
|
||||
@@ -23,7 +23,7 @@ test("importing the bundle resolves while the broker keeps refusing the binds",
|
||||
await import(${JSON.stringify(index)});
|
||||
console.log("imported");
|
||||
process.exit(0);`;
|
||||
const env = { ...process.env, MESH_GITEA_TOKEN: "x", MESH_GITEA_URL: "http://127.0.0.1:1" };
|
||||
const env = { ...process.env, GITEA_TOKEN: "x", MESH_GITEA_URL: "http://127.0.0.1:1" };
|
||||
for (const k of ["MESH_GITEA_STATE_DIR", "MESH_GITEA_ADMIN_USER", "MESH_GITEA_ADMIN_PASSWORD", "MESH_SERVED_MODULE"]) delete env[k];
|
||||
const child = spawn(process.execPath, ["--input-type=module", "-e", script], { cwd: join(here, ".."), env, stdio: ["ignore", "pipe", "pipe"] });
|
||||
let out = "";
|
||||
@@ -86,7 +86,7 @@ test("a bundle whose read of the checker's owed is granted late writes the lates
|
||||
await new Promise((r) => setTimeout(r, 200));
|
||||
console.log(JSON.stringify({ tools, bound, watched, statuses }));
|
||||
process.exit(0);`;
|
||||
const env = { ...process.env, MESH_GITEA_TOKEN: "x" };
|
||||
const env = { ...process.env, GITEA_TOKEN: "x" };
|
||||
for (const k of ["MESH_GITEA_STATE_DIR", "MESH_GITEA_ADMIN_USER", "MESH_GITEA_ADMIN_PASSWORD", "MESH_SERVED_MODULE", "MESH_GITEA_URL"]) delete env[k];
|
||||
const child = spawn(process.execPath, ["--input-type=module", "-e", script], { cwd: join(here, ".."), env, stdio: ["ignore", "pipe", "pipe"] });
|
||||
let out = "";
|
||||
@@ -141,7 +141,7 @@ test("importing the bundle resolves, its tools registered, while the forge never
|
||||
const { collectTools } = await import("@novox/mesh-sdk/tools");
|
||||
console.log(JSON.stringify({ imported: true, tools: collectTools().flatMap((r) => r.tools.map((t) => t.name)) }));
|
||||
process.exit(0);`;
|
||||
const env = { ...process.env, MESH_GITEA_TOKEN: "x", MESH_GITEA_ADMIN_USER: "admin", MESH_GITEA_HOOK_URL: "http://127.0.0.1:1" };
|
||||
const env = { ...process.env, GITEA_TOKEN: "x", MESH_GITEA_ADMIN_USER: "admin", MESH_GITEA_HOOK_URL: "http://127.0.0.1:1" };
|
||||
for (const k of ["MESH_SERVED_MODULE", "MESH_GITEA_ADMIN_PASSWORD", "MESH_GITEA_CONFIG_FILE"]) delete env[k];
|
||||
const child = spawn(process.execPath, ["--input-type=module", "-e", script], { cwd: join(here, ".."), env, stdio: ["ignore", "pipe", "pipe"] });
|
||||
let out = "";
|
||||
|
||||
@@ -1,56 +0,0 @@
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
import { grant, holds, PACKAGE_TEAM, READ_TEAM, teamFor, type Forge } from "../provisioner/teams.ts";
|
||||
|
||||
// Defends novox/hq issue 624: the lab reads the forge with a credential of its own that can write no
|
||||
// package. A consumer that contributes {"access": "read"} joins a team with code and package read only.
|
||||
|
||||
function aForge(): { forge: Forge; did: string[] } {
|
||||
const did: string[] = [];
|
||||
const teams: Record<string, number> = {};
|
||||
const forge: Forge = {
|
||||
async ensureOrg(org) { did.push(`org ${org}`); },
|
||||
async ensureTeam(org, team, packageWrite) {
|
||||
did.push(`team ${org}/${team} packages:${packageWrite ? "write" : "read"}`);
|
||||
return (teams[team] ??= Object.keys(teams).length + 1);
|
||||
},
|
||||
async ensureUser(username) { did.push(`user ${username}`); },
|
||||
async addUserToTeam(id, username) { did.push(`add ${username} to ${id}`); },
|
||||
async removeUserFromTeam(id, username) { did.push(`remove ${username} from ${id}`); },
|
||||
async holdsTeamMember(org, team, username) { did.push(`holds ${username} in ${org}/${team}`); return true; },
|
||||
};
|
||||
return { forge, did };
|
||||
}
|
||||
|
||||
test("a consumer that asks nothing, or write, publishes: the package team", () => {
|
||||
assert.deepEqual(teamFor({}), { team: PACKAGE_TEAM, packageWrite: true });
|
||||
assert.deepEqual(teamFor({ access: "write" }), { team: PACKAGE_TEAM, packageWrite: true });
|
||||
});
|
||||
|
||||
test("a consumer that asks to read joins the readers, who write no package", () => {
|
||||
assert.deepEqual(teamFor({ access: "read" }), { team: READ_TEAM, packageWrite: false });
|
||||
assert.throws(() => teamFor({ access: "admin" }), /not "read" or "write"/);
|
||||
});
|
||||
|
||||
test("a reader is made, put on the readers team, and taken off the package team", async () => {
|
||||
const { forge, did } = aForge();
|
||||
await grant(forge, { as: "lab-shanks", password: "pw", values: { access: "read" } });
|
||||
assert.deepEqual(did, [
|
||||
"org novox",
|
||||
"team novox/readers packages:read",
|
||||
"user lab-shanks",
|
||||
"add lab-shanks to 1",
|
||||
"team novox/packages packages:write",
|
||||
"remove lab-shanks from 2",
|
||||
]);
|
||||
did.length = 0;
|
||||
await holds(forge, { as: "lab-shanks", password: "pw", values: { access: "read" } });
|
||||
assert.deepEqual(did, ["holds lab-shanks in novox/readers"]);
|
||||
});
|
||||
|
||||
test("a publisher is made and put on the package team, as before", async () => {
|
||||
const { forge, did } = aForge();
|
||||
await grant(forge, { as: "build-agent-ace", password: "pw", values: {} });
|
||||
assert.deepEqual(did, ["org novox", "team novox/packages packages:write", "user build-agent-ace", "add build-agent-ace to 1"]);
|
||||
});
|
||||
@@ -293,11 +293,11 @@ test("first start: mints with the admin account, keeps the token at 0600, asks f
|
||||
assert.equal(repos[0]?.full_name, "novox/hq");
|
||||
assert.equal(forge.mints, 1);
|
||||
// Never write:admin (novox/hq ADR 0266): the forge's administration is the operator's, not a tool's.
|
||||
assert.deepEqual(forge.lastScopes, ["write:repository", "write:issue", "read:user"]);
|
||||
assert.deepEqual(forge.lastScopes, ["write:repository", "write:issue", "read:user", "read:package"]);
|
||||
assert.deepEqual(forge.lastScopes, [...TOKEN_SCOPES]);
|
||||
const token = forge.tokens.get("mesh-tools")!;
|
||||
assert.equal(await readFile(file, "utf8"), token + "\n");
|
||||
assert.equal(await readFile(file + ".scopes", "utf8"), "read:user,write:issue,write:repository\n");
|
||||
assert.equal(await readFile(file + ".scopes", "utf8"), "read:package,read:user,write:issue,write:repository\n");
|
||||
assert.equal((await stat(file)).mode & 0o777, 0o600);
|
||||
// Said that it minted, and where it keeps it — never what it is.
|
||||
assert.ok(logs.some((l) => l.startsWith("minted a token")), logs.join("\n"));
|
||||
@@ -500,16 +500,18 @@ test("a configured token wins, and is reported rather than minted over when the
|
||||
const { env } = await delivered(forge);
|
||||
const before = forge.mints;
|
||||
|
||||
const client = GiteaClient.fromEnv({ ...env, MESH_GITEA_TOKEN: "one-somebody-pasted-in" });
|
||||
const client = GiteaClient.fromEnv({ ...env, GITEA_TOKEN: "one-somebody-pasted-in" });
|
||||
|
||||
await assert.rejects(client.listRepos(), /rejected the configured Gitea token \(401\)/);
|
||||
assert.equal(forge.mints, before);
|
||||
// MESH_GITEA_TOKEN is no word of the mesh's (nothing sets it, checks/environment): it is not read, the module mints.
|
||||
await assert.doesNotReject(GiteaClient.fromEnv({ ...env, MESH_GITEA_TOKEN: "one-somebody-pasted-in" }).listRepos());
|
||||
});
|
||||
|
||||
test("nothing to mint with and no token: the client says what is missing", async () => {
|
||||
assert.throws(
|
||||
() => GiteaClient.fromEnv({ MESH_GITEA_URL: forge.url, MESH_GITEA_ADMIN_USER: ADMIN }),
|
||||
/set MESH_GITEA_TOKEN, or MESH_GITEA_ADMIN_PASSWORD_FILE, MESH_GITEA_STATE_DIR/,
|
||||
/set GITEA_TOKEN, or MESH_GITEA_ADMIN_PASSWORD_FILE, MESH_GITEA_STATE_DIR/,
|
||||
);
|
||||
});
|
||||
|
||||
|
||||
@@ -14,9 +14,10 @@ import { mkdtemp, readFile, writeFile, mkdir } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
|
||||
import { MintedToken } from "../dist/token.js";
|
||||
import { MintedToken, TOKEN_SCOPES } from "../dist/token.js";
|
||||
import { GiteaClient } from "../dist/client.js";
|
||||
import { TOOLS_ACCOUNT, TOOLS_TEAM_UNITS, BUILDER_ACCOUNT, AGENT_ACCOUNT, meshAccounts } from "../dist/tools-account.js";
|
||||
import { PACKAGES_ACCOUNT, PACKAGES_TEAM_UNITS, PACKAGES_TOKEN_REFUSED, keepPackagesToken, packagesToken, renewOnRefusal } from "../dist/packages-account.js";
|
||||
|
||||
const ADMIN = "mesh-admin";
|
||||
const PASSWORD = "the-vault-minted-this";
|
||||
@@ -116,6 +117,13 @@ function fakeForge() {
|
||||
}
|
||||
}
|
||||
if (req.method === "GET" && p === "/user") return json(res, 200, { login: me, is_admin: isAdmin(me) });
|
||||
// The organisation's packages: a token whose scope reaches packages, of a member of a team writing them.
|
||||
if (req.method === "GET" && p === "/packages/novox") {
|
||||
const scopes = tokens.find((t) => `token ${t.sha1}` === req.headers.authorization)?.scopes ?? [];
|
||||
if (!scopes.some((s) => /:package$/.test(s))) return json(res, 403, { message: "token does not have at least one of required scope(s): [read:package]" });
|
||||
if (!teams.some((t) => t.members.has(me) && t.units.units_map?.["repo.packages"] === "write")) return json(res, 403, { message: "no access to the packages" });
|
||||
return json(res, 200, []);
|
||||
}
|
||||
m = p.match(/^\/users\/([^/]+)$/);
|
||||
if (m && req.method === "GET") {
|
||||
const u = users.get(decodeURIComponent(m[1]));
|
||||
@@ -359,6 +367,44 @@ test("the manifest hands the tools their account, and the Go forge bundle holds
|
||||
const forgeBundle = manifest.build.artifacts.find((a: any) => a.name === "forge");
|
||||
assert.ok(forgeBundle.env.MESH_GITEA_TOKEN_FILE);
|
||||
assert.equal(forgeBundle.env.MESH_GITEA_ADMIN_PASSWORD_FILE, undefined);
|
||||
// The package registry's bundle reads with the tools' token and writes with the packages account's (design 51).
|
||||
const packages = manifest.build.artifacts.find((a: any) => a.name === "package-registry");
|
||||
assert.equal(packages.env.MESH_GITEA_TOKEN_FILE, forgeBundle.env.MESH_GITEA_TOKEN_FILE);
|
||||
assert.equal(packages.env.MESH_GITEA_PACKAGES_USER, PACKAGES_ACCOUNT);
|
||||
assert.match(packages.env.MESH_GITEA_PACKAGES_TOKEN_FILE, /^\$\{dir:runtime-state\}\/packages-token$/);
|
||||
assert.equal(packages.env.MESH_GITEA_ADMIN_PASSWORD_FILE, undefined);
|
||||
assert.equal(code.env.MESH_GITEA_STATE_DIR, "${dir:runtime-state}", "the watcher mints the packages token where the bundle reads it");
|
||||
});
|
||||
|
||||
test("the package seats write as an account of their own: plain, in a team writing only packages, its token write:package alone, renewed when the forge rejects it", async () => {
|
||||
const env = await environment();
|
||||
const source = packagesToken(forge.url, env, () => {});
|
||||
assert.equal(await keepPackagesToken(source, forge.url), "the package seats' write token holds");
|
||||
const account = forge.users.get(PACKAGES_ACCOUNT);
|
||||
assert.ok(account, "the module made the packages account");
|
||||
assert.equal(account!.admin, false, "the packages account is no administrator");
|
||||
const held = forge.tokens.filter((t) => t.name === "mesh-packages");
|
||||
assert.deepEqual(held.map((t) => [t.user, t.scopes]), [[PACKAGES_ACCOUNT, ["write:package"]]]);
|
||||
assert.equal(await readFile(join(env.MESH_GITEA_STATE_DIR, "packages-token"), "utf8"), held[0].sha1 + "\n");
|
||||
assert.equal(await readFile(join(env.MESH_GITEA_STATE_DIR, "packages-token.scopes"), "utf8"), "write:package for mesh-packages\n");
|
||||
const team = forge.teams.find((t) => t.name === "mesh-packages");
|
||||
assert.ok(team?.members.has(PACKAGES_ACCOUNT));
|
||||
assert.deepEqual(team!.units.units_map, PACKAGES_TEAM_UNITS.units_map, "the team writes packages and nothing else");
|
||||
// The forge drops the token behind the module's back: the Go bundle's refusal (the finding's put) mints it afresh and
|
||||
// clears the finding; no timer looks at it, and another key of the findings state is not its own.
|
||||
forge.tokens.splice(forge.tokens.indexOf(held[0]), 1);
|
||||
const cleared: string[] = [];
|
||||
const said: string[] = [];
|
||||
const renew = renewOnRefusal(source, forge.url, { delete: async (k) => { cleared.push(k); } }, (l) => said.push(l));
|
||||
await renew({ key: "seat.gitea.other", op: "put" });
|
||||
await renew({ key: PACKAGES_TOKEN_REFUSED, op: "delete" });
|
||||
assert.equal(forge.tokens.filter((t) => t.name === "mesh-packages").length, 0, "nothing renews but the refusal");
|
||||
await renew({ key: PACKAGES_TOKEN_REFUSED, op: "put" });
|
||||
assert.deepEqual(said, ["the forge rejected the package seats' write token; minted afresh"]);
|
||||
assert.deepEqual(cleared, [PACKAGES_TOKEN_REFUSED]);
|
||||
assert.equal(forge.tokens.filter((t) => t.name === "mesh-packages").length, 1);
|
||||
// The tools' token reads packages too: read:package (design 51's read-only tool token).
|
||||
assert.ok(TOKEN_SCOPES.includes("read:package"));
|
||||
});
|
||||
|
||||
// Review of issue 469: what reads the rules' list is moved to the branch, and nothing leaves the tools without a token.
|
||||
|
||||
@@ -47,6 +47,9 @@ export const TOKEN_NAME = "mesh-tools";
|
||||
* The least the fifteen tools and the watcher need (gitea's route groups, 1.20+ scoped tokens):
|
||||
* write:repository — create/delete repositories, pull requests (list/get/open/merge);
|
||||
* write:issue — issues, comments, labels;
|
||||
* read:package — the package registry's reads by the Go bundle (cmd/package-registry): the package seats'
|
||||
* held, versions and the retention's survey (design 51: a read-only tool token; the
|
||||
* writes go with the packages account's own token, packages-account.ts);
|
||||
* read:user — GET /user/repos, which the watcher's poll and gitea_list_repos both call.
|
||||
* It sits under the `user` category despite listing repositories, not `repository`
|
||||
* — confirmed against the running forge (1.27.3), which answered
|
||||
@@ -62,7 +65,7 @@ export const TOKEN_NAME = "mesh-tools";
|
||||
* `403 token does not have at least one of required scope(s)`, and the client treats that like a
|
||||
* 401 — the source re-mints by name, with the whole list, and the call is retried once.
|
||||
*/
|
||||
export const TOKEN_SCOPES: readonly string[] = ["write:repository", "write:issue", "read:user"];
|
||||
export const TOKEN_SCOPES: readonly string[] = ["write:repository", "write:issue", "read:user", "read:package"];
|
||||
|
||||
/** Where a client's token comes from, and what to do when the forge says it is wrong. */
|
||||
export interface TokenSource {
|
||||
@@ -82,7 +85,7 @@ export class ConfiguredToken implements TokenSource {
|
||||
|
||||
async renew(): Promise<string> {
|
||||
throw new Error(
|
||||
"the forge rejected the configured Gitea token (401). It was set explicitly (settings or MESH_GITEA_TOKEN), " +
|
||||
"the forge rejected the configured Gitea token (401). It was set explicitly (settings or GITEA_TOKEN), " +
|
||||
"so the module does not mint over it — fix it, or unset it and the module mints its own",
|
||||
);
|
||||
}
|
||||
@@ -173,7 +176,7 @@ export class MintedToken implements TokenSource {
|
||||
stateDir ? null : "MESH_GITEA_STATE_DIR",
|
||||
].filter((v): v is string => v !== null);
|
||||
if (missing.length) {
|
||||
throw new Error(`no Gitea token, and nothing to mint one with — set MESH_GITEA_TOKEN, or ${missing.join(", ")}`);
|
||||
throw new Error(`no Gitea token, and nothing to mint one with — set GITEA_TOKEN, or ${missing.join(", ")}`);
|
||||
}
|
||||
// The tools' own account, never the administrator (novox/hq ADR 0365 §2): MESH_GITEA_TOOLS_USER names it.
|
||||
const account = env.MESH_GITEA_TOOLS_USER || TOOLS_ACCOUNT;
|
||||
|
||||
@@ -24,8 +24,9 @@ import {
|
||||
/** A head named for a merge: at least 12 hex characters of its sha (novox/hq issue 381), up to a sha-256 commit's 64. */
|
||||
const REVIEWED_HEAD = /^[0-9a-f]{12,64}$/;
|
||||
|
||||
/** The forge's container, where its repositories and git are: the note is written there (novox/hq ADR 0239). */
|
||||
const forgeContainer = process.env.MESH_GITEA_CONTAINER || "gitea";
|
||||
/** The forge's container, where its repositories and git are: the note is written there (novox/hq ADR 0239). Its name
|
||||
* is the manifest's container's; no variable names it (a MESH_* word nothing sets is refused, checks/environment). */
|
||||
const forgeContainer = "gitea";
|
||||
|
||||
/** A protection rule as a person reads it: what it guards, not every field the forge keeps. */
|
||||
export function summarised(p: BranchProtection) {
|
||||
|
||||
@@ -8,5 +8,5 @@
|
||||
"skipLibCheck": true,
|
||||
"noEmit": true
|
||||
},
|
||||
"include": ["client.ts", "token.ts", "pulls.ts", "protection.ts", "index.ts", "provisioner/index.ts", "provisioner/teams.ts", "tools/index.ts"]
|
||||
"include": ["client.ts", "token.ts", "pulls.ts", "protection.ts", "index.ts", "tools/index.ts"]
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user