Compare commits

...
Author SHA1 Message Date
mesh-tools 05e21fd8a3 Merge pull request 'A put without a hold ends once the node runs its fingerprint healthy' (#192) from fix/1966-put-ends into main 2026-11-09 19:37:53 +00:00
jschoubben b71358e2cc A put without a hold ends once the node runs its fingerprint healthy, and assigner_puts says a put older than the newest build
mesh/check being carried to b71358e2cc14 on 05e21fd8a382 | from fc21f2d1bd56
mesh/delivery merged: source.main moved to novox/mesh-delivery@05e21fd8a382
A put with hold false waited for the node's composition digest to equal its own and never ended when the node composes its own,
pinning the module's want (issue 1966, F21).

Claude-Session: https://claude.ai/code/session_01JG73KEq4zoDfM9WtYpcQgZ
2026-11-09 19:54:16 +01:00
mesh-tools fc21f2d1bd Merge pull request 'The checker keeps a composition request in pieces and packs it in the compose ask (issue 1966, recovery; group with mesh-catalog)' (#189) from fix/1966-checker-compositions-objects into main 2026-11-09 16:31:25 +00:00
jschoubben bf7540cd9e The build step part's request names both forms; the three build seats' contracts regenerated (issue 1966)
mesh/delivery merged: source.main moved to novox/mesh-delivery@fc21f2d1bd56
mesh/delivery-group group fix/1966-checker-compositions-objects merged: every member is merged
mesh/check checking bf7540cd9ec2 on fc21f2d1bd56 | novox/mesh-delivery#189 in group fix/1966-checker-compositions-objects
The compose job's `request` description now says it is the request object or the JSON string "gzip+base64:" followed by the
base64 of the gzip-compressed object, as ADR 0634 §1 puts a seat's messages in its contract. The field stays `any`;
seatgen regenerated node-build-go, node-build-node and node-build-image (Go and TypeScript), and its -check is clean.
ask.go's comment says the checker packs every request it sends.

Delivery: an old holder of node-build-go refuses the packed ask loudly (it holds no request object) until it runs the new
mesh-build-go; the checker is built by mesh-build-go after it (module graph edge), and every check composing ace or novox
fails today anyway.

Claude-Session: https://claude.ai/code/session_01NcbgfFfQ4Z9UE5jK32PmE5
2026-11-09 16:41:56 +01:00
jschoubben 173c49fc87 The compose ask carries its request gzip-packed; requests of another version are refused by name (issue 1966)
A large node's request is 1.2 MB, over the bus's 1 MiB message limit, so the compose ask failed once keeping no longer did.
The checker packs the request (a JSON string "gzip+base64:..." in the part's `any` request, no contract change) and
buildstep.ComposeJob.RequestBytes reads either form. A kept request of another version is refused naming it, and listing
no longer reads pieces. An older mesh-build-go refuses a packed ask loudly (it holds no request object): the check fails
naming it, so the catalog change is delivered with this one as a group.

Quick fix (chosen as the recovery): pack the request, ~290 KB for the largest node. Thorough fix (milestone G): the request
named by digest and served on a seat verb, as RunInput is.

Claude-Session: https://claude.ai/code/session_01NcbgfFfQ4Z9UE5jK32PmE5
2026-11-09 16:41:56 +01:00
jschoubben 3f1d9cecba Checker keeps a composition request in pieces, each under the state's value limit (issue 1966)
mesh-host #219's check failed: the request kept for g14 was 300127 bytes compressed, over the 262144 the state takes.
The key of the check and node now holds a pointer (digest, size, pieces) and each piece of the gzip bytes is a value of
its own under pieces/<digest>/<i>, at most 128 KiB; checker_composition_requests follows the pointer and checks the digest.

Quick fix: a larger cap or a better compression: still a limit, met again as the mesh grows. Thorough fix (chosen, and
bounded only by the bucket): pieces. Not an object store: the checker is a foundation part, whose data stays in the old
sections (ADR 0619 §2), and the tool runner serves no old-section object read to its tools. No manifest change.

Claude-Session: https://claude.ai/code/session_01NcbgfFfQ4Z9UE5jK32PmE5
2026-11-09 16:41:56 +01:00
mesh-tools a072d60cae Merge pull request 'The indexer writes no fingerprint from a packages state it did not read, and computes fingerprints again by hand (issue 1966, recovery)' (#191) from fix/1966-indexer-packages-read into main 2026-11-09 14:48:20 +00:00
jschoubben efdfe1f755 A bucket the bus cannot reach is its error, not nothing published; only a bucket not found is not yet (issue 1966, review round 3)
mesh/delivery merged: source.main moved to novox/mesh-delivery@a072d60cae1d
mesh/check checking efdfe1f75518 on a072d60cae1d
Claude-Session: https://claude.ai/code/session_01NcbgfFSnn24M3GcqttoU7i
2026-11-09 15:43:52 +01:00
jschoubben 9073190646 A packages state not on the bus yet is nothing published, not a failed read (issue 1966, check of #191)
Claude-Session: https://claude.ai/code/session_01NcbgfFSnn24M3GcqttoU7i
2026-11-09 15:40:44 +01:00
jschoubben 3a63813b3d Fingerprints computed again by hand run a merge's recompute; the tests join existing ones (issue 1966, review round 1)
mesh/delivery closed: a newer head of the same pull request
mesh/check failed (6): integration: mesh-delivery-indexer/cmd/mesh-delivery-indexer.TestIndexerWritesModule…; and 5 more | 3a63813b3d75 on fb69c8dee691
Claude-Session: https://claude.ai/code/session_01NcbgfFSnn24M3GcqttoU7i
2026-11-09 15:24:30 +01:00
jschoubben 03164e853e The indexer writes no fingerprint from a packages state it did not read, keeps no import form beside a version line, and computes fingerprints again by hand (issue 1966)
An indexer that started while the bus's storage was down read no published package,
took every import for unpublished and wrote the version line over ~130 modules'
fingerprints, which a later pass kept through the build's shortcut. A failed or empty
read of the packages state, where held modules were computed from published ones,
now fails the pass and says why; a version line clears the import form it was
computed beside; indexer_fingerprints_again computes the import form again by hand,
acting first and recording the hand act after, with a bound.

Claude-Session: https://claude.ai/code/session_01NcbgfFSnn24M3GcqttoU7i
2026-11-09 15:15:26 +01:00
jschoubben 2819b05838 The assigner's integration test answers the verbs' asks of the indexer's first views from the indexer's buckets (issue 1966)
The verbs ask mesh-delivery-indexer.modules and .manifests through the tool runner
since the asks moved off the indexer's state, but the test's stand-in answered every
ask {ok:true}, which is no answer: propose failed on main with 'unexpected end of
JSON input'.

Claude-Session: https://claude.ai/code/session_01NcbgfFSnn24M3GcqttoU7i
2026-11-09 15:10:32 +01:00
34 changed files with 736 additions and 69 deletions
+1 -1
View File
@@ -485,7 +485,7 @@ type StepAskCompose struct {
Fetch Fetch `json:"fetch"`
Package string `json:"package"`
Reader string `json:"reader,omitempty"`
// the compose-candidate request
// the compose-candidate request, in one of two forms: the request as a JSON object, or, for one near the bus's message limit, a JSON string "gzip+base64:" followed by the base64 of the request object gzip-compressed; a holder that does not know the packed form refuses it, naming that it holds no request object
Request any `json:"request"`
}
@@ -485,7 +485,7 @@ type StepAskCompose struct {
Fetch Fetch `json:"fetch"`
Package string `json:"package"`
Reader string `json:"reader,omitempty"`
// the compose-candidate request
// the compose-candidate request, in one of two forms: the request as a JSON object, or, for one near the bus's message limit, a JSON string "gzip+base64:" followed by the base64 of the request object gzip-compressed; a holder that does not know the packed form refuses it, naming that it holds no request object
Request any `json:"request"`
}
+1 -1
View File
@@ -485,7 +485,7 @@ type StepAskCompose struct {
Fetch Fetch `json:"fetch"`
Package string `json:"package"`
Reader string `json:"reader,omitempty"`
// the compose-candidate request
// the compose-candidate request, in one of two forms: the request as a JSON object, or, for one near the bus's message limit, a JSON string "gzip+base64:" followed by the base64 of the request object gzip-compressed; a holder that does not know the packed form refuses it, naming that it holds no request object
Request any `json:"request"`
}
+1 -1
View File
@@ -161,7 +161,7 @@
"type": "string"
},
"request": {
"description": "the compose-candidate request"
"description": "the compose-candidate request, in one of two forms: the request as a JSON object, or, for one near the bus's message limit, a JSON string \"gzip+base64:\" followed by the base64 of the request object gzip-compressed; a holder that does not know the packed form refuses it, naming that it holds no request object"
},
"reader": {
"type": "string"
@@ -293,7 +293,7 @@ export interface StepAskCompose {
"fetch": Fetch;
"package": string;
"reader"?: string;
/** the compose-candidate request */
/** the compose-candidate request, in one of two forms: the request as a JSON object, or, for one near the bus's message limit, a JSON string "gzip+base64:" followed by the base64 of the request object gzip-compressed; a holder that does not know the packed form refuses it, naming that it holds no request object */
"request": unknown;
}
@@ -293,7 +293,7 @@ export interface StepAskCompose {
"fetch": Fetch;
"package": string;
"reader"?: string;
/** the compose-candidate request */
/** the compose-candidate request, in one of two forms: the request as a JSON object, or, for one near the bus's message limit, a JSON string "gzip+base64:" followed by the base64 of the request object gzip-compressed; a holder that does not know the packed form refuses it, naming that it holds no request object */
"request": unknown;
}
@@ -293,7 +293,7 @@ export interface StepAskCompose {
"fetch": Fetch;
"package": string;
"reader"?: string;
/** the compose-candidate request */
/** the compose-candidate request, in one of two forms: the request as a JSON object, or, for one near the bus's message limit, a JSON string "gzip+base64:" followed by the base64 of the request object gzip-compressed; a holder that does not know the packed form refuses it, naming that it holds no request object */
"request": unknown;
}
@@ -1233,7 +1233,18 @@ func TestAProposalIsAnsweredApprovedAndWritten(t *testing.T) {
t.Fatal("anchor in the form before was not converted")
}
}
verbs := newSeatTools(bedStore{b}, func(string, any) (json.RawMessage, error) { return json.RawMessage(`{"ok":true}`), nil })
// The verbs ask the indexer's first views of modules and manifests (ADR 0615 §1), as the tool runner answers them from the
// indexer's buckets; every other ask the controller's, answered ok.
verbs := newSeatTools(bedStore{b}, func(tool string, _ any) (json.RawMessage, error) {
if state, ok := strings.CutPrefix(tool, buckets.IndexerModule+"."); ok && indexerFirstViews[state] != "" {
a, err := views.FirstView(buckets.IndexerModule, state, bedSource{b.ctx, b.buckets[indexerFirstViews[state]]}, nil, "", time.Now())
if err != nil {
return nil, err
}
return json.Marshal(map[string]any{"result": a})
}
return json.RawMessage(`{"ok":true}`), nil
})
propose := func() mma.Proposal {
t.Helper()
got, err := verbs.doPropose(map[string]any{"why": "move web", "changes": `[{"op":"take-off","node":"anchor","module":"web"},` +
@@ -458,6 +458,13 @@ func putOfTheBus(t *testing.T) {
if got := step(); got.Step != puts.Released {
t.Fatalf("released, the put stands at %s", got.Step)
}
// A put that stands at an older version than the newest says so (issue 1966): it pins the node's want.
okey, _ := puts.Key("g14", "mesh-catalog")
older, _ := puts.Encode(puts.Put{FP: meshcatalogclient.SourcePrefix2 + strings.Repeat("b", 64), Why: "old"}.Said(now, puts.Wanted, "wanted"))
putsKV.values[okey] = older
if l, _ := s.doPuts(map[string]any{"module": "mesh-catalog"}); l.(map[string]any)["puts"].([]putAnswer)[0].Older != fp {
t.Errorf("a put older than the newest build says %+v", l)
}
listed, _ := s.doPuts(map[string]any{"module": "nats"})
if l := listed.(map[string]any)["puts"].([]putAnswer); len(l) != 1 || l[0].Step != puts.Released || len(l[0].Steps) < 6 {
t.Errorf("puts lists %+v", l)
@@ -130,6 +130,9 @@ type putAnswer struct {
Module string `json:"module"`
puts.Put
Follow string `json:"follow,omitempty"`
// Older says a put that has not ended stands at an older version than the newest build, and which that is: it pins the
// node's want until it ends or is released.
Older string `json:"older-than-newest,omitempty"`
}
// doPut is the tool assigner_put.
@@ -249,6 +252,8 @@ func (s *putTools) doPuts(args map[string]any) (any, error) {
}
sort.Strings(keys)
out := []putAnswer{}
newest := map[string]string{} // the indexer's newest fingerprint by module, asked once and only for a put that stands
asked := false
for _, key := range keys {
node, module, err := puts.ParseKey(key)
if err != nil || (arg(args, "node") != "" && node != arg(args, "node")) || (arg(args, "module") != "" && module != arg(args, "module")) {
@@ -262,7 +267,23 @@ func (s *putTools) doPuts(args map[string]any) (any, error) {
if err != nil {
continue
}
out = append(out, putAnswer{Node: node, Module: module, Put: p})
a := putAnswer{Node: node, Module: module, Put: p}
if !puts.Ended(p.Step) {
if !asked {
asked = true
if mods, err := indexerView(s.ask, indexerModules); err == nil {
for name, raw := range mods {
if m, err := meshcatalogclient.DecodeModule(raw); err == nil {
newest[name] = m.Fp2
}
}
}
}
if n := newest[module]; n != "" && n != p.FP {
a.Older = n
}
}
out = append(out, a)
}
return map[string]any{"puts": out}, nil
}
@@ -157,7 +157,15 @@ func StepPut(w PutWorld) PutNext {
// applied is a wanted put once the node's have says how it went: holding or done when it runs the composition healthy.
func applied(w PutWorld, p puts.Put) puts.Put {
h := w.Have
if h == nil || h.Fp != p.FP || h.Composed != p.Composed {
if h == nil || h.Fp != p.FP {
return p
}
// A put without a hold ends on the fingerprint alone: a node that composes its own digests no composition of the put's, and
// a put that waits for that one pins the module's want for good (issue 1966). A held put keeps its composition.
if !p.Hold && h.Phase == nodeengineclient.PhaseRunning && h.Health == nodeengineclient.HealthOK {
return p.Said(w.Now, puts.Done, fmt.Sprintf("%s runs %s at %s healthy; the rules write its want again", w.Node, w.Module, short(p.FP)))
}
if h.Composed != p.Composed {
return p
}
switch {
@@ -933,6 +933,21 @@ func stepsOfAPut(t *testing.T) {
w.Have = &nodeengine.HaveByNodeModuleValue{Fp: fNew, Composed: d.Digest, Phase: nodeengineclient.PhaseRunning, Health: nodeengineclient.HealthOK}
return w
}, puts.Done, "the rules write its want again", false, false, false},
{"wanted without hold, the node composes its own: done at the fingerprint", func() PutWorld {
w := bus()
w.Put.Hold = false
w.Put = w.Put.Said(now, puts.Wanted, "wanted")
w.Put.Composed = d.Digest
w.Have = &nodeengine.HaveByNodeModuleValue{Fp: fNew, Composed: dig(fOld), Phase: nodeengineclient.PhaseRunning, Health: nodeengineclient.HealthOK}
return w
}, puts.Done, "the rules write its want again", false, false, false},
{"wanted with hold, the node composes its own: stays wanted", func() PutWorld {
w := bus()
w.Put = w.Put.Said(now, puts.Wanted, "wanted")
w.Put.Composed = d.Digest
w.Have = &nodeengine.HaveByNodeModuleValue{Fp: fNew, Composed: dig(fOld), Phase: nodeengineclient.PhaseRunning, Health: nodeengineclient.HealthOK}
return w
}, puts.Wanted, "wanted", false, false, false},
{"released: nothing more", func() PutWorld { w := bus(); w.Put = w.Put.Said(now, puts.Released, "released"); return w }, puts.Released, "released", false, false, false},
}
for _, c := range cases {
@@ -378,7 +378,7 @@ func RequestTools(r check.StateRead) []stdio.Tool {
Description: "The composition request a check sent to a node: the composer's whole input for that node (the node-engine's composition input, the case's " +
"manifests and verdicts, the seat list), kept exactly as sent so a session reproduces the check's composition (the composer run by hand over the same " +
"request gives the check's answer). Kept in the checker's state compositions, per check key (group.heads, the 12-digit heads hash) and node, " +
"about 430 KB each (kept compressed, so a value fits the state's 256 KiB), for 3 days (the state's ttl-seconds). With a check and a node, that request whole; with a check, " +
"0.7 to 0.8 MB each (kept compressed, in pieces that each fit the state's 256 KiB value), for 3 days (the state's ttl-seconds). With a check and a node, that request whole; with a check, " +
"the nodes it kept one for; with neither, every kept request, each with its size, digest and when it was kept. Read-only.",
Input: map[string]any{"type": "object", "properties": map[string]any{
"check": map[string]any{"type": "string", "description": "the check's key, <group>.<heads hash>, as the verdict and composition name it"},
@@ -5,6 +5,7 @@ import (
"encoding/base64"
"encoding/json"
"errors"
"math/rand"
"os"
"regexp"
"slices"
@@ -48,7 +49,36 @@ func TestTheToolsServedAreTheFiveReadTools(t *testing.T) {
t.Fatal(err)
}
}
// Issue 1966: a request whose compressed bytes pass the state's value limit is kept in pieces and read back whole; the
// pieces are no requests in the list.
rnd := make([]byte, 600<<10)
_, _ = rand.New(rand.NewSource(1)).Read(rnd)
big := []byte(`{"input":"` + base64.StdEncoding.EncodeToString(rnd) + `"}`)
if err := keeper.Keep(context.Background(), "big.0123456789ab", "anchor", big); err != nil {
t.Fatalf("a request of %d bytes: %v", len(big), err)
}
rt := RequestTools(check.StateRead{Keys: store.Keys, Get: store.Get})
if one, err := rt[0].Run(map[string]any{"check": "big.0123456789ab", "node": "anchor"}); err != nil || !strings.Contains(func() string { b, _ := json.Marshal(one); return string(b) }(), base64.StdEncoding.EncodeToString(rnd)[:64]) {
t.Errorf("the big request read back: %v", err)
}
// Listing reads no piece, and a request of another version is refused by name.
store.vals["v1.0123456789ab/anchor"] = json.RawMessage(`{"v":1,"check":"v1.0123456789ab","node":"anchor","gzip":"x"}`)
noPieces := check.StateRead{Keys: store.Keys, Get: func(k string) (json.RawMessage, error) {
if strings.HasPrefix(k, "pieces/") {
return nil, errors.New("a piece was read")
}
return store.Get(k)
}}
if _, err := RequestTools(noPieces)[0].Run(map[string]any{}); err != nil {
t.Errorf("listing: %v", err)
}
if _, err := rt[0].Run(map[string]any{"check": "v1.0123456789ab", "node": "anchor"}); err == nil || !strings.Contains(err.Error(), "version 1") {
t.Errorf("a version 1 request: %v", err)
}
delete(store.vals, "v1.0123456789ab/anchor")
if all, _ := rt[0].Run(map[string]any{}); strings.Count(func() string { b, _ := json.Marshal(all); return string(b) }(), `"node"`) != 4 {
t.Errorf("the list holds a piece as a request: %v", all)
}
if len(rt) != 1 || rt[0].Name != check.RequestsReadTool || !strings.Contains(rt[0].Description, "composition request a check sent to a node") {
t.Fatalf("request tools = %+v", rt)
}
@@ -312,6 +342,9 @@ func TestTheTryToolsAskAndReadATry(t *testing.T) { // ADR 0501 §5, the agreed i
type memPut struct{ m *memStore }
func (p memPut) Put(_ context.Context, key string, value []byte) (uint64, error) {
if len(value) > 262144 { // the state's value limit, headers included (issue 1966)
return 0, errors.New("the value is over the 262144 bytes the bucket takes")
}
p.m.vals[key] = value
return 1, nil
}
@@ -228,6 +228,10 @@ func (e EngineCompositions) inSandbox(ctx context.Context, c *Case, head, node s
}
job := buildstep.ComposeJob{Fetch: buildstep.Fetch{Repository: nodeEngineRepo, Commit: head, Packages: packagesFor(e.Published, c, c.Tagged, nodeEngineRepo)}, Dir: meshcatalogclient.RootPath,
Package: composer.Package, Request: raw, Reader: "the node-engine head of " + c.Group}
// The ask travels on the bus, whose message limit a whole request passes (1.2 MB for a large node): it goes packed.
if err := job.PackRequest(raw); err != nil {
return a, err
}
verdict, err := e.Sandbox.Compose(ctx, job)
if err != nil {
return a, fmt.Errorf("the composer at the node-engine's head %s did not answer for %s: %w", head, node, err)
@@ -59,7 +59,11 @@ type fakeSandbox struct {
func (f *fakeSandbox) Compose(_ context.Context, job buildstep.ComposeJob) (composer.Verdict, error) {
var req composer.Request
if err := json.Unmarshal(job.Request, &req); err != nil {
body, err := job.RequestBytes()
if err != nil {
return composer.Verdict{}, err
}
if err := json.Unmarshal(body, &req); err != nil {
return composer.Verdict{}, err
}
f.mu.Lock()
@@ -206,7 +210,11 @@ func TestTheCompositionIsMadeFromTheEnginesAnswers(t *testing.T) {
}
job := sb.jobs[0]
var req composer.Request
_ = json.Unmarshal(job.Request, &req)
if len(job.Request) == 0 || job.Request[0] != '"' {
t.Errorf("the request went whole, not packed (issue 1966): %.60s", job.Request)
}
body, _ := job.RequestBytes()
_ = json.Unmarshal(body, &req)
if job.Fetch.Repository != nodeEngineRepo || job.Fetch.Commit != head2 || job.Package != composer.Package ||
string(req.Input) != `{"nodes":[{"name":"anchor"}]}` || req.Cases[0].Manifests["web"].Digest == "" ||
len(req.Seats) != 1 || string(req.Seats[0]) != string(host.Seats[0]) {
@@ -220,8 +228,9 @@ func TestTheCompositionIsMadeFromTheEnginesAnswers(t *testing.T) {
t.Fatal(err)
}
keyOfHost, _ := nodeengineclient.CompositionKey(host.Group, host.Heads)
lastBody, _ := sb.jobs[len(sb.jobs)-1].RequestBytes()
got, err := ReadRequests(store.read(), keyOfHost, "anchor")
if b, _ := json.Marshal(got); err != nil || !strings.Contains(string(b), `"request":`+string(sb.jobs[len(sb.jobs)-1].Request)) {
if b, _ := json.Marshal(got); err != nil || !strings.Contains(string(b), `"request":`+string(lastBody)) {
t.Errorf("kept %s, %v: want the request of the last job under %s/anchor", b, err, keyOfHost)
}
store.err = errors.New("the state is full")
@@ -311,7 +320,8 @@ func TestTheCompositionIsMadeFromTheEnginesAnswers(t *testing.T) {
t.Fatalf("jobs %+v: want one job at the node-engine's trunk", sb.jobs)
}
var req composer.Request
_ = json.Unmarshal(sb.jobs[0].Request, &req)
b0, _ := sb.jobs[0].RequestBytes()
_ = json.Unmarshal(b0, &req)
if got := string(req.Cases[0].Verdicts["mesh-build-go"]); got != string(brings.Judged["mesh-build-go"]) || len(req.Cases[0].Verdicts) != 2 {
t.Errorf("the request's verdicts %s: want the check's judge's for mesh-build-go and the indexer's for mail", req.Cases[0].Verdicts)
}
@@ -321,7 +331,8 @@ func TestTheCompositionIsMadeFromTheEnginesAnswers(t *testing.T) {
t.Fatal(err)
}
var none composer.Request
_ = json.Unmarshal(sb.jobs[len(sb.jobs)-1].Request, &none)
bn, _ := sb.jobs[len(sb.jobs)-1].RequestBytes()
_ = json.Unmarshal(bn, &none)
if _, kept := none.Cases[0].Verdicts["mesh-build-go"]; kept {
t.Errorf("the running indexer's verdict on the change's manifest was kept: %s", none.Cases[0].Verdicts)
}
@@ -20,20 +20,28 @@ import (
// session reproduces a check's composition exactly. The checker is a foundation part (novox/hq ADR 0619 §1-3), so its
// kept data is a section of its manifest's old form, here `state`, and not an object store asked in `requires`.
//
// **What bounds it.** A request is the node-engine's whole composition input of one node, about 430 KB as JSON. A
// state's value holds at most 256 KiB (kvlimit.ModuleStateValueBytes), so it is kept gzip-compressed (JSON shrinks about
// tenfold) in a document of its own; a request whose compressed bytes still do not fit is refused by the put, which
// fails the node's composition naming the size (never dropped). The manifest's ttl-seconds is its lifetime (3 days):
// the bus drops a value past it, so nothing waits on a timer here. A check kept again overwrites the key.
// **What bounds it.** A request is the node-engine's whole composition input of one node, 0.7 to 0.8 MB as JSON and
// growing with the mesh. A state's value holds at most 256 KiB (kvlimit.ModuleStateValueBytes, headers included), and
// the compressed request already passed it (issue 1966: 300127 bytes), so it is kept gzip-compressed in pieces: the key
// of the check and node holds a small document naming the request's digest, size and number of pieces, and each piece is
// a value of its own under RequestChunkKey, of at most RequestChunkBytes. The kept data is the checker's, a foundation
// part's, so it stays in the old section `state` (ADR 0619 §2) and is read through the tool runner's state read, which
// serves no object store of the old sections. The manifest's ttl-seconds is the lifetime (3 days) of the pointer and of
// every piece: the bus drops a value past it, so nothing waits on a timer here. A check kept again overwrites the key;
// a request whose pieces cannot all be put is refused naming the size (never dropped, never half kept: the pointer is
// put after its pieces).
const (
// RequestsStateName is the checker's state of kept requests and RequestsBucket its bucket.
RequestsStateName = "compositions"
RequestsBucket = CheckerModule + "_" + RequestsStateName
// RequestsReadTool is the tool that reads a kept request.
RequestsReadTool = "checker_composition_requests"
// RequestChunkBytes is the most compressed bytes one piece holds: base64 of it, in its document, stays well under
// the state's 256 KiB value.
RequestChunkBytes = 128 << 10
)
// KeptRequest is a request as the state holds it: whose it is, and the request's JSON, compressed.
// KeptRequest is the pointer to a request as the state holds it: whose it is, and what its pieces make.
type KeptRequest struct {
V int `json:"v"`
Check string `json:"check"`
@@ -42,8 +50,32 @@ type KeptRequest struct {
// Size and Digest are the request's, uncompressed: its bytes and their sha256.
Size int `json:"size"`
Digest string `json:"digest"`
// Gzip is the request's bytes, gzip-compressed and base64-encoded.
Gzip string `json:"gzip"`
// Pieces is how many pieces (RequestChunkKey of the digest, 0 to Pieces-1) make the request's gzip bytes.
Pieces int `json:"pieces"`
}
// KeptPiece is one piece of a request's gzip bytes, base64-encoded.
type KeptPiece struct {
V int `json:"v"`
Digest string `json:"digest"`
Index int `json:"index"`
Gzip string `json:"gzip"`
}
// isPieceKey is whether a key is a piece's: under pieces/ and a sha256 digest, which no check's key (group.heads/node) is.
func isPieceKey(k string) bool {
rest, ok := strings.CutPrefix(k, "pieces/")
if !ok {
return false
}
digest, _, _ := strings.Cut(rest, "/")
return len(digest) == 64
}
// RequestChunkKey is the key of a request's piece: under the request's digest, so a request kept again for the same
// bytes writes the same pieces.
func RequestChunkKey(digest string, index int) string {
return fmt.Sprintf("pieces/%s/%d", digest, index)
}
// RequestKey is the key a check's request to a node is kept under: the check's key (group.heads) and the node.
@@ -75,14 +107,28 @@ func (k RequestKeeper) Keep(ctx context.Context, key, node string, body []byte)
now = k.Now
}
sum := sha256.Sum256(body)
doc, err := json.Marshal(KeptRequest{V: 1, Check: key, Node: node, KeptAt: now().UTC(), Size: len(body),
Digest: hex.EncodeToString(sum[:]), Gzip: base64.StdEncoding.EncodeToString(packed.Bytes())})
digest := hex.EncodeToString(sum[:])
pieces := (packed.Len() + RequestChunkBytes - 1) / RequestChunkBytes
fail := func(err error) error {
return fmt.Errorf("the composition request of check %s to %s (%d bytes, %d compressed in %d pieces) could not be kept in the state %s: %w",
key, node, len(body), packed.Len(), pieces, RequestsStateName, err)
}
for i := 0; i < pieces; i++ {
end := min((i+1)*RequestChunkBytes, packed.Len())
doc, err := json.Marshal(KeptPiece{V: 2, Digest: digest, Index: i, Gzip: base64.StdEncoding.EncodeToString(packed.Bytes()[i*RequestChunkBytes : end])})
if err != nil {
return err
}
if _, err := k.Store.Put(ctx, RequestChunkKey(digest, i), doc); err != nil {
return fail(err)
}
}
doc, err := json.Marshal(KeptRequest{V: 2, Check: key, Node: node, KeptAt: now().UTC(), Size: len(body), Digest: digest, Pieces: pieces})
if err != nil {
return err
}
if _, err := k.Store.Put(ctx, RequestKey(key, node), doc); err != nil {
return fmt.Errorf("the composition request of check %s to %s (%d bytes, %d compressed) could not be kept in the state %s: %w",
key, node, len(body), packed.Len(), RequestsStateName, err)
return fail(err)
}
return nil
}
@@ -104,7 +150,7 @@ func ReadRequests(s StateRead, key, node string) (any, error) {
if len(raw) == 0 {
return nil, fmt.Errorf("no composition request of check %s to %s is kept (none was sent, or it is past its lifetime)", key, node)
}
kept, body, err := unpackRequest(raw)
kept, body, err := unpackRequest(s, raw)
if err != nil {
return nil, fmt.Errorf("the kept request %s: %w", RequestKey(key, node), err)
}
@@ -120,6 +166,9 @@ func ReadRequests(s StateRead, key, node string) (any, error) {
if key != "" && !strings.HasPrefix(k, key+"/") {
continue
}
if isPieceKey(k) {
continue // a piece of a request, not a request: never read here
}
raw, err := s.Get(k)
if err != nil {
return nil, err
@@ -134,15 +183,30 @@ func ReadRequests(s StateRead, key, node string) (any, error) {
return map[string]any{"kept": listed}, nil
}
// unpackRequest reads a kept document and its request's bytes, which must match the digest they were kept with.
func unpackRequest(raw []byte) (KeptRequest, []byte, error) {
// unpackRequest reads a kept pointer's pieces and the request's bytes, which must match the digest they were kept with.
func unpackRequest(s StateRead, raw []byte) (KeptRequest, []byte, error) {
var kept KeptRequest
if err := json.Unmarshal(raw, &kept); err != nil {
return kept, nil, fmt.Errorf("is no kept request: %w", err)
}
packed, err := base64.StdEncoding.DecodeString(kept.Gzip)
if err != nil {
return kept, nil, fmt.Errorf("holds no base64: %w", err)
if kept.V != 2 {
return kept, nil, fmt.Errorf("is a kept request of version %d, which this reader does not read (it reads version 2; an older one is past its lifetime within 3 days)", kept.V)
}
var packed []byte
for i := 0; i < kept.Pieces; i++ {
raw, err := s.Get(RequestChunkKey(kept.Digest, i))
if err != nil {
return kept, nil, err
}
var piece KeptPiece
if err := json.Unmarshal(raw, &piece); err != nil || len(raw) == 0 {
return kept, nil, fmt.Errorf("lacks its piece %d of %d (past its lifetime, or never kept)", i, kept.Pieces)
}
part, err := base64.StdEncoding.DecodeString(piece.Gzip)
if err != nil {
return kept, nil, fmt.Errorf("holds no base64 in its piece %d: %w", i, err)
}
packed = append(packed, part...)
}
zr, err := gzip.NewReader(bytes.NewReader(packed))
if err != nil {
@@ -278,6 +278,10 @@ func main() {
tools = append(tools, catalogTools()...)
tools = append(tools, viewTools(os.Getenv, keptSource, time.Now)...)
tools = append(tools, index.PendingTools(keptRead{stdio.State(index.PendingState)}, keptRead{stdio.State(index.OwnBuildStateName)}, asks, time.Now)...)
tools = append(tools, index.FingerprintsTools(asks, time.Now, func(what, why string) error {
_, err := acts.Record(meshhandactsclient.Request{What: what, Why: why, Cause: index.FingerprintsAgainTool, Verb: "fingerprints-again"})
return err
}, 10*time.Second)...)
tools = append(tools, index.TryCandidateTools(keptRead{stdio.State("candidates")}, keptRead{stdio.State(index.TryRecoveryState)}, asks, time.Now)...)
tools = append(tools, configReadersTool(stdio.NewWatchedStates()), busStateTool(keptSource), requirementsTool(keptSource),
packageUsersTool(keptSource), seatDefinitionTool(keptSource), outOfStepTool(keptSource),
@@ -52,7 +52,14 @@ func main() {
// awaitedStore is a state bound when it appears (process.AwaitBucket) as the indexer's store: until then every call is
// process.ErrNotYet.
type awaitedStore struct{ b *process.AwaitedBucket }
type awaitedStore struct {
b *process.AwaitedBucket
// js and name, when set, tell a bucket that does not exist from one that cannot be reached: the awaited bucket is nil for
// both (the SDK drops the bus's error), and only the first is "not yet". The packages state is read this way: an unreached
// state is no package published, and the version line written over every import form (issue 1966).
js process.BucketOpener
name string
}
// expiringKeys is index.Bounds on a state whose keys take a time to live (key-ttl in the manifest): a key armed expires
// at its end, which its watcher hears as a purge; one standing keeps its end.
@@ -88,6 +95,11 @@ func (e expiringKeys) Disarm(ctx context.Context, key string) error {
func (s awaitedStore) store() (kvStore, error) {
kv := s.b.KV()
if kv == nil {
if s.js != nil {
if _, err := s.js.KeyValue(context.Background(), s.name); err != nil && !errors.Is(err, jetstream.ErrBucketNotFound) {
return kvStore{}, fmt.Errorf("the bucket %s cannot be reached: %w", s.name, err)
}
}
return kvStore{}, process.ErrNotYet
}
return kvStore{kv}, nil
@@ -411,17 +423,17 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
seatBus := &seatBus{defs: awaitedSource{ctx, seatDefs}, marks: awaitedSource{ctx, marks}}
seatReader, saySeats := followSeatList(seatBus, saySeatDefinition(js, cred), logger.Printf)
readWhenBound(seatReader, seatDefs, marks)
ix := &index.Indexer{Findings: awaitedStore{findings}, OldSections: awaitedStore{oldSections}, Requirements: awaitedStore{requirements}, Store: kvStore{candidates}, Modules: kvStore{modules}, Manifests: kvStore{manifests}, Taken: awaitedStore{taken}, Builds: known,
Verdicts: awaitedStore{verdicts}, SayRefused: sayRefused(conn, cred, logger.Printf), SayAccepted: sayAccepted(conn, cred),
Predefined: awaitedStore{predefined}, ConfigCandidates: awaitedStore{configCandidates}, Logf: logger.Printf,
ix := &index.Indexer{Findings: awaitedStore{b: findings}, OldSections: awaitedStore{b: oldSections}, Requirements: awaitedStore{b: requirements}, Store: kvStore{candidates}, Modules: kvStore{modules}, Manifests: kvStore{manifests}, Taken: awaitedStore{b: taken}, Builds: known,
Verdicts: awaitedStore{b: verdicts}, SayRefused: sayRefused(conn, cred, logger.Printf), SayAccepted: sayAccepted(conn, cred),
Predefined: awaitedStore{b: predefined}, ConfigCandidates: awaitedStore{b: configCandidates}, Logf: logger.Printf,
Named: namedVia(calls.Call), Trunk: trunkVia(calls.Call), Contains: containsVia(calls.Call), Read: readVia(calls.Call),
ReadOpen: openVia(calls.Call), Packages: awaitedStore{packages}, Files: filesVia(calls.Call), GoSum: goSumVia(calls.Call), GoMod: goModVia(calls.Call),
OwnBuild: ownBuildVia(calls.Call, cred.Node), OpenTries: openTriesVia(calls.Call), OwnBuildState: awaitedStore{ownBuildState}, TryRecovery: awaitedStore{tryRecovery}, Graph: awaitedStore{graph},
SeatDefinitions: awaitedStore{seatDefs}, ReadFile: readFileVia(calls.Call), SaySeatDefinition: saySeats,
ReadOpen: openVia(calls.Call), Packages: awaitedStore{b: packages, js: js, name: index.PackagesBucket}, Files: filesVia(calls.Call), GoSum: goSumVia(calls.Call), GoMod: goModVia(calls.Call),
OwnBuild: ownBuildVia(calls.Call, cred.Node), OpenTries: openTriesVia(calls.Call), OwnBuildState: awaitedStore{b: ownBuildState}, TryRecovery: awaitedStore{b: tryRecovery}, Graph: awaitedStore{b: graph},
SeatDefinitions: awaitedStore{b: seatDefs}, ReadFile: readFileVia(calls.Call), SaySeatDefinition: saySeats,
Message: messageVia(calls.Call),
// The take-in step (ADR 0633): its state pending and its stream take-in, both this delivery's, bound when they appear;
// until then every merge is taken in at once, as before.
Pending: awaitedStore{process.AwaitBucket(ctx, js, conn, index.PendingBucket, "no merge is held for the indexer its revision names: each is taken in at once", logger.Printf)},
Pending: awaitedStore{b: process.AwaitBucket(ctx, js, conn, index.PendingBucket, "no merge is held for the indexer its revision names: each is taken in at once", logger.Printf)},
SayTakeIn: sayTakeIn(js, cred)}
// Which build this program is, fixed here once, before it hears any merge and never asked again (index.FixOwnBuild): the
// node-engine names the successor while it replaces this process, and an old process that learnt it later would take
@@ -483,7 +495,7 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
// The holds, new with ADR 0632 (issue 1790's kind), bound when they appear: until then a merge that would hold a
// manifest is left for redelivery, said, rather than write what a counted engine cannot read.
holds := process.AwaitBucket(ctx, js, conn, index.HoldsBucket, "a merge whose manifest every node reads, needing what not every counted node-engine states, is left for redelivery", logger.Printf)
ix.Holds = awaitedStore{holds}
ix.Holds = awaitedStore{b: holds}
holds.Ready(func(jetstream.KeyValue) { ix.KickHolds(ctx) })
// Each standing hold's bound, a key that expires at its end (ADR 0632 §2, the indexer's own state): its expiry weighs
// the holds again, which raises manifest-held; until the state is bound a hold past its bound is raised at its next
@@ -696,7 +708,7 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
taken.Ready(func(jetstream.KeyValue) { ix.KickFill(ctx) })
// What the indexer heard of FORGE (issue 1126 note 76) is bound when its state appears, as every state a delivery adds
// (issue 1790): until then no message is written there, and the rest of the indexer runs.
heard := awaitedStore{process.AwaitBucket(ctx, js, conn, index.HeardBucket, "no FORGE message the indexer handles is written for indexer_heard", logger.Printf)}
heard := awaitedStore{b: process.AwaitBucket(ctx, js, conn, index.HeardBucket, "no FORGE message the indexer handles is written for indexer_heard", logger.Printf)}
// Each change of the states its readers read is said on its views family (ADR 0615 §5), from the mark each state was
// said up to; a first view (the tools bundle) answers at that mark.
startViews(ctx, js, cred, marks,
@@ -4,6 +4,7 @@ import (
"bytes"
"context"
"encoding/json"
"errors"
"log"
"strings"
"testing"
@@ -151,8 +152,28 @@ func TestAMessageIsIndexedAndAcked(t *testing.T) { // 1238/AC2 (unit part)
t.Errorf("forge/group's merged message read as a group's end: %q", logs.String())
}
})
// Issue 1966: a bucket the bus has not made is "not yet"; one it cannot reach (jetstream not enabled, no responders) is the
// bus's own error, which stops a pass instead of reading as nothing published.
t.Run("a bucket not found is not yet, one the bus cannot reach is its error", func(t *testing.T) {
for _, c := range []struct {
name string
err error
notYet bool
}{{"not found", jetstream.ErrBucketNotFound, true}, {"not reachable", jetstream.ErrJetStreamNotEnabled, false}} {
b := process.AwaitBucket(context.Background(), opener{c.err}, nil, "packages", "none", t.Logf)
_, err := awaitedStore{b: b, js: opener{c.err}, name: "packages"}.Keys(context.Background())
if errors.Is(err, process.ErrNotYet) != c.notYet || err == nil || (!c.notYet && !errors.Is(err, c.err)) {
t.Errorf("%s: Keys answered %v, want not-yet %v", c.name, err, c.notYet)
}
}
})
}
// opener is a bus that answers every bucket with err.
type opener struct{ err error }
func (o opener) KeyValue(context.Context, string) (jetstream.KeyValue, error) { return nil, o.err }
func TestTheSourceMainKeyIsTakenAtItsRevision(t *testing.T) { // 1344/AC1 (the watch's side: key, revision, value)
var logs bytes.Buffer
logger := log.New(&logs, "", 0)
@@ -24,6 +24,7 @@ type takerIn interface {
TakeInMerge(ctx context.Context, repository, merge string, seq uint64, by, why string) index.TakenIn
DropTryCandidate(ctx context.Context, id, by, why string) index.TakenIn
RecoverTriesByHand(ctx context.Context, by, why string) index.TakenIn
FingerprintsAgain(ctx context.Context, module string) index.TakenIn
}
// takenIn answers one value of the takeins state (the tool indexer_take_in, issue 1945): a value not yet answered takes the
@@ -42,7 +43,9 @@ func takenIn(ctx context.Context, ix takerIn, kv jetstream.KeyValue, e jetstream
return
}
asked, r := "take-in", index.TakenIn{}
if a.RecoverTries {
if a.Fingerprints {
asked, r = "computing of fingerprints again", ix.FingerprintsAgain(ctx, a.Module)
} else if a.RecoverTries {
asked, r = "recovery of the tries", ix.RecoverTriesByHand(ctx, a.By, a.Why)
a.Module = "tries"
} else if a.TryCandidate != "" && a.Drop {
@@ -150,13 +150,15 @@ type Indexer struct {
retrying atomic.Bool // a RetryUnread runs in the background (kickRetry)
retryAgain atomic.Bool // an event asked for one more RetryUnread after the running one
retryWG sync.WaitGroup // the background RetryUnread, for tests to wait on
recompute sync.Mutex // one Recompute at a time
untaken map[string]string // repository to the latest merge applied while the taken state was missing (taken.go)
filling atomic.Bool // a FillTaken runs in the background (KickFill)
fillAgain atomic.Bool // a kick arrived while it ran
atHeads map[string]map[string]meshcatalog.PackagesByModuleValue // a member head's published Go modules, by repository@head (candidatePublished)
saidAt map[string]string // the head of a repository whose failed read was said, by repository: said once per head
failedAt map[string]string // the head of a repository whose tree lacks a directory asked, by repository: not asked again until the head changes
fresh map[string]bool // modules whose import form is computed again by hand (FingerprintsAgain), "*" for every one; under freshMu
freshMu sync.Mutex
recompute sync.Mutex // one Recompute at a time
untaken map[string]string // repository to the latest merge applied while the taken state was missing (taken.go)
filling atomic.Bool // a FillTaken runs in the background (KickFill)
fillAgain atomic.Bool // a kick arrived while it ran
atHeads map[string]map[string]meshcatalog.PackagesByModuleValue // a member head's published Go modules, by repository@head (candidatePublished)
saidAt map[string]string // the head of a repository whose failed read was said, by repository: said once per head
failedAt map[string]string // the head of a repository whose tree lacks a directory asked, by repository: not asked again until the head changes
// OwnBuild answers the fingerprint of the build this program runs, from the node-engine's seat; nil knows none.
OwnBuild func(ctx context.Context) (string, error)
underSeats bool // a dry run inside judge.WithSeats: its judge passes do not hold the seat list (verdicts.go)
@@ -0,0 +1,167 @@
package index
import (
"context"
"encoding/json"
"errors"
"fmt"
"maps"
"slices"
"strings"
"time"
"git.novox.be/novox/mesh-delivery/contracts/go/clients/meshcatalogclient"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// FingerprintsAgainTool computes the fingerprints again now (issue 1966; the instruction "tools are how a module is spoken
// to"): the indexer reads the packages state, takes no held fingerprint for the build's, and writes the import form for
// every module whose imports are all published, as a merge's recompute would. For the module whose fingerprint a pass
// wrote while the packages state was unread (the version line over the import form) and that nothing else moves.
const FingerprintsAgainTool = "indexer_fingerprints_again"
// FingerprintMove is one module's fingerprint before and after a recompute by hand.
type FingerprintMove struct {
Module string `json:"module"`
Old string `json:"old"`
New string `json:"new"`
}
// FingerprintsAgain computes the fingerprints of module again ("all" or empty: every held module) from the published
// packages as they are read now, and answers each fingerprint that moved. A read of the packages state that failed
// writes nothing, and says so.
func (ix *Indexer) FingerprintsAgain(ctx context.Context, module string) TakenIn {
module = strings.TrimSpace(module)
if module == "" {
module = "all"
}
out := TakenIn{Module: module}
if ix.Modules == nil || ix.Manifests == nil {
out.Answer = "the indexer writes no fingerprints"
return out
}
before, err := ix.fingerprintsHeld(ctx)
if err != nil {
out.Answer = "the modules state cannot be read: " + err.Error()
return out
}
if module != "all" {
if _, ok := before[module]; !ok {
out.Answer = "the indexer holds no fingerprint of " + module
return out
}
}
key := module
if module == "all" {
key = "*"
}
ix.freshMu.Lock()
ix.fresh = map[string]bool{key: true}
ix.freshMu.Unlock()
// The path of a merge's recompute: the modules, then the candidates their fingerprints reach (recomputeWith), then the
// pending merges a changed fingerprint settles (PendingAgain, as after a toolchain report).
err = ix.recomputeWith(ctx, nil, nil)
ix.freshMu.Lock()
ix.fresh = nil
ix.freshMu.Unlock()
if err != nil {
out.Answer = "nothing was written: " + err.Error()
return out
}
if err := ix.PendingAgain(ctx); err != nil {
ix.logf("the pending merges are not evaluated again after the fingerprints were computed again: %v", err)
}
after, err := ix.fingerprintsHeld(ctx)
if err != nil {
out.Answer = "computed, but the modules state cannot be read back: " + err.Error()
return out
}
for _, m := range slices.Sorted(maps.Keys(after)) {
if module != "all" && m != module {
continue
}
if before[m] != after[m] {
out.Fingerprints = append(out.Fingerprints, FingerprintMove{Module: m, Old: before[m], New: after[m]})
}
}
out.Stored = true
out.Answer = fmt.Sprintf("%d fingerprint(s) moved", len(out.Fingerprints))
return out
}
// isFresh says the module's import form is computed again by hand in this pass.
func (ix *Indexer) isFresh(module string) bool {
ix.freshMu.Lock()
defer ix.freshMu.Unlock()
return ix.fresh[module] || ix.fresh["*"]
}
// fingerprintsHeld is each held module's fingerprint of the new form.
func (ix *Indexer) fingerprintsHeld(ctx context.Context) (map[string]string, error) {
keys, err := ix.Modules.Keys(ctx)
if err != nil {
return nil, err
}
out := map[string]string{}
for _, k := range keys {
value, _, found, err := ix.Modules.Get(ctx, k)
if err != nil {
return nil, err
}
if !found {
continue
}
if m, err := meshcatalogclient.DecodeModule(value); err == nil {
out[k] = m.Fp2
}
}
return out, nil
}
// FingerprintsTools is the tool indexer_fingerprints_again over ask (the takeins state). It acts first and says the hand
// act after: record is told what was done and answers an error when it could not be recorded, which the answer says and
// which never refuses or delays the recovery beyond bound.
func FingerprintsTools(ask func(key string, value []byte) (TakeInAsk, error), now func() time.Time, record func(what, why string) error, bound time.Duration) []stdio.Tool {
return []stdio.Tool{{Name: FingerprintsAgainTool,
Description: "Compute the fingerprints of one module (or all) again now from the published packages as the indexer reads them: " +
"the import form is written for every module whose imports are all published, whatever fingerprint the module holds. For the " +
"fingerprints a pass wrote while the packages state was unread (the version line over the import form: builds then fail on an " +
"older package). Answers each module whose fingerprint moved, old and new; when the packages state cannot be read it writes " +
"nothing and says why. Runs as a merge's recompute does (the modules, the candidates they reach, the pending merges they settle). Acts first, then records the hand act through mesh-hand-acts; when that cannot be recorded the answer says so.",
Input: map[string]any{"type": "object", "required": []string{"why"}, "properties": map[string]any{
"module": map[string]any{"type": "string", "description": "the module's name, or all (the default)"},
"why": map[string]any{"type": "string", "description": "why it is computed again now, in plain words"}}},
Run: func(args map[string]any) (any, error) {
module, _ := args["module"].(string)
why, _ := args["why"].(string)
module, why = strings.TrimSpace(module), strings.TrimSpace(why)
if why == "" {
return nil, errors.New("say why: a hand act is said in the journal")
}
if module == "" {
module = "all"
}
data, err := json.Marshal(TakeInAsk{V: 1, Module: module, Fingerprints: true, Why: why, By: stdio.CallerOf(args), At: now().UTC()})
if err != nil {
return nil, err
}
a, err := ask("fingerprints."+module+"."+now().UTC().Format("20060102T150405.000000000"), data)
if err != nil {
return nil, err
}
res := map[string]any{"result": a.Result}
done := make(chan error, 1)
go func() { done <- record("computed the fingerprints of "+module+" again by hand", why) }()
wait, cancel := context.WithTimeout(context.Background(), bound) // a context's bound, no timer (ADR 0437 §8)
defer cancel()
select {
case err := <-done:
if err != nil {
res["hand-act"] = "not recorded: " + err.Error()
}
case <-wait.Done(): // a bound on the record, never on the recovery
res["hand-act"] = fmt.Sprintf("not recorded within %s: mesh-hand-acts did not answer", bound)
}
return res, nil
}}}
}
@@ -57,6 +57,9 @@ func formOf(module, repository string, kept *meshcatalog.ModulesByModuleValueInp
logf("%s: no published package holds %s, so its fingerprint takes the version line", module, strings.Join(r.Missing, ", "))
}
}
// The fingerprint written here is the version line's, not the import form's: no import form is held for the inputs it was
// computed from, so a later pass whose import form computes the same does not take this one for the build's (issue 1966).
kept.ImportForm = ""
others := fingerprint.OtherPackages(kept.Packages, repository)
versions, unpublished := fingerprint.HeldVersionsOf(others, held)
for p, v := range pins { // an accepted series' newest published version (acceptedSeries, DLV-I6)
@@ -2,13 +2,16 @@ package index
import (
"context"
"errors"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-delivery/contracts/go/clients/meshcatalogclient"
"git.novox.be/novox/mesh-delivery/contracts/go/seats/meshcatalog"
"git.novox.be/novox/mesh-delivery/contracts/go/seats/meshdeliverychecker"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/builds"
"git.novox.be/novox/mesh-sdk/go/process"
)
const meshSDK = "git.novox.be/novox/mesh-sdk/go"
@@ -51,6 +54,7 @@ func (m *missingDir) Error() string { return "no directory " + m.dir }
func TestImportsReportedMoveTheFingerprintOnlyWithAnImportedPackage(t *testing.T) {
t.Run("a build of an older fingerprint keeps no imports", buildOfAnOlderFingerprintKeepsNoImports)
t.Run("a change of a sibling directory alone moves its importer", aSiblingDirectoryAloneMovesItsImporter)
t.Run("an unread packages state writes no fingerprint, and they are computed again by hand (issue 1966)", anUnreadPackagesStateWritesNoFingerprintAndFingerprintsAreComputedAgain)
t.Run("an accepted series' newest published version pins it, by that version's contents", anAcceptedSeriesPinsItsNewest)
ctx := context.Background()
ix, mods := newFormIndexer(t, mod{"avahi", catalogueManifest(t, "avahi")})
@@ -270,3 +274,114 @@ func anAcceptedSeriesPinsItsNewest(t *testing.T) {
t.Errorf("a trunk merge moved avahi off its accepted series' newest: %q %v", m.Fp2, m.Packages)
}
}
// Issue 1966 (the indexer started while the bus's storage was down): a read of the packages state that failed, or that
// answered no published package where held modules were computed from published ones, writes no fingerprint (the version
// line over every import form is what an empty state computes); a version line written leaves no import form behind
// that a later pass takes for the build's; and indexer_fingerprints_again computes the import form again by hand.
func anUnreadPackagesStateWritesNoFingerprintAndFingerprintsAreComputedAgain(t *testing.T) {
ctx := context.Background()
// avahi at the import form, pinned to mesh-sdk at c3, as the test above leaves it.
setup := func(t *testing.T) (*Indexer, *memStore, meshcatalog.ModulesByModuleValue) {
ix, mods := newFormIndexer(t, mod{"avahi", catalogueManifest(t, "avahi")})
reportToolchain(t, ix, "mesh-build-go", "novox", "go", "go1.27.1")
ix.Packages, ix.Files = newMem(), forge{c1: sdkFiles("a1", "b1"), c3: sdkFiles("a2", "b2")}.files
setPackages(t, mods, "avahi", []string{meshSDK})
raw, err := meshcatalogclient.EncodeModule(meshcatalog.ModulesByModuleValue{Repository: "novox/mesh-sdk", Path: ".", Commit: c1, None2: "a TypeScript package"})
if err != nil {
t.Fatal(err)
}
mods.put("mesh-sdk", raw)
held, _ := ix.heldModules(ctx)
if err := ix.publish(ctx, "novox/mesh-sdk", c1, held, false); err != nil {
t.Fatal(err)
}
if err := ix.Recompute(ctx); err != nil {
t.Fatal(err)
}
first, _ := moduleOf(t, mods, "avahi")
if err := ix.ApplyBuildEnd(ctx, buildEnd(t, "avahi", first.Fp2, meshSDK, meshSDK+"/announce")); err != nil {
t.Fatal(err)
}
held, _ = ix.heldModules(ctx)
if err := ix.publish(ctx, "novox/mesh-sdk", c3, held, false); err != nil {
t.Fatal(err)
}
if err := ix.Recompute(ctx); err != nil {
t.Fatal(err)
}
m, _ := moduleOf(t, mods, "avahi")
if m.Fp2 == first.Fp2 || m.Fp2 != m.Inputs2.ImportForm {
t.Fatalf("setup: fp2 %q (was %q), import form %q; want the import form", m.Fp2, first.Fp2, m.Inputs2.ImportForm)
}
return ix, mods, m
}
t.Run("a pass whose read of the packages state failed or answered nothing writes no fingerprint", func(t *testing.T) {
for _, c := range []struct {
name string
bad func(ix *Indexer)
why string
}{
{"the listing fails", func(ix *Indexer) { ix.Packages.(*memStore).keysErr = errors.New("jetstream not enabled") }, "cannot be listed"},
{"the state is not on the bus yet", func(ix *Indexer) { ix.Packages.(*memStore).keysErr = process.ErrNotYet }, "answers no published package"}, // a mesh with no import form computes as before: TestIndexerWritesModulesAtAMerge
{"the state answers no package", func(ix *Indexer) { ix.Packages = newMem() }, "answers no published package"},
} {
t.Run(c.name, func(t *testing.T) {
ix, mods, _ := setup(t)
was, _ := moduleOf(t, mods, "avahi")
c.bad(ix)
err := ix.Recompute(ctx)
if err == nil || !strings.Contains(err.Error(), c.why) {
t.Fatalf("recompute with an unread packages state: err %v, want one saying %q", err, c.why)
}
if now, _ := moduleOf(t, mods, "avahi"); now.Fp2 != was.Fp2 {
t.Fatalf("a pass over an unread packages state moved avahi's fingerprint to %q (was %q)", now.Fp2, was.Fp2)
}
})
}
})
t.Run("a version line written keeps no import form, and a fingerprint computed again by hand is the import form", func(t *testing.T) {
ix, mods, good := setup(t)
// What the pass that read nothing wrote on novox: the version line's fingerprint, the import form still kept.
bad := good
bad.Fp2 = "src2:" + strings.Repeat("d", 64)
raw, err := meshcatalogclient.EncodeModule(bad)
if err != nil {
t.Fatal(err)
}
mods.put("avahi", raw)
if err := ix.Recompute(ctx); err != nil {
t.Fatal(err)
}
if now, _ := moduleOf(t, mods, "avahi"); now.Fp2 != bad.Fp2 {
t.Fatalf("the held version line is not kept by the build's shortcut (fp2 %q); the case the tool recovers has changed", now.Fp2)
}
r := ix.FingerprintsAgain(ctx, "avahi")
if !r.Stored || len(r.Fingerprints) != 1 || r.Fingerprints[0].Old != bad.Fp2 || r.Fingerprints[0].New != good.Fp2 {
t.Fatalf("fingerprints again: %+v, want avahi from %q to %q", r, bad.Fp2, good.Fp2)
}
if now, _ := moduleOf(t, mods, "avahi"); now.Fp2 != good.Fp2 || ix.fresh != nil {
t.Fatalf("avahi holds %q after it, want %q (fresh %v)", now.Fp2, good.Fp2, ix.fresh)
}
// It runs a merge's recompute, candidates included: a candidate store that cannot be listed fails it, said.
ix.Store, ix.previews = &memStore{keysErr: errors.New("no bus")}, map[string]meshdeliverychecker.PreviewsByRepositoryHeadTrunkValue{"k": {}}
if r := ix.FingerprintsAgain(ctx, "avahi"); r.Stored || !strings.Contains(r.Answer, "listing the candidates") {
t.Fatalf("fingerprints again over candidates that cannot be listed: %+v", r)
}
if r := ix.FingerprintsAgain(ctx, "nothing"); r.Stored {
t.Fatalf("fingerprints again of a module not held answered stored: %+v", r)
}
})
t.Run("formOf clears the import form when it writes the version line", func(t *testing.T) {
for _, c := range []struct {
name string
imports []string
}{{"an import no package holds", []string{meshSDK + "/announce"}}, {"no import reported", nil}} {
kept := meshcatalog.ModulesByModuleValueInputs2{Tree: "t", Imports: c.imports, ImportForm: "src2:" + strings.Repeat("a", 64)}
f := formOf("avahi", "novox/mesh-catalog", &kept, nil, meshcatalog.ModulesByModuleValue{}, nil, nil, nil, nil)
if kept.ImportForm != "" {
t.Errorf("%s: kept import form %q after the version line %q", c.name, kept.ImportForm, f.fp)
}
}
})
}
@@ -4,6 +4,7 @@ import (
"context"
"errors"
"fmt"
"git.novox.be/novox/mesh-sdk/go/process"
"maps"
"path"
"reflect"
@@ -31,21 +32,40 @@ import (
// publishedAll is every value of the packages state, by Go module path; empty with no state or one that does not answer.
func (ix *Indexer) publishedAll(ctx context.Context) map[string]meshcatalog.PackagesByModuleValue {
out := map[string]meshcatalog.PackagesByModuleValue{}
if ix.Packages == nil {
return out
}
keys, err := ix.Packages.Keys(ctx)
out, err := ix.publishedRead(ctx)
if err != nil {
ix.logf("the packages state cannot be listed, so every fingerprint takes the version line: %v", err)
return out
}
return out
}
// publishedRead is publishedAll with a read that failed said as an error, never as an empty map: a pass that takes an
// unread state for "no package is published" writes the version line over every module's fingerprint (issue 1966, the
// indexer started while the bus's storage was down). No state wired is no error: nothing is published then.
func (ix *Indexer) publishedRead(ctx context.Context) (map[string]meshcatalog.PackagesByModuleValue, error) {
out := map[string]meshcatalog.PackagesByModuleValue{}
if ix.Packages == nil {
return out, nil
}
keys, err := ix.Packages.Keys(ctx)
if errors.Is(err, process.ErrNotYet) {
return out, nil // the state is not on the bus yet: nothing is published, as before
}
if err != nil {
return out, fmt.Errorf("the packages state cannot be listed: %w", err)
}
for _, key := range keys {
if IsSeriesKey(key) {
continue // a package's series (series.go), not a Go module's digests
}
value, _, found, err := ix.Packages.Get(ctx, key)
if err != nil || !found {
if errors.Is(err, process.ErrNotYet) {
continue
}
if err != nil {
return out, fmt.Errorf("packages.%s cannot be read: %w", key, err)
}
if !found {
continue
}
p, err := DecodePublishedPackages(value)
@@ -55,7 +75,32 @@ func (ix *Indexer) publishedAll(ctx context.Context) map[string]meshcatalog.Pack
}
out[p.Module] = p
}
return out
return out, nil
}
// publishedForPass is the packages state a recompute computes fingerprints from, or why it must write none: a read that
// failed, or that answered no published module where the held modules show packages were published before (a module
// whose fingerprint was computed from the import form, or that holds resolved package versions). The first start of a
// mesh, whose held modules show none, computes from an empty state as before.
func (ix *Indexer) publishedForPass(ctx context.Context, held map[string]meshcatalog.ModulesByModuleValue) (map[string]meshcatalog.PackagesByModuleValue, error) {
published, err := ix.publishedRead(ctx)
if err != nil {
return nil, err
}
if len(published) > 0 {
return published, nil
}
var had []string
for key, m := range held {
if m.Inputs2 != nil && m.Inputs2.ImportForm != "" { // not Packages: a version line holds those too, with nothing published
had = append(had, key)
}
}
if len(had) > 0 {
slices.Sort(had)
return nil, fmt.Errorf("the packages state answers no published package, though %d held modules (%s ...) were computed from published ones", len(had), had[0])
}
return published, nil
}
// reached are, by repository, the mesh Go module paths the held modules' go.sum names and the imports their builds
@@ -135,6 +135,40 @@ func sha(c byte) string { return strings.Repeat(string(c), 40) }
func TestTakeIn(t *testing.T) { // 1585/T3-T13 (ADR 0633)
ctx := context.Background()
t.Run("indexer_fingerprints_again acts before it records the hand act and says a failed record", func(t *testing.T) { // issue 1966
for _, c := range []struct {
name string
record func(what, why string) error
said string
}{
{"recorded", func(string, string) error { return nil }, ""},
{"record refused", func(string, string) error { return errors.New("no ledger") }, "not recorded: no ledger"},
{"record never answers", func(string, string) error { select {} }, "not recorded within"},
} {
t.Run(c.name, func(t *testing.T) {
var asked []string
ask := func(key string, _ []byte) (TakeInAsk, error) {
asked = append(asked, key)
return TakeInAsk{Result: &TakenIn{Module: "all", Stored: true}}, nil
}
tool := FingerprintsTools(ask, time.Now, c.record, 10*time.Millisecond)[0]
if _, err := tool.Run(map[string]any{"module": "avahi"}); err == nil {
t.Fatal("a hand act with no why was done")
}
if len(asked) != 0 {
t.Fatalf("asked the indexer without a why: %v", asked)
}
got, err := tool.Run(map[string]any{"module": "avahi", "why": "the packages read came back empty"})
if err != nil || len(asked) != 1 || !strings.HasPrefix(asked[0], "fingerprints.avahi.") {
t.Fatalf("run: %v, asked %v", err, asked)
}
said, _ := got.(map[string]any)["hand-act"].(string)
if !strings.Contains(said, c.said) || (c.said == "") != (said == "") {
t.Fatalf("hand-act %q, want one saying %q", said, c.said)
}
})
}
})
t.Run("T6 a merge while the own build is unknown writes nothing", func(t *testing.T) {
b := newTakeInBed(t)
b.ix.ownFP, b.ix.ownFixed = "", false // this process is started again and asks the engine, which does not answer
@@ -220,6 +220,8 @@ type TakenIn struct {
// and deleted.
Handed []string `json:"handed,omitempty"`
Dropped []string `json:"dropped,omitempty"`
// Fingerprints are the modules whose fingerprint a recompute by hand moved (FingerprintsAgain), each old and new.
Fingerprints []FingerprintMove `json:"fingerprints,omitempty"`
}
// TakeIn reads a registered module's manifest now, whatever was tried before, and writes it as a merge would (the tool
@@ -56,6 +56,9 @@ type TakeInAsk struct {
Sequence uint64 `json:"sequence,omitempty"`
// TryCandidate asks the candidate of this try's group to be dropped (indexer_drop_try_candidate), trycandidates.go.
TryCandidate string `json:"try-candidate,omitempty"`
// Fingerprints asks the fingerprints of Module ("all": every module) to be computed again now, from the published
// packages as they are read (indexer_fingerprints_again, fingerprints.go).
Fingerprints bool `json:"fingerprints,omitempty"`
// RecoverTries asks the open tries' candidates to be recovered now (indexer_recover_tries).
RecoverTries bool `json:"recover-tries,omitempty"`
Drop bool `json:"drop,omitempty"`
@@ -77,7 +80,7 @@ func DecodeTakeIn(data []byte) (TakeInAsk, error) {
if a.V != 1 {
return a, fmt.Errorf("a takeins value of version %d, not 1", a.V)
}
if a.Module == "" && a.Repository == "" && a.Pending == "" && a.Merge == "" && a.TryCandidate == "" && !a.RecoverTries {
if a.Module == "" && a.Repository == "" && a.Pending == "" && a.Merge == "" && a.TryCandidate == "" && !a.RecoverTries && !a.Fingerprints {
return a, errors.New("a takeins value naming no module, repository, pending wait, merge or try")
}
return a, nil
@@ -223,7 +223,13 @@ func (ix *Indexer) recomputeLocked(ctx context.Context, overlay map[string]meshc
all[key] = &known{key: key, cur: cur, manifest: m, languages: languagesOf(m), bases: basesOf(m), contexts: hasContext(m), held: before}
}
tools := ix.agreed()
published := ix.publishedAll(ctx)
published, err := ix.publishedForPass(ctx, held)
if err != nil {
// No module's fingerprint is written from a state that was not read: the version line over every import form is
// what an empty packages state would compute (issue 1966). The next pass, or indexer_fingerprints_again, computes them.
ix.logf("no fingerprint is written by this pass: %v", err)
return fmt.Errorf("no fingerprint is written: %w", err)
}
series := ix.SeriesAll(ctx) // each package's series, the newest a user accepting it takes (DLV-I6)
type result struct {
fp, none string
@@ -258,6 +264,9 @@ func (ix *Indexer) recomputeLocked(ctx context.Context, overlay map[string]meshc
chain = append(chain, module)
kept := *k.cur.Inputs2
kept.Bases, kept.Builder = nil, ""
if ix.isFresh(module) { // indexer_fingerprints_again: the import form is written again, the held one stands for nothing
kept.ImportForm = ""
}
var missing []string
for _, b := range k.bases {
if r := of(b, chain); r.fp != "" {
@@ -175,8 +175,8 @@ func TestManifestDeclarations(t *testing.T) { // 1238/AC1, 1444 (reads the build
want = append(want, "indexer_first_views", "indexer_first_view_again", index.ConfigReadersTool, index.BusStateTool, index.RequirementsTool, index.PackageUsersTool, index.SayAcceptedTool, // DLV-F3r, DLV-I10, DLV-I11, DLV-I7, DLV-I8, DLV-I9c
index.SeatDefinitionTool, index.SeatDefinitionsAgainTool, index.OutOfStepTool, // DLV-I19
index.HoldsTool, index.EngineReadsTool, // ADR 0632 §2: the mesh-wide holds and the engines' reads they weigh
index.PendingTool, index.TakeInPendingTool, index.DropPendingTool, index.TryCandidatesTool, index.DropTryCandidateTool, index.RecoverTriesTool, index.TakeInMergeTool) // 1585/T12, T13 (ADR 0633 §5)
if !reflect.DeepEqual(m.Tools, want) || len(want) != 38 {
index.PendingTool, index.TakeInPendingTool, index.DropPendingTool, index.TryCandidatesTool, index.DropTryCandidateTool, index.RecoverTriesTool, index.TakeInMergeTool, index.FingerprintsAgainTool) // issue 1966; 1585/T12, T13 (ADR 0633 §5)
if !reflect.DeepEqual(m.Tools, want) || len(want) != 39 {
t.Errorf("tools = %v, want %v", m.Tools, want)
}
// Every read tool of the indexer a reader of this repository invokes is one it serves (the checker's seventh first
+2 -1
View File
@@ -54,7 +54,8 @@
"indexer_try_candidates",
"indexer_drop_try_candidate",
"indexer_recover_tries",
"indexer_take_in_merge"
"indexer_take_in_merge",
"indexer_fingerprints_again"
],
"consumes": [
"go-package-registry.published",
+54 -4
View File
@@ -1,8 +1,12 @@
package buildstep
import (
"bytes"
"compress/gzip"
"encoding/base64"
"encoding/json"
"fmt"
"io"
"path"
"slices"
"sort"
@@ -281,13 +285,55 @@ type CheckJob struct {
// ComposeJob asks one compose-candidate build.
type ComposeJob struct {
Fetch Fetch `json:"fetch"`
Dir string `json:"dir"`
Package string `json:"package"`
Fetch Fetch `json:"fetch"`
Dir string `json:"dir"`
Package string `json:"package"`
// Request is the composer's request: a JSON object, or a JSON string PackedRequestPrefix + the base64 of the object
// gzip-compressed (issue 1966; the checker packs every request it sends, and
// the part's `request` describes both forms). The part types it `any`, so both fit it; a
// build module that does not know the packed form refuses the ask naming that it holds no request object.
Request json.RawMessage `json:"request"`
Reader string `json:"reader,omitempty"`
}
// PackedRequestPrefix opens a compose job's request that travels gzip-compressed.
const PackedRequestPrefix = "gzip+base64:"
// PackRequest sets the job's request from the JSON object raw, packed.
func (c *ComposeJob) PackRequest(raw []byte) error {
var b bytes.Buffer
zw := gzip.NewWriter(&b)
if _, err := zw.Write(raw); err != nil {
return err
}
if err := zw.Close(); err != nil {
return err
}
packed, err := json.Marshal(PackedRequestPrefix + base64.StdEncoding.EncodeToString(b.Bytes()))
c.Request = packed
return err
}
// RequestBytes is the job's request as a JSON object, unpacked when it travels packed.
func (c ComposeJob) RequestBytes() ([]byte, error) {
var packed string
if len(c.Request) == 0 || c.Request[0] != '"' {
return c.Request, nil
}
if err := json.Unmarshal(c.Request, &packed); err != nil || !strings.HasPrefix(packed, PackedRequestPrefix) {
return nil, fmt.Errorf("buildstep: a compose job's request is a string that is no packed request")
}
raw, err := base64.StdEncoding.DecodeString(strings.TrimPrefix(packed, PackedRequestPrefix))
if err != nil {
return nil, fmt.Errorf("buildstep: the packed request of a compose job is no base64: %w", err)
}
zr, err := gzip.NewReader(bytes.NewReader(raw))
if err != nil {
return nil, fmt.Errorf("buildstep: the packed request of a compose job is no gzip: %w", err)
}
return io.ReadAll(zr)
}
// AskOf is any seat's ask (or this form) as the neutral StepAsk: the one conversion, since the three seats' types
// are the part's one layout. A value that does not fit the layout is refused.
func AskOf[T any](seatAsk T) (StepAsk, error) {
@@ -378,7 +424,11 @@ func (c ComposeJob) check(id string) error {
if c.Package == "" {
return fmt.Errorf("buildstep: the compose ask %s names no package to build", id)
}
if !json.Valid(c.Request) || len(c.Request) == 0 || c.Request[0] != '{' {
req, err := c.RequestBytes()
if err != nil {
return err
}
if !json.Valid(req) || len(req) == 0 || req[0] != '{' {
return fmt.Errorf("buildstep: the compose ask %s holds no request object", id)
}
return nil
+22
View File
@@ -144,6 +144,28 @@ func TestAskCheckAndRoundTrip(t *testing.T) {
if _, err := DecodeStepAsk([]byte(`{"v":2}`)); !errors.Is(err, ErrUnknownVersion) {
t.Errorf("another version: %v", err)
}
// Issue 1966: a compose job's request travels packed when it passes the bus's message limit, and reads back as sent.
t.Run("1966: a compose request travels packed", func(t *testing.T) {
obj := []byte(`{"v":1,"input":{"nodes":[]}}`)
job := ComposeJob{Fetch: Fetch{Repository: "novox/mesh-host", Commit: strings.Repeat("a", 40)}, Dir: ".", Package: "p"}
if err := job.PackRequest(obj); err != nil {
t.Fatal(err)
}
if got, err := job.RequestBytes(); err != nil || string(got) != string(obj) {
t.Errorf("read back %s, %v", got, err)
}
if err := job.check("ask"); err != nil {
t.Errorf("a packed request is refused: %v", err)
}
job.Request = json.RawMessage(`"not packed"`)
if err := job.check("ask"); err == nil {
t.Errorf("a string that is no packed request is accepted")
}
job.Request = obj
if got, err := job.RequestBytes(); err != nil || string(got) != string(obj) {
t.Errorf("a whole request read back %s, %v", got, err)
}
})
}
// The seat's value and the neutral form are the same JSON: AskOf takes any seat's type (here its shape, with a