Give each node a bucket of its own for the files that travel by digest, and keep the sweep from removing an object a send relies on (issues 1696, 1697; review of ADR 0486 step 3)

A literal * in a published subject matches the grant's *, so in one shared bucket a node could read every node's files. Each node now has declaration-files-<node>, which its account reads by name and the controller writes; the sweep judges each bucket against that node's latest declaration, under the node's lock with the send, and an object the sweep marked is put again by the next send. Putting that fails send after send raises a condition.
2026-10-21 15:43:36 +02:00
parent ceaa616d01
commit da364c2730
15 changed files with 544 additions and 242 deletions
@@ -231,10 +231,6 @@ func (j *JetStream) EnsureControllerBuckets() error {
 	if err := EnsureLeaseBucket(ctx, js); err != nil {
 		return err
 	}
-	// And the object bucket a declaration's large files travel in (novox/hq ADR 0486 §4): it exists whenever the bus does.
-	if err := EnsureDeclarationFiles(ctx, js); err != nil {
-		return err
-	}
 	if _, err := js.CreateOrUpdateKeyValue(ctx, jetstream.KeyValueConfig{
 		Bucket: CallsBucket,
 		Description: "the calls of the mesh's own verbs and what came of each (novox/hq to-be 45 §6, issue " +
@@ -9,81 +9,93 @@ import (
 	"github.com/nats-io/nats.go/jetstream"
 )

-// The object bucket a declaration's large files travel in (novox/hq ADR 0486, issues 1696 and 1697).
+// The object buckets a declaration's large files travel in (novox/hq ADR 0486, issues 1696 and 1697): **one bucket for each
+// node**, `declaration-files-<node>`.
 //
-// **The controller alone writes it, and a node-engine reads an object by its name** (ADR 0486 §2): its account may
-// neither list the bucket nor write it, and no other account has any access. A bucket of the bus's object store is a
-// stream, OBJ_<bucket>, whose messages are the objects' metadata (`$O.<bucket>.M.<base64 of the name>`) and chunks
-// (`$O.<bucket>.C.<nuid>`).
+// **The controller alone writes them, and a node-engine reads an object by its name from its own bucket** (ADR 0486 §2): its
+// account may neither list the bucket nor write it, no account may touch another node's, and no other account has any access.
+// Why a bucket each: a subject pattern cannot tell a name from a wildcard. A grant of `...M.*` matches a publish to the literal
+// subject `...M.*` too (probed on nats-server 2.11.17 by the review of this issue: a node created a consumer filtered on
+// `$O.<bucket>.C.*` and read every object of the bucket), so in one shared bucket a node could read every other node's files.
+// With a bucket each, the same trick reads only the node's own files, which are the files of its own declaration.
+//
+// A bucket of the bus's object store is a stream, OBJ_<bucket>, whose messages are the objects' metadata
+// (`$O.<bucket>.M.<base64 of the name>`) and chunks (`$O.<bucket>.C.<nuid>`).

-// DeclarationFilesBucket is the bucket's name, which mesh-sdk fixes for the node-engine and the controller alike.
+// DeclarationFilesBucket is the prefix of the buckets' names, which mesh-sdk fixes for the node-engine and the controller alike.
 const DeclarationFilesBucket = declaration.FileBucket

-// DeclarationFilesStream is the stream under the bucket.
-const DeclarationFilesStream = "OBJ_" + DeclarationFilesBucket
+// DeclarationFilesBucketOf is the bucket a node's files are in. One function, so the name of a node's bucket is composed in one
+// place.
+func DeclarationFilesBucketOf(node string) string { return DeclarationFilesBucket + "-" + node }

-// DeclarationFilesMaxBytes is the bucket's size: the controller's objects are immutable and regenerable, and the
-// largest declaration it composes holds well under a hundred megabytes of them.
+// DeclarationFilesStreamOf is the stream under a node's bucket.
+func DeclarationFilesStreamOf(node string) string { return "OBJ_" + DeclarationFilesBucketOf(node) }
+
+// DeclarationFilesMaxBytes is a bucket's size: the objects are immutable and regenerable, and one node's hold well under a hundred
+// megabytes.
 const DeclarationFilesMaxBytes = 1 << 30

 // DeclarationFilesKeptFor is how long an object stays after no node's latest declaration names it (ADR 0486 §3).
 const DeclarationFilesKeptFor = 24 * time.Hour

-// DeclarationFilesWrites is the one pattern the controller may publish to write the bucket: every object's metadata
-// and chunks. The server's API for the stream is the controller's `$JS.API.>`.
-const DeclarationFilesWrites = "$O." + DeclarationFilesBucket + ".>"
+// DeclarationFilesWrites is the one pattern the controller may publish to write a node's bucket: every object's metadata and
+// chunks. The server's API for the stream is the controller's `$JS.API.>`.
+func DeclarationFilesWrites(node string) string { return "$O." + DeclarationFilesBucketOf(node) + ".>" }

-// DeclarationFilesReads is what a node-engine's account may publish to read an object by its name, and nothing more.
-// The set is the one node-engine's own call (mesh-host internal/link FileObject) makes, for B the bucket and S its
+// DeclarationFilesReads is what a node-engine's account may publish to read an object of its own bucket by its name, and nothing
+// more. The set is the one node-engine's own call (mesh-host internal/link FileObject) makes, for B the node's bucket and S its
 // stream (issue 1697, notes 4 and 5):
 //
 //   - the object's metadata, by a direct get of the last message of the one subject that names it. **Only the suffixed
-//     form**: the bare `$JS.API.DIRECT.GET.S` takes a sequence number and would read any message of the stream, every
-//     object's name and chunks.
+//     form**: the bare `$JS.API.DIRECT.GET.S` takes a sequence number and would read any message of the stream.
 //   - the ordered consumer on that object's chunks: `$JS.API.CONSUMER.CREATE.S.<name>.$O.B.C.<nuid>`. **Not `S.>`**:
-//     a consumer filtered on `$O.B.M.>` would read every object's name, the list that is refused. The chunks'
-//     subject carries the nuid the metadata gives, which only a reader of the metadata knows.
+//     a consumer filtered on `$O.B.M.>` would read every object's name.
 //   - the flow-control replies of that consumer, without which a large object stalls at the flow-control window.
 //   - the consumer's deletion, best effort: the consumer's inactivity threshold removes it if this is refused.
 //
-// **Never granted:** STREAM.INFO (a `subjects_filter` of `>` lists every name), STREAM.MSG.GET (any message by its
-// sequence), a publish to `$O.B.>` (a write).
-func DeclarationFilesReads() []string {
+// **Never granted:** STREAM.INFO (a `subjects_filter` of `>` lists every name), STREAM.MSG.GET (any message by its sequence), a
+// publish to `$O.B.>` (a write), and any subject of another node's stream. A literal `*` in a subject matches the grant's `*`: it
+// reaches the objects of this node's own bucket and no other (above).
+func DeclarationFilesReads(node string) []string {
+	stream, bucket := DeclarationFilesStreamOf(node), DeclarationFilesBucketOf(node)
 	return []string{
-		"$JS.API.DIRECT.GET." + DeclarationFilesStream + ".$O." + DeclarationFilesBucket + ".M.*",
-		"$JS.API.CONSUMER.CREATE." + DeclarationFilesStream + ".*.$O." + DeclarationFilesBucket + ".C.*",
-		"$JS.API.CONSUMER.DELETE." + DeclarationFilesStream + ".*",
-		"$JS.FC." + DeclarationFilesStream + ".>",
+		"$JS.API.DIRECT.GET." + stream + ".$O." + bucket + ".M.*",
+		"$JS.API.CONSUMER.CREATE." + stream + ".*.$O." + bucket + ".C.*",
+		"$JS.API.CONSUMER.DELETE." + stream + ".*",
+		"$JS.FC." + stream + ".>",
 	}
 }

-// EnsureDeclarationFiles creates the bucket if it is absent and brings its options to match. The bucket exists
-// whenever the bus does, so a node that is starting finds it (ADR 0486 §4). An update, never a delete.
-func (j *JetStream) EnsureDeclarationFiles() error {
+// EnsureDeclarationFiles creates the buckets of these nodes if absent and brings their options to match. A bucket exists whenever the
+// bus does, so a node that is starting finds it (ADR 0486 §4). An update, never a delete.
+func (j *JetStream) EnsureDeclarationFiles(nodes []string) error {
 	js, err := jetstream.New(j.conn)
 	if err != nil {
 		return err
 	}
-	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
 	defer cancel()
-	return EnsureDeclarationFiles(ctx, js)
+	return EnsureDeclarationFiles(ctx, js, nodes...)
 }

 // EnsureDeclarationFiles is the same on a JetStream the caller holds.
-func EnsureDeclarationFiles(ctx context.Context, js jetstream.JetStream) error {
-	if _, err := js.CreateOrUpdateObjectStore(ctx, DeclarationFilesConfig()); err != nil {
-		return fmt.Errorf("asserting the object bucket %s: %w", DeclarationFilesBucket, err)
+func EnsureDeclarationFiles(ctx context.Context, js jetstream.JetStream, nodes ...string) error {
+	for _, node := range nodes {
+		if _, err := js.CreateOrUpdateObjectStore(ctx, DeclarationFilesConfig(node)); err != nil {
+			return fmt.Errorf("asserting the object bucket %s: %w", DeclarationFilesBucketOf(node), err)
+		}
 	}
 	return nil
 }

-// DeclarationFilesConfig is the bucket as the controller asserts it.
-func DeclarationFilesConfig() jetstream.ObjectStoreConfig {
+// DeclarationFilesConfig is a node's bucket as the controller asserts it.
+func DeclarationFilesConfig(node string) jetstream.ObjectStoreConfig {
 	return jetstream.ObjectStoreConfig{
-		Bucket: DeclarationFilesBucket,
-		Description: "the contents of a declaration's files over 64 KiB, each under the hex of its sha256 digest " +
-			"(novox/hq ADR 0486): written by the controller alone, read by a node-engine by name, removed a day after " +
-			"no node's latest declaration names it",
+		Bucket: DeclarationFilesBucketOf(node),
+		Description: "the contents of the files over 64 KiB of the declaration of " + node + ", each under the hex of its sha256 " +
+			"digest (novox/hq ADR 0486): written by the controller alone, read by the node-engine of that node by name, removed " +
+			"a day after the node's latest declaration no longer names it",
 		MaxBytes: DeclarationFilesMaxBytes,
 		Storage:  jetstream.FileStorage,
 	}
@@ -25,13 +25,13 @@ import (
 // engine makes it: the object's metadata by a direct get of one subject, then its chunks through an ordered consumer
 // bound to the stream. It is copied here so the grants are judged by the call that has to work, and its subjects are the
 // ones the grant names (issue 1697, notes 4 and 5); the node-engine's own integration test holds the other copy.
-func nodeEnginesFileObject(ctx context.Context, nc *nats.Conn, name string) ([]byte, error) {
+func nodeEnginesFileObject(ctx context.Context, nc *nats.Conn, node, name string) ([]byte, error) {
 	js, err := nc.JetStream(nats.MaxWait(10 * time.Second))
 	if err != nil {
 		return nil, err
 	}
-	stream := "OBJ_" + DeclarationFilesBucket
-	meta, err := js.GetLastMsg(stream, "$O."+DeclarationFilesBucket+".M."+base64.URLEncoding.EncodeToString([]byte(name)),
+	stream, bucket := DeclarationFilesStreamOf(node), DeclarationFilesBucketOf(node)
+	meta, err := js.GetLastMsg(stream, "$O."+bucket+".M."+base64.URLEncoding.EncodeToString([]byte(name)),
 		nats.DirectGet(), nats.Context(ctx))
 	if err != nil {
 		return nil, err
@@ -43,7 +43,7 @@ func nodeEnginesFileObject(ctx context.Context, nc *nats.Conn, name string) ([]b
 	if err := json.Unmarshal(meta.Data, &info); err != nil {
 		return nil, err
 	}
-	sub, err := js.SubscribeSync("$O."+DeclarationFilesBucket+".C."+info.NUID, nats.OrderedConsumer(), nats.BindStream(stream))
+	sub, err := js.SubscribeSync("$O."+bucket+".C."+info.NUID, nats.OrderedConsumer(), nats.BindStream(stream))
 	if err != nil {
 		return nil, err
 	}
@@ -59,16 +59,18 @@ func nodeEnginesFileObject(ctx context.Context, nc *nats.Conn, name string) ([]b
 	return body, nil
 }

-// **The bucket's grants on a real server, judged by the node-engine's own call** (novox/hq ADR 0486 §2, issue 1697,
-// 1697/AC2): the controller's account writes an object, a node-engine's account reads it by its name, a large object
-// included, and is refused a list of the bucket (a stream info asking for every subject), a message get by sequence, a
-// direct get by sequence, a consumer on the names, and a write; another module's account is refused all of them.
-func TestNatsANodeEngineReadsAnObjectByNameAndIsRefusedTheListAndTheWrite(t *testing.T) { // 1697/AC2
+// **The buckets' grants on a real server, judged by the node-engine's own call** (novox/hq ADR 0486 §2, issue 1697, 1697/AC2): the
+// controller's account writes an object into each node's bucket, a node-engine's account reads one of its own bucket by its name,
+// a large object included, and is refused a list of the bucket (a stream info asking for every subject), a message get by
+// sequence, a direct get by sequence, a consumer on the names, and a write; it is refused every one of those and the read by name on
+// another node's bucket, **the literal-`*` forms included** (a published subject `...M.*` or `...C.*` matches the grant's `*`: on
+// its own bucket that reaches its own files, on another's it is refused); another module's account is refused all of them.
+func TestNatsANodeEngineReadsAnObjectOfItsOwnBucketByNameAndNothingElse(t *testing.T) { // 1697/AC2
 	hash, err := bcrypt.GenerateFromPassword([]byte("pw"), bcrypt.MinCost)
 	if err != nil {
 		t.Fatal(err)
 	}
-	users, err := Users(Records{Nodes: []string{"alpha"}, Assigned: map[string][]Declared{
+	users, err := Users(Records{Nodes: []string{"alpha", "beta"}, Assigned: map[string][]Declared{
 		"alpha": {{Module: RuntimeModule}, {Module: "other-module"}},
 	}})
 	if err != nil {
@@ -101,7 +103,7 @@ func TestNatsANodeEngineReadsAnObjectByNameAndIsRefusedTheListAndTheWrite(t *tes
 	}
 	t.Cleanup(s.Shutdown)

-	ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
+	ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
 	defer cancel()
 	connect := func(user string) (*nats.Conn, chan error) {
 		t.Helper()
@@ -115,26 +117,29 @@ func TestNatsANodeEngineReadsAnObjectByNameAndIsRefusedTheListAndTheWrite(t *tes
 		return nc, refused
 	}

-	// The controller makes the bucket (as it does at its start) and writes two objects: a small one and one of
-	// several chunks, which needs the consumer's flow control.
+	// The controller makes the buckets (as it does at its start) and writes into each: for alpha a small object and one of
+	// several chunks, which needs the consumer's flow control; for beta one that holds a marker alpha must never read.
 	controller, _ := connect(ControllerName)
 	cjs, err := jetstream.New(controller)
 	if err != nil {
 		t.Fatal(err)
 	}
-	if err := EnsureDeclarationFiles(ctx, cjs); err != nil {
-		t.Fatalf("the controller cannot make %s: %v", DeclarationFilesBucket, err)
+	if err := EnsureDeclarationFiles(ctx, cjs, "alpha", "beta"); err != nil {
+		t.Fatalf("the controller cannot make the buckets: %v", err)
 	}
-	store, err := cjs.ObjectStore(ctx, DeclarationFilesBucket)
-	if err != nil {
-		t.Fatal(err)
+	if err := EnsureDeclarationFiles(ctx, cjs, "alpha", "beta"); err != nil {
+		t.Fatalf("asserting them again: %v", err)
 	}
-	put := func(content []byte) string {
+	put := func(node string, content []byte) string {
 		t.Helper()
+		store, err := cjs.ObjectStore(ctx, DeclarationFilesBucketOf(node))
+		if err != nil {
+			t.Fatal(err)
+		}
 		sum := sha256.Sum256(content)
 		name := hex.EncodeToString(sum[:])
 		if _, err := store.PutBytes(ctx, name, content); err != nil {
-			t.Fatalf("the controller cannot write the object %s: %v", name[:8], err)
+			t.Fatalf("the controller cannot write the object %s of %s: %v", name[:8], node, err)
 		}
 		return name
 	}
@@ -143,62 +148,71 @@ func TestNatsANodeEngineReadsAnObjectByNameAndIsRefusedTheListAndTheWrite(t *tes
 	for i := range large {
 		large[i] = byte(i*7 + i>>8)
 	}
-	smallName, largeName := put(small), put(large)
+	secret := bytes.Repeat([]byte("BETA-SECRET "), 50)
+	smallName, largeName, betaName := put("alpha", small), put("alpha", large), put("beta", secret)

-	node, nodeRefused := connect("node.alpha")
+	alpha, alphaRefused := connect("node.alpha")
 	for name, want := range map[string][]byte{smallName: small, largeName: large} {
-		got, err := nodeEnginesFileObject(ctx, node, name)
+		got, err := nodeEnginesFileObject(ctx, alpha, "alpha", name)
 		if err != nil {
-			t.Fatalf("a node-engine cannot read the object %s by its name: %v", name[:8], err)
+			t.Fatalf("a node-engine cannot read the object %s of its own bucket by its name: %v", name[:8], err)
 		}
 		if !bytes.Equal(got, want) {
 			t.Fatalf("the node-engine read %d bytes of the object %s, want %d", len(got), name[:8], len(want))
 		}
 	}
+	beta, _ := connect("node.beta")
+	if got, err := nodeEnginesFileObject(ctx, beta, "beta", betaName); err != nil || !bytes.Equal(got, secret) {
+		t.Fatalf("beta's node-engine reads its own object: %v", err)
+	}
 	select {
-	case err := <-nodeRefused:
+	case err := <-alphaRefused:
 		t.Fatalf("a node-engine's own read was refused something: %v", err)
 	default:
 	}

 	// What each account is refused, judged by the server.
+	alphaStream, alphaBucket := DeclarationFilesStreamOf("alpha"), DeclarationFilesBucketOf("alpha")
+	betaStream, betaBucket := DeclarationFilesStreamOf("beta"), DeclarationFilesBucketOf("beta")
 	listing := []byte(`{"subjects_filter":">"}`)
 	bySeq := []byte(`{"seq":1}`)
-	refusals := []struct {
+	type refusal struct {
 		what string
 		do   func(nc *nats.Conn)
-	}{
-		{"a list of the bucket (stream info asking for every subject)", func(nc *nats.Conn) {
-			_, _ = nc.Request("$JS.API.STREAM.INFO."+DeclarationFilesStream, listing, 300*time.Millisecond)
-		}},
-		{"a message get by sequence", func(nc *nats.Conn) {
-			_, _ = nc.Request("$JS.API.STREAM.MSG.GET."+DeclarationFilesStream, bySeq, 300*time.Millisecond)
-		}},
-		{"a direct get by sequence", func(nc *nats.Conn) {
-			_, _ = nc.Request("$JS.API.DIRECT.GET."+DeclarationFilesStream, bySeq, 300*time.Millisecond)
-		}},
-		{"a consumer on every object's name", func(nc *nats.Conn) {
-			js, _ := nc.JetStream(nats.MaxWait(500 * time.Millisecond))
-			sub, err := js.SubscribeSync("$O."+DeclarationFilesBucket+".M.>", nats.OrderedConsumer(), nats.BindStream(DeclarationFilesStream))
-			if err == nil {
-				sub.Unsubscribe()
-			}
-		}},
-		{"a write of an object's metadata", func(nc *nats.Conn) {
-			_ = nc.Publish("$O."+DeclarationFilesBucket+".M.abc", []byte(`{}`))
-			_ = nc.Flush()
-		}},
-		{"a write of a chunk", func(nc *nats.Conn) {
-			_ = nc.Publish("$O."+DeclarationFilesBucket+".C.abc", []byte("x"))
-			_ = nc.Flush()
-		}},
 	}
-	refusedWith := func(who string, nc *nats.Conn, ch chan error, what string, do func(*nats.Conn)) {
+	onBucket := func(stream, bucket, which string) []refusal {
+		return []refusal{
+			{which + ": a list of the bucket (stream info asking for every subject)", func(nc *nats.Conn) {
+				_, _ = nc.Request("$JS.API.STREAM.INFO."+stream, listing, 300*time.Millisecond)
+			}},
+			{which + ": a message get by sequence", func(nc *nats.Conn) {
+				_, _ = nc.Request("$JS.API.STREAM.MSG.GET."+stream, bySeq, 300*time.Millisecond)
+			}},
+			{which + ": a direct get by sequence", func(nc *nats.Conn) {
+				_, _ = nc.Request("$JS.API.DIRECT.GET."+stream, bySeq, 300*time.Millisecond)
+			}},
+			{which + ": a consumer on every object's name", func(nc *nats.Conn) {
+				js, _ := nc.JetStream(nats.MaxWait(500 * time.Millisecond))
+				if sub, err := js.SubscribeSync("$O."+bucket+".M.>", nats.OrderedConsumer(), nats.BindStream(stream)); err == nil {
+					sub.Unsubscribe()
+				}
+			}},
+			{which + ": a write of an object's metadata", func(nc *nats.Conn) {
+				_ = nc.Publish("$O."+bucket+".M.abc", []byte(`{}`))
+				_ = nc.Flush()
+			}},
+			{which + ": a write of a chunk", func(nc *nats.Conn) {
+				_ = nc.Publish("$O."+bucket+".C.abc", []byte("x"))
+				_ = nc.Flush()
+			}},
+		}
+	}
+	refusedWith := func(who string, nc *nats.Conn, ch chan error, r refusal) {
 		t.Helper()
 		for len(ch) > 0 {
 			<-ch
 		}
-		do(nc)
+		r.do(nc)
 		_ = nc.Flush()
 		deadline := time.After(3 * time.Second)
 		for {
@@ -208,32 +222,70 @@ func TestNatsANodeEngineReadsAnObjectByNameAndIsRefusedTheListAndTheWrite(t *tes
 					return
 				}
 			case <-deadline:
-				t.Errorf("the server did not refuse %s %s", who, what)
+				t.Errorf("the server did not refuse %s %s", who, r.what)
 				return
 			}
 		}
 	}
-	for _, r := range refusals {
+	// Alpha's account on its own bucket (list, sequence reads, names, writes) and on beta's (all of that, and the read by name,
+	// and the literal-`*` probes of the review).
+	reachBeta := []refusal{
+		{"beta's bucket: a read of one object by its name", func(nc *nats.Conn) {
+			_, _ = nc.Request("$JS.API.DIRECT.GET."+betaStream+".$O."+betaBucket+".M."+
+				base64.URLEncoding.EncodeToString([]byte(betaName)), nil, 300*time.Millisecond)
+		}},
+		{"beta's bucket: a direct get with a literal `*` for the name", func(nc *nats.Conn) {
+			_, _ = nc.Request("$JS.API.DIRECT.GET."+betaStream+".$O."+betaBucket+".M.*", nil, 300*time.Millisecond)
+		}},
+		{"beta's bucket: a consumer with a literal `*` for the chunks", func(nc *nats.Conn) {
+			js, _ := nc.JetStream(nats.MaxWait(500 * time.Millisecond))
+			if sub, err := js.SubscribeSync("$O."+betaBucket+".C.*", nats.OrderedConsumer(), nats.BindStream(betaStream)); err == nil {
+				sub.Unsubscribe()
+			}
+		}},
+	}
+	all := append(append(onBucket(alphaStream, alphaBucket, "its own bucket"), onBucket(betaStream, betaBucket, "beta's bucket")...), reachBeta...)
+	for _, r := range all {
 		nodeNC, nodeCh := connect("node.alpha")
-		refusedWith("a node-engine", nodeNC, nodeCh, r.what, r.do)
+		refusedWith("a node-engine", nodeNC, nodeCh, r)
+	}
+	// Another module is refused all of that on both buckets, and the reads by name on both.
+	for _, r := range append(append(onBucket(alphaStream, alphaBucket, "alpha's bucket"), reachBeta...),
+		refusal{"alpha's bucket: a read of one object by its name", func(nc *nats.Conn) {
+			_, _ = nc.Request("$JS.API.DIRECT.GET."+alphaStream+".$O."+alphaBucket+".M."+
+				base64.URLEncoding.EncodeToString([]byte(smallName)), nil, 300*time.Millisecond)
+		}}) {
 		otherNC, otherCh := connect("alpha.other-module")
-		refusedWith("another module", otherNC, otherCh, r.what, r.do)
+		refusedWith("another module", otherNC, otherCh, r)
 	}
-	// And another module cannot read an object by name either.
-	other, otherCh := connect("alpha.other-module")
-	refusedWith("another module", other, otherCh, "a read of one object by its name", func(nc *nats.Conn) {
-		_, _ = nc.Request("$JS.API.DIRECT.GET."+DeclarationFilesStream+".$O."+DeclarationFilesBucket+".M."+
-			base64.URLEncoding.EncodeToString([]byte(smallName)), nil, 300*time.Millisecond)
-	})

-	// Nothing was written by the refused: the bucket holds the two objects.
-	var names int
-	lister, err := store.List(ctx)
+	// The literal `*` on alpha's own bucket reaches alpha's own files and never beta's: the chunks it is given hold no marker of beta's.
+	js, err := alpha.JetStream(nats.MaxWait(2 * time.Second))
 	if err != nil {
 		t.Fatal(err)
 	}
-	names = len(lister)
-	if names != 2 {
-		t.Errorf("the bucket holds %d objects, want the 2 the controller wrote", names)
+	if sub, err := js.SubscribeSync("$O."+alphaBucket+".C.*", nats.OrderedConsumer(), nats.BindStream(alphaStream)); err == nil {
+		for i := 0; i < 20; i++ {
+			msg, err := sub.NextMsg(300 * time.Millisecond)
+			if err != nil {
+				break
+			}
+			if bytes.Contains(msg.Data, []byte("BETA-SECRET")) {
+				t.Fatal("a literal `*` on alpha's bucket gave it a chunk of beta's file")
+			}
+		}
+		sub.Unsubscribe()
+	}
+
+	// Nothing was written by the refused: each bucket holds what the controller wrote.
+	for node, want := range map[string]int{"alpha": 2, "beta": 1} {
+		store, err := cjs.ObjectStore(ctx, DeclarationFilesBucketOf(node))
+		if err != nil {
+			t.Fatal(err)
+		}
+		infos, err := store.List(ctx)
+		if err != nil || len(infos) != want {
+			t.Errorf("the bucket of %s holds %d objects (%v), want the %d the controller wrote", node, len(infos), err, want)
+		}
 	}
 }
@@ -6,60 +6,72 @@ import (
 	"testing"
 )

-// 1697/AC1: the composed accounts give a node-engine exactly the four reads of one object by its name, the controller the
-// one write, and no other principal anything of the bucket.
-func TestTheDeclarationBucketIsWrittenByTheControllerAndReadByNameByANodeEngine(t *testing.T) { // 1697/AC1
+// 1697/AC1: the composed accounts give a node-engine exactly the four reads of one object by its name, on its own node's bucket and
+// no other's; the controller the write of every node's bucket; and no other principal anything of any bucket.
+func TestEachNodesBucketIsWrittenByTheControllerAndReadByNameByThatNodesEngine(t *testing.T) { // 1697/AC1
 	node, err := PermissionsFor(Principal{Kind: KindNode, Node: "alpha", PasswordHash: "x"})
 	if err != nil {
 		t.Fatal(err)
 	}
 	var mine []string
 	for _, s := range node.Publish {
-		if strings.Contains(s, DeclarationFilesStream) || strings.Contains(s, "$O."+DeclarationFilesBucket) {
+		if strings.Contains(s, "declaration-files") {
 			mine = append(mine, s)
 		}
 	}
 	want := []string{
-		"$JS.API.CONSUMER.CREATE.OBJ_declaration-files.*.$O.declaration-files.C.*",
-		"$JS.API.CONSUMER.DELETE.OBJ_declaration-files.*",
-		"$JS.API.DIRECT.GET.OBJ_declaration-files.$O.declaration-files.M.*",
-		"$JS.FC.OBJ_declaration-files.>",
+		"$JS.API.CONSUMER.CREATE.OBJ_declaration-files-alpha.*.$O.declaration-files-alpha.C.*",
+		"$JS.API.CONSUMER.DELETE.OBJ_declaration-files-alpha.*",
+		"$JS.API.DIRECT.GET.OBJ_declaration-files-alpha.$O.declaration-files-alpha.M.*",
+		"$JS.FC.OBJ_declaration-files-alpha.>",
 	}
 	if !slices.Equal(mine, want) {
-		t.Fatalf("a node-engine may publish %q of the bucket, want exactly %q", mine, want)
+		t.Fatalf("a node-engine may publish %q of the buckets, want exactly %q", mine, want)
 	}
-	// What would make a list or a write: none of it, in any pattern a node holds. (The real server's judgement of
-	// the wildcard forms is the live test's.)
+	// What would make a list, a write, or a read of another node's files: none of it, in any pattern a node holds. (The real
+	// server's judgement of the wildcard forms is the live test's.)
 	for _, refused := range []string{
-		"$JS.API.STREAM.INFO.OBJ_declaration-files",
-		"$JS.API.STREAM.MSG.GET.OBJ_declaration-files",
-		"$JS.API.DIRECT.GET.OBJ_declaration-files",
-		"$JS.API.DIRECT.GET.OBJ_declaration-files.$O.declaration-files.M.abc.def",
-		"$JS.API.CONSUMER.CREATE.OBJ_declaration-files.x.$O.declaration-files.M.YWJj",
-		"$JS.API.CONSUMER.CREATE.OBJ_declaration-files.x.$O.declaration-files.M.YWJj.more",
-		"$JS.API.CONSUMER.CREATE.OBJ_declaration-files",
-		"$O.declaration-files.M.abc",
-		"$O.declaration-files.C.abc",
+		"$JS.API.STREAM.INFO.OBJ_declaration-files-alpha",
+		"$JS.API.STREAM.MSG.GET.OBJ_declaration-files-alpha",
+		"$JS.API.DIRECT.GET.OBJ_declaration-files-alpha",
+		"$JS.API.DIRECT.GET.OBJ_declaration-files-alpha.$O.declaration-files-alpha.M.abc.def",
+		"$JS.API.CONSUMER.CREATE.OBJ_declaration-files-alpha.x.$O.declaration-files-alpha.M.YWJj",
+		"$JS.API.CONSUMER.CREATE.OBJ_declaration-files-alpha.x.$O.declaration-files-alpha.M.YWJj.more",
+		"$JS.API.CONSUMER.CREATE.OBJ_declaration-files-alpha",
+		"$O.declaration-files-alpha.M.abc",
+		"$O.declaration-files-alpha.C.abc",
+		// another node's bucket, however it is named
+		"$JS.API.DIRECT.GET.OBJ_declaration-files-beta.$O.declaration-files-beta.M.YWJj",
+		"$JS.API.CONSUMER.CREATE.OBJ_declaration-files-beta.h4sh.$O.declaration-files-beta.C.NUID",
+		"$JS.API.CONSUMER.DELETE.OBJ_declaration-files-beta.h4sh",
+		"$JS.FC.OBJ_declaration-files-beta.x",
+		// and the old shared bucket
+		"$JS.API.DIRECT.GET.OBJ_declaration-files.$O.declaration-files.M.YWJj",
 	} {
 		if MayPublish(node, refused) {
-			t.Errorf("a node-engine may publish %s, which lists or writes the bucket", refused)
+			t.Errorf("a node-engine may publish %s, which lists or writes a bucket or reads another node's", refused)
 		}
 	}
 	for _, kept := range []string{
-		"$JS.API.DIRECT.GET.OBJ_declaration-files.$O.declaration-files.M.YWJj",
-		"$JS.API.CONSUMER.CREATE.OBJ_declaration-files.h4sh.$O.declaration-files.C.NUID",
+		"$JS.API.DIRECT.GET.OBJ_declaration-files-alpha.$O.declaration-files-alpha.M.YWJj",
+		"$JS.API.CONSUMER.CREATE.OBJ_declaration-files-alpha.h4sh.$O.declaration-files-alpha.C.NUID",
 	} {
 		if !MayPublish(node, kept) {
-			t.Errorf("a node-engine may not publish %s, which reads one object by its name", kept)
+			t.Errorf("a node-engine may not publish %s, which reads one object of its own bucket by its name", kept)
 		}
 	}

-	controller, err := PermissionsFor(Principal{Kind: KindController, PasswordHash: "x"})
+	controller, err := PermissionsFor(Principal{Kind: KindController, PasswordHash: "x", FilesFor: []string{"alpha", "beta"}})
 	if err != nil {
 		t.Fatal(err)
 	}
-	if !MayPublish(controller, "$O.declaration-files.M.abc") || !MayPublish(controller, "$O.declaration-files.C.nuid") {
-		t.Error("the controller may not write the bucket")
+	for _, s := range []string{"$O.declaration-files-alpha.M.abc", "$O.declaration-files-beta.C.nuid"} {
+		if !MayPublish(controller, s) {
+			t.Errorf("the controller may not write %s", s)
+		}
+	}
+	if MayPublish(controller, "$O.declaration-files-gamma.M.abc") {
+		t.Error("the controller may write the bucket of a node it was not given")
 	}

 	for _, p := range []Principal{
@@ -78,3 +90,20 @@ func TestTheDeclarationBucketIsWrittenByTheControllerAndReadByNameByANodeEngine(
 		}
 	}
 }
+
+// The users the controller composes carry each node's bucket to the controller and to that node alone.
+func TestTheComposedUsersGiveEachNodeItsOwnBucketAndTheControllerAll(t *testing.T) {
+	users, err := Users(Records{Nodes: []string{"alpha", "beta"}})
+	if err != nil {
+		t.Fatal(err)
+	}
+	var controller Principal
+	for _, u := range users {
+		if u.Kind == KindController {
+			controller = u
+		}
+	}
+	if !slices.Equal(controller.FilesFor, []string{"alpha", "beta"}) {
+		t.Fatalf("the controller writes the buckets of %v", controller.FilesFor)
+	}
+}
@@ -104,6 +104,10 @@ type Principal struct {
 	// Username and inbox are <node>.<module>.<process id>.
 	Process string

+	// FilesFor are the nodes whose declaration-file buckets this principal writes, for the controller alone (novox/hq ADR 0486 §2):
+	// every node of the mesh, one bucket each.
+	FilesFor []string
+
 	Emits    []string
 	Consumes []string
 	Serves   []string
@@ -392,12 +396,6 @@ var DegradableGrants = []DegradableGrant{
 		Reason: "carried only at genesis: the controller composes its own permissions and pushes them to the bus at " +
 			"its first push, which is when it acknowledges a merge (ADR 0407 decision 3, issue 973)",
 	},
-	{
-		Subject: DeclarationFilesWrites,
-		Reason: "carried only at genesis: the controller composes its own permissions and pushes them to the bus at its " +
-			"first push, and puts a declaration's large files in the bucket only for a node that reports it reads them, " +
-			"after that (ADR 0486 §1, §2, issue 1697)",
-	},
 	{
 		Subject: "$KV." + MergeWindowBucket + ".>",
 		Reason: "refused: the controller closes a batch's window by a timer of its own for that batch, as the " +
@@ -744,7 +742,9 @@ func PermissionsFor(p Principal) (Permissions, error) {
 		}
 		// **And the object bucket a declaration's large files travel in** (novox/hq ADR 0486 §2, issue 1697): the
 		// controller alone writes it, as a publish to the bucket's own subjects, which `$JS.API.>` does not cover.
-		pub = append(pub, DeclarationFilesWrites)
+		for _, node := range p.FilesFor {
+			pub = append(pub, DeclarationFilesWrites(node))
+		}
 		// **And what the bus says about itself, read-only** (novox/hq to-be 45 §3, S9): a durable
 		// consumer that gave up on a message, or one that was deleted. The server already publishes
 		// both in the mesh's own account; the controller says each as a condition in the mesh's words.
@@ -846,7 +846,7 @@ func PermissionsFor(p Principal) (Permissions, error) {
 		pub = append(pub, haveKeys...)
 		// And an object of the declaration bucket by its name, and nothing of the bucket's list or writing
 		// (novox/hq ADR 0486 §2, issue 1697).
-		pub = append(pub, DeclarationFilesReads()...)
+		pub = append(pub, DeclarationFilesReads(p.Node)...)
 		// And, while the vault serves it, the vault's move-in in its own name and nobody else's (issue 676): the
 		// values sealed to this machine, which only it opens, handed in sealed to the vault's key for the call.
 		if p.MovesIntoTheVault {
@@ -29,7 +29,7 @@ accounts {
    ]
    users = [
      { user: "controller", password: "$2a$11$cccccccccccccccccccccc", permissions: {
-        publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.DEAD_LETTER_NOTICES.controller.>", "$JS.ACK.EVENTS.controller-merges.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>", "$KV.SEAT_NODE_BUILD_AGENT_cancelled.>", "$KV.mesh-controller_asked.>", "$KV.mesh-controller_assignments.>", "$KV.mesh-controller_builds.>", "$KV.mesh-controller_calls.>", "$KV.mesh-controller_checks-running.>", "$KV.mesh-controller_condition-history.>", "$KV.mesh-controller_conditions.>", "$KV.mesh-controller_declared-config.>", "$KV.mesh-controller_delivery-merges.>", "$KV.mesh-controller_hand-acts.>", "$KV.mesh-controller_lease.>", "$KV.mesh-controller_merge-window.>", "$KV.mesh-controller_predefined.>", "$KV.mesh-controller_rollout-silent.>", "$KV.mesh-controller_rollouts.>", "$O.declaration-files.>", "$SRV.INFO", "_INBOX.enrol.>", "mesh.again.>", "mesh.assignment.>", "mesh.events.dead.>", "mesh.mod.*.tool.>", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>", "mesh.seat.mesh-build-machine.tool.>", "mesh.seat.mesh-controller.event.applied", "mesh.seat.mesh-controller.event.assignment-changed", "mesh.seat.mesh-controller.event.built-before", "mesh.seat.mesh-controller.event.checked", "mesh.seat.mesh-controller.event.condition-changed", "mesh.seat.mesh-controller.event.condition-cleared", "mesh.seat.mesh-controller.event.condition-raised", "mesh.seat.mesh-controller.event.declared-config-changed", "mesh.seat.mesh-controller.event.doctor-heartbeat", "mesh.seat.mesh-controller.event.healer-acted", "mesh.seat.mesh-controller.event.plan-moved", "mesh.seat.mesh-controller.event.predefined-changed", "mesh.seat.mesh-controller.event.refused", "mesh.seat.mesh-controller.event.rolled-back", "mesh.seat.mesh-controller.event.secret-replaced", "mesh.seat.mesh-controller.event.sent", "mesh.seat.mesh-controller.tool.plans", "mesh.seat.mesh-delivery.tool.close", "mesh.seat.mesh-delivery.tool.groups", "mesh.seat.mesh-delivery.tool.release", "mesh.seat.mesh-delivery.tool.stalled", "mesh.seat.mesh-delivery.tool.stop", "mesh.seat.mesh-delivery.tool.times", "mesh.seat.node-backup.tool.backed-up.*", "mesh.seat.node-backup.tool.now.*", "mesh.seat.node-build-agent.accept.>", "mesh.seat.node-build-agent.check.>", "mesh.seat.node-build-agent.tool.>", "mesh.seat.node-intrusion-prevention.tool.banned.*", "mesh.seat.node-launcher.tool.secret.*", "mesh.seat.node-service-manager.tool.restart.*", "mesh.seat.operator-channel.tool.tell"] }
+        publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.DEAD_LETTER_NOTICES.controller.>", "$JS.ACK.EVENTS.controller-merges.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>", "$KV.SEAT_NODE_BUILD_AGENT_cancelled.>", "$KV.mesh-controller_asked.>", "$KV.mesh-controller_assignments.>", "$KV.mesh-controller_builds.>", "$KV.mesh-controller_calls.>", "$KV.mesh-controller_checks-running.>", "$KV.mesh-controller_condition-history.>", "$KV.mesh-controller_conditions.>", "$KV.mesh-controller_declared-config.>", "$KV.mesh-controller_delivery-merges.>", "$KV.mesh-controller_hand-acts.>", "$KV.mesh-controller_lease.>", "$KV.mesh-controller_merge-window.>", "$KV.mesh-controller_predefined.>", "$KV.mesh-controller_rollout-silent.>", "$KV.mesh-controller_rollouts.>", "$SRV.INFO", "_INBOX.enrol.>", "mesh.again.>", "mesh.assignment.>", "mesh.events.dead.>", "mesh.mod.*.tool.>", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>", "mesh.seat.mesh-build-machine.tool.>", "mesh.seat.mesh-controller.event.applied", "mesh.seat.mesh-controller.event.assignment-changed", "mesh.seat.mesh-controller.event.built-before", "mesh.seat.mesh-controller.event.checked", "mesh.seat.mesh-controller.event.condition-changed", "mesh.seat.mesh-controller.event.condition-cleared", "mesh.seat.mesh-controller.event.condition-raised", "mesh.seat.mesh-controller.event.declared-config-changed", "mesh.seat.mesh-controller.event.doctor-heartbeat", "mesh.seat.mesh-controller.event.healer-acted", "mesh.seat.mesh-controller.event.plan-moved", "mesh.seat.mesh-controller.event.predefined-changed", "mesh.seat.mesh-controller.event.refused", "mesh.seat.mesh-controller.event.rolled-back", "mesh.seat.mesh-controller.event.secret-replaced", "mesh.seat.mesh-controller.event.sent", "mesh.seat.mesh-controller.tool.plans", "mesh.seat.mesh-delivery.tool.close", "mesh.seat.mesh-delivery.tool.groups", "mesh.seat.mesh-delivery.tool.release", "mesh.seat.mesh-delivery.tool.stalled", "mesh.seat.mesh-delivery.tool.stop", "mesh.seat.mesh-delivery.tool.times", "mesh.seat.node-backup.tool.backed-up.*", "mesh.seat.node-backup.tool.now.*", "mesh.seat.node-build-agent.accept.>", "mesh.seat.node-build-agent.check.>", "mesh.seat.node-build-agent.tool.>", "mesh.seat.node-intrusion-prevention.tool.banned.*", "mesh.seat.node-launcher.tool.secret.*", "mesh.seat.node-service-manager.tool.restart.*", "mesh.seat.operator-channel.tool.tell"] }
        subscribe: { allow: ["$JS.API.>", "$JS.EVENT.ADVISORY.CONSUMER.DELETED.>", "$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>", "$SRV.INFO", "$SRV.INFO.mesh-controller", "$SRV.INFO.mesh-controller.>", "$SRV.PING", "$SRV.PING.mesh-controller", "$SRV.PING.mesh-controller.>", "$SRV.STATS", "$SRV.STATS.mesh-controller", "$SRV.STATS.mesh-controller.>", "_DELIVER.controller", "_DELIVER.controller.>", "_INBOX.controller.>", "mesh.control.>", "mesh.mod.*.event.provisioner.failing", "mesh.mod.*.event.provisioner.recovered", "mesh.mod.*.event.provisioner.retirement", "mesh.mod.gitea.event.pull.merged", "mesh.mod.gitea.event.pull.updated", "mesh.mod.mesh-catalog.event.catching-up", "mesh.mod.mesh-catalog.event.upgraded", "mesh.seat.mesh-build-machine.event.built", "mesh.seat.mesh-controller.tool.>", "mesh.seat.node-build-agent.event.built", "mesh.seat.operator-channel.event.decided.mesh-controller"] }
        allow_responses: { max: 1, ttl: "1m" }
      } }
@@ -38,7 +38,7 @@ accounts {
        subscribe: { allow: ["_INBOX.enrol.one.>"] }
      } }
      { user: "node.one", password: "$2a$11$nnnnnnnnnnnnnnnnnnnnnn", permissions: {
-        publish: { allow: ["$JS.ACK.NODES.one.>", "$JS.API.CONSUMER.CREATE.KV_node-engine_have.*.$KV.node-engine_have.one.>", "$JS.API.CONSUMER.CREATE.OBJ_declaration-files.*.$O.declaration-files.C.*", "$JS.API.CONSUMER.DELETE.KV_node-engine_have.>", "$JS.API.CONSUMER.DELETE.OBJ_declaration-files.*", "$JS.API.CONSUMER.INFO.NODES.one", "$JS.API.DIRECT.GET.KV_node-engine_have.$KV.node-engine_have.one.>", "$JS.API.DIRECT.GET.OBJ_declaration-files.$O.declaration-files.M.*", "$JS.API.STREAM.INFO.KV_node-engine_have", "$JS.FC.KV_node-engine_have.>", "$JS.FC.OBJ_declaration-files.>", "$KV.node-engine_have.one.>", "$SRV.PING.node-tools.one", "mesh.control.one.>"] }
+        publish: { allow: ["$JS.ACK.NODES.one.>", "$JS.API.CONSUMER.CREATE.KV_node-engine_have.*.$KV.node-engine_have.one.>", "$JS.API.CONSUMER.CREATE.OBJ_declaration-files-one.*.$O.declaration-files-one.C.*", "$JS.API.CONSUMER.DELETE.KV_node-engine_have.>", "$JS.API.CONSUMER.DELETE.OBJ_declaration-files-one.*", "$JS.API.CONSUMER.INFO.NODES.one", "$JS.API.DIRECT.GET.KV_node-engine_have.$KV.node-engine_have.one.>", "$JS.API.DIRECT.GET.OBJ_declaration-files-one.$O.declaration-files-one.M.*", "$JS.API.STREAM.INFO.KV_node-engine_have", "$JS.FC.KV_node-engine_have.>", "$JS.FC.OBJ_declaration-files-one.>", "$KV.node-engine_have.one.>", "$SRV.PING.node-tools.one", "mesh.control.one.>"] }
        subscribe: { allow: ["_DELIVER.one", "_DELIVER.one.>", "_INBOX.node.one.>", "mesh.node.one.ask.hand-over", "mesh.node.one.ask.judge-directories", "mesh.node.one.ask.move-secret", "mesh.node.one.ask.report", "mesh.node.one.ask.setuid-search", "mesh.node.one.declare"] }
        allow_responses: { max: 1, ttl: "1m" }
      } }
@@ -135,7 +135,7 @@ type Records struct {
 // correct.
 func Users(r Records) ([]Principal, error) {
 	out := []Principal{{Kind: KindController, Uses: asksSeatOf(r), AsksIssued: issuedOf(r), AsksTheVault: vaultVerbsOf(r),
-		AsksTheConfig: servedOf(r, VerbsTheControllerAsksTheConfig)}}
+		AsksTheConfig: servedOf(r, VerbsTheControllerAsksTheConfig), FilesFor: sortedCopy(r.Nodes)}}

 	for _, node := range sortedCopy(r.Nodes) {
 		witness := false
@@ -222,10 +222,10 @@ var WritersTable = []WriterRow{
 	// The contents of a declaration's files over 64 KiB (novox/hq ADR 0486 §2): the controller alone writes the bucket, each
 	// object under the digest of its content; a node-engine reads one by its name, with no list and no write.
 	{State: "a declaration's large files", Writer: "controller, each object under the sha256 digest of its content",
-		KeptIn: "the object bucket " + DeclarationFilesBucket + " (stream " + DeclarationFilesStream + "), an object removed a " +
-			"day after no node's latest declaration names it",
-		Others:   "a node-engine reads an object by its name and may neither list the bucket nor write it",
-		Subjects: []string{DeclarationFilesWrites}, Writes: isController},
+		KeptIn: "the object bucket " + DeclarationFilesBucket + "-<node> of each node, an object removed a day after the " +
+			"node's latest declaration no longer names it",
+		Others:   "a node-engine reads an object of its own bucket by its name and may neither list the bucket nor write it",
+		Subjects: []string{"$O.*.>"}, Writes: isController},
 	{State: "the declared-config mirror", Writer: "controller (lease holder), from its store, only on a change",
 		KeptIn: "key-value " + DeclaredConfigBucket, Others: "read through `reads` as mesh-controller." + MirrorDeclaredConfig,
 		Subjects: kvOf(DeclaredConfigBucket), Writes: isController},