Compare commits

..
37 Commits
Author SHA1 Message Date
mesh-tools 32e1ed6c55 Merge pull request 'The Node build module's integration tests supply the SDK the skipped fetch would place (design 47 G §571)' (#732) from fix/47-H1-build-node-tests-under-load into main 2026-11-21 21:44:31 +00:00
mesh-tools d7b23f50b4 Merge pull request 'Packet filter control verbs: the nftables holder serves the verbs, keeps allowances in its state and arms expiries (ADR 0683)' (#731) from feat/packet-filter-allow into main 2026-11-21 21:36:38 +00:00
jschoubben 6527ebd077 mesh-build-node: the real-toolchain tests' fixture holds the SDK where the fetch places it
Since the toolchain lock pins no SDK (design 47 G §571), only the fetch puts
@novox/mesh-sdk in a module's node_modules. The three tests that build with
the real tsc and esbuild skip the fetch, so they read the SDK from an older
installed toolchain that still carried one. Once a build node installed the
current lock (shanks 19:26, then g14), they failed with TS2307 (main.137,
main.155). The fixture now lays a minimal SDK of its own where the fetch
would, so the tests no longer depend on which toolchains a node still keeps.

Builds: design 47 G §571
2026-11-21 22:25:38 +01:00
jschoubben 31abcad907 nftables: test that a start waits for the watch's expiry; refuse a stale node-engine package
Review round 2 (1): the start's wait for the expiries the watch handed over
had no test that failed without it; the start test gains a store whose
deletes wait, and without the wait the key is said expired twice. The note:
a node-engine package naming no verb converged refuses the start loudly
instead of asking seat:node-engine.undefined.

Builds: design 33 control-verbs §10, §11, §14; design 47 H H-A-312
2026-11-21 22:05:47 +01:00
jschoubben f589b5af5c nftables: hold the verbs until the start ends; until with seconds; the seat's own forms
Review round 1 (1)-(3): an until without seconds was accepted, which the
node-engine's RFC 3339 read refuses, withholding every later compose; the
verbs answered before started was said and before the watch's expiries
ended; and the holder kept its own copies of the seat's names, key forms
and value forms, which had drifted from H-A-324. The holder now takes them
from the seat's generated package, the tools' inputs are checked against
its input types, and every seat verb waits for the start's end.

Builds: design 33 control-verbs §2, §3, §11, §12, §14; design 47 H H-A-324, H-A-330
2026-11-21 21:42:45 +01:00
jschoubben 38ea9ce25c nftables: allow names the outward link a kept allowance opens
An allowance with no interface never opens an outward link; one named in
interfaces does, and allow now says so: a kept allowance whose interface is
in the outward links of the converged answer asked for the call answers
with outward, that link. The holder reads no shape of the loaded table to
tell an outward link from the tunnel; the event carries no outward.

Builds: design 33 control-verbs §4, §14; design 47 H H-A-324, H-A-325
2026-11-21 21:19:03 +01:00
jschoubben 09d5758343 nftables: ask converged through the node-engine seat's generated package
The holder's ask of the node-engine's converged takes the seat's name, the
verb's name and the answer's form from the node-engine seat's TypeScript
package, as the holder says its own event through its seat's package.

Builds: design 33 control-verbs §10, §12; design 47 H H-A-316, H-A-321
2026-11-21 21:04:16 +01:00
jschoubben 83482a4513 nftables: allowances per machine, refused unproven or adopted, written in the hand-act log
The security and seams reviews of the packet filter's control verbs: the
allowances state is one bucket per machine whose keys carry the node, so a
holder lists and writes its own node's only; allow and disallow refuse an
unproven caller, an adopted node and a converged that does not answer within
10 s, and are written in the hand-act log before the state is; allow refuses a
port the loaded mesh table accepts from given sources only unless a source is
given, and more than the setting allowances-max; allowed answers none on an
adopted node and asks converged again while none answered; rules marks loaded
only on an equal at and lists loaded allowance comments nobody keeps. The event
is said through the seat's generated package; the claim says it, so emits
stays without it.

Builds: design 33 control-verbs §1-§11, §14; ADR 0683; design 47 H H-A-314 (3), (5), (6), H-A-315 (1)-(3), (5), (7), H-A-316, H-A-317, H-A-318, H-A-319
2026-11-21 20:52:34 +01:00
jschoubben 7bd1e74086 nftables: serve allow, disallow and allowed, keeping allowances as its own state
The operator, 2026-11-21 20:35: a firewall module has tools to control the
firewall beside the provisioning from declarations. The holder keeps one
allowance per protocol, port and interface in its state allowances, says every
change as allowances-changed, removes an allowance at its until by a timer on
its own state (and every expired one at its start, before started), and rules
marks each allowance loaded by its comment in the mesh's table. It writes no
rule: the node-engine renders the allowances.

Builds: ADR 0683; design 33 control-verbs §1-§7, §9; design 47 H H-A-304..310, H-A-312 (2)
2026-11-21 20:40:35 +01:00
mesh-tools 553bfb0ccf Merge pull request 'The forge's whole-seat test ends NOT RUN when its npm tooling is missing (H-A-294)' (#730) from fix/47-H1-gitea-whole-test-not-run into main 2026-11-21 18:30:10 +00:00
mesh-tools 3c27ea7a11 Merge pull request 'The zfs keeper test's race, and a non-root peer whose write is cut is told only root (A-37)' (#729) from fix/47-H1-zfs-keeper-test into main 2026-11-21 18:29:59 +00:00
jschoubben e2571a6468 gitea: the seat git's whole test says NOT RUN where node, npm or its npm packages are missing
The Go integration level fetches the forge module's tree without node_modules and
without network, so TestTheSeatGitWholeOverARealBus failed every pipeline carrying
the forge. It now checks its needs first and skips by name, as the Node build
module's realMachine does; with its needs present it runs whole as before.

Issue 1960 (H1).
Builds: design 47 H H-A-294
2026-11-21 19:26:31 +01:00
jschoubben 94bcc14a82 zfs: the keeper's socket test no longer races, and a refused peer is told so
The test switched the peer's uid in a plain variable the keeper's serving
goroutines read, a data race that fails it under -race; it is an atomic now.
A peer the keeper refuses could also get "broken pipe" instead of the
refusal, when the keeper closed before the request was written: the client
now reads the refusal that waits on the socket. A request larger than the
socket's buffer makes that case happen every time.

Builds: A-37 (only root opens the keeper's socket; a non-root peer is told so); design 47 H section 10 rows 165 and 167 (hq branch design/47-H1-answers-59, 81fa1ee4)
2026-11-21 19:26:14 +01:00
mesh-tools 461868b9fe Merge pull request 'Design 47 H1: a trunk build places every mesh package it names; the bus module's whole test waits for a fill pass (H-A-267)' (#728) from fix/47-H1-catalog-trunk-packages-and-nats-test into main 2026-11-21 14:49:34 +00:00
jschoubben 4fb3ab6df5 mesh-build-node: say a placed release outside the package file's range (H-A-276)
The latest release is placed whatever the range a package.json names
(ADR 0573 §2); one outside that range is now said in the journal, for
the SDK and every other mesh package, so a module built against a
release its range excludes is visible. A range of a form not read is
said as such.

Builds: design 47 H H-A-267, H-A-276
2026-11-21 15:47:46 +01:00
jschoubben a5976cf299 mesh-build-node: hold the member's-build precedence over the registry in a test (H-A-267)
A fetch naming a member's build of a mesh package the registry lacks
lays the build and waits on nothing; without the skip in
placeMeshPackages it waits for the registry's release.

Builds: design 47 H H-A-267
2026-11-21 15:45:10 +01:00
jschoubben 820840076f nats: the whole test waits for the first fill pass, not a first view's answer (H-A-268)
The keeper never marks a first view listing no module as answered
(A-367), so the subtest waited for what cannot come and hid every later
subtest. It waits for the first fill pass and takes it as the first.

Builds: design 47 H H-A-268
2026-11-21 15:37:46 +01:00
jschoubben 83802dc4a6 mesh-build-node: place every mesh package a module names at its latest release (H-A-267)
A trunk build put only the SDK into a module's node_modules; any other
@novox/mesh-* package its package.json names was left out of the install
and only a group or a member's build brought it, so the forge module's
build failed with "Cannot find module" for its contracts package. Each
such package is now placed at the latest release the registry holds, as
the SDK is; a group's packed package or a member's build still takes
precedence.

Builds: design 47 H H-A-267
2026-11-21 15:35:26 +01:00
mesh-tools efc8c52e68 Merge pull request 'Design 47 H1: the artifact sweep runs one pending pass from a followed view of the builds (H-A-250)' (#727) from fix/47-H1-artifact-sweep-one-pass into main 2026-11-21 11:08:08 +00:00
jschoubben 72872b045a Design 47 H1: a build's end comes from the builder's log; an unread end or view holds the pass loudly (H-A-258, H-A-260)
A done build whose history read failed was left out of the view for the whole process, so later passes judged from an
incomplete keep set, and every build cost one history read. A build's end is now taken from the build-ended entries of
the builder's log the pass already reads once, and only a build the log lacks is read from its history. While any done
key's end is unread, or the view is not yet taken in, no pass judges: it ends with the finding artifact-builds-unread,
written once and cleared by the next pass that judges. Every change heard while a watch's handover is gathered now goes
into that handover, so a build finished between the handover and its taking in is not lost by the swap (H-A-257).

Builds: design 47 H §H-A-255 (1), §H-A-257, §H-A-258, §H-A-260
2026-11-21 12:06:18 +01:00
jschoubben 603d4c7557 Design 47 H1: a watch of the builds asked again replaces the followed view whole (H-A-257)
A watch asked again after an error handed over the current values on top of the old view, so a key deleted while no
watch ran stayed in the keep set. Each handover is now gathered apart and replaces the view whole once it is in, and
the ends of keys it no longer holds are forgotten.

Builds: design 47 H §H-A-255 (1), §H-A-257
2026-11-21 12:01:58 +01:00
jschoubben 8d1c0d7d1d Design 47 H1: the artifact sweep judges from a followed view of the builds, not pages (H-A-255)
A page read costs the whole bucket per page, so the builder's builds are now a followed view: the watch's first view
at start, then its updates, kept in memory with each build's end read once from its history. A pass reads nothing of
the builds on the bus and judges nothing before the first view is in, so it never sweeps against an empty keep set.

Builds: design 47 H §H-A-250 (2), §H-A-255 (1)
2026-11-21 11:59:56 +01:00
jschoubben c01a816385 Design 47 H1: the artifact sweep runs one pending pass from one paged read, and stops on an unread list (H-A-250)
A finished build or a want change started a sweep and an observation each, every one reading the builder's builds key
by key and asking the registry's list again, queued behind one lock: a third of the night's bus flood. Now an event
marks the one pending pass, a pass running when marked runs once more, the builds are read in pages of the state with
each build's end from the page, and a list the registry seats cannot answer ends the pass with artifact-list-unread,
written once and cleared by a pass that reads it.

Builds: design 47 H §H-A-250 (1), (2), (3)
2026-11-21 11:54:46 +01:00
mesh-tools eb443cfc24 Merge pull request 'Design 47 H1: the bus module's warrant asks at approve level until a channel carries destroy; the router names the hold (H-A-237)' (#726) from fix/47-H1-nats-warrant-approve into main 2026-11-21 10:28:07 +00:00
jschoubben 7c6cf18532 Design 47 H1: a held destroy-level ask says its level; an offer with no channel named says so
The router held the bus module's destroy-level asks with the bare words
"no channel can carry it now", so the asker could not tell why. holdWhy
now names the destroy level and that no channel carries a destroy answer
yet. The bus module's tool says "offered, channel not yet named" for an
open record with no tickets, never "offered on :".

Builds: design 47 H H-A-237 (3), H-A-239 (2)
2026-11-21 11:26:19 +01:00
jschoubben a8f2316d08 Design 47 H1: the keeper's warrant asks at approve level; the tool says the router's standing
The bus module's drop and rebuild asks were at the destroy level, which no
channel carries yet, so the router held every one unshown until it expired
while the tool answered "the operator is asked". The ask is now at the
approve level for the caller's expires_minutes (5 to 1430, default 60,
refused outside), the keeper accepts an approve-level choice for these two
acts and bounds a request by its ask's own expiry, and the tool makes one
read of the router's record after the publish and says offered, held or
refused as it finds it; the request read shows that standing too.

Builds: design 47 H H-A-237 (1) (2), H-A-239 (1) (2)
2026-11-21 11:17:58 +01:00
mesh-tools af004a8ef6 Merge pull request 'Design 47 H1, proof 1 (second run): one line in feh' (#725) from proof/47-H1-feh-2 into main 2026-11-21 10:08:05 +00:00
jschoubben e1974b770e Design 47 H1, proof 1 (second run): one line in feh
Design lines: H §5 step 7. The first run's pipeline failed on the validate ask before the assigner fix.
2026-11-21 11:07:41 +01:00
mesh-tools c9beea9a38 Merge pull request 'Design 47 H1: the forge answers not-found for a missing directory in a trees listing (H-A-232)' (#724) from fix/47-H1-gitea-trees-not-found into main 2026-11-21 02:09:21 +00:00
jschoubben 74488f3e66 Design 47 H1: the seat git answers a listing's missing directory not-found (H-A-232)
A directory of a trees listing's files the commit does not hold was
thrown as a plain error and answered with the kind failed, so the
indexer took it for a forge that failed, never ran its one-directory
fallback (issue 1585), and published nothing of the repository.
commitTrees throws DirectoryNotInTree for it, and the seat's failure
answers it with the kind not-found, the error naming the directory.

Builds: design 47 H H-A-232 (2), H-A-234; issue 1585
2026-11-21 02:59:05 +01:00
mesh-tools 49cc5b3d7d Merge pull request 'Design 47 H1: the bus module counts every manifest's old-section entries as asked for (H-A-229)' (#723) from fix/47-H1-nats-old-sections into main 2026-11-21 01:33:10 +00:00
jschoubben 08ec25e367 Design 47 H1: a kept old-section entry keeps only the objects the keeper made; a kept state stays in the cap sum
Review round 1. keepOld took the old plan's objects whole, so a state's `<state>-objects` store
the grants part made beside a bucket the keeper made alone would have been compared, grown and,
on a recreate, deleted by the keeper: it now keeps only the old plan's objects the made record
already holds. CapSum skipped every kept entry as counted in the old sections' caps, which count
objects, logs and streams but never a state: it now skips only a kept objects, log or stream
entry, so a kept state's bucket stays in the sum.

At the delivery the plan digest changes once for every module with old sections (most modules,
since state counts): each plans again once, quietly, as its entries are compared with what the
bus holds and nothing is created or said where they agree.

Builds: design 47 H H-A-229, H-A-230, H-A-231; G §547
2026-11-21 02:32:01 +01:00
jschoubben ec305f08e6 Design 47 H1: the bus module counts every manifest's old-section entries as asked for (H-A-229)
The keeper read only a manifest's new-form asks, so the entries it had made for the artifact
registry and the forge, now declared in their old sections, stood as orphans with live data: a
rebuild refused, a drop offered. The keeper now reads every manifest's old sections (state,
objects, logs, streams) by the SDK's naming per section, as the grants part composes them. An
entry it made under one of those bus names is no orphan, is compared and grown to the declared
cap and rebuilt on request; it creates none of them and says no created for one, and counts a
kept one's cap once, in the old sections' sum.

Builds: design 47 H H-A-229, H-A-230, H-A-231
2026-11-21 02:26:20 +01:00
mesh-tools c050f23860 Merge pull request 'Design 47 H1, proof 1: one comment in feh, delivered by its pipeline' (#722) from proof/47-H1-feh into main 2026-11-21 00:43:52 +00:00
jschoubben ae9bb17e0f feh: name H1's proof 1 in the wallpaper file, the single-module change its pipeline delivers
A one-line comment in ~/.fehbg (the files archive) moves feh's fingerprint and
is readable on every machine that holds feh once it is delivered, with no hand act.

Design 47 H §5 runbook step 7: "Prove it. One single-module change (a comment in
a module with lint, unit and composition steps only), merged to main through a
merge call (merge-pull, design 47 H §734), followed from take-in.taken through
pipeline-ended to the node-engine's running and the gate's ended record, with no
hand act".
G §370: "The version reaches every artifact's build": the delivered files
archive carries the version the pipeline gave it.

Builds: design 47 H §5 step 7, G §370
2026-11-21 01:39:08 +01:00
mesh-tools be6eb0358f Merge pull request 'Design 47 H1: the Go build module takes a same-repository requirement from the tree at any version and reads the repository from the fetch (H-A-202, H-A-198, H-A-204)' (#721) from fix/47-H1-build-go-same-repo into main 2026-11-20 23:52:00 +00:00
jschoubben e1d2af8d6e mesh-build-go: build a same-repository requirement from the tree at any version (H item 6)
A requirement of the checked repository at a real version was fetched from the Go seat while the
indexer's fingerprint takes the tree, so the build and its key disagreed; and "same repository" was
read from the owner name novox/ written into the code. The fetch now takes every requirement of its
own repository from the fetched tree before anything is downloaded, at whatever version go.mod
names, keeps those directories in the narrowed tree, and judges "same repository" by the fetch's
repository as the ask names it.

Builds: design 47 H S31, S3, L8 (H-pins-spec-and-gaps section 8 item 6); H-A-204, H-A-202, H-A-198
2026-11-20 23:22:53 +01:00
44 changed files with 2566 additions and 276 deletions
+13
View File
@@ -119,3 +119,16 @@ removed for `retention` on its seat (`release`, `remove`). It reads a reference
and `<seat>/<module>/<artifact>@<digest>`, as one artifact, and says artifacts-gone in the form the build named it. A
call a seat refuses stops the sweep and writes the finding `sweep-refused` in the module's findings; the next sweep no
seat refuses clears it (`sweepseats.go`).
### One pending pass
A finished build or a changed want marks the one pending pass (design 47 H, H-A-250); a pass running when one is marked
runs once more when it ends, however many events marked it. The builder's builds are a followed view: the watch's first view
at start, then its updates, kept in memory (H-A-255), each build's end taken from the builder's log's build-ended entries the
pass reads once, and only for a build the log lacks from its key's history (H-A-258); a watch asked again after an error replaces the view whole with its handover (H-A-257). A pass reads nothing
of them on the bus, judges nothing before the first view is in, and sweeps and observes S25
from that one judgement. A list the registry
seats cannot answer ends the pass with the finding `artifact-list-unread`, written once; nothing asks the list again
until the next event, and a pass that reads it clears the finding. A pass that cannot judge the builds (the view not yet taken in, or a
done build's end unread) ends the same way with the finding `artifact-builds-unread` (H-A-260) (`holder.go`,
`sweepseats.go`).
@@ -116,12 +116,16 @@ func (runtimeBus) Publish(subject string, body any) error {
return stdio.SeatPublish(subject, body, "")
}
// follow watches the builder's done keys and the wants: a build's end sweeps and
// observes again; a want changed observes again. Event-driven: nothing reads them on a timer.
// follow watches the builder's done keys and the wants: the builds are a followed view, the watch's first view then its
// updates kept in memory (design 47 H, H-A-255); a build's end or a want changed marks the one pending pass (H-A-250 (1)),
// which sweeps and observes from the views. Event-driven: nothing reads them on a timer.
func follow(h *Holder, v *views.Reader) {
<-stdio.Served()
watch := func(state string, handle func(stdio.StateChange) error) {
for {
if state == BuildsRead {
h.Watching() // each handover, the first or one asked again, replaces the view whole (H-A-257)
}
if err := stdio.State(state).Watch("", handle); err == nil {
return
} else {
@@ -130,12 +134,8 @@ func follow(h *Holder, v *views.Reader) {
time.Sleep(10 * time.Second)
}
}
observe := func() {
if err := h.Observe(context.Background()); err != nil {
h.logf("image-missing could not be observed: %v", err)
}
}
watch(BuildsRead, func(c stdio.StateChange) error {
h.Heard(c)
if c.Current || c.Op != stdio.OpPut {
return nil
}
@@ -143,10 +143,12 @@ func follow(h *Holder, v *views.Reader) {
State string `json:"state"`
}
if json.Unmarshal(c.Value, &b) == nil && b.State == "done" {
go func() { h.AfterBuild(c.Key); observe() }()
h.Wanted("after the build " + c.Key)
}
return nil
})
// The watch hands every current value over before it returns: the first view is in.
h.TakenIn()
// The wants are the assigner's: the first view then its events (ADR 0615). What Watch hands over before it returns
// is the first view, not a change.
var current atomic.Bool
@@ -154,7 +156,7 @@ func follow(h *Holder, v *views.Reader) {
for {
err := v.Watch(WantsRead, func(string, []byte, bool) {
if !current.Load() {
go observe()
h.Wanted("a want changed")
}
})
if err == nil {
@@ -164,5 +166,5 @@ func follow(h *Holder, v *views.Reader) {
time.Sleep(10 * time.Second)
}
current.Store(false)
observe()
h.Wanted("the first view of the wants")
}
@@ -38,6 +38,10 @@ import (
"git.novox.be/novox/mesh-sdk/go/views"
)
// ErrBuildsUnread is a judgement refused because the builds cannot be judged: the followed view not yet taken in, or a
// done key's end unread (H-A-258, H-A-260).
var ErrBuildsUnread = errors.New("the builder's builds cannot be judged")
// What the holder reads and keeps.
const (
WantsRead = assignerModule + ".wants"
@@ -95,8 +99,19 @@ type Holder struct {
sweep sync.Mutex // one sweep at a time
mu sync.Mutex
fmu sync.Mutex // one pass of Observe at a time
ended map[string]time.Time // a done key's end, read once from its history
// pmu guards the pending pass (H-A-250 (1)): running while a pass runs, wanted when an event marked it since.
pmu sync.Mutex
running bool
wanted bool
wantedBy string
fmu sync.Mutex // one pass of Observe at a time
ended map[string]time.Time // a done key's end, read once from its history
// done is the followed view of the builder's builds (design 47 H, H-A-255): the done builds by key, taken in from the
// watch's first view and its updates (Heard); taken says the first view is in. A pass judges from it, reading nothing.
done map[string]Build
taken bool
// next is the handover a watch asked (again) is giving, which replaces done whole once it is in (H-A-257).
next map[string]Build
}
// Build is one done build of a module, as its key holds it, with when it ended.
@@ -108,39 +123,81 @@ type Build struct {
Manifest string
}
// Builds is every done build the builder's keys hold, each with its end.
// Builds is every done build of the followed view, by key, read from memory, never from the bus (design 47 H, H-A-250
// (2), H-A-255): each with the end known for it, zero where none is known yet (withEnds gives it). Before the view's first
// view is taken in there is none to judge from, and it says so, so no pass sweeps against an empty keep set.
func (h *Holder) Builds() ([]Build, error) {
keys, err := h.Bus.Keys(BuildsRead)
if err != nil {
return nil, fmt.Errorf("the builder's builds: %w", err)
h.mu.Lock()
defer h.mu.Unlock()
if !h.taken {
return nil, fmt.Errorf("%w: they are not taken in yet, their first view has not come", ErrBuildsUnread)
}
var out []Build
for _, k := range keys {
module, fp, err := builds.ParseBuilderBuildKey(k)
if err != nil {
continue
}
raw, found, err := h.Bus.Get(BuildsRead, k)
if err != nil {
return nil, fmt.Errorf("the builder's build %s: %w", k, err)
}
if !found {
continue
}
b, err := builds.DecodeBuilderBuild(raw)
if err != nil || b.State != builds.BuildDone {
continue
}
at, err := h.endOf(k)
if err != nil {
return nil, err
}
out = append(out, Build{Module: module, FP: fp, Key: k, At: at, Artifacts: b.Artifacts, Manifest: b.Manifest})
out := make([]Build, 0, len(h.done))
for _, b := range h.done {
b.At = h.ended[b.Key]
out = append(out, b)
}
sort.Slice(out, func(a, b int) bool { return out[a].Key < out[b].Key })
return out, nil
}
// endOf is when a done key was last written done, read once from its history: a done key is not written again.
// Heard takes one change of the builder's builds into the followed view: a done value is kept, anything else drops the
// key. Its end is not read here: withEnds takes it from the builder's log at the next pass (H-A-258); a key written done
// anew forgets the end it had. While a watch's handover is gathered (Watching), every change goes into it, current or
// not, so nothing heard before TakenIn is lost by the swap (H-A-257).
func (h *Holder) Heard(c stdio.StateChange) {
module, fp, err := builds.ParseBuilderBuildKey(c.Key)
if err != nil {
return
}
var b builds.BuilderBuild
if c.Op == stdio.OpPut {
b, err = builds.DecodeBuilderBuild(c.Value)
}
h.mu.Lock()
defer h.mu.Unlock()
if !c.Current {
delete(h.ended, c.Key) // written anew or gone: an end it had is no longer its
}
if h.next == nil && h.done == nil {
h.done = map[string]Build{}
}
into := h.done
if h.next != nil {
into = h.next
}
if c.Op != stdio.OpPut || err != nil || b.State != builds.BuildDone {
delete(into, c.Key)
return
}
into[c.Key] = Build{Module: module, FP: fp, Key: c.Key, Artifacts: b.Artifacts, Manifest: b.Manifest}
}
// Watching says a watch of the builds is asked, first or again after an error: its handover of current values is
// gathered apart and replaces the view whole at TakenIn, so a key deleted while no watch ran leaves it (H-A-257).
func (h *Holder) Watching() {
h.mu.Lock()
h.next = map[string]Build{}
h.mu.Unlock()
}
// TakenIn says the watch's handover is in: every current value was Heard. It replaces the view whole, and forgets the
// end of every key it no longer holds.
func (h *Holder) TakenIn() {
h.mu.Lock()
defer h.mu.Unlock()
if h.next != nil {
h.done, h.next = h.next, nil
for k := range h.ended {
if _, kept := h.done[k]; !kept {
delete(h.ended, k)
}
}
}
h.taken = true
}
// endOf is when a done key was last written done, read from its history and kept: for a build the builder's log lacks.
func (h *Holder) endOf(key string) (time.Time, error) {
h.mu.Lock()
at, known := h.ended[key]
@@ -320,6 +377,8 @@ type judged struct {
mirrors map[string]*Ref
// mirrorsHeld says why no copy is eligible now, when one is not: a kept build whose bases the log does not name.
mirrorsHeld string
// unread is why the registry seats' list could not be read, nil when it was (or none is read).
unread error
}
func (h *Holder) judge(ctx context.Context) (*judged, error) {
@@ -327,6 +386,15 @@ func (h *Holder) judge(ctx context.Context) (*judged, error) {
if err != nil {
return nil, err
}
// The builder's log, read once: what each build stood on, and each build's end (H-A-258).
against, ends, logErr := h.logged()
if logErr != nil {
h.logf("the builder's log could not be read, so every copy is kept as unrecorded and each end is read from its history: %v", logErr)
against, ends = nil, nil
}
if builds, err = h.withEnds(builds, ends); err != nil {
return nil, err
}
wants, err := h.Wants()
if err != nil {
return nil, err
@@ -335,15 +403,10 @@ func (h *Holder) judge(ctx context.Context) (*judged, error) {
if h.Listed != nil {
if j.view, err = h.Listed(ctx); err != nil {
h.logf("the store's listing could not be read, so nothing is said gone: %v", err)
j.view = nil
j.view, j.unread = nil, err
}
}
if j.view != nil {
against, err := h.against()
if err != nil {
h.logf("the builder's log could not be read, so every copy is kept as unrecorded: %v", err)
against = nil
}
j.mirrors, j.mirrorsHeld = Mirrors(j.view, builds, keptBuilds(builds, wants), against)
for r, m := range j.mirrors {
if len(m.Builds) > 0 {
@@ -354,13 +417,47 @@ func (h *Holder) judge(ctx context.Context) (*judged, error) {
return j, nil
}
// against is what each build the builder's log says ended stood on, by build key: the bases' references.
func (h *Holder) against() (map[string][]string, error) {
// withEnds gives each build its end: the one known, else the builder's log's build-ended entry, else its key's history,
// each kept once read (H-A-258). A done key whose end can be read from neither stays unread, and then nothing is judged:
// never from a view missing a build.
func (h *Holder) withEnds(built []Build, ends map[string]time.Time) ([]Build, error) {
var unread []string
var why error
for i := range built {
if !built[i].At.IsZero() {
continue
}
if at, logged := ends[built[i].Key]; logged {
h.mu.Lock()
if h.ended == nil {
h.ended = map[string]time.Time{}
}
h.ended[built[i].Key] = at
h.mu.Unlock()
built[i].At = at
continue
}
at, err := h.endOf(built[i].Key)
if err != nil {
unread, why = append(unread, built[i].Key), err
continue
}
built[i].At = at
}
if len(unread) > 0 {
return nil, fmt.Errorf("%w: the end of %d done builds could not be read (%s first), so nothing is judged: %v", ErrBuildsUnread, len(unread), unread[0], why)
}
return built, nil
}
// logged is what each build the builder's log says ended stood on, by build key (the bases' references), and when it
// ended: one read of the log.
func (h *Holder) logged() (map[string][]string, map[string]time.Time, error) {
entries, err := h.Bus.LogSince(builds.BuildLogRead, h.now().Add(-builds.BuilderLogDays*24*time.Hour))
if err != nil {
return nil, err
return nil, nil, err
}
out := map[string][]string{}
out, ends := map[string][]string{}, map[string]time.Time{}
for _, raw := range entries {
e, err := builds.DecodeBuildLogEntry(raw.Value)
if err != nil || e.Event != builds.BuildEndedEvent {
@@ -371,8 +468,11 @@ func (h *Holder) against() (map[string][]string, error) {
continue
}
out[key] = e.Against
if e.At.After(ends[key]) {
ends[key] = e.At
}
}
return out, nil
return out, ends, nil
}
// Mirrors judges each copy in the store's mirror repositories (upstream/<host>/<path>, <module>/on-<argument>; ADR
@@ -514,6 +614,11 @@ func (h *Holder) Sweep(ctx context.Context, s Sweeping) (*CollectAnswer, error)
if err != nil {
return nil, err
}
return h.sweepFrom(ctx, j, s)
}
// sweepFrom is a sweep from a judgement already read; the caller holds h.sweep.
func (h *Holder) sweepFrom(ctx context.Context, j *judged, s Sweeping) (*CollectAnswer, error) {
a := &CollectAnswer{DryRun: s.DryRun, WouldLetGo: []string{}, LetGo: []string{}}
var keptArchives, eligible []string
for _, r := range j.sorted() {
@@ -658,15 +763,73 @@ func namedBy(b Build, r string) string {
return r
}
// AfterBuild is the sweep at a build's end.
func (h *Holder) AfterBuild(key string) {
a, err := h.Sweep(context.Background(), Sweeping{Most: MostPerSweep, By: "after the build " + key})
// Wanted marks the pass wanted (design 47 H, H-A-250 (1)): a finished build or a want change asks it, by naming what
// asked. With no pass running one starts; a pass running when it is marked runs once more when it ends, however many
// events marked it, never once per event.
func (h *Holder) Wanted(by string) {
if h.mark(by) {
go h.passes(by)
}
}
// mark sets the pending mark, and says whether no pass ran, so the caller starts one.
func (h *Holder) mark(by string) bool {
h.pmu.Lock()
defer h.pmu.Unlock()
if h.running {
h.wanted, h.wantedBy = true, by
return false
}
h.running = true
return true
}
// passes runs the pass, then once more for as long as an event marked it during the one that ran.
func (h *Holder) passes(by string) {
for {
h.Pass(context.Background(), by)
h.pmu.Lock()
if !h.wanted {
h.running = false
h.pmu.Unlock()
return
}
h.wanted, by = false, h.wantedBy
h.pmu.Unlock()
}
}
// Pass is one pass of the seat: the keep set judged from one read of the builds, the wants and the store's list, then
// the sweep and S25's observation from that one judgement. A list the registry seats cannot answer ends the pass with
// the finding artifact-list-unread, asked again only at the next event (H-A-250 (3)); a list read again clears it.
func (h *Holder) Pass(ctx context.Context, by string) {
h.sweep.Lock()
j, err := h.judge(ctx)
if err != nil {
h.logf("the sweep after %s: %v", key, err)
h.sweep.Unlock()
h.logf("the pass %s: %v", by, err)
if errors.Is(err, ErrBuildsUnread) {
h.unreadFinding(BuildsUnread, err) // H-A-260
}
return
}
if len(a.LetGo) > 0 || a.Stopped != "" || a.Missing > 0 {
h.logf("the sweep after %s let go of %d (%d left, %d missing): %s", key, len(a.LetGo), a.Left, a.Missing, a.Stopped)
h.readAgain(BuildsUnread)
if j.unread != nil {
h.sweep.Unlock()
h.unreadFinding(ListUnread, j.unread)
return
}
h.readAgain(ListUnread)
a, err := h.sweepFrom(ctx, j, Sweeping{Most: MostPerSweep, By: by})
h.sweep.Unlock()
switch {
case err != nil:
h.logf("the sweep %s: %v", by, err)
case len(a.LetGo) > 0 || a.Stopped != "" || a.Missing > 0:
h.logf("the sweep %s let go of %d (%d left, %d missing): %s", by, len(a.LetGo), a.Left, a.Missing, a.Stopped)
}
if err := h.observeFrom(ctx, j); err != nil {
h.logf("image-missing could not be observed: %v", err)
}
}
@@ -857,10 +1020,16 @@ func inStoreAt(image, address string) string {
// want's machine, written once and cleared once it no longer holds. Nothing is said without a listing. What stands is what
// the held writer holds (design 47 G A-32, A-38), so a restart neither drops nor rewrites it.
func (h *Holder) Observe(ctx context.Context) error {
now, err := h.observed(ctx)
j, err := h.judge(ctx)
if err != nil {
return err
}
return h.observeFrom(ctx, j)
}
// observeFrom is Observe from a judgement already read.
func (h *Holder) observeFrom(ctx context.Context, j *judged) error {
now := h.observed(j)
if now == nil {
return nil
}
@@ -899,13 +1068,9 @@ type Findings interface {
}
// observed is S25's findings now by key, nil without a listing.
func (h *Holder) observed(ctx context.Context) (map[string]meshconditions.Finding, error) {
j, err := h.judge(ctx)
if err != nil {
return nil, err
}
func (h *Holder) observed(j *judged) map[string]meshconditions.Finding {
if j.view == nil {
return nil, nil
return nil
}
address := ""
if d, err := h.Door(); err == nil {
@@ -944,7 +1109,7 @@ func (h *Holder) observed(ctx context.Context) (map[string]meshconditions.Findin
break
}
}
return now, nil
return now
}
// imageMissing is S25's finding, in the controller's words (cmd/mesh-controller/image_missing.go watchMissingImages); its id
@@ -200,6 +200,15 @@ func TestTheSeatSweepsAtABuildsEndOnARealBusWithNoLease(t *testing.T) {
}
h := &Holder{Bus: findingsAt{realBus: bus}, Findings: findingsThrough{findingsAt{realBus: bus}}, Door: mem.Door, Listed: mem.Listed, Seats: ask, Now: time.Now, Logf: t.Logf}
h.Acts = meshhandactsclient.Recorder{Ask: ask}
// The followed view of the builds: the six on the bus handed over as the watch's first view does.
h.Watching()
if keys, err := bus.Keys(BuildsRead); err == nil {
for _, k := range keys {
raw, _, _ := bus.Get(BuildsRead, k)
h.Heard(stdio.StateChange{State: BuildsRead, Key: k, Op: stdio.OpPut, Value: raw, Current: true})
}
}
h.TakenIn()
// A build's end heard from the builds bucket, as follow hands it over.
kv, _ := bus.kv(BuildsRead)
@@ -215,7 +224,8 @@ func TestTheSeatSweepsAtABuildsEndOnARealBusWithNoLease(t *testing.T) {
}
select {
case e := <-w.Updates():
h.AfterBuild(e.Key())
h.Heard(stdio.StateChange{State: BuildsRead, Key: e.Key(), Op: stdio.OpPut, Value: e.Value()})
h.Pass(ctx, "after the build "+e.Key())
case <-ctx.Done():
t.Fatal("the build's end was not heard")
}
@@ -34,6 +34,17 @@ type memBus struct {
history map[string][]stdio.StateRevision
log []stdio.LogEntry
published []string
// historyErr fails the history read of a key.
historyErr map[string]error
// calls counts each read by "<verb> <state>": a pass reads nothing of the builds, judged from the followed view.
calls map[string]int
}
func (b *memBus) count(verb, state string) {
if b.calls == nil {
b.calls = map[string]int{}
}
b.calls[verb+" "+state]++
}
func (b *memBus) LogSince(read string, at time.Time) ([]stdio.LogEntry, error) {
@@ -49,6 +60,7 @@ func (b *memBus) LogSince(read string, at time.Time) ([]stdio.LogEntry, error) {
func (b *memBus) Keys(state string) ([]string, error) {
b.mu.Lock()
defer b.mu.Unlock()
b.count("keys", state)
var out []string
for k := range b.states[state] {
out = append(out, k)
@@ -60,11 +72,18 @@ func (b *memBus) Keys(state string) ([]string, error) {
func (b *memBus) Get(state, key string) ([]byte, bool, error) {
b.mu.Lock()
defer b.mu.Unlock()
b.count("get", state)
v, ok := b.states[state][key]
return v, ok, nil
}
func (b *memBus) History(state, key string) ([]stdio.StateRevision, error) {
b.mu.Lock()
defer b.mu.Unlock()
b.count("history", state)
if err := b.historyErr[key]; err != nil {
return nil, err
}
return b.history[state+" "+key], nil
}
@@ -229,7 +248,7 @@ func aMesh(t *testing.T) (*Holder, *memBus, *memDoor, *[]string) {
want, _ := json.Marshal(meshwants.WantsByWantNodeModuleValue{V: 1, Fp: fp(1), Self: true})
_ = b.Put(WantsRead, "want.ace.letta", json.RawMessage(want))
var acts []string
h := &Holder{Bus: b, Findings: findingsThrough{b}, Door: func() (Door, error) { return door, nil }, Seats: door.seat,
h := &Holder{Logf: t.Logf, Bus: b, Findings: findingsThrough{b}, Door: func() (Door, error) { return door, nil }, Seats: door.seat,
Listed: func(context.Context) (*View, error) { return &View{L: listing}, nil },
Now: func() time.Time { return day.Add(48 * time.Hour) },
Acts: meshhandactsclient.Recorder{Ask: func(key string, body any) (json.RawMessage, error) {
@@ -237,9 +256,49 @@ func aMesh(t *testing.T) (*Holder, *memBus, *memDoor, *[]string) {
acts = append(acts, key+" "+string(raw))
return json.RawMessage(`{}`), nil
}}}
takeIn(h, b)
return h, b, door, &acts
}
// takeIn hands the builds over as the watch's first view does, then forgets the reads that took: a pass's are counted.
func takeIn(h *Holder, b *memBus) {
handOver(h, b)
h.TakenIn()
b.calls = nil
}
// handOver is a watch's handover of every current value, not yet taken in.
func handOver(h *Holder, b *memBus) {
h.Watching()
keys, _ := b.Keys(BuildsRead)
for _, k := range keys {
raw, _, _ := b.Get(BuildsRead, k)
h.Heard(stdio.StateChange{State: BuildsRead, Key: k, Op: stdio.OpPut, Value: raw, Current: true})
}
}
// keyOf is letta's n-th build's key.
func keyOf(n int) string { k, _ := builds.BuilderBuildKey("letta", fp(n)); return k }
// viewKeys are the keys the followed view holds.
func viewKeys(t *testing.T, h *Holder) []string {
t.Helper()
h.mu.Lock()
defer h.mu.Unlock()
var out []string
for k := range h.done {
out = append(out, k)
}
sort.Strings(out)
return out
}
// putBuild writes a build's key and hands it over as the watch's update does.
func putBuild(h *Holder, b *memBus, key string, raw []byte) {
_ = b.Put(BuildsRead, key, json.RawMessage(raw))
h.Heard(stdio.StateChange{State: BuildsRead, Key: key, Op: stdio.OpPut, Value: raw})
}
// ADR 0571 clause 2, 1126/A4/AC2: the keep set from the store's listing, the wants and the five newest builds; with no
// lease key, a build's end and a confirmed collect let go of the eligible within the bound, leave an index for a
// confirmed collect, record each deletion as a hand act and say artifacts-gone.
@@ -312,7 +371,7 @@ func TestTheSeatKeepsTheFiveNewestAndTheWantedAndSweepsWithNoLease(t *testing.T)
raw, _ := builds.EncodeBuilderBuild(builds.BuilderBuild{State: builds.BuildDone, Manifest: strings.Repeat("d", 64), Artifacts: []builds.BuiltArtifact{
{Name: "image", Kind: "image", Digest: digestOf(imageRef("letta", n)), Reference: artifactref.SeatReference(reg.Seat, "letta", "image", digestOf(imageRef("letta", n)))},
{Name: "files", Kind: "archive", Digest: digestOf(archiveRef("letta", n)), Reference: artifactref.SeatReference(ociseat.Seat, "letta", "files", digestOf(archiveRef("letta", n)))}}})
_ = b.Put(BuildsRead, key, json.RawMessage(raw))
putBuild(h, b, key, raw)
}
}
if c.refuse != "" {
@@ -323,7 +382,7 @@ func TestTheSeatKeepsTheFiveNewestAndTheWantedAndSweepsWithNoLease(t *testing.T)
}
switch c.how {
case "build":
h.AfterBuild("letta.x")
h.Pass(context.Background(), "after the build letta.x")
var gone []string
for _, x := range door.asked {
if r, ok := strings.CutPrefix(x, "let go "); ok {
@@ -420,6 +479,7 @@ func TestAWantedImageTheStoreLacksIsAFindingAndImagesNamesWantsAndHaves(t *testi
view.L.Repos[repo].Revisions[digest] = true
// A restart: a holder made afresh holds no finding of its own, and clears what the writer holds (design 47 G A-32, A-38).
h = &Holder{Bus: h.Bus, Findings: h.Findings, Door: h.Door, Listed: h.Listed, Seats: h.Seats, Now: h.Now, Acts: h.Acts, Logf: h.Logf}
takeIn(h, b)
if err := h.Observe(context.Background()); err != nil {
t.Fatal(err)
}
@@ -482,3 +542,154 @@ func (f findingsThrough) Held() []meshconditions.Finding {
func (f findingsThrough) Clear(_ context.Context, scope, id, kind, _ string) error {
return f.bus.Delete(FindingsState, views.FindingKey(scope, id, kind))
}
// writesCounted counts the findings written through the test's writer.
type writesCounted struct {
findingsThrough
writes *int
}
func (w writesCounted) Write(ctx context.Context, f meshconditions.Finding) error {
*w.writes++
return w.findingsThrough.Write(ctx, f)
}
// Design 47 H, H-A-250: the starts fold into one pending pass, a pass reads the builds in one page, and a list the
// registry seats cannot answer ends the pass with artifact-list-unread, written once and asked again only at an event.
func TestOnePendingPassReadsTheBuildsOnceAndEndsOnAnUnreadList(t *testing.T) {
t.Run("three events during a pass run exactly one more", func(t *testing.T) {
h, _, _, _ := aMesh(t)
listed, inner := 0, h.Listed
h.Listed = func(ctx context.Context) (*View, error) {
listed++
if listed == 1 {
for _, by := range []string{"after the build a", "after the build b", "a want changed"} {
h.Wanted(by) // a pass runs: each only marks it
}
}
return inner(ctx)
}
if !h.mark("after the build letta.x") {
t.Fatal("no pass ran, and the mark did not start one")
}
h.passes("after the build letta.x")
if listed != 2 {
t.Fatalf("three events during a pass ran %d passes, want 2", listed)
}
if h.running || h.wanted {
t.Fatalf("after the passes running is %v and wanted %v", h.running, h.wanted)
}
})
t.Run("a pass reads nothing of the builds and their ends come from the log", func(t *testing.T) {
h, b, door, _ := aMesh(t)
handOver(h, b) // a fresh take-in, counted: no history read for a build the log holds (H-A-258)
h.TakenIn()
h.Pass(context.Background(), "after the build letta.x")
if n := b.calls["keys "+BuildsRead] - 1 + b.calls["get "+BuildsRead] - 7 + b.calls["history "+BuildsRead]; n != 0 {
t.Fatalf("a take-in and a pass read the builds by %v", b.calls)
}
if !slices.Contains(door.asked, "let go "+imageRef("letta", 2)) {
t.Fatalf("the pass swept nothing: %v", door.asked)
}
// Only a build the log lacks is read from its history, once.
h, b, _, _ = aMesh(t)
b.log = append(b.log[:2:2], b.log[3:]...) // letta's 3rd
h.Pass(context.Background(), "a")
h.Pass(context.Background(), "b")
if b.calls["history "+BuildsRead] != 1 {
t.Fatalf("two passes read the builds' history %d times, want once for the build the log lacks", b.calls["history "+BuildsRead])
}
})
t.Run("no pass judges before the first view, and says so", func(t *testing.T) {
h, b, door, _ := aMesh(t)
h.taken = false
h.Pass(context.Background(), "after the build letta.x")
_, found, _ := b.Get(FindingsState, views.FindingKey("module", Module, BuildsUnread))
if len(door.asked) != 0 || !found {
t.Fatalf("a pass before the builds were taken in asked %v; artifact-builds-unread is written: %v", door.asked, found)
}
})
t.Run("an unread end holds every pass until it is read", func(t *testing.T) {
h, b, door, _ := aMesh(t)
writes := 0
h.Findings = writesCounted{findingsThrough{b}, &writes}
b.log = append(b.log[:2:2], b.log[3:]...) // letta's 3rd: the log lacks it, and its history fails
b.historyErr = map[string]error{keyOf(3): errors.New("no responders")}
h.Pass(context.Background(), "a")
h.Pass(context.Background(), "b")
key := views.FindingKey("module", Module, BuildsUnread)
if _, found, _ := b.Get(FindingsState, key); !found || writes != 1 || len(door.asked) != 0 {
t.Fatalf("with an end unread the passes asked %v; the finding is written %v, %d times", door.asked, found, writes)
}
b.historyErr = nil
h.Pass(context.Background(), "c")
if _, found, _ := b.Get(FindingsState, key); found || !slices.Contains(door.asked, "let go "+imageRef("letta", 2)) {
t.Fatalf("once the end was read the pass asked %v and left the finding: %v", door.asked, found)
}
})
t.Run("a change heard during the handover is in the view it gives", func(t *testing.T) {
h, b, _, _ := aMesh(t)
handOver(h, b)
raw, _, _ := b.Get(BuildsRead, keyOf(1))
h.Heard(stdio.StateChange{State: BuildsRead, Key: keyOf(8), Op: stdio.OpPut, Value: raw}) // after the handover, before TakenIn
h.Heard(stdio.StateChange{State: BuildsRead, Key: keyOf(3), Op: stdio.OpDelete})
h.TakenIn()
got := viewKeys(t, h)
if !slices.Contains(got, keyOf(8)) || slices.Contains(got, keyOf(3)) {
t.Fatalf("the view after the handover holds %v", got)
}
})
t.Run("a key deleted or no longer done leaves the view; one written done again takes its new end", func(t *testing.T) {
h, b, _, _ := aMesh(t)
h.Pass(context.Background(), "a") // every end known
building, _ := builds.EncodeBuilderBuild(builds.BuilderBuild{State: "building"})
h.Heard(stdio.StateChange{State: BuildsRead, Key: keyOf(3), Op: stdio.OpDelete})
h.Heard(stdio.StateChange{State: BuildsRead, Key: keyOf(4), Op: stdio.OpPut, Value: building})
if got := viewKeys(t, h); slices.Contains(got, keyOf(3)) || slices.Contains(got, keyOf(4)) {
t.Fatalf("the view holds %v", got)
}
later := b.log[len(b.log)-1].At.Add(time.Hour)
e := builds.BuildLogEntry{Event: builds.BuildEndedEvent, Module: "letta", FP: fp(5), Repository: "novox/x", Commit: "c", Node: "ace", At: later, Outcome: "done"}
entry, _ := builds.EncodeBuildLogEntry(e)
b.log = append(b.log, stdio.LogEntry{Sequence: 99, Key: "letta", Value: entry, At: later})
raw, _, _ := b.Get(BuildsRead, keyOf(5))
h.Heard(stdio.StateChange{State: BuildsRead, Key: keyOf(5), Op: stdio.OpPut, Value: raw})
h.Pass(context.Background(), "b")
if got, _ := h.Builds(); !slices.ContainsFunc(got, func(x Build) bool { return x.Key == keyOf(5) && x.At.Equal(later) }) {
t.Fatalf("letta's 5th written done again does not end at %v: %+v", later, got)
}
})
t.Run("a watch asked again replaces the view whole", func(t *testing.T) {
h, b, _, _ := aMesh(t)
gone, _ := builds.BuilderBuildKey("letta", fp(3))
_ = b.Delete(BuildsRead, gone) // deleted while no watch ran: no change of it is heard
takeIn(h, b)
got, err := h.Builds()
if err != nil || len(got) != 6 || slices.ContainsFunc(got, func(x Build) bool { return x.Key == gone }) {
t.Fatalf("after the watch was asked again the view holds %d builds (%v), %s among them", len(got), err, gone)
}
})
t.Run("a list unread ends the pass with its finding once", func(t *testing.T) {
h, b, door, _ := aMesh(t)
writes, listed, inner := 0, 0, h.Listed
h.Findings = writesCounted{findingsThrough{b}, &writes}
h.Listed = func(context.Context) (*View, error) {
listed++
return nil, errors.New("the answer exceeds the bus's one megabyte")
}
h.Pass(context.Background(), "after the build a")
h.Pass(context.Background(), "a want changed")
key := views.FindingKey("module", Module, ListUnread)
if raw, found, _ := b.Get(FindingsState, key); !found || writes != 1 || !strings.Contains(string(raw), "one megabyte") {
t.Fatalf("the finding is %s (found %v), written %d times", raw, found, writes)
}
if listed != 2 || len(door.asked) != 0 || len(b.published) != 0 {
t.Fatalf("two passes listed %d times, asked %v and said %v", listed, door.asked, b.published)
}
h.Listed = inner
h.Pass(context.Background(), "after the build b")
if _, found, _ := b.Get(FindingsState, key); found || len(door.asked) == 0 {
t.Fatalf("a pass that read the list left the finding (%v) or swept nothing (%v)", found, door.asked)
}
})
}
@@ -32,7 +32,7 @@ func main() {
h.Acts = meshhandactsclient.Recorder{Ask: stdio.Ask}
// The module's one held writer (design 47 G §689, A-16, A-29, A-32): the holder's findings and the reader's go through it.
w, err := views.NewStdioHeldWriter(views.WriterConfig[meshconditions.Finding, meshconditions.FindingCleared, meshconditions.FindingsStated, meshconditions.FindingsAskedEvent]{
Asked: meshconditions.SubjectFindingsAsked, Module: Module, State: stdio.State("findings"), Kinds: []string{ImageMissing, SweepRefused}})
Asked: meshconditions.SubjectFindingsAsked, Module: Module, State: stdio.State("findings"), Kinds: []string{ImageMissing, SweepRefused, ListUnread, BuildsUnread}})
if err != nil {
logf("%v", err)
os.Exit(1)
@@ -27,6 +27,9 @@ import (
// SweepRefused is the sweep's finding: a seat refused a hold or a remove the sweep asked.
const SweepRefused = "sweep-refused"
// ListUnread is the pass's finding: the registry seats' list could not be read, so the pass ended (design 47 H, H-A-250 (3)).
const ListUnread = "artifact-list-unread"
// canonical is a kept reference in the older form, `artifact-store://<path>`, whichever form it came in; another is as
// it was.
func canonical(reference string) string {
@@ -140,6 +143,53 @@ func (h *Holder) sweepClean() {
}
}
// BuildsUnread is the pass's finding: the builder's builds could not be judged, so the pass ended (design 47 H, H-A-260).
const BuildsUnread = "artifact-builds-unread"
// unreadFinding writes a pass's finding of what it could not read (ListUnread, BuildsUnread) once: a pass that fails the
// same way while the writer holds it writes nothing more. Nothing asks again until the next event marks a pass.
func (h *Holder) unreadFinding(kind string, err error) {
for _, was := range h.Findings.Held() {
if was.Kind == kind {
return
}
}
at := h.now().UTC()
f := meshconditions.Finding{Scope: "module", ID: keys.EscapeName(Module), Kind: kind,
Subject: meshconditions.Subject{Scope: "module", ID: keys.EscapeName(Module)},
Severity: meshconditions.FindingSeverityWarning, Resolver: "operator",
Summary: "the artifact sweep stopped: the registry seats' list could not be read (the evidence names why)",
Headline: "The artifact sweep cannot read the store's list",
Explanation: "The sweep judges what to keep against the store's list, which it asks the registry seats. The list " +
"could not be read, so the sweep removed nothing and observed nothing; what it kept stays kept.",
Needs: "read the registry seat holder's journal for why its list failed; the next finished build or want change asks again.",
Evidence: []meshconditions.Evidence{{At: at, Said: err.Error()}},
Source: "CAT-AS1", Observer: Module, At: at}
if kind == BuildsUnread {
f.Summary = "the artifact sweep stopped: the builder's builds could not be judged (the evidence names why)"
f.Headline = "The artifact sweep cannot judge the builds"
f.Explanation = "The sweep keeps each module's newest builds and every wanted one, judged from the builder's builds " +
"and when each ended. Those could not all be read, so the sweep removed nothing and observed nothing; what " +
"it kept stays kept."
f.Needs = "read this module's journal for which build's end is unread; the next finished build or want change asks again."
}
if werr := h.Findings.Write(context.Background(), f); werr != nil {
h.logf("the pass's finding %s (%v) could not be written: %v", kind, err, werr)
}
}
// readAgain clears a pass's finding of kind, when the held writer holds it, once a pass read what it could not.
func (h *Holder) readAgain(kind string) {
for _, was := range h.Findings.Held() {
if was.Kind != kind {
continue
}
if err := h.Findings.Clear(context.Background(), was.Scope, was.ID, was.Kind, "a pass read it again"); err != nil {
h.logf("the pass's finding %s could not be cleared: %v", kind, err)
}
}
}
// keptWhy is why a kept reference is held, as the seat's hold takes it.
func keptWhy(r *Ref) string {
if r == nil || len(r.Why) == 0 {
+1 -1
View File
@@ -2,5 +2,5 @@
# The wallpaper (module feh, novox/hq ADR 0208). Owned by the mesh: replaced at every delivery. The
# session's start runs it, and so may anything that wants the declared wallpaper back. The image is
# the module's own, in ~/.local/share/feh/wallpapers. feh_set changes the wallpaper for a session
# without touching this file (the second single-module proof, 2026-11-17).
# without touching this file (design 47 H1, proof 1 (second run): delivered by its pipeline, 2026-11-21).
feh --no-fehbg --bg-fill "$HOME/.local/share/feh/wallpapers/default.jpg"
+10 -1
View File
@@ -182,6 +182,15 @@ export const FORGE_REQUEST_TIMEOUT_MS = 20_000;
/** The most a tool's answer carries of one file: the bus's message bound is 1 MiB and a file travels as text (issue 1577). */
export const MAX_FILE_BYTES = 512 * 1024;
/** A directory of a listing's `files` the commit does not hold: the seat git answers it with the kind not-found, the error
* naming the directory (seat.ts failure; novox/hq design 47 H H-A-232). */
export class DirectoryNotInTree extends Error {
constructor(message: string) {
super(message);
this.name = "DirectoryNotInTree";
}
}
/** One request to the forge, cut at the limit and failing loud naming the request. */
async function timedFetch(url: string, init: RequestInit, label: string, timeoutMs: number): Promise<Response> {
try {
@@ -634,7 +643,7 @@ export class GiteaClient {
// is whole (a cut one failed above), so a directory absent from it is absent at the commit.
filesOf = {};
for (const dir of files) {
if (dir !== "" && dirOf.get(dir) === undefined) throw new Error(`the directory ${dir} is not in ${owner}/${repo} at ${sha}`);
if (dir !== "" && dirOf.get(dir) === undefined) throw new DirectoryNotInTree(`the directory ${dir} is not in ${owner}/${repo} at ${sha}`);
const prefix = dir === "" ? "" : `${dir}/`;
const blobs: Record<string, string> = {};
for (const e of entries) {
@@ -344,14 +344,27 @@ func numbers(v any) []float64 {
return out
}
// needTheModulesPackages ends the test NOT RUN, never failed, where what it builds and runs with is missing: node and npm
// on the PATH and the module's installed SDK package under node_modules. The Go integration level is given the tree
// without its npm packages and no network (issue 1960, H-A-294), as the Node build module's realMachine skips by name.
func needTheModulesPackages(t *testing.T, dir string) {
t.Helper()
for _, p := range []string{"node", "npm"} {
if _, err := exec.LookPath(p); err != nil {
t.Skipf("NOT RUN: no %s installed where this check runs; proven in a working copy of the module after npm install", p)
}
}
if _, err := os.Stat(filepath.Join(dir, "node_modules", "@novox", "mesh-sdk")); err != nil {
t.Skipf("NOT RUN: the forge module's npm packages are not installed in %s (%v); proven in a working copy of the module after npm install", dir, err)
}
}
func TestTheSeatGitWholeOverARealBus(t *testing.T) {
dir, err := filepath.Abs(filepath.Join("..", ".."))
if err != nil {
t.Fatal(err)
}
if _, err := os.Stat(filepath.Join(dir, "node_modules", "@novox", "mesh-sdk")); err != nil {
t.Fatalf("the forge module's packages are not installed in %s (npm install first): %v", dir, err)
}
needTheModulesPackages(t, dir)
build := exec.Command("npm", "run", "build")
build.Dir = dir
if out, err := build.CombinedOutput(); err != nil {
+4 -1
View File
@@ -21,7 +21,7 @@ import {
type TagAnswer, type TagsAnswer, type TreesAnswer, type ViewAnswer,
} from "@novox/mesh-delivery-contracts-git";
import type { CommitStatus, GiteaClient, GiteaComment, GiteaPull } from "./client.js";
import { MAX_FILE_BYTES } from "./client.js";
import { DirectoryNotInTree, MAX_FILE_BYTES } from "./client.js";
import { containedFrom, newestByContext } from "./checks.js";
import { Refused, checkName } from "./guard.js";
import { appendNote, viewBody, viewComment, type Runner } from "./delivery.js";
@@ -64,6 +64,9 @@ export function failure(err: unknown): { error: string; "error-kind": ErrorKind
const text = err instanceof Error ? err.message : String(err);
// The forge's 404: what was asked about (a repository, a commit, a pull request, a branch, a tag) does not exist.
if (/^Gitea API [^:]*: 404 /.test(text)) return { error: text, "error-kind": "not-found" };
// A directory of `files` the commit does not hold (client.ts commitTrees): not-found, the error naming the directory, so the
// indexer reads the others alone and leaves the gone one out (novox/hq design 47 H H-A-232).
if (err instanceof DirectoryNotInTree) return { error: text, "error-kind": "not-found" };
const refused = err instanceof Refused || /^Gitea API [^:]*: 4\d\d /.test(text) || /^Gitea API: /.test(text);
return { error: text, "error-kind": refused ? "refused" : "failed" };
}
+13 -5
View File
@@ -50,8 +50,9 @@ async function serve(pages: any[]) {
res.end(JSON.stringify(pages[page - 1] ?? { sha: COMMIT, tree: [], truncated: false, page, total_count: 0 }));
});
await new Promise<void>((r) => server.listen(0, r));
const client = new GiteaClient(`http://127.0.0.1:${(server.address() as any).port}`, "t");
return { client, server, asked };
const url = `http://127.0.0.1:${(server.address() as any).port}`;
const client = new GiteaClient(url, "t");
return { client, server, asked, url };
}
test("a commit's trees: the root and the tree of each directory holding a module.json", async () => {
@@ -238,12 +239,19 @@ test("files: every blob under each asked directory, recursively, as its path rel
});
test("a directory of files the commit lacks, or spelled badly, or a listing the forge cut, fails and answers nothing", async () => {
const { client, server } = await serve([{ sha: COMMIT, url: "u", tree: LISTING, truncated: false, page: 1, total_count: LISTING.length }]);
const { client, server, url } = await serve([{ sha: COMMIT, url: "u", tree: LISTING, truncated: false, page: 1, total_count: LISTING.length }]);
try {
await assert.rejects(client.commitTrees("novox", "hq", COMMIT, undefined, ["modules/a", "modules/gone"]), (err: Error) => {
assert.match(err.message, /modules\/gone/);
// The seat answers it with the kind not-found naming the directory (novox/hq design 47 H H-A-232), never failed: the
// holder's own client and failure, as the module runs them.
const { failure } = await import("../dist/seat.js");
const { GiteaClient } = await import("../dist/client.js");
await assert.rejects(new GiteaClient(url, "t").commitTrees("novox", "hq", COMMIT, undefined, ["modules/a", "modules/gone"]), (err: Error) => {
const said = failure(err);
assert.equal(said["error-kind"], "not-found", JSON.stringify(said));
assert.match(said.error, /the directory modules\/gone is not in novox\/hq/);
return true;
});
await assert.rejects(client.commitTrees("novox", "hq", COMMIT, undefined, ["modules/a", "modules/gone"]), /modules\/gone/);
await assert.rejects(client.commitTrees("novox", "hq", COMMIT, undefined, ["README.md"]), /README\.md/, "a file is no directory");
for (const bad of ["../x", "/abs", "a//b", "modules/a/", "."]) {
await assert.rejects(client.commitTrees("novox", "hq", COMMIT, undefined, [bad]), /files/, bad);
@@ -208,11 +208,16 @@ func (m *Module) fetchInto(ctx context.Context, req sdkstep.FetchRequest, id str
go func() {
defer wg.Done()
defer func() { <-slots }()
// The checked repository's own modules come from the fetched tree, at whatever version go.mod names them
// (pins.go takeOwnRequirements; design 47 H, S31).
if taken[i], errs[i] = m.takeOwnRequirements(root, repository, filepath.Join(tree, d), d); errs[i] != nil {
return
}
// The asked versions of the mesh's packages go into the modfile before anything is downloaded (versions.go).
if taken[i], errs[i] = m.takeVersions(ctx, root, filepath.Join(tree, d), d, req.Packages, &pins); errs[i] != nil {
more, err := m.takeVersions(ctx, root, repository, filepath.Join(tree, d), d, req.Packages, &pins)
if taken[i], errs[i] = append(taken[i], more...), err; errs[i] != nil {
return
}
var more []TakenVersion
// Then what only a module the directory replaces names, which the download would not miss (pins.go).
more, errs[i] = m.takeReplacedZeroPins(ctx, root, repository, filepath.Join(tree, d), d, req.Packages, taken[i], &pins)
if taken[i] = append(taken[i], more...); errs[i] != nil {
@@ -296,8 +296,9 @@ func TestAVerifyThatFailsEndsTheStepNamingTheDirectoryAndTheLine(t *testing.T) {
return forgekey.Ran{}
}}
m, states := fetchModule(t, f)
// Asked for the module of tools, as a build is for one module (A-447b).
if _, err := fetchTool(t, m)(map[string]any{"repository": "novox/mesh-sdk", "commit": commitSHA, "dir": "tools"}); err != nil {
// Asked for the module of tools, as a build is for one module (A-447b), of a repository other than mesh-sdk's,
// whose packages its go.mod requires: one of its own repository is built from the tree (S31).
if _, err := fetchTool(t, m)(map[string]any{"repository": "novox/mesh-member", "commit": commitSHA, "dir": "tools"}); err != nil {
t.Fatal(err)
}
st := states.waitEnd(t)
@@ -22,7 +22,7 @@ var narrowedKeptAtRoot = []string{"module.json", "go.sum", "package-lock.json"}
// narrowTree narrows the fetched tree for a build of an artifact that names its paths (ADR 0499 clause 4,
// steprun.Narrow). Beside the artifact's paths and the module's manifest, the workspace holds every generated-from
// directory an artifact of the module names, and the directory (with its go.mod) of every package of the same repository
// the artifact's go.mod requires at the zero version, found by the manifests of that repository's modules; nothing else
// the artifact's go.mod requires at any version (S31), found by the manifests of that repository's modules; nothing else
// (design 47 H, lead answer H-A-136). A package of another repository comes published, as before (ADR 0573 §4).
func narrowTree(tree, repository string, req sdkstep.FetchRequest) error {
dir := cleanDir(req.Dir)
@@ -30,7 +30,7 @@ func narrowTree(tree, repository string, req sdkstep.FetchRequest) error {
return steprun.Narrow(tree, req) // refused there: named paths without the module's directory
}
extra := generatedFromDirs(tree, dir)
extra = append(extra, ownZeroPackageDirs(tree, repository, dir, req.Paths)...)
extra = append(extra, ownPackageDirs(tree, repository, dir, req.Paths)...)
var outside []string
inside := slices.Clone(req.Paths)
for _, e := range extra {
@@ -127,18 +127,19 @@ func generatedFromDirs(tree, dir string) []string {
return out
}
// ownZeroPackageDirs are the directories (from the tree's root) of the packages of the checked repository that a go.mod
// under the artifact's named paths requires at the zero version (H-A-136; ADR 0573 §2), transitively: the go.mod of each
// such directory is read in turn for the packages of the repository it requires at the zero version, each directory
// once (design 47 H, H-A-202, H-A-198). Each is found by the manifests of the repository's modules: a module's directory
// ownPackageDirs are the directories (from the tree's root) of the packages of the checked repository (the fetch's
// repository, sameRepository) that a go.mod under the artifact's named paths requires, at whatever version it names them
// (H-A-136; ADR 0573 §2; design 47 H, S31), transitively: the go.mod of each such directory is read in turn for the
// packages of the repository it requires, each directory once (design 47 H, H-A-202, H-A-198). Each is found by the manifests of the repository's modules: a module's directory
// or the directory of one of its artifacts' paths whose go.mod declares the required module path. One no manifest leads
// to is left to the fetch's resolution, which fails naming it.
func ownZeroPackageDirs(tree, repository, dir string, paths []string) []string {
func ownPackageDirs(tree, repository, dir string, paths []string) []string {
want := map[string]bool{}
addReqs := func(raw []byte) {
for _, pn := range forgePins(string(raw)) {
if pn.short == zeroShort && "novox/"+pn.repo == repository {
want[pn.module] = true
required, _ := goModRequirements(string(raw))
for module := range required {
if sameRepository(module, repository) {
want[module] = true
}
}
}
@@ -33,6 +33,7 @@ func TestANarrowedWorkspaceHoldsTheGeneratedFromDirectoriesAndItsRepositorysZero
zero := " v0.0.0-00010101000000-" + zeroShort
for _, tc := range []struct {
name, dir string
repository string // the fetch's repository, novox/mesh-delivery when empty
paths []string
files map[string]string
kept, gone []string
@@ -114,6 +115,25 @@ func TestANarrowedWorkspaceHoldsTheGeneratedFromDirectoriesAndItsRepositorysZero
kept: []string{"modules/a/state/s.go", "modules/b/builds/b.go", "modules/c/client/c.go"},
gone: []string{"modules/b/other", "modules/c/other", "modules/d"},
modules: []string{"modules/a/state"}},
// S31, L8: a package of the repository required at a real version comes along too, transitively, and "the
// repository" is the fetch's, under any owner; a package of another repository at a real version does not.
{name: "the repository's packages at a real version, under another owner", repository: "kin/app", dir: "modules/a", paths: []string{"state"},
files: map[string]string{
"modules/a/module.json": `{"module":"a","build":{"artifacts":[{"name":"state","kind":"package","language":"go","paths":["state"]}]}}`,
"modules/a/state/go.mod": "module git.novox.be/kin/app/modules/a/state\n\ngo 1.26\n\nrequire (\n\tgit.novox.be/kin/app/modules/b/builds v0.1.3\n\tgit.novox.be/novox/app/modules/d/x v0.1.3\n)\n",
"modules/a/state/s.go": "package state\n",
"modules/b/module.json": `{"module":"b","build":{"artifacts":[{"name":"builds","kind":"package","language":"go","paths":["builds"]}]}}`,
"modules/b/builds/go.mod": "module git.novox.be/kin/app/modules/b/builds\n\ngo 1.26\n\nrequire git.novox.be/kin/app/modules/c/client v0.2.0\n",
"modules/b/builds/b.go": "package builds\n",
"modules/c/module.json": `{"module":"c","build":{"artifacts":[{"name":"client","kind":"package","language":"go","paths":["client"]}]}}`,
"modules/c/client/go.mod": "module git.novox.be/kin/app/modules/c/client\n\ngo 1.26\n",
"modules/c/client/c.go": "package client\n",
"modules/d/module.json": `{"module":"d","build":{"artifacts":[{"name":"x","kind":"package","language":"go","paths":["x"]}]}}`,
"modules/d/x/go.mod": "module git.novox.be/novox/app/modules/d/x\n\ngo 1.26\n",
},
kept: []string{"modules/a/state/s.go", "modules/b/builds/b.go", "modules/c/client/c.go"},
gone: []string{"modules/d"},
modules: []string{"modules/a/state"}},
// A named path that leaves the repository is refused.
{name: "a named path that leaves the repository", dir: "mesh-grants", paths: []string{".", "../../outside"},
files: map[string]string{"mesh-grants/module.json": `{"module":"mesh-grants"}`, "mesh-grants/go.mod": "module mesh-grants\n\ngo 1.26\n"},
@@ -153,7 +173,11 @@ func TestANarrowedWorkspaceHoldsTheGeneratedFromDirectoriesAndItsRepositorysZero
}
}
}
req := sdkstep.FetchRequest{Repository: "novox/mesh-delivery", Commit: commitSHA, Dir: tc.dir, Paths: tc.paths}
repository := tc.repository
if repository == "" {
repository = "novox/mesh-delivery"
}
req := sdkstep.FetchRequest{Repository: repository, Commit: commitSHA, Dir: tc.dir, Paths: tc.paths}
err := narrowTree(tree, req.Repository, req)
if tc.refused != "" {
if err == nil || !strings.Contains(err.Error(), tc.refused) {
@@ -25,7 +25,19 @@ import (
dforgekey "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/forgekey"
)
const forgePrefix = "git.novox.be/novox/"
// goHost is the host the forge's Go modules are named under: a module of the repository <owner>/<name> has the path
// goHost+"<owner>/<name>" or one under it.
const goHost = "git.novox.be/"
const forgePrefix = goHost + "novox/"
// sameRepository says whether the Go module path lies in the repository the fetch is of (its owner/name, as the ask
// names it): the repository's root module or one nested in it (design 47 H, S31, L8: read from the fetch, never from an
// owner name written into the code).
func sameRepository(module, repository string) bool {
prefix := goHost + repository
return repository != "" && (module == prefix || strings.HasPrefix(module, prefix+"/"))
}
// zeroShort is the 12 digits a requirement at the zero version v0.0.0-00010101000000-000000000000 ends in: no commit.
const zeroShort = "000000000000"
@@ -381,10 +393,16 @@ func (m *Module) takeZeroRequirements(ctx context.Context, root, repository, dir
}
}
}
replacedBefore := ""
if modfile, ok := m.pinModfile(dir); ok {
if b, err := os.ReadFile(modfile); err == nil {
replacedBefore = string(b)
}
}
var taken []TakenVersion
for _, r := range reqs {
p := r.pin
if v := resolved[p.module]; v != "" && !strings.HasSuffix(v, zeroShort) {
if v := resolved[p.module]; (v != "" && !strings.HasSuffix(v, zeroShort)) || strings.Contains(replacedBefore, "replace "+p.module+" =>") {
continue // taken before
}
named := "v0.0.0-00010101000000-" + zeroShort
@@ -392,10 +410,10 @@ func (m *Module) takeZeroRequirements(ctx context.Context, root, repository, dir
if r.by != "" {
namedBy = "; " + namedBySaid(r.by)
}
if "novox/"+p.repo == repository {
if sameRepository(p.module, repository) {
// A module of the checked repository itself: it is in the fetched tree and nowhere else, since nothing of the
// repository is published before its merge. Every module of the tree goes in, as one may require another.
own, err := m.useOwnModules(dir, root, p.repo)
own, err := m.useOwnModules(dir, root, repository)
if err != nil {
return taken, fmt.Errorf("the requirement of %s at the zero version: %w", p.module, err)
}
@@ -431,12 +449,79 @@ func (m *Module) takeZeroRequirements(ctx context.Context, root, repository, dir
// namedBySaid says that a module dir replaces by the directory by names a package, and dir's go.mod does not.
func namedBySaid(by string) string { return "named by " + by + ", not by this go.mod" }
// ownRequirement is a requirement of a module of the checked repository in the module graph of a directory, at the
// version its go.mod names (any version): by as in zeroRequirement.
type ownRequirement struct {
module, named, by string
}
// ownRequirementsOf are the requirements of dir's module graph (its go.mod, then the go.mod of each module it replaces by
// a directory, as zeroRequirementsOf reads them) that lie in the repository the fetch is of, at whatever version they are
// named, each once, own first. One dir's go.mod replaces is left out, since its replace decides it.
func ownRequirementsOf(dir, repository string) []ownRequirement {
raw, err := os.ReadFile(filepath.Join(dir, "go.mod"))
if err != nil {
return nil
}
_, replaced := goModRequirements(string(raw))
seen := map[string]bool{}
var out []ownRequirement
add := func(gomod, by string) {
required, _ := goModRequirements(gomod)
for _, module := range slices.Sorted(maps.Keys(required)) {
if !sameRepository(module, repository) || seen[module] || replaced[module] {
continue
}
seen[module] = true
out = append(out, ownRequirement{module: module, named: required[module], by: by})
}
}
add(string(raw), "")
local := localReplaces(string(raw))
for _, module := range slices.Sorted(maps.Keys(local)) {
if b, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(local[module]), "go.mod")); err == nil {
add(string(b), local[module])
}
}
return out
}
// takeOwnRequirements takes, before anything is downloaded, every requirement of dir's module graph that lies in the
// checked repository from the fetched tree, at whatever version go.mod names it, the zero version or a real one (design
// 47 H, S3, S31; H-A-198, H-A-202): the indexer never pins it and its fingerprint takes the tree's objects, so the build
// takes the tree too and never the Go seat's published version. One no module of the tree declares fails the fetch,
// naming it.
func (m *Module) takeOwnRequirements(root, repository, dir, d string) ([]TakenVersion, error) {
reqs := ownRequirementsOf(dir, repository)
if len(reqs) == 0 {
return nil, nil
}
own, err := m.useOwnModules(dir, root, repository)
if err != nil {
return nil, fmt.Errorf("the requirements of %s: %w", repository, err)
}
var taken []TakenVersion
for _, r := range reqs {
if !slices.Contains(own, r.module) {
return taken, fmt.Errorf("the requirement of %s at %s: no module of that path is in the fetched tree of %s, and a module of the checked repository is built from its tree", r.module, r.named, repository)
}
said := "a module of the checked repository: built from its tree"
if r.by != "" {
said += "; " + namedBySaid(r.by)
}
if m.Say != nil {
m.Say("%s: %s (go.mod names %s) is built from the fetched tree of %s", d, r.module, r.named, repository)
}
taken = append(taken, TakenVersion{Dir: d, Package: r.module, Named: r.named, Built: "the fetched tree", Said: said})
}
return taken, nil
}
// useOwnModules replaces, in dir's temporary modfile, every module of the fetched tree whose path lies in the repository
// repo by its directory in the tree (once), and answers the paths of those modules. A replace of a module dir does not
// require is harmless to Go; one module of a repository requires another of it at the zero version.
func (m *Module) useOwnModules(dir, root, repo string) ([]string, error) {
// the fetch is of (sameRepository) by its directory in the tree (once), and answers the paths of those modules. A
// replace of a module dir does not require is harmless to Go; one module of a repository requires another of it.
func (m *Module) useOwnModules(dir, root, repository string) ([]string, error) {
tree := filepath.Join(root, "tree")
prefix := forgePrefix + repo
type found struct{ module, where string }
var mods []found
err := filepath.WalkDir(tree, func(path string, e fs.DirEntry, err error) error {
@@ -458,7 +543,7 @@ func (m *Module) useOwnModules(dir, root, repo string) ([]string, error) {
}
for _, line := range strings.Split(string(raw), "\n") {
if f := strings.Fields(line); len(f) == 2 && f[0] == "module" {
if mod := strings.Trim(f[1], `"`); (mod == prefix || strings.HasPrefix(mod, prefix+"/")) && filepath.Dir(path) != dir {
if mod := strings.Trim(f[1], `"`); sameRepository(mod, repository) && filepath.Dir(path) != dir {
mods = append(mods, found{mod, filepath.Dir(path)})
}
break
@@ -82,7 +82,9 @@ func newPinHarness(t *testing.T, gomod string, vendor bool, pulls string) (*pinH
func (h *pinHarness) fetch(states *keptStates) (FetchResult, sdkstep.Step) {
h.t.Helper()
if _, err := fetchTool(h.t, h.m)(fetchArgs); err != nil {
// A member of a repository other than mesh-sdk's, whose packages it pins: one of its own repository is built from the
// tree (S31).
if _, err := fetchTool(h.t, h.m)(map[string]any{"repository": "novox/mesh-member", "commit": commitSHA}); err != nil {
h.t.Fatal(err)
}
st := states.waitEnd(h.t)
@@ -405,55 +407,64 @@ func TestARequirementAtTheZeroVersionIsTakenFromItsPinOrRefused(t *testing.T) {
}
}
// A requirement at the zero version of a module of the checked repository itself (a nested module not yet published, which
// another module of the repository requires) is built from the fetched tree, and never asked of Go or the forge (issue 1921).
// A requirement of a module of the checked repository itself (a nested module not yet published, which another module of
// the repository requires) is built from the fetched tree, and never asked of Go or the forge (issue 1921), at whatever
// version go.mod names it: the zero version or a real one, which the Go seat would otherwise serve while the indexer's
// fingerprint takes the tree (design 47 H, S31; H-A-198, H-A-202). "The checked repository" is the fetch's repository as
// the ask names it, under any owner, never an owner written into the code (L8).
// Narrowed: a fetch of one artifact of a tree shaped like mesh-delivery's (modules/mesh-delivery-contracts/go/clients/<c>
// requiring modules/mesh-delivery-sdk at the zero version), asked with the module's directory and the artifact's paths,
// keeps the sdk's directory, found by its manifest, and resolves it there (design 47 H, H-A-136; live 2026-11-20,
// pipeline main.3: "no module of that path is in the fetched tree of novox/mesh-delivery").
func TestARequirementAtTheZeroVersionOfTheCheckedRepositorysOwnModuleIsTakenFromItsTree(t *testing.T) {
// requiring modules/mesh-delivery-sdk), asked with the module's directory and the artifact's paths, keeps the sdk's
// directory, found by its manifest, and resolves it there (design 47 H, H-A-136; live 2026-11-20, pipeline main.3: "no
// module of that path is in the fetched tree of novox/mesh-delivery").
func TestARequirementOfTheCheckedRepositorysOwnModuleIsTakenFromItsTree(t *testing.T) {
zero := "v0.0.0-00010101000000-" + zeroShort
for _, tc := range []struct {
name, own, ownDir, gomods string
args map[string]any
narrowed bool
name, own, ownDir, gomods, version string
args map[string]any
narrowed bool
}{
{name: "whole tree", own: "git.novox.be/novox/mesh-delivery/nested/go", ownDir: "nested/go", gomods: "go.mod\n",
{name: "whole tree", own: "git.novox.be/novox/mesh-delivery/nested/go", ownDir: "nested/go", gomods: "go.mod\n", version: zero,
args: map[string]any{"repository": "novox/mesh-delivery", "commit": commitSHA}},
{name: "whole tree, at a real version", own: "git.novox.be/novox/mesh-delivery/nested/go", ownDir: "nested/go", gomods: "go.mod\n",
version: "v0.1.3", args: map[string]any{"repository": "novox/mesh-delivery", "commit": commitSHA}},
{name: "a repository of another owner, judged by the ask's repository", own: "git.novox.be/kin/app/nested/go", ownDir: "nested/go",
gomods: "go.mod\n", version: zero, args: map[string]any{"repository": "kin/app", "commit": commitSHA}},
{name: "narrowed to an artifact's paths", own: "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk",
ownDir: "modules/mesh-delivery-sdk", gomods: "modules/mesh-delivery-contracts/go/clients/meshcatalogclient/go.mod\n",
ownDir: "modules/mesh-delivery-sdk", gomods: "modules/mesh-delivery-contracts/go/clients/meshcatalogclient/go.mod\n", version: zero,
args: map[string]any{"repository": "novox/mesh-delivery", "commit": commitSHA, "dir": "modules/mesh-delivery-contracts",
"paths": []any{"go/clients/meshcatalogclient"}}, narrowed: true},
{name: "narrowed to an artifact's paths, at a real version", own: "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk",
ownDir: "modules/mesh-delivery-sdk", gomods: "modules/mesh-delivery-contracts/go/clients/meshcatalogclient/go.mod\n", version: "v0.1.3",
args: map[string]any{"repository": "novox/mesh-delivery", "commit": commitSHA, "dir": "modules/mesh-delivery-contracts",
"paths": []any{"go/clients/meshcatalogclient"}}, narrowed: true},
} {
t.Run(tc.name, func(t *testing.T) {
zeroMod := "module fixture/member\n\ngo 1.26\n\nrequire (\n\t" + tc.own + " v0.0.0-00010101000000-" + zeroShort + "\n)\n"
h, states := newPinHarness(t, zeroMod, false, pullList)
ownMod := "module fixture/member\n\ngo 1.26\n\nrequire (\n\t" + tc.own + " " + tc.version + "\n)\n"
h, states := newPinHarness(t, ownMod, false, pullList)
h.f.gomods = tc.gomods
base := h.m.Run
h.m.Run = func(ctx context.Context, env []string, name string, args ...string) forgekey.Ran {
line := name + " " + strings.Join(args, " ")
if name == "git" && strings.Contains(line, " checkout ") && tc.narrowed {
if name == "git" && strings.Contains(line, " checkout ") {
r := base(ctx, env, name, args...)
tree := args[slices.Index(args, "-C")+1]
for f, body := range map[string]string{
"modules/mesh-delivery-sdk/go.mod": "module " + tc.own + "\n\ngo 1.26\n",
"modules/mesh-delivery-sdk/module.json": `{"module":"mesh-delivery-sdk","build":{"artifacts":[{"name":"go-sdk","kind":"package","language":"go"}]}}`,
"modules/mesh-delivery-contracts/module.json": `{"module":"mesh-delivery-contracts","build":{"artifacts":[{"name":"go-meshcatalogclient","kind":"package","language":"go","paths":["go/clients/meshcatalogclient"]}]}}`,
"modules/mesh-delivery-view/go.mod": "module git.novox.be/novox/mesh-delivery/modules/mesh-delivery-view\n\ngo 1.26\n",
} {
files := map[string]string{tc.ownDir + "/go.mod": "module " + tc.own + "\n\ngo 1.26\n"}
if tc.narrowed {
files["modules/mesh-delivery-sdk/module.json"] = `{"module":"mesh-delivery-sdk","build":{"artifacts":[{"name":"go-sdk","kind":"package","language":"go"}]}}`
files["modules/mesh-delivery-contracts/module.json"] = `{"module":"mesh-delivery-contracts","build":{"artifacts":[{"name":"go-meshcatalogclient","kind":"package","language":"go","paths":["go/clients/meshcatalogclient"]}]}}`
files["modules/mesh-delivery-view/go.mod"] = "module git.novox.be/novox/mesh-delivery/modules/mesh-delivery-view\n\ngo 1.26\n"
}
for f, body := range files {
os.MkdirAll(filepath.Join(tree, filepath.Dir(f)), 0o755)
os.WriteFile(filepath.Join(tree, f), []byte(body), 0o644)
}
return r
}
if name == "go" && strings.Contains(line, " mod download") && !strings.Contains(strings.Join(env, " "), "-modfile=") {
if !tc.narrowed {
trees, _ := filepath.Glob(filepath.Join(h.m.Work, "*", "tree"))
for _, tree := range trees {
os.MkdirAll(filepath.Join(tree, tc.ownDir), 0o755)
os.WriteFile(filepath.Join(tree, tc.ownDir, "go.mod"), []byte("module "+tc.own+"\n\ngo 1.26\n"), 0o644)
}
if tc.version != zero {
return forgekey.Ran{} // the Go seat serves the published version: what a build of the tree must never take
}
return forgekey.Ran{Status: 1, Stderr: "go: " + tc.own + "@v0.0.0-00010101000000-" + zeroShort + ": invalid version: unknown revision " + zeroShort + "\n"}
return forgekey.Ran{Status: 1, Stderr: "go: " + tc.own + "@" + zero + ": invalid version: unknown revision " + zeroShort + "\n"}
}
return base(ctx, env, name, args...)
}
@@ -485,6 +496,11 @@ func TestARequirementAtTheZeroVersionOfTheCheckedRepositorysOwnModuleIsTakenFrom
if h.ran(" mod download -json ") || h.ran("ls-remote") {
t.Error("a module of the checked repository was asked of Go or looked for among the pull request heads")
}
var res FetchResult
json.Unmarshal(st.Result, &res)
if len(res.Packages) != 1 || res.Packages[0].Package != tc.own || res.Packages[0].Built != "the fetched tree" || res.Packages[0].Named != tc.version {
t.Errorf("the taken versions = %+v, want %s named %s, built from the fetched tree", res.Packages, tc.own, tc.version)
}
})
}
}
@@ -58,10 +58,10 @@ type downloadInfo struct {
}
// takeVersions writes the asked versions into the modfile of dir (d in the tree) and answers what it took, one entry per
// asked package its go.mod requires. A package go.mod replaces is built as go.mod says, and said. A version that can be
// asked package its go.mod requires that is not of the checked repository repository. A package go.mod replaces is built as go.mod says, and said. A version that can be
// neither fetched nor found among the forge's pull request heads fails the fetch, naming it. pins serialises the
// laid copies a workspace's directories share.
func (m *Module) takeVersions(ctx context.Context, root, dir, d string, asked sdkstep.PackageVersions, pins *sync.Mutex) ([]TakenVersion, error) {
func (m *Module) takeVersions(ctx context.Context, root, repository, dir, d string, asked sdkstep.PackageVersions, pins *sync.Mutex) ([]TakenVersion, error) {
raw, err := os.ReadFile(filepath.Join(dir, "go.mod"))
if err != nil {
return nil, nil
@@ -77,8 +77,8 @@ func (m *Module) takeVersions(ctx context.Context, root, dir, d string, asked sd
var taken []TakenVersion
for _, pkg := range slices.Sorted(maps.Keys(asked)) {
named, ok := required[pkg]
if !ok {
continue // the module does not use it
if !ok || sameRepository(pkg, repository) {
continue // the module does not use it, or it is of the checked repository: built from its tree (takeOwnRequirements)
}
tv := TakenVersion{Dir: d, Package: pkg, Asked: asked[pkg], Named: named}
switch {
@@ -58,17 +58,33 @@ func realMachine(t *testing.T) (*Module, string) {
t.Fatalf("the toolchain copied from %s cannot be used: %v", from, err)
}
for _, id := range []string{"w1", "w2", "w3"} {
for p, body := range map[string]string{
files := map[string]string{
"modules/m/module.json": `{"module":"m","build":{"artifacts":[{"name":"tools","kind":"bundle","language":"typescript","entrypoints":["tools/index.js"]}]}}`,
"modules/m/package.json": `{"name":"m","type":"module","scripts":{"test":"node -e \"process.exit(0)\""}}`,
"modules/m/tools/index.ts": "import { serveRegisteredOverStdio } from \"@novox/mesh-sdk/stdio\";\nexport const f: () => Promise<void> = serveRegisteredOverStdio as () => Promise<void>;\n",
} {
}
for p, body := range fetchedSDK {
files["modules/m/node_modules/@novox/mesh-sdk/"+p] = body
}
for p, body := range files {
write(t, filepath.Join(work, id, "tree", filepath.FromSlash(p)), body, 0o644)
}
}
return m, work
}
// fetchedSDK is the SDK the fixture's workspaces hold where the fetch step places it, in the module's node_modules
// (fetch.go, placeSDK: the latest published release, ADR 0573 §2; design 47 G §571), made here because the integration
// level has no registry to fetch it from. The toolchain's lock names no SDK since §571, so a fixture that leaves it out
// reads one only from a toolchain installed before (main.137 and main.155: TS2307 once a build node installed the
// lock). It holds the one export the fixture and the launchers import.
var fetchedSDK = map[string]string{
"package.json": `{"name":"@novox/mesh-sdk","version":"9.9.9","type":"module",` +
`"exports":{"./stdio":{"types":"./dist/stdio.d.ts","default":"./dist/stdio.js"}}}`,
"dist/stdio.js": "export async function serveRegisteredOverStdio() {}\n",
"dist/stdio.d.ts": "export declare function serveRegisteredOverStdio(): Promise<void>;\n",
}
// 1490/AC2, 1491 and issue 1869, in one test so the toolchain is copied once: the real type-check and test script of the
// fixture pass the unit level; a package whose test imports its own build output (as mesh-sdk's do) passes the unit level,
// the test step emitting it with the real tsc, shut off, before the test script runs; and the fixture TypeScript bundle
@@ -245,6 +245,11 @@ func (m *Module) fetchInto(ctx context.Context, req sdkstep.FetchRequest, group
return nil, err
}
}
// Every other mesh package it names, at the latest release (design 47 H, H-A-267); the group's and the
// members' builds are placed over them.
if err := m.placeMeshPackages(ctx, dir, d, served, members); err != nil {
return nil, err
}
if err := placeGroup(dir, served); err != nil {
return nil, err
}
@@ -2,6 +2,8 @@ package main
import (
"context"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"slices"
@@ -97,9 +99,17 @@ func TestAModuleNamingOnlyTheSdkAndTypesInstallsNothing(t *testing.T) {
}
// 47/G A-72: a module naming another @novox/mesh-* package beside the SDK has no package beyond the SDK, so it installs
// nothing and is not refused for want of a lock.
// nothing and is not refused for want of a lock. 47/H H-A-267 (ADR 0573 §2): a fetch with no group and no members' builds
// places that package at the latest release the registry holds, as it places the SDK, whatever the module's lock names.
func TestAMeshPackageIsNotBeyondTheSdk(t *testing.T) {
m, f := fetchMachine(t, typeScriptModule(`{"dependencies":{"@novox/mesh-sdk":"^0.1.6","@novox/mesh-tools":"^0.1.6"}}`), nil)
const contracts = "@novox/mesh-delivery-contracts-git"
files := typeScriptModule(`{"dependencies":{"@novox/mesh-sdk":"^0.1.6","` + contracts + `":"^0.1.2"}}`)
files["modules/m/package-lock.json"] = `{"lockfileVersion":3,"packages":{"node_modules/` + contracts + `":{"version":"0.1.2","integrity":"sha512-old"}}}`
m, f := fetchMachine(t, files, nil)
fakeRegistryOf(t, m, map[string]map[string][]byte{
sdkPackage: {"0.1.15": tgzOf(t, sdkPackage, "0.1.15")},
contracts: {"0.1.2": tgzOf(t, contracts, "0.1.2"), "0.1.4": tgzOf(t, contracts, "0.1.4"), "0.1.5-feat.1": tgzOf(t, contracts, "0.1.5-feat.1")},
})
out, err := m.fetch(context.Background(), fetchArgs)
if err != nil {
t.Fatal(err)
@@ -108,6 +118,42 @@ func TestAMeshPackageIsNotBeyondTheSdk(t *testing.T) {
if len(npmCalls(f)) != 0 || len(r.Installed) != 0 || len(r.Refused) != 0 {
t.Errorf("result = %+v, npm calls %d, want nothing installed and nothing refused", r, len(npmCalls(f)))
}
dir := filepath.Join(m.Work, r.Workspace, "tree", "modules", "m")
got, err := readPackageJSON(filepath.Join(dir, "node_modules", filepath.FromSlash(contracts)))
if err != nil || got.Version != "0.1.4" {
t.Errorf("%s placed %q (%v), want the latest release 0.1.4", contracts, got.Version, err)
}
if v := placedSDK(dir); v != "0.1.15" || r.SDK["modules/m"] != "0.1.15" {
t.Errorf("the SDK placed %q, answered %q, want the latest release 0.1.15", v, r.SDK["modules/m"])
}
// A member's build of a mesh package takes precedence: it is laid, and the registry, holding none of it, is not
// waited on.
const member = "@novox/mesh-tools"
m, _ = fetchMachine(t, typeScriptModule(`{"dependencies":{"@novox/mesh-sdk":"^0.1.6","`+member+`":"^0.1.6"}}`), nil)
tgz := tgzOf(t, member, "0.0.0-member")
digest := digestOf(tgz)
store := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/v2/tools/pkg/blobs/"+digest {
http.NotFound(w, r)
return
}
w.Write(tgz)
}))
defer store.Close()
m.ArtifactStoreFile = filepath.Join(t.TempDir(), "artifact-store.json")
write(t, m.ArtifactStoreFile, `{"at":"`+strings.TrimPrefix(store.URL, "http://")+`"}`, 0o600)
args := map[string]any{"repository": fetchArgs["repository"], "commit": fetchArgs["commit"],
"builds": []any{map[string]any{"package": member, "module": "tools", "artifact": "pkg", "digest": digest}}}
out, err = m.fetch(context.Background(), args)
if err != nil {
t.Fatalf("a fetch with a member's build of %s: %v", member, err)
}
r = out.(FetchResult)
got, err = readPackageJSON(filepath.Join(m.Work, r.Workspace, "tree", "modules", "m", "node_modules", filepath.FromSlash(member)))
if err != nil || got.Version != "0.0.0-member" {
t.Errorf("%s placed %q (%v), want the member's build", member, got.Version, err)
}
}
// 1489/AC2: a module naming lodash with a lock gets npm ci --omit=dev --ignore-scripts (fake runner), its install put
@@ -364,9 +364,9 @@ func placeGroup(dir string, served map[string]*packed) error {
return nil
}
// sdkLocked is what a module's lock names of the SDK: its version, its integrity and where it was resolved from; ok is
// false when the lock names none (the latest release is then placed, placeSDK).
func sdkLocked(dir string) (version, integrity string, ok bool, err error) {
// locked is what a module's lock names of the package name: its version and its integrity; ok is false when the lock
// names none (the latest release is then placed, placeLatest).
func locked(dir, name string) (version, integrity string, ok bool, err error) {
raw, err := os.ReadFile(filepath.Join(dir, "package-lock.json"))
if os.IsNotExist(err) {
return "", "", false, nil
@@ -383,7 +383,7 @@ func sdkLocked(dir string) (version, integrity string, ok bool, err error) {
if err := json.Unmarshal(raw, &lock); err != nil {
return "", "", false, fmt.Errorf("the package-lock.json is not JSON: %w", err)
}
p, ok := lock.Packages["node_modules/"+sdkPackage]
p, ok := lock.Packages["node_modules/"+name]
if !ok || p.Version == "" {
return "", "", false, nil
}
@@ -398,51 +398,197 @@ func sdkLocked(dir string) (version, integrity string, ok bool, err error) {
// none. The lock's integrity is checked when the lock names the registry's version placed; a registry holding no
// release is a wait (`waits for:`).
func (m *Module) placeSDK(ctx context.Context, dir, rel string, group *packed) error {
version, integrity, _, err := sdkLocked(dir)
if group == nil {
return m.placeLatest(ctx, dir, rel, sdkPackage)
}
version, _, _, err := locked(dir, sdkPackage)
if err != nil {
return err
}
var tgz []byte
if group != nil {
if m.Say != nil && version != "" && version != group.Version {
m.Say("%s: its lock names %s@%s; the group's %s is placed (design 47 G A-284)", rel, sdkPackage, version, group.Version)
}
tgz = group.tgz
} else {
latest, err := m.registryLatest(ctx, sdkPackage)
if err != nil {
return err
}
if latest == "" {
return fmt.Errorf("%s%s", builds.WaitsFor, sdkPackage)
}
if version != latest {
if m.Say != nil && version != "" {
m.Say("%s: its lock names %s@%s; the latest published %s is placed (ADR 0573 §2)", rel, sdkPackage, version, latest)
}
version, integrity = latest, ""
}
want := sdkPackage + "@" + version
got, registered, held, err := m.registryPackageChecked(ctx, sdkPackage, version)
if err != nil {
return err
}
if !held {
return fmt.Errorf("%s%s", builds.WaitsFor, want)
}
if integrity == "" {
integrity = registered
}
if integrity != "" && integrityOf(got) != integrity {
return fmt.Errorf("the registry's %s is %s, and the lock of %s names %s", want, integrityOf(got), rel, integrity)
}
tgz = got
if m.Say != nil && version != "" && version != group.Version {
m.Say("%s: its lock names %s@%s; the group's %s is placed (design 47 G A-284)", rel, sdkPackage, version, group.Version)
}
into := filepath.Join(dir, "node_modules", filepath.FromSlash(sdkPackage))
if err := os.RemoveAll(into); err != nil {
return err
}
return unpackPackage(tgz, into)
return unpackPackage(group.tgz, into)
}
// placeMeshPackages places in the module's node_modules each mesh package beyond the SDK (@novox/mesh-*) its
// package.json names at the latest release the registry holds, as placeSDK does for the SDK (ADR 0573 §2; design 47 H,
// H-A-267): the install leaves them out (ownDependencies). One the group serves or a member's build lays is left to
// placeGroup and layBuilds, which take precedence; the module itself is not placed.
func (m *Module) placeMeshPackages(ctx context.Context, dir, rel string, served map[string]*packed, members []storedBuild) error {
self, _ := readPackageJSON(dir)
var names []string
for name := range namedPackages(dir) {
if name == sdkPackage || !strings.HasPrefix(name, "@novox/mesh-") || name == self.Name || served[name] != nil || namesPackage(members, name) {
continue
}
names = append(names, name)
}
sort.Strings(names)
for _, name := range names {
if err := m.placeLatest(ctx, dir, rel, name); err != nil {
return err
}
}
return nil
}
// placeLatest places in the module's node_modules the latest release of the mesh package name the registry holds,
// whatever the module's lock names (ADR 0573 §2), checked against the lock's integrity when the lock names that
// version; a registry holding no release is a wait (`waits for:`).
func (m *Module) placeLatest(ctx context.Context, dir, rel, name string) error {
version, integrity, _, err := locked(dir, name)
if err != nil {
return err
}
latest, err := m.registryLatest(ctx, name)
if err != nil {
return err
}
if latest == "" {
return fmt.Errorf("%s%s", builds.WaitsFor, name)
}
if version != latest {
if m.Say != nil && version != "" {
m.Say("%s: its lock names %s@%s; the latest published %s is placed (ADR 0573 §2)", rel, name, version, latest)
}
version, integrity = latest, ""
}
if spec := namedRange(dir, name); spec != "" && m.Say != nil {
// Said, never refused: the latest release is placed whatever the range (ADR 0573 §2; design 47 H, H-A-276).
switch in, read := inRange(spec, version); {
case !read:
m.Say("%s: its package.json names %s at %q, a range not read; the latest published %s is placed (ADR 0573 §2)", rel, name, spec, version)
case !in:
m.Say("%s: the latest published %s@%s lies outside the range %q its package.json names; it is placed (ADR 0573 §2)", rel, name, version, spec)
}
}
want := name + "@" + version
got, registered, held, err := m.registryPackageChecked(ctx, name, version)
if err != nil {
return err
}
if !held {
return fmt.Errorf("%s%s", builds.WaitsFor, want)
}
if integrity == "" {
integrity = registered
}
if integrity != "" && integrityOf(got) != integrity {
return fmt.Errorf("the registry's %s is %s, and the lock of %s names %s", want, integrityOf(got), rel, integrity)
}
into := filepath.Join(dir, "node_modules", filepath.FromSlash(name))
if err := os.RemoveAll(into); err != nil {
return err
}
return unpackPackage(got, into)
}
// namedRange is the range the module's package.json names of name: its dependencies first, then its peer, optional and
// dev dependencies; "" when it names none.
func namedRange(dir, name string) string {
raw, err := os.ReadFile(filepath.Join(dir, "package.json"))
if err != nil {
return ""
}
var p map[string]any
if json.Unmarshal(raw, &p) != nil {
return ""
}
for _, k := range []string{"dependencies", "peerDependencies", "optionalDependencies", "devDependencies"} {
deps, _ := p[k].(map[string]any)
if spec, ok := deps[name].(string); ok {
return strings.TrimSpace(spec)
}
}
return ""
}
// inRange says whether the release v (major.minor.patch) lies in the npm range spec; read is false for a range of a
// form it does not read (a union, a hyphen range, a comparator, a tag other than latest), never taken as in or out. It
// reads what a package file of the mesh names: an exact version, an x-range, a caret and a tilde range, as npm does.
func inRange(spec, v string) (in, read bool) {
spec = strings.TrimSpace(spec)
if spec == "" || spec == "*" || spec == "latest" {
return true, true
}
if strings.ContainsAny(spec, " |<>") {
return false, false
}
op := ""
if spec[0] == '^' || spec[0] == '~' {
op, spec = spec[:1], spec[1:]
}
spec = strings.TrimPrefix(strings.TrimPrefix(spec, "="), "v")
if strings.Contains(spec, "-") || strings.Contains(spec, "+") {
return false, false // a pre-release or build range
}
var low [3]int
given := 0
for i, part := range strings.Split(spec, ".") {
if i > 2 {
return false, false
}
if part == "x" || part == "X" || part == "*" {
break
}
n, err := strconv.Atoi(part)
if err != nil || n < 0 {
return false, false
}
low[i] = n
given = i + 1
}
if given == 0 {
return true, true // x, x.x
}
var at [3]int
for i, part := range strings.Split(v, ".") {
if i > 2 {
return false, false
}
n, err := strconv.Atoi(part)
if err != nil {
return false, false
}
at[i] = n
}
// bump is the part of low raised by one for the exclusive upper bound, the parts after it zero.
bump := given - 1
switch op {
case "^":
switch {
case low[0] > 0 || given == 1:
bump = 0
case low[1] > 0 || given == 2:
bump = 1
default:
bump = 2
}
case "~":
bump = 0
if given > 1 {
bump = 1
}
}
high := low
high[bump]++
for i := bump + 1; i < 3; i++ {
high[i] = 0
}
less := func(a, b [3]int) bool {
for i := range a {
if a[i] != b[i] {
return a[i] < b[i]
}
}
return false
}
return !less(at, low) && less(at, high), true
}
// placedSDK is the version of the SDK placed in the module's node_modules ("" when none is).
@@ -6,6 +6,7 @@ import (
"compress/gzip"
"context"
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"os"
@@ -33,34 +34,40 @@ func tgzOf(t *testing.T, name, version string) []byte {
return buf.Bytes()
}
// fakeRegistry serves the packument and the packed files of the versions it holds; asked counts its answers.
// fakeRegistry serves the packument and the packed files of the SDK's versions it holds; asked counts its answers.
func fakeRegistry(t *testing.T, m *Module, held map[string][]byte) *atomic.Int32 {
t.Helper()
return fakeRegistryOf(t, m, map[string]map[string][]byte{sdkPackage: held})
}
// fakeRegistryOf serves the packument and the packed files of each package's versions it holds (by name); asked counts
// its answers.
func fakeRegistryOf(t *testing.T, m *Module, held map[string]map[string][]byte) *atomic.Int32 {
t.Helper()
var asked atomic.Int32
var srv *httptest.Server
srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
asked.Add(1)
switch {
case r.URL.Path == "/npm/@novox%2fmesh-sdk" || r.URL.Path == "/npm/@novox/mesh-sdk":
if len(held) == 0 {
http.NotFound(w, r)
return
}
versions := map[string]any{}
for v, tgz := range held {
versions[v] = map[string]any{"dist": map[string]any{"tarball": srv.URL + "/files/" + v + ".tgz", "integrity": integrityOf(tgz)}}
}
json.NewEncoder(w).Encode(map[string]any{"versions": versions})
case strings.HasPrefix(r.URL.Path, "/files/"):
v := strings.TrimSuffix(strings.TrimPrefix(r.URL.Path, "/files/"), ".tgz")
if tgz, ok := held[v]; ok {
if f, ok := strings.CutPrefix(r.URL.Path, "/files/"); ok {
name, v, _ := strings.Cut(strings.TrimSuffix(f, ".tgz"), "@@")
if tgz, ok := held[strings.ReplaceAll(name, "~", "/")][v]; ok {
w.Write(tgz)
return
}
http.NotFound(w, r)
default:
http.NotFound(w, r)
return
}
name, ok := strings.CutPrefix(r.URL.Path, "/npm/")
name = strings.ReplaceAll(name, "%2f", "/")
if !ok || len(held[name]) == 0 {
http.NotFound(w, r)
return
}
versions := map[string]any{}
for v, tgz := range held[name] {
versions[v] = map[string]any{"dist": map[string]any{"tarball": srv.URL + "/files/" + strings.ReplaceAll(name, "/", "~") + "@@" + v + ".tgz", "integrity": integrityOf(tgz)}}
}
json.NewEncoder(w).Encode(map[string]any{"versions": versions})
}))
t.Cleanup(srv.Close)
m.Toolchain.Registry = func() (Registry, error) { return Registry{URL: srv.URL + "/npm/"}, nil }
@@ -183,3 +190,46 @@ func TestTheGroupPackagesArgumentIsRefusedWhenItIsNotOne(t *testing.T) {
t.Errorf("a member's root and a member's directory: %+v %v", gs, err)
}
}
// design 47 H, H-A-276 (amending H-A-267): the latest release placed outside the range the package file names is said in
// the journal, and placed all the same; one inside it is not said; a range of a form not read is said as such.
func TestALatestReleaseOutsideItsRangeIsSaid(t *testing.T) {
for _, c := range []struct {
spec, v string
in, read bool
}{
{"^0.1.0", "0.1.9", true, true}, {"^0.1.0", "0.2.0", false, true}, {"^1.2.3", "1.9.0", true, true},
{"^1.2.3", "2.0.0", false, true}, {"^0.0.3", "0.0.4", false, true}, {"^0.x", "0.9.0", true, true},
{"~0.1.2", "0.1.9", true, true}, {"~0.1.2", "0.2.0", false, true}, {"~1", "1.5.0", true, true},
{"0.1.4", "0.1.4", true, true}, {"0.1.4", "0.1.5", false, true}, {"0.1.x", "0.1.7", true, true},
{"0.1", "0.2.0", false, true}, {"*", "9.9.9", true, true}, {"^0.1.0", "0.0.9", false, true},
{">=0.1.0", "0.2.0", false, false}, {"^0.1.0 || ^0.2.0", "0.2.0", false, false}, {"file:../x", "0.1.0", false, false},
} {
if in, read := inRange(c.spec, c.v); in != c.in || read != c.read {
t.Errorf("inRange(%q, %q) = %v, %v; want %v, %v", c.spec, c.v, in, read, c.in, c.read)
}
}
m, _, _ := newMachine(t)
const name = "@novox/mesh-delivery-contracts-git"
fakeRegistryOf(t, m, map[string]map[string][]byte{name: {"0.1.9": tgzOf(t, name, "0.1.9"), "0.2.0": tgzOf(t, name, "0.2.0")}})
for _, c := range []struct{ spec, said string }{
{"^0.1.0", `the latest published ` + name + `@0.2.0 lies outside the range "^0.1.0"`},
{"^0.2.0", ""},
{">=0.1.0", `at ">=0.1.0", a range not read`},
} {
var said []string
m.Say = func(f string, a ...any) { said = append(said, fmt.Sprintf(f, a...)) }
dir := t.TempDir()
write(t, filepath.Join(dir, "package.json"), `{"dependencies":{"`+name+`":"`+c.spec+`"}}`, 0o644)
if err := m.placeLatest(context.Background(), dir, "modules/m", name); err != nil {
t.Fatalf("%s: %v", c.spec, err)
}
if p, _ := readPackageJSON(filepath.Join(dir, "node_modules", filepath.FromSlash(name))); p.Version != "0.2.0" {
t.Errorf("%s: placed %q, want the latest release 0.2.0", c.spec, p.Version)
}
joined := strings.Join(said, "\n")
if c.said == "" && joined != "" || c.said != "" && !strings.Contains(joined, c.said) {
t.Errorf("%s: said %q, want %q", c.spec, joined, c.said)
}
}
}
+3
View File
@@ -649,6 +649,9 @@ func (h *Holder) holdWhy(a oc.Ask) string {
}
parts = append(parts, why)
}
case answerLevel(a) == occ.Destroy:
// Design 47 H-A-237 (3): the held ask names its level, which no channel carries yet (carriers).
parts = append(parts, Display(k.Kind)+": the ask is at the destroy level, and no channel carries a destroy answer yet (it needs a code)")
}
}
if len(parts) == 0 {
@@ -149,6 +149,12 @@ func TestAFailedRootAskIsSaidAndKeepsAFreshWord(t *testing.T) {
if why := h.holdWhy(heldAsk(c, "d-9")); !strings.Contains(why, "no responders") {
t.Errorf("a held ask does not say why: %q", why)
}
// Design 47 H-A-237 (3): an ask held at the destroy level says its level, which no channel carries yet.
destroy := heldAsk(c, "d-10")
destroy.Options[0].Level = occ.Destroy
if why := h.holdWhy(destroy); !strings.Contains(why, "destroy level") || !strings.Contains(why, "no channel carries a destroy answer yet") {
t.Errorf("a destroy ask held does not name its level: %q", why)
}
}
// H3: a message rendered while root is not free offers no approval; a word kept for rendering is never the
+9 -5
View File
@@ -45,11 +45,15 @@ func AsksOf(e catalog.CatalogEntry) (Asks, error) {
return Asks{}, fmt.Errorf("the manifest of %s is not readable: %w", e.Module, err)
}
state := m.BusState
// The old sections' entries are asked for too (design 47 H H-A-229, H-A-230), every manifest's, named and capped as
// the grants part composes them.
m.Module = cmp.Or(m.Module, e.Module)
old := keeper.OldPlansOf(m)
if e.Verdict != "" && e.Verdict != "accepted" {
state = nil
state, old = nil, nil
}
sum := sha256.Sum256(mustJSON(state))
out := Asks{Accepted: keeper.Accepted{Module: e.Module, Digest: hex.EncodeToString(sum[:]), BusState: state},
out := Asks{Accepted: keeper.Accepted{Module: e.Module, Digest: hex.EncodeToString(sum[:]), BusState: state, Old: old},
EmitsOwn: len(m.Emits) > 0 || len(e.Streams) > 0}
for _, c := range m.Claims {
out.Claims = append(out.Claims, c.Name)
@@ -81,7 +85,7 @@ type Branch struct {
// branches in its registration's order (order: the registration's branches; a branch it does not name comes after, in
// the order given) whose entry asks it; another branch asking that local name with other settings is a drift naming
// the branch, and plans nothing. The module sends when a branch's entry says so; it is a foundation part, and its old
// sections' caps are read, from the first branch.
// sections' caps and entries are read, from the first branch (design 47 H H-A-229).
func ViewOf(branches []Branch, order func(module string) []string) (keeper.CatalogView, error) {
modules := map[string]bool{}
for _, b := range branches {
@@ -117,8 +121,8 @@ func ViewOf(branches []Branch, order func(module string) []string) (keeper.Catal
return keeper.CatalogView{}, fmt.Errorf("the branch %s: %w", b.Name, err)
}
if i == 0 {
cm = keeper.CatalogModule{Accepted: keeper.Accepted{Module: module, Digest: b.Entries[module].ManifestDigest, Revision: uint64(max(b.Revision, 0))},
Foundation: a.Foundation, OldBytes: a.OldBytes}
cm = keeper.CatalogModule{Accepted: keeper.Accepted{Module: module, Digest: b.Entries[module].ManifestDigest, Revision: uint64(max(b.Revision, 0)),
Old: a.Accepted.Old}, Foundation: a.Foundation, OldBytes: a.OldBytes}
}
cm.EmitsOwn = cm.EmitsOwn || a.EmitsOwn
for _, c := range a.Claims {
@@ -1,9 +1,14 @@
package main
import (
"encoding/json"
"fmt"
"slices"
"testing"
catalog "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"nats-tools/internal/keeper"
)
func TestAsksOf(t *testing.T) {
@@ -22,6 +27,42 @@ func TestAsksOf(t *testing.T) {
if b.Accepted.Digest != a.Accepted.Digest {
t.Errorf("the digest is of the bus-state asked alone: %s vs %s", b.Accepted.Digest, a.Accepted.Digest)
}
// Design 47 H H-A-229, H-A-230: the old sections of the artifact registry and the forge, as their manifests declare
// them, are asked for under the bus names the grants part gives them, the names the live orphans of 2026-11-21 hold;
// a manifest not accepted asks none.
for _, row := range []struct{ module, sections, want string }{
{"distribution", `{"state": ["transfers", {"name": "idle", "key-ttl": true}, "holds", "windows", "mirrors", "findings"],
"streams": [{"name": "distribution-events", "events": ["old-name-closed"], "max-age-days": 7, "max-mib": 16}],
"objects": [{"name": "incoming", "max-mib": 8192}]}`,
"[KV_distribution_transfers KV_distribution_idle KV_distribution_holds KV_distribution_windows KV_distribution_mirrors KV_distribution_findings OBJ_distribution_incoming DISTRIBUTION_EVENTS]"},
{"gitea", `{"state": ["bound", {"name": "idle", "key-ttl": true}, "transfers"], "objects": [{"name": "incoming", "max-mib": 2048}],
"logs": [{"name": "calls", "max-mib": 8}]}`,
"[KV_gitea_bound KV_gitea_idle KV_gitea_transfers OBJ_gitea_incoming LOG_gitea_calls]"},
} {
var m map[string]any
if err := json.Unmarshal([]byte(row.sections), &m); err != nil {
t.Fatal(err)
}
m["module"] = row.module
a, err := AsksOf(catalog.CatalogEntry{Module: row.module, Verdict: "accepted", Manifest: m})
if err != nil {
t.Fatal(err)
}
var names []string
for _, p := range a.Accepted.Old {
names = append(names, p.Objects[0].Stream())
}
if fmt.Sprint(names) != row.want {
t.Errorf("%s asks %v of its old sections, want %s", row.module, names, row.want)
}
i := slices.IndexFunc(a.Accepted.Old, func(p keeper.Plan) bool { return p.Local == "incoming" })
if i < 0 || a.Accepted.Old[i].Objects[0].Settings.MaxBytes != int64(m["objects"].([]any)[0].(map[string]any)["max-mib"].(float64))*keeper.MiB {
t.Errorf("%s's incoming is not capped as declared: %+v", row.module, a.Accepted.Old)
}
if n, _ := AsksOf(catalog.CatalogEntry{Module: row.module, Verdict: "refused", Manifest: m}); len(n.Accepted.Old) != 0 {
t.Errorf("a refused manifest of %s asks %v", row.module, n.Accepted.Old)
}
}
if _, err := AsksOf(catalog.CatalogEntry{}); err == nil {
t.Error("an entry with no module was read")
}
+7 -7
View File
@@ -21,7 +21,7 @@ import (
// newest revision lists, each local name from the first of its watched branches in its registration's order whose entry
// asks it, another branch asking it with other settings a drift naming that branch; a branch the registration does not
// name after; a verdict not accepted asks nothing; the claims of every branch; a foundation part and its old sections'
// caps from the first branch.
// caps and entries from the first branch (design 47 H H-A-229).
func TestTheCatalogHalfIsReadPerWatchedBranch(t *testing.T) {
entryOf := func(verdict string, manifest map[string]any) catalog.CatalogEntry {
manifest["module"] = "m"
@@ -44,17 +44,17 @@ func TestTheCatalogHalfIsReadPerWatchedBranch(t *testing.T) {
want string
}{
{"one branch", []Branch{{Name: "main", Revision: 3, Entries: map[string]catalog.CatalogEntry{"m": main}}}, []string{"main"},
"m 3 [drafts/key-value/state] [a] false 0;"},
"m 3 [drafts/key-value/state] [a] false 0 0;"},
{"a feature branch asking a local name with other settings drifts, its new one planned",
[]Branch{{Name: "feature", Revision: 9, Entries: map[string]catalog.CatalogEntry{"m": feature}}, {Name: "main", Revision: 3, Entries: map[string]catalog.CatalogEntry{"m": main}}},
[]string{"main", "feature"}, "m 3 [drafts/key-value/state log/stream/log] [a b] false 0; drifted m drafts feature from main"},
[]string{"main", "feature"}, "m 3 [drafts/key-value/state log/stream/log] [a b] false 0 0; drifted m drafts feature from main"},
{"the registration's order decides", []Branch{{Name: "main", Revision: 3, Entries: map[string]catalog.CatalogEntry{"m": main}}, {Name: "feature", Revision: 9, Entries: map[string]catalog.CatalogEntry{"m": feature}}},
[]string{"feature"}, "m 9 [drafts/key-value/state log/stream/log] [b a] false 0; drifted m drafts main from feature"},
[]string{"feature"}, "m 9 [drafts/key-value/state log/stream/log] [b a] false 0 0; drifted m drafts main from feature"},
{"a verdict not accepted asks nothing", []Branch{{Name: "main", Revision: 3, Entries: map[string]catalog.CatalogEntry{"m": entryOf("refused", map[string]any{"bus-state": []any{state("")}})}}}, []string{"main"},
"m 3 [] [] false 0;"},
"m 3 [] [] false 0 0;"},
{"a foundation part, its old sections counted", []Branch{{Name: "main", Revision: 3, Entries: map[string]catalog.CatalogEntry{"m": entryOf("accepted", map[string]any{
"objects": []any{map[string]any{"name": "sent-objects", "max-mib": 256}}, "logs": []any{map[string]any{"name": "l", "max-mib": 4}}, "streams": []any{map[string]any{"name": "s"}}})}}}, []string{"main"},
"m 3 [] [] true 1284;"},
"m 3 [] [] true 1284 3;"},
} {
t.Run(row.name, func(t *testing.T) {
v, err := ViewOf(row.branches, func(string) []string { return row.order })
@@ -67,7 +67,7 @@ func TestTheCatalogHalfIsReadPerWatchedBranch(t *testing.T) {
for _, e := range m.BusState {
es = append(es, e.Name+"/"+e.Storage+"/"+e.Template)
}
got += fmt.Sprintf("%s %d [%s] %v %v %d;", m.Module, m.Revision, strings.Join(es, " "), m.Claims, m.Foundation, m.OldBytes/keeper.MiB)
got += fmt.Sprintf("%s %d [%s] %v %v %d %d;", m.Module, m.Revision, strings.Join(es, " "), m.Claims, m.Foundation, m.OldBytes/keeper.MiB, len(m.Old))
}
for _, d := range v.Drifts {
got += fmt.Sprintf(" drifted %s %s %s from %s", d.Module, d.Local, d.Branch, d.From)
@@ -316,7 +316,9 @@ func TestTheBusModuleRunsWhole(t *testing.T) {
_ = Keep(kctx, deaf, func(context.Context, func(string, []byte) error) error { return errors.New("no consumer yet") }, nil, nil, t.Logf)
close(done)
}()
eventually(t, "the first view answered", func() bool { return !deaf.Status().FirstView.Answered.IsZero() })
// A first view listing no module is none yet and never marked answered (A-367): the first fill pass is waited
// for instead, then taken as the first (design 47 H, H-A-268).
eventually(t, "the first fill pass", func() bool { return !deaf.Status().Pass.IsZero() })
first := deaf.Status().Pass
eventually(t, "a fill pass while the consumer is bound again", func() bool { return deaf.Status().Pass.After(first) })
stopDeaf()
@@ -397,8 +399,8 @@ func TestTheBusModuleRunsWhole(t *testing.T) {
}
put(keeper.Request{ID: "r1", Verb: keeper.VerbDrop, Module: "m", Local: "drafts", Caller: "person.jo", Why: "unused"})
eventually(t, "the drop with no ask refused", func() bool { return strings.Contains(get("r1").Error, "operator's warrant") })
r := keeper.Request{ID: "r2", Verb: keeper.VerbDrop, Module: "m", Local: "drafts", Caller: "person.jo", Why: "unused", At: time.Now()}
a, _ := askFor(r, time.Now())
r := keeper.Request{ID: "r2", Verb: keeper.VerbDrop, Module: "m", Local: "drafts", Caller: "person.jo", Why: "unused", At: time.Now(), AskExpires: time.Now().Add(keeper.AskLasts)}
a, _ := askFor(r)
r.Ask = a.ID
record := func(w *oc.Warrant) {
raw, _ := json.Marshal(map[string]any{"asker": keeper.Module, "ask": a, "state": "open", "warrant": w})
@@ -415,7 +417,7 @@ func TestTheBusModuleRunsWhole(t *testing.T) {
// The operator chooses: the router records the warrant and says decided; the keeper acts on decided, on its
// durable consumer, with no timer between and no call of its pass by the test (seams F1).
w := &oc.Warrant{Ask: a.ID, Asker: keeper.Module, Outcome: occ.OutcomeChosen, By: &oc.Person{Who: occ.Operator},
Option: "go", Label: "Remove", Level: occ.Destroy, At: occ.FormatTime(time.Now()), AskDigest: occ.Digest(a)}
Option: "go", Label: "Remove", Level: occ.Approve, At: occ.FormatTime(time.Now()), AskDigest: occ.Digest(a)}
record(w)
decided := time.Now()
body, _ := json.Marshal(w)
+114 -11
View File
@@ -40,6 +40,9 @@ type requestStore struct {
Ended func(id string, within time.Duration) *keeper.Request
// Record records a hand act through the seat mesh-hand-acts.
Record Recorded
// Standing reads the router's record of one of this module's asks, by its id, or nil while it holds none (design 47
// H-A-237, H-A-239).
Standing func(askID string) (*stdio.StateEntry, error)
}
// RequestsInState is the store through the runtime that launched this bundle: one watch of the state requests for the
@@ -80,10 +83,11 @@ func RequestsInState() requestStore {
}
return out, nil
},
Ask: func(a oc.Ask) error { return stdio.SeatPublish(oc.AcceptSubjectAsk(keeper.Module), a, a.ID) },
Now: func() time.Time { return time.Now().UTC() },
Ended: ends.wait,
Record: RecordedBySeat,
Ask: func(a oc.Ask) error { return stdio.SeatPublish(oc.AcceptSubjectAsk(keeper.Module), a, a.ID) },
Now: func() time.Time { return time.Now().UTC() },
Ended: ends.wait,
Record: RecordedBySeat,
Standing: func(id string) (*stdio.StateEntry, error) { return stdio.SeatRecord("", id) },
}
}
@@ -146,36 +150,129 @@ func requestID(now time.Time) string {
return now.Format("20060102t150405") + "-" + hex.EncodeToString(b[:])
}
// askFor is the operator's ask for a recreate or a drop of one entry, its approve option binding exactly that act.
func askFor(r keeper.Request, now time.Time) (oc.Ask, error) {
// The bounds of the caller's expiry of a recreate's or a drop's ask, in minutes (design 47 H-A-237, H-A-239): the
// channel's own, the longest under the day the state requests keeps a key.
const (
askLeastMinutes = 5
askMostMinutes = 1430
)
// askExpiry is the caller's expiry of the ask, expires_minutes, or the default AskLasts: a value outside its bounds is
// refused, naming them, never clamped (design 47 H-A-239).
func askExpiry(a map[string]any) (time.Duration, error) {
v, given := a["expires_minutes"]
if !given || v == nil {
return keeper.AskLasts, nil
}
m, ok := v.(float64)
if !ok || m < askLeastMinutes || m > askMostMinutes {
return 0, fmt.Errorf("expires_minutes is %d to %d minutes, and %v is not: nothing is asked", askLeastMinutes, askMostMinutes, v)
}
return time.Duration(m * float64(time.Minute)), nil
}
// askFor is the operator's ask for a recreate or a drop of one entry, its approve option binding exactly that act: at
// the approve level until a channel carries a destroy answer, ending at the request's AskExpires, the caller's (design
// 47 H-A-237).
func askFor(r keeper.Request) (oc.Ask, error) {
binds, err := occ.ActDigest(r.Act())
if err != nil {
return oc.Ask{}, err
}
what := map[string]string{keeper.VerbDrop: "remove", keeper.VerbRecreate: "rebuild"}[r.Verb]
name := keeper.BusName(r.Module, r.Local)
return oc.Ask{ID: "bus-" + r.ID, Who: occ.Operator, Expires: occ.FormatTime(now.Add(keeper.AskLasts)),
return oc.Ask{ID: "bus-" + r.ID, Who: occ.Operator, Expires: occ.FormatTime(r.AskExpires),
OnExpiry: "nothing is removed and the entry stays as it is",
About: keeper.ConditionKey(map[string]string{keeper.VerbDrop: "orphaned", keeper.VerbRecreate: "drifted"}[r.Verb], r.Module, r.Local),
Headline: fmt.Sprintf("%s's kept data %s: %s it?", r.Module, r.Local, what),
Explanation: fmt.Sprintf("%s asks to %s %s on the bus, %s. A snapshot of it is taken first; its data is lost from the bus.",
r.Caller, what, name, r.Why),
Options: []oc.Option{
{ID: "go", Label: strings.ToUpper(what[:1]) + what[1:], Does: fmt.Sprintf("%s %s after a snapshot", what, name), Level: occ.Destroy, Binds: binds},
{ID: "go", Label: strings.ToUpper(what[:1]) + what[1:], Does: fmt.Sprintf("%s %s after a snapshot", what, name), Level: occ.Approve, Binds: binds},
{ID: "keep", Label: "Keep it", Does: "nothing is removed", Level: occ.Acknowledge},
}}, nil
}
// notTakenIn is what the tool says of an ask the router holds no record of yet (design 47 H-A-239).
const notTakenIn = "raised, not yet taken in by the router: read the request for its standing"
// standingOf is the router's standing of an ask, in words, from one read of its record (design 47 H-A-237, H-A-239):
// offered (where), held (why), refused (why), or how it ended; "the operator is asked" is said only for an offered ask.
func standingOf(rs requestStore, askID string) string {
if rs.Standing == nil {
return notTakenIn
}
e, err := rs.Standing(askID)
if err != nil {
return "raised; the router's record of it cannot be read now (" + err.Error() + "): read the request for its standing"
}
if e == nil || len(e.Value) == 0 || string(e.Value) == "null" {
return notTakenIn
}
var rec struct {
State string `json:"state"`
Held string `json:"held"`
Tickets map[string]struct {
Kind string `json:"kind"`
} `json:"tickets"`
Warrant *oc.Warrant `json:"warrant"`
}
if err := json.Unmarshal(e.Value, &rec); err != nil {
return "raised; the router's record of it is not readable (" + err.Error() + "): read the request for its standing"
}
switch rec.State {
case "open":
var kinds []string
for _, t := range rec.Tickets {
if !slices.Contains(kinds, t.Kind) {
kinds = append(kinds, t.Kind)
}
}
sort.Strings(kinds)
where := "offered on " + strings.Join(kinds, ", ")
if len(kinds) == 0 {
where = "offered, channel not yet named"
}
return where + ": the operator is asked; the keeper acts once they approve, after a snapshot"
case "held":
return "held by the router, not shown to the operator: " + rec.Held
case string(occ.OutcomeRefused):
words := ""
if rec.Warrant != nil {
words = rec.Warrant.Words
}
return "refused by the router, so nothing will be removed: " + words
}
words := ""
if rec.Warrant != nil {
words = ": " + occ.Says(*rec.Warrant)
}
return "ended " + rec.State + words
}
func requestTools(rs requestStore) []stdio.Tool {
entry := map[string]any{"module": str("the module, e.g. mesh-knowledge"), "local": str("its local name, e.g. records"),
"why": str("why, in plain words: kept with the act"), "request": str("a request this tool made before: its end is read, nothing new is asked")}
warranted := map[string]any{"expires_minutes": num(fmt.Sprintf("how long the operator's ask waits for an answer, %d to %d minutes (default %d)",
askLeastMinutes, askMostMinutes, int(keeper.AskLasts/time.Minute)))}
for k, v := range entry {
warranted[k] = v
}
tool := func(verb, description string) stdio.Tool {
return stdio.Tool{Name: "nats_" + verb, Description: description, Input: entry, Run: func(a map[string]any) (any, error) {
input := entry
if verb == keeper.VerbRecreate || verb == keeper.VerbDrop {
input = warranted
}
return stdio.Tool{Name: "nats_" + verb, Description: description, Input: input, Run: func(a map[string]any) (any, error) {
if id := arg(a, "request"); id != "" {
r, err := rs.Get(id)
if err != nil || r == nil {
return nil, fmt.Errorf("no request %s is held (%v)", id, err)
}
// The router's standing of its ask beside the keeper's (design 47 H-A-239).
if r.Ask != "" {
return map[string]any{"request": r, "router": standingOf(rs, r.Ask)}, nil
}
return r, nil
}
r := keeper.Request{Verb: verb, Module: arg(a, "module"), Local: arg(a, "local"), Why: strings.TrimSpace(arg(a, "why")),
@@ -186,7 +283,12 @@ func requestTools(rs requestStore) []stdio.Tool {
}
r.ID = requestID(r.At)
if verb == keeper.VerbRecreate || verb == keeper.VerbDrop {
ask, err := askFor(r, r.At)
lasts, err := askExpiry(a)
if err != nil {
return nil, err
}
r.AskExpires = r.At.Add(lasts)
ask, err := askFor(r)
if err != nil {
return nil, err
}
@@ -199,7 +301,8 @@ func requestTools(rs requestStore) []stdio.Tool {
if err := rs.Put(r.ID, r); err != nil {
return nil, fmt.Errorf("the request cannot be handed to the keeper, so nothing will be removed whatever the operator answers: %w", err)
}
return map[string]any{"request": r.ID, "ask": ask.ID, "state": "the operator is asked; the keeper acts once they approve, after a snapshot",
// One read of the router's record right after the publish, no timer (design 47 H-A-237, H-A-239).
return map[string]any{"request": r.ID, "ask": ask.ID, "state": standingOf(rs, ask.ID),
"read": fmt.Sprintf("nats_%s with request %s", verb, r.ID)}, nil
}
if err := rs.Put(r.ID, r); err != nil {
+58 -14
View File
@@ -52,6 +52,10 @@ func TestAPersonsRequestIsHandedToTheKeeperAsTheProvedCaller(t *testing.T) {
Ask: func(a oc.Ask) error { raised = append(raised, a); return nil },
Now: func() time.Time { return time.Date(2026, 11, 4, 12, 0, 0, 0, time.UTC) },
Record: func(q meshhandactsclient.Request) error { acts = append(acts, q); return nil },
// Design 47 H-A-237: the router held the ask (no channel carries its level), as it held every one live.
Standing: func(id string) (*stdio.StateEntry, error) {
return &stdio.StateEntry{Key: keeper.Module + "." + id, Value: []byte(`{"asker":"nats","state":"held","held":"telegram: Destroy needs a code"}`)}, nil
},
}
tools := map[string]stdio.Tool{}
for _, tl := range requestTools(rs) {
@@ -74,14 +78,21 @@ func TestAPersonsRequestIsHandedToTheKeeperAsTheProvedCaller(t *testing.T) {
t.Fatalf("raised %+v, held %+v: the ask raised, the request waiting for it", raised, r)
}
binds, _ := occ.ActDigest(occ.Act{"verb": "drop", "module": "m", "local": "old"})
if o, _ := occ.OptionOf(raised[0], "go"); o.Level != occ.Destroy || o.Binds != binds || !strings.Contains(raised[0].Explanation, "snapshot") {
t.Errorf("the ask is %+v: its option destroys exactly this drop, after a snapshot", raised[0])
// Design 47 H-A-237: the ask is at the approve level, naming the entry, the removal and the snapshot, for the default
// sixty minutes; an ask the router held is said held, with its why, never "the operator is asked".
if o, _ := occ.OptionOf(raised[0], "go"); o.Level != occ.Approve || o.Binds != binds || !strings.Contains(raised[0].Explanation, "snapshot") ||
!strings.Contains(raised[0].Explanation, "remove m_old") || raised[0].Expires != occ.FormatTime(rs.Now().Add(time.Hour)) || !r.AskExpires.Equal(rs.Now().Add(time.Hour)) {
t.Errorf("the ask is %+v: its option approves exactly this drop, after a snapshot, for sixty minutes", raised[0])
}
if st := answer["state"].(string); !strings.Contains(st, "held by the router") || !strings.Contains(st, "Destroy needs a code") || strings.Contains(st, "operator is asked") {
t.Errorf("the tool answered %q for an ask the router held", st)
}
if _, err := tools["nats_drop"].Run(map[string]any{"module": "m", "local": "old"}); err == nil || !strings.Contains(err.Error(), "says why") {
t.Errorf("a drop without why: %v", err)
}
if got, _ := tools["nats_drop"].Run(map[string]any{"request": r.ID}); got.(*keeper.Request).ID != r.ID {
t.Errorf("reading the request back answered %+v", got)
if got, _ := tools["nats_drop"].Run(map[string]any{"request": r.ID}); got.(map[string]any)["request"].(*keeper.Request).ID != r.ID ||
!strings.Contains(got.(map[string]any)["router"].(string), "held by the router") {
t.Errorf("reading the request back answered %+v: the request, and the router's standing of its ask beside it", got)
}
// Review round 1 (tools F3): nats_requests reads what waits, what ended and each warrant spent, and drops a waiting
// request by hand, recorded; an ended one is not dropped.
@@ -146,20 +157,53 @@ func TestAPersonsRequestIsHandedToTheKeeperAsTheProvedCaller(t *testing.T) {
}
}
// The ask for a recreate or a drop lasts what its highest option level allows (design 47 G §570), so the asker client
// takes it: an expiry at the bound passes occ.Check, and the fixed 12 hours it was is refused.
func TestTheAskLastsWhatItsLevelAllows(t *testing.T) {
// The ask for a recreate or a drop is at the approve level, for the caller's expires_minutes, 5 to 1430, a value outside
// refused naming its bounds (design 47 H-A-237, H-A-239); the tool's answer is the router's standing as one read finds
// it, "the operator is asked" only for an offered ask.
func TestTheAskLastsTheCallersExpiryAndTheToolSaysTheRoutersStanding(t *testing.T) {
now := time.Now()
r := keeper.Request{ID: "r1", Verb: keeper.VerbDrop, Module: "m", Local: "drafts", Why: "unused", Caller: "person.jo"}
a, err := askFor(r, now)
r := keeper.Request{ID: "r1", Verb: keeper.VerbDrop, Module: "m", Local: "drafts", Why: "unused", Caller: "person.jo", AskExpires: now.Add(1430 * time.Minute)}
a, err := askFor(r)
if err != nil {
t.Fatal(err)
}
if err := occ.Check(a, now); err != nil {
t.Errorf("the ask is refused by the asker client: %v", err)
if err := occ.Check(a, now); err != nil || occ.Highest(a) != occ.Approve {
t.Errorf("the ask at %s is refused by the asker client (%v), or not at the approve level", occ.Highest(a), err)
}
a.Expires = occ.FormatTime(now.Add(12 * time.Hour))
if err := occ.Check(a, now); err == nil || !strings.Contains(err.Error(), "at most") {
t.Errorf("12 hours is held to the level's bound: %v", err)
for _, c := range []struct {
minutes any
want time.Duration
refused bool
}{{nil, time.Hour, false}, {5.0, 5 * time.Minute, false}, {1430.0, 1430 * time.Minute, false}, {4.0, 0, true}, {1431.0, 0, true}, {"60", 0, true}} {
a := map[string]any{}
if c.minutes != nil {
a["expires_minutes"] = c.minutes
}
got, err := askExpiry(a)
if c.refused != (err != nil) || got != c.want || (c.refused && !strings.Contains(err.Error(), "5 to 1430")) {
t.Errorf("expires_minutes %v: %s, %v", c.minutes, got, err)
}
}
for _, c := range []struct {
record string
err error
want string
}{
{"", nil, "not yet taken in by the router"},
{`{"state":"open","tickets":{"a":{"kind":"telegram"},"b":{"kind":"telegram"},"c":{"kind":"desktop"}}}`, nil, "offered on desktop, telegram: the operator is asked"},
{`{"state":"open"}`, nil, "offered, channel not yet named: the operator is asked"},
{`{"state":"held","held":"no channel carries destroy"}`, nil, "held by the router, not shown to the operator: no channel carries destroy"},
{`{"state":"refused","warrant":{"ask":"bus-r1","asker":"nats","outcome":"refused","words":"its words carry a path"}}`, nil, "refused by the router, so nothing will be removed: its words carry a path"},
{"", errors.New("no responders"), "cannot be read now (no responders)"},
} {
rs := requestStore{Standing: func(string) (*stdio.StateEntry, error) {
if c.record == "" {
return nil, c.err
}
return &stdio.StateEntry{Value: []byte(c.record)}, nil
}}
if got := standingOf(rs, "bus-r1"); !strings.Contains(got, c.want) || (!strings.HasPrefix(c.want, "offered") && strings.Contains(got, "operator is asked")) {
t.Errorf("the record %s (%v) is said %q, not %q", c.record, c.err, got, c.want)
}
}
}
+71 -4
View File
@@ -8,6 +8,7 @@ import (
"encoding/json"
"errors"
"fmt"
"slices"
"sort"
"strings"
"sync"
@@ -84,11 +85,14 @@ type EventFill struct {
// Accepted is one module's entry as the catalog half's watched branches give it (design 47 G §344, §433, §532): the
// module, its manifest's digest, the catalog revision of the branch it was read from, and its bus-state entries whole.
// Old are the entries its old sections declare, as OldPlansOf names and caps them (design 47 H H-A-229, H-A-230): asked
// for too, but never made by the keeper.
type Accepted struct {
Module string `json:"module"`
Digest string `json:"digest"`
Revision uint64 `json:"revision"`
BusState []catalogue.BusStateEntry `json:"bus-state"`
Old []Plan `json:"old-sections,omitempty"`
}
// CatalogModule is one module of the catalog half for the bus module (design 47 G §344, §532, §533, §547): what its
@@ -146,7 +150,9 @@ type Asked struct {
Digest string `json:"digest,omitempty"`
Revision uint64 `json:"revision"`
Entries []catalogue.BusStateEntry `json:"entries"`
Made map[string]Made `json:"made,omitempty"`
// Old are the entries its old sections declare, as last planned (Accepted's Old; design 47 H H-A-229, H-A-230).
Old []Plan `json:"old-sections,omitempty"`
Made map[string]Made `json:"made,omitempty"`
// Drifts are the local names a later watched branch asks with other settings, as last said (design 47 G §532).
Drifts map[string]Drift `json:"drifts,omitempty"`
At time.Time `json:"at"`
@@ -259,7 +265,8 @@ type Limits struct {
}
// CapSum is the sum, in bytes, of the caps of every object the entries made hold, but for the entry of skipModule and
// skipLocal (design 47 G §547): what the bus's limits are read against. An entry never made, or refused, holds none.
// skipLocal (design 47 G §547): what the bus's limits are read against. An entry never made, or refused, holds none, and
// one an old objects, log or stream section keeps is counted once, in the old sections' caps (design 47 H H-A-229).
func CapSum(held map[string]Asked, skipModule, skipLocal string) int64 {
var sum int64
for module, a := range held {
@@ -267,6 +274,11 @@ func CapSum(held map[string]Asked, skipModule, skipLocal string) int64 {
if (module == skipModule && local == skipLocal) || !(m.Created || m.Orphaned) {
continue
}
// A kept entry of an objects, log or stream section is in the old sections' caps already; a state's is not
// (design 47 G §547, H H-A-229).
if p, kept := oldKept(a.Old, local, m); kept && p.Storage != catalogue.StorageKeyValue {
continue
}
for _, o := range m.Plan.Objects {
sum += o.Settings.MaxBytes
}
@@ -596,6 +608,15 @@ func (k *Keeper) Heard(a Accepted) {
held.Made = map[string]Made{}
}
digest := EntriesDigest(a.BusState)
if len(a.Old) > 0 {
// The old sections' entries are planned too (design 47 H H-A-229): a module that declares none keeps its digest.
raw, _ := json.Marshal(struct {
BusState []catalogue.BusStateEntry `json:"bus-state"`
Old []Plan `json:"old-sections"`
}{a.BusState, a.Old})
sum := sha256.Sum256(raw)
digest = "sha256:" + hex.EncodeToString(sum[:])
}
if p, ok := k.planned(a.Module); ok && found && p.Digest == digest {
if held.Digest != a.Digest || held.Revision != a.Revision {
held.Digest, held.Revision, held.At = a.Digest, a.Revision, k.now()
@@ -603,13 +624,18 @@ func (k *Keeper) Heard(a Accepted) {
}
return
}
held.Digest, held.Entries, held.Revision, held.At = a.Digest, a.BusState, a.Revision, k.now()
held.Digest, held.Entries, held.Old, held.Revision, held.At = a.Digest, a.BusState, a.Old, a.Revision, k.now()
defer k.writePlan(a.Module, digest, held.Revision, a.BusState)
asked := map[string]bool{}
for _, e := range a.BusState {
asked[e.Name] = true
k.makeEntry(&held, e)
}
for _, local := range sortedKeys(held.Made) {
if !asked[local] && k.keepOld(&held, local) {
asked[local] = true
}
}
for _, local := range sortedKeys(held.Made) {
m := held.Made[local]
if asked[local] || m.Orphaned || local == SentLocal {
@@ -650,6 +676,35 @@ func (k *Keeper) Sends(module string) {
k.write(held)
}
// keepOld keeps an entry the keeper made that an old section of the module declares under the same bus name (design 47
// H H-A-229, H-A-230, H-A-231): never an orphan (one that was is asked for again, its condition cleared), compared with
// the bus and grown to the declared cap, and rebuilt on request; the keeper creates none of its objects the bus lacks and
// says no created for it, as the grants part makes and binds them. It answers whether an old section keeps it. An entry
// never made is none of the keeper's: every other old-section entry is the grants part's alone.
func (k *Keeper) keepOld(held *Asked, local string) bool {
m := held.Made[local]
p, ok := oldKept(held.Old, local, m)
if !ok || !m.Created || local == SentLocal {
return false
}
if m.Orphaned {
m.Orphaned = false
k.clear(conditionOrphaned, held.Module, local)
}
// Only the objects the keeper made are its own: one the old plan names beside them (a state's `<state>-objects` the
// grants part made) is never compared, grown or rebuilt by the keeper (H-A-231).
var objects []Object
for _, o := range p.Objects {
if slices.ContainsFunc(m.Plan.Objects, func(made Object) bool { return made.Stream() == o.Stream() }) {
objects = append(objects, o)
}
}
m.Plan.Objects = objects
k.ensureMade(held, &m, false)
held.Made[local] = m
return true
}
// makeEntry makes one asked entry where the bus lacks it, compares it where it holds it, and says what it did.
func (k *Keeper) makeEntry(held *Asked, e catalogue.BusStateEntry) {
module := held.Module
@@ -679,7 +734,11 @@ func (k *Keeper) makeEntry(held *Asked, e catalogue.BusStateEntry) {
// ensure brings one planned entry onto the bus: each object the bus lacks created, a held one compared. A difference
// that only grows is applied; any other is a drift, said and never fixed quietly.
func (k *Keeper) ensure(held *Asked, m *Made) {
func (k *Keeper) ensure(held *Asked, m *Made) { k.ensureMade(held, m, true) }
// ensureMade is ensure; with create false an object the bus lacks is left to the part that makes it (an old section's,
// the grants part: design 47 H H-A-230) and nothing is created.
func (k *Keeper) ensureMade(held *Asked, m *Made, create bool) {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
asked, holds := map[string]Settings{}, map[string]Settings{}
@@ -694,6 +753,9 @@ func (k *Keeper) ensure(held *Asked, m *Made) {
m.Waiting = fmt.Sprintf("%s cannot be read: %v", o.Stream(), err)
return
}
if !found && !create {
continue
}
if !found {
if err := k.capRoom(ctx, held, m); err != nil {
if errors.Is(err, ErrRetry) {
@@ -794,11 +856,16 @@ func (k *Keeper) CheckAll() []CheckLine {
out = append(out, lineOf(module, local, m))
continue
}
asked := false
for _, e := range held.Entries {
if e.Name == local {
asked = true
k.makeEntry(&held, e)
}
}
if !asked {
k.keepOld(&held, local)
}
out = append(out, lineOf(module, local, held.Made[local]))
}
k.write(held)
+93 -6
View File
@@ -872,10 +872,11 @@ func TestTheKeeperMakesWhatAVerdictAsksForAndSaysIt(t *testing.T) {
t.Fatalf("acted with no warrant: %+v, deleted %v", got, bus.deleted)
}
binds := func(act occ.Act) string { d, _ := occ.ActDigest(act); return d }
// Design 47 H-A-237: the ask's option is at the approve level, and an approve-level choice warrants the drop.
choose := func(act occ.Act) {
ask.Options = []oc.Option{{ID: "go", Label: "Remove", Level: occ.Destroy, Binds: binds(act)}}
ask.Options = []oc.Option{{ID: "go", Label: "Remove", Level: occ.Approve, Binds: binds(act)}, {ID: "keep", Label: "Keep it", Does: "nothing is removed", Level: occ.Acknowledge}}
w = &oc.Warrant{Ask: "bus-1", Asker: Module, Outcome: occ.OutcomeChosen, By: &oc.Person{Who: occ.Operator},
Option: "go", Label: "Remove", Level: occ.Destroy, At: occ.FormatTime(k.now()), AskDigest: occ.Digest(ask)}
Option: "go", Label: "Remove", Level: occ.Approve, At: occ.FormatTime(k.now()), AskDigest: occ.Digest(ask)}
}
choose(occ.Act{"verb": "drop", "module": "m", "local": "log"})
if got := put(Request{ID: "r1", Verb: VerbDrop, Module: "m", Local: "drafts", Ask: "bus-1", Caller: "person.jo"}); !strings.Contains(got.Error, "bound") || len(bus.deleted) != 0 {
@@ -890,8 +891,7 @@ func TestTheKeeperMakesWhatAVerdictAsksForAndSaysIt(t *testing.T) {
}
refused(Request{ID: "r1a", Verb: VerbDrop, Module: "m", Local: "drafts", Caller: "person.jo"}, "names no ask")
choose(occ.Act{"verb": "drop", "module": "m", "local": "drafts"})
w.Level, w.Label, ask.Options[0].Level = occ.Approve, "Keep", occ.Approve
w.AskDigest = occ.Digest(ask)
w.Option, w.Level, w.Label = "keep", occ.Acknowledge, "Keep it"
refused(Request{ID: "r1b", Verb: VerbDrop, Module: "m", Local: "drafts", Ask: "bus-1", Caller: "person.jo"}, "nothing is removed")
choose(occ.Act{"verb": "drop", "module": "m", "local": "drafts"})
w.Asker = "mesh-grants"
@@ -1112,6 +1112,90 @@ func TestTheKeeperMakesWhatAVerdictAsksForAndSaysIt(t *testing.T) {
t.Errorf("the hand acts: %v", acts)
}
}},
// Design 47 H H-A-229, H-A-230, H-A-231: an entry the keeper made, now declared in the module's old sections under
// the same bus name, is asked for: its orphan cleared, grown to the declared cap, rebuilt on request; the keeper
// creates nothing an old section declares and says no created for it.
{"an entry an old section declares is no orphan and is rebuilt, nothing of the old sections made", func(t *testing.T, k *Keeper, bus *fakeBus, out *said) {
store := k.Store.(*memStore)
incoming := entry("incoming", catalogue.StorageObjectStore, catalogue.TemplateObjects, map[string]string{"max-mib": "8"})
k.Heard(Accepted{Module: "m", Revision: 1, BusState: []catalogue.BusStateEntry{drafts, incoming}})
k.Heard(Accepted{Module: "m", Revision: 2})
if a := k.Held()["m"]; !a.Made["incoming"].Orphaned || !a.Made["drafts"].Orphaned {
t.Fatalf("held %+v: both orphans once no section asks for them", a.Made)
}
delete(bus.held, "OBJ_m_drafts-objects")
made := fmt.Sprint(bus.created)
out.reset()
old := OldPlansOf(catalogue.Manifest{Module: "m", State: []catalogue.StateDeclaration{{Name: "drafts"}, {Name: "bound"}},
Objects: []catalogue.ObjectsDeclaration{{Name: "incoming", MaxMiB: 16}}})
k.Heard(Accepted{Module: "m", Revision: 3, Old: old})
a := k.Held()["m"]
if a.Made["incoming"].Orphaned || a.Made["drafts"].Orphaned || len(a.Made) != 2 {
t.Errorf("held %+v: incoming and drafts asked for by the old sections, bound none of the keeper's", a.Made)
}
if fmt.Sprint(bus.created) != made || len(out.events) != 0 || bus.held["OBJ_m_incoming"].MaxBytes != 16*MiB {
t.Errorf("created %v (was %s), said %v, incoming held %+v: nothing made or said, grown to the declared 16 MiB",
bus.created, made, out.events, bus.held["OBJ_m_incoming"])
}
if fmt.Sprint(out.conds) != "[cleared nats.orphaned.m_drafts cleared nats.orphaned.m_incoming]" {
t.Errorf("conditions %v: both orphans cleared", out.conds)
}
k.CheckAll()
if fmt.Sprint(bus.created) != made {
t.Errorf("the check created %v (was %s): the grants part makes an old section's objects", bus.created, made)
}
binds, _ := occ.ActDigest(occ.Act{"verb": "recreate", "module": "m", "local": "incoming"})
ask := oc.Ask{ID: "bus-9", Who: occ.Operator, Expires: occ.FormatTime(k.now().Add(time.Hour)), Options: []oc.Option{{ID: "go", Label: "Rebuild", Level: occ.Destroy, Binds: binds}}}
w := &oc.Warrant{Ask: "bus-9", Asker: Module, Outcome: occ.OutcomeChosen, By: &oc.Person{Who: occ.Operator},
Option: "go", Label: "Rebuild", Level: occ.Destroy, At: occ.FormatTime(k.now()), AskDigest: occ.Digest(ask)}
k.Warrant = func(string) (*oc.Ask, *oc.Warrant, error) { return &ask, w, nil }
out.reset()
raw, _ := json.Marshal(Request{ID: "r9", Verb: VerbRecreate, Module: "m", Local: "incoming", Ask: "bus-9"})
_, _ = store.Put(StateRequests, "r9", raw)
k.ActOnRequests()
raw, _, _, _ = store.Get(StateRequests, "r9")
var got Request
_ = json.Unmarshal(raw, &got)
if got.Error != "" || fmt.Sprint(out.events) != "[recreated m incoming]" || bus.held["OBJ_m_incoming"].MaxBytes != 16*MiB {
t.Errorf("request %+v, said %v, held %+v: rebuilt at the declared cap", got, out.events, bus.held["OBJ_m_incoming"])
}
}},
// Design 47 H H-A-231, G §547: a state the keeper made as its bucket alone keeps only that bucket; the store beside it
// the grants part made is neither compared, grown nor rebuilt by the keeper; a kept state's cap stays in the sum, a
// kept objects section's is counted in the old sections' caps alone.
{"a kept state's store made by the grants part is left alone, its bucket's cap summed", func(t *testing.T, k *Keeper, bus *fakeBus, out *said) {
store := k.Store.(*memStore)
incoming := entry("incoming", catalogue.StorageObjectStore, catalogue.TemplateObjects, map[string]string{"max-mib": "8"})
k.Heard(Accepted{Module: "m", Revision: 1, BusState: []catalogue.BusStateEntry{incoming}})
old := OldPlansOf(catalogue.Manifest{Module: "m", State: []catalogue.StateDeclaration{{Name: "findings"}},
Objects: []catalogue.ObjectsDeclaration{{Name: "incoming", MaxMiB: 8}}})
bucket := old[0].Objects[0]
bus.held[bucket.Stream()] = bucket.Settings
bus.held["OBJ_m_findings-objects"] = Settings{MaxBytes: 8 * MiB, Discard: DiscardNew}
held := k.Held()["m"]
held.Made["findings"] = Made{Plan: Plan{Module: "m", Local: "findings", Storage: catalogue.StorageKeyValue, Template: catalogue.TemplateState,
Objects: []Object{bucket}}, Created: true, Orphaned: true}
k.write(held)
k.Heard(Accepted{Module: "m", Revision: 2, Old: old})
k.CheckAll()
if m := k.Held()["m"].Made["findings"]; m.Orphaned || len(m.Plan.Objects) != 1 || len(bus.updated) != 0 || m.Drift != "" {
t.Errorf("findings %+v, updated %v: kept, its bucket alone, the grants part's store not grown", m, bus.updated)
}
if sum := CapSum(k.Held(), "", ""); sum != 64*MiB {
t.Errorf("the sum is %d MiB, want 64: the kept state's bucket, not the kept objects section's store", sum/MiB)
}
binds, _ := occ.ActDigest(occ.Act{"verb": "recreate", "module": "m", "local": "findings"})
ask := oc.Ask{ID: "bus-8", Who: occ.Operator, Expires: occ.FormatTime(k.now().Add(time.Hour)), Options: []oc.Option{{ID: "go", Label: "Rebuild", Level: occ.Destroy, Binds: binds}}}
w := &oc.Warrant{Ask: "bus-8", Asker: Module, Outcome: occ.OutcomeChosen, By: &oc.Person{Who: occ.Operator},
Option: "go", Label: "Rebuild", Level: occ.Destroy, At: occ.FormatTime(k.now()), AskDigest: occ.Digest(ask)}
k.Warrant = func(string) (*oc.Ask, *oc.Warrant, error) { return &ask, w, nil }
raw, _ := json.Marshal(Request{ID: "r8", Verb: VerbRecreate, Module: "m", Local: "findings", Ask: "bus-8"})
_, _ = store.Put(StateRequests, "r8", raw)
k.ActOnRequests()
if fmt.Sprint(bus.deleted) != "[KV_m_findings]" || bus.held["OBJ_m_findings-objects"].MaxBytes != 8*MiB {
t.Errorf("deleted %v, store held %+v: the recreate rebuilds the bucket alone", bus.deleted, bus.held["OBJ_m_findings-objects"])
}
}},
{"a recreate rebuilds a drifted entry after a snapshot", func(t *testing.T, k *Keeper, bus *fakeBus, out *said) {
store := k.Store.(*memStore)
k.Heard(Accepted{Module: "m", Revision: 1, BusState: []catalogue.BusStateEntry{log}})
@@ -1189,9 +1273,12 @@ func TestTheKeeperMakesWhatAVerdictAsksForAndSaysIt(t *testing.T) {
_, _ = store.Put(StateRequests, "r7", raw)
raw, _ = json.Marshal(Request{ID: "r8", Verb: VerbDrop, Module: "m", Local: "drafts", Ask: "bus-8", Caller: "person.jo", At: k.now().Add(-AskLasts - time.Minute)})
_, _ = store.Put(StateRequests, "r8", raw)
// Design 47 H-A-239: the ask's own expiry as asked bounds it, past the default.
raw, _ = json.Marshal(Request{ID: "r7a", Verb: VerbDrop, Module: "m", Local: "drafts", Ask: "bus-7a", Caller: "person.jo", At: k.now().Add(-2 * AskLasts), AskExpires: k.now().Add(time.Hour)})
_, _ = store.Put(StateRequests, "r7a", raw)
k.ActOnRequests()
if got := get("r7"); !got.Done.IsZero() {
t.Errorf("a request whose ask still runs ended: %+v", got)
if got, longer := get("r7"), get("r7a"); !got.Done.IsZero() || !longer.Done.IsZero() {
t.Errorf("a request whose ask still runs ended: %+v, %+v", got, longer)
}
if got := get("r8"); got.Done.IsZero() || !strings.Contains(got.Error, "ended at") || len(bus.deleted) != 0 {
t.Errorf("a request whose ask ended unanswered: %+v, deleted %v", got, bus.deleted)
+67
View File
@@ -423,3 +423,70 @@ func Differs(asked, held Settings) string {
add("key-ttl", asked.KeyTTL, held.KeyTTL)
return strings.Join(out, "; ")
}
// OldPlansOf are the entries a manifest declares in its old sections (state, objects, logs, streams), each named on the
// bus and capped as the grants part composes it, by the SDK's one naming per section (design 47 H H-A-229, H-A-230): a
// state `<module>_<state>` with its object store `<state>-objects` beside it (none for a state only read, or one whose
// store's name is one of the module's own object buckets), an objects section `<module>_<name>`, a log
// `LOG_<module>_<name>` and a stream its local name in capitals. They are asked for: an entry the keeper made under one
// of their bus names is never an orphan, is kept, compared and grown to the declared cap, and rebuilt on request; the
// keeper creates none of them and says no created for one (the grants part makes and binds them). Pure.
func OldPlansOf(m catalogue.Manifest) []Plan {
module := m.Module
var out []Plan
own := map[string]bool{}
for _, o := range m.Objects {
own[o.Name] = true
}
for _, s := range m.State {
capMiB := int64(s.MaxMiB)
if capMiB <= 0 {
capMiB = StateCap / MiB
}
p := Plan{Module: module, Local: s.Name, Storage: catalogue.StorageKeyValue, Template: catalogue.TemplateState,
Objects: []Object{{Kind: KindBucket, Name: buscompose.BucketName(module, s.Name),
Settings: Settings{MaxBytes: capMiB * MiB, MaxAgeSeconds: int64(s.TTLSeconds), History: int64(max(s.History, 1)),
Discard: DiscardNew, MaxMsgSize: ValueMost, KeyTTL: s.KeyTTL}}}}
if objects := catalogue.StateObjectsName(s.Name); !s.ReadOnly && !own[objects] {
objCap := int64(s.MaxMiB)
if objCap <= 0 {
objCap = StateObjectsCapMiB
}
objCap = min(max(objCap, catalogue.ObjectsLeastMiB), catalogue.ObjectsMostMiB)
p.Objects = append(p.Objects, Object{Kind: KindObjects, Name: catalogue.ObjectsBucket(module, objects),
Settings: Settings{MaxBytes: objCap * MiB, Discard: DiscardNew}})
}
out = append(out, p)
}
for _, o := range m.Objects {
out = append(out, Plan{Module: module, Local: o.Name, Storage: catalogue.StorageObjectStore, Template: catalogue.TemplateObjects,
Objects: []Object{{Kind: KindObjects, Name: o.Bucket(module), Settings: Settings{MaxBytes: int64(o.MaxMiB) * MiB, Discard: DiscardNew}}}})
}
for _, l := range m.Logs {
out = append(out, Plan{Module: module, Local: l.Name, Storage: catalogue.StorageStream, Template: catalogue.TemplateLog,
Objects: []Object{{Kind: KindStream, Name: buscompose.LogStreamName(module, l.Name), Subjects: []string{buscompose.LogSubject(module, l.Name) + ".>"},
Settings: Settings{MaxBytes: int64(l.Cap()) * MiB, MaxAgeSeconds: days(int64(l.MaxAgeDays)), Discard: DiscardNew, MaxMsgSize: buscompose.LogMaxMessageBytes}}}})
}
for _, s := range m.Streams {
discard := DiscardOld
if s.KeepsHistory() {
discard = DiscardNew
}
out = append(out, Plan{Module: module, Local: s.Name, Storage: catalogue.StorageStream,
Objects: []Object{{Kind: KindStream, Name: s.BusName(), Settings: Settings{MaxBytes: int64(s.Cap()) * MiB,
MaxAgeSeconds: days(int64(s.Days())), Discard: discard, WorkQueue: s.Retention == catalogue.StreamRetentionWorkQueue}}}})
}
return out
}
// oldKept is the old-section plan that keeps the entry made under local, or false: one of olds of the same local name
// whose objects are on the bus under the same names as the entry's (design 47 H H-A-229, H-A-230).
func oldKept(olds []Plan, local string, m Made) (Plan, bool) {
for _, p := range olds {
if p.Local != local || len(p.Objects) == 0 || len(m.Plan.Objects) == 0 || p.Objects[0].Stream() != m.Plan.Objects[0].Stream() {
continue
}
return p, true
}
return Plan{}, false
}
+18 -8
View File
@@ -40,6 +40,9 @@ type Request struct {
At time.Time `json:"at"`
// Ask is the id of the operator's ask the tool raised for a recreate or a drop; the keeper acts only on its warrant.
Ask string `json:"ask,omitempty"`
// AskExpires is when that ask ends unanswered, as its caller asked it (design 47 H-A-237, H-A-239): the keeper's
// bound where the router holds no record of the ask.
AskExpires time.Time `json:"ask-expires,omitempty"`
// Done is when the keeper ended it, Outcome what it did, Error why not.
Done time.Time `json:"done,omitempty"`
Outcome string `json:"outcome,omitempty"`
@@ -68,11 +71,12 @@ type Request struct {
Unrecorded string `json:"unrecorded,omitempty"`
}
// AskLasts is how long the operator's ask for a recreate or a drop waits for an answer: the bound of its highest option
// level, destroy (design 47 G §570: no asker asks for longer than its level allows; the asker client refuses more). It is
// shorter than the day the state requests keeps a key, so a request whose ask ended unanswered is ended by the keeper,
// saying so, before its key ages out, and the warrant's acted key (ActedKey) outlives the ask it spends.
const AskLasts = occ.DestroyLasts
// AskLasts is how long the operator's ask for a recreate or a drop waits for an answer when its caller names no expiry:
// sixty minutes (design 47 H-A-237: the ask is at the approve level until a channel carries a destroy answer, with the
// asker's expiry). It is shorter than the day the state requests keeps a key, so a request whose ask ended unanswered is
// ended by the keeper, saying so, before its key ages out, and the warrant's acted key (ActedKey) outlives the ask it
// spends.
const AskLasts = 60 * time.Minute
// ActedPrefix started the spend record's key while it was kept in StateRequests (`acted.<ask id>`); a key of that form
// still met there is no request and is skipped.
@@ -287,7 +291,7 @@ func (k *Keeper) act(r *Request) (string, error) {
}
// warranted holds a recreate or a drop to the operator's warrant on the ask its tool raised: chosen by the operator,
// on that ask, binding exactly this act.
// on that ask, at the approve level or above, binding exactly this act.
func (k *Keeper) warranted(r *Request) error {
if r.Ask == "" || k.Warrant == nil {
return fmt.Errorf("a %s acts only on the operator's warrant, and this request names no ask", r.Verb)
@@ -303,8 +307,12 @@ func (k *Keeper) warranted(r *Request) error {
}
// An ask that ended unanswered ends its request, said: never a request that waits until its key ages out. And a
// warrant is not acted on once its ask ended, so a request handed after its acted key aged out acts on nothing.
// Where the router holds no record, the ask's own expiry as asked bounds it (design 47 H-A-239), else the default.
var expires time.Time
if !r.At.IsZero() {
switch {
case !r.AskExpires.IsZero():
expires = r.AskExpires
case !r.At.IsZero():
expires = r.At.Add(AskLasts)
}
if ask != nil {
@@ -323,7 +331,9 @@ func (k *Keeper) warranted(r *Request) error {
if err != nil {
return fmt.Errorf("no warrant: %w", err)
}
if o.Level != occ.Destroy {
// An approve-level choice warrants a recreate or a drop (design 47 H-A-237): no channel carries a destroy answer yet,
// so the ask is at the approve level; a destroy-level choice still warrants it.
if occ.Rank(o.Level) < occ.Rank(occ.Approve) {
return fmt.Errorf("the operator chose %s, so nothing is removed", o.Label)
}
return occ.Performs(o, r.Act())
+464
View File
@@ -0,0 +1,464 @@
// The operator's allowances (ADR 0683; design 33 control-verbs): a port opened by hand on this machine, beside the
// ports derived from the modules' listens. An allowance is this module's own state on the bus, one key per protocol,
// port and interface (allow.<protocol>.<port>[.<interface>]), kept one bucket per machine under its node's keys
// (<node>.allow.…, H-A-317), so a holder reads and writes its own node's only; the node-engine asks `allowed` at every compose of a
// converged node's filter and renders it into the inet mesh table's input chain (design 08 control-verbs), so this
// code writes no rule. Every change is said as the seat's event allowances-changed, which the node-engine hears to
// compose the filter again; at its until an allowance is removed by a timer this module keeps per key from watching
// its own state, said and journalled, so no expired entry is kept or answered.
import { isIP } from "node:net";
import type { ConvergedAnswer } from "@novox/mesh-contracts-node-engine";
import {
allowancesByNodeAllowProtocolPortInterfaceKey, allowancesByNodeAllowProtocolPortKey,
parseAllowancesByNodeAllowProtocolPortInterfaceKey, parseAllowancesByNodeAllowProtocolPortKey, states,
type Allowance, type AllowanceValue, type AllowancesChanged, type Kept,
} from "@novox/mesh-connectivity-contracts-node-packet-filter";
// The seat's names, key forms and value forms are its generated package's (design 33 control-verbs §2, §7, §12): this
// module writes no copy of them.
export type { Allowance, AllowanceValue, AllowancesChanged, Kept };
/** The state the allowances are kept in, as the seat's definition names it (§7). */
export const STATE = states.allowances;
/** The most allowances one node keeps when the setting allowances-max says none (§3, H-A-314 (5)). */
export const MOST = 100;
/** How long the holder waits for the node-engine's converged (§10; a value, H-A-330). */
export const CONVERGED_BOUND_MS = 10_000;
/** The caller of a call the runtime named no caller for (the SDK's callers.UNPROVEN). */
export const UNPROVEN = "unproven";
type Protocol = AllowanceValue["protocol"];
/** The node's convergence state as the node-engine seat's converged answers it (§10, H-A-316, H-A-321, H-A-325). */
export type Convergence = ConvergedAnswer;
/** One act by hand, as the seat mesh-hand-acts' hand-act takes it (ADR 0565 §10): the SDK's views.HandAct. */
export interface HandAct {
what: string;
why: string;
cause: string;
verb: string;
}
/** What the holder asks beside its state: the node's convergence, the hand-act log and its setting allowances-max. */
export interface Asks {
converged: () => Promise<Convergence>;
handAct: (act: HandAct) => Promise<void>;
most: () => Promise<number>;
/** The mesh's table as loaded (`nft list table inet mesh`). */
meshTable: () => Promise<string>;
}
/** What the allowances need of the module's state: the SDK's `state()` serves it, keys from a watch of the node's own. */
export interface Store {
get(key: string): Promise<{ value: AllowanceValue } | null>;
put(key: string, value: AllowanceValue): Promise<number>;
delete(key: string): Promise<void>;
keys(): Promise<string[]>;
}
/** One change of the state as its watch hands it over: the SDK's `StateChange`. */
export interface Watched {
key: string;
op: "put" | "delete";
value?: AllowanceValue;
}
/** Says the event. */
export type Say = (body: AllowancesChanged) => Promise<void>;
/** Runs `fire` after `ms` milliseconds, answering how to cancel it: `setTimeout` on the machine, a fake in tests. */
export type Schedule = (ms: number, fire: () => void) => () => void;
/** The longest one timer waits (Node's setTimeout holds 2^31-1 ms); a later until is waited for in steps. */
export const LONGEST_WAIT_MS = 2 ** 31 - 1;
export const timers: Schedule = (ms, fire) => {
const t = setTimeout(fire, ms);
return () => clearTimeout(t);
};
/** The state's key of an allowance on node, in the seat's key forms (§7, H-A-317): <node>.allow.<protocol>.<port>, and
* .<interface> when it names one. */
function stateKeyOf(node: string, protocol: Protocol, port: number, iface?: string): string {
return iface
? allowancesByNodeAllowProtocolPortInterfaceKey(node, protocol, String(port), iface)
: allowancesByNodeAllowProtocolPortKey(node, protocol, String(port));
}
/** An allowance's key in every answer, the event and the rule's comment: its state's key without the node (§7). */
export function keyOf(protocol: Protocol, port: number, iface?: string): string {
return stateKeyOf("", protocol, port, iface).slice(1);
}
const IFACE = /^[A-Za-z0-9_.-]{1,15}$/;
function refuse(argument: string, why: string): never {
throw new Error(`${argument}: ${why}; nothing is written`);
}
function only(args: Record<string, unknown>, known: string[]): void {
for (const k of Object.keys(args)) {
if (!known.includes(k)) refuse(k, `not an argument of this verb (${known.join(", ") || "none"})`);
}
}
function portOf(v: unknown): number {
if (typeof v !== "number" || !Number.isInteger(v) || v < 1 || v > 65535) {
refuse("port", `${JSON.stringify(v)} is not an integer from 1 to 65535`);
}
return v;
}
function protocolOf(v: unknown): Protocol {
if (v === undefined) return "tcp";
if (v === "tcp" || v === "udp") return v;
refuse("protocol", `${JSON.stringify(v)} is neither tcp (the default) nor udp`);
}
function interfacesOf(v: unknown): (string | undefined)[] {
if (v === undefined) return [undefined];
if (!Array.isArray(v)) refuse("interfaces", "a list of interface names");
if (v.length === 0) return [undefined];
for (const i of v) {
if (typeof i !== "string" || !IFACE.test(i)) {
refuse("interfaces", `${JSON.stringify(i)} is not 1 to 15 letters, digits, _ . or -`);
}
}
return [...new Set(v as string[])];
}
function sourceOf(v: unknown): string | undefined {
if (v === undefined) return undefined;
const s = typeof v === "string" ? v : "";
const [addr, bits, more] = s.split("/");
const family = isIP(addr);
const max = family === 4 ? 32 : 128;
if (family === 0 || more !== undefined || (bits !== undefined && !(/^\d+$/.test(bits) && Number(bits) <= max))) {
refuse("source", `${JSON.stringify(v)} is not one IPv4 or IPv6 CIDR`);
}
return s;
}
function untilOf(v: unknown, now: Date): string | undefined {
if (v === undefined) return undefined;
const s = typeof v === "string" ? v : "";
const t = Date.parse(s);
// A full RFC 3339 time, seconds included: the node-engine reads it with Go's time.RFC3339, which refuses a time
// without them, and would withhold every later compose of the filter (§2).
if (!/^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\d(\.\d+)?(Z|[+-]\d\d:\d\d)$/.test(s) || Number.isNaN(t)) {
refuse("until", `${JSON.stringify(v)} is not an RFC 3339 time with seconds, such as 2026-11-22T08:00:00Z`);
}
if (t <= now.getTime()) refuse("until", `${s} is not in the future`);
return s;
}
function whyOf(v: unknown): string {
if (typeof v !== "string" || v.length < 1 || v.length > 500) refuse("why", "required, 1 to 500 characters");
return v;
}
function answered(key: string, value: AllowanceValue): Allowance {
return { key, ...value };
}
export class Allowances {
private readonly pending = new Map<string, () => void>();
private readonly store: Store;
private readonly node: string;
private readonly say: Say;
private readonly now: () => Date;
private readonly schedule: Schedule;
private readonly journal: (line: string) => void;
private readonly asks: Asks;
/** The last answer of converged; undefined while none answered. */
private convergence: Convergence | undefined;
/** The expiries running now (track). */
private readonly expiring = new Set<Promise<boolean>>();
constructor(
store: Store,
node: string,
say: Say,
asks: Asks,
now: () => Date = () => new Date(),
schedule: Schedule = timers,
journal: (line: string) => void = (line) => console.log(`[nftables] ${line}`),
) {
this.store = store;
this.node = node;
this.say = say;
this.asks = asks;
this.now = now;
this.schedule = schedule;
this.journal = journal;
}
/** The state's key of an allowance's key: under this node's (H-A-317). */
private stored(key: string): string {
return `${this.node}.${key}`;
}
/** The allowance's key of a state's key of this node, or undefined for another node's or another form. */
private own(stateKey: string): string | undefined {
const parts = parseAllowancesByNodeAllowProtocolPortInterfaceKey(stateKey) ?? parseAllowancesByNodeAllowProtocolPortKey(stateKey);
return parts?.node === this.node ? stateKey.slice(this.node.length + 1) : undefined;
}
/** Ask converged, bound 10 s, keeping its answer; refused, naming why, when it does not answer (§10). */
private async askConverged(verb: string): Promise<Convergence> {
let stop: () => void = () => {};
const late = new Promise<never>((_, no) => {
stop = this.schedule(CONVERGED_BOUND_MS, () => no(new Error(`no answer within ${CONVERGED_BOUND_MS / 1000} s`)));
});
try {
const c = await Promise.race([this.asks.converged(), late]);
if (c?.state !== "adopted" && c?.state !== "converged") throw new Error(`it answered no convergence state (${JSON.stringify(c)})`);
this.convergence = c;
return c;
} catch (err) {
throw new Error(`${verb}: refused: the node-engine's converged did not answer (${err instanceof Error ? err.message : String(err)}), ` +
"and the holder never assumes the node converged; nothing is written");
} finally {
stop();
}
}
private refuseAdopted(verb: string, c: Convergence): void {
if (c.state !== "adopted") return;
const about = [c.found ? `the found firewall ${c.found}` : "", c.since ? `since ${c.since}` : ""].filter(Boolean).join(", ");
throw new Error(`${verb}: refused: this node is adopted${about ? ` (${about})` : ""}, and the mesh speaks the found firewall's terms: ` +
"no allowance is kept on an adopted node; nothing is written");
}
/** The caller as the runtime proved it; an unproven one is refused (§3, H-A-315 (3)). */
private proven(verb: string, by: string): void {
if (!by || by === UNPROVEN) throw new Error(`${verb}: refused: the request carries no caller the tool runner proved; nothing is written`);
}
/** Every allowance kept for this node, by its key, sorted. */
private async ownKeys(): Promise<string[]> {
return (await this.store.keys()).map((k) => this.own(k)).filter((k): k is string => k !== undefined).sort();
}
/** Said once each time the holder starts serving the seat, with every allowance (§8): converged asked first, every
* key past its until removed first (§11). */
async started(): Promise<void> {
try {
await this.askConverged("start");
} catch (err) {
this.journal(`${err instanceof Error ? err.message : String(err)}; allowed asks it again`);
}
// Every expiry the watch handed over ends before started is said (§11).
await Promise.all([...this.expiring]);
const allowances = await this.kept();
await this.say({ node: this.node, change: "started", allowances: this.convergence?.state === "adopted" ? [] : allowances, at: this.now().toISOString() });
}
/** One change of the module's own state, from its watch: a key with an until gets its timer, a removed key loses it. */
async watched(change: Watched): Promise<void> {
const key = this.own(change.key);
if (key === undefined) return;
this.pending.get(key)?.();
this.pending.delete(key);
if (change.op !== "put" || !change.value?.until) return;
this.arm(key, change.value.until);
}
private arm(key: string, until: string): void {
const left = Date.parse(until) - this.now().getTime();
if (left <= 0) {
this.track(this.expire(key, until));
return;
}
const step = Math.min(left, LONGEST_WAIT_MS);
this.pending.set(key, this.schedule(step, () => {
this.pending.delete(key);
if (step < left) this.arm(key, until);
else this.track(this.expire(key, until));
}));
}
/** An expiry the watch or a timer started, kept until it ends so a start waits for it (§11). */
private track(expiring: Promise<boolean>): void {
this.expiring.add(expiring);
expiring.finally(() => this.expiring.delete(expiring)).catch(() => {});
}
/** Remove a key at its until, when it still holds that until (a later allow replaced it otherwise) (§4). */
async expire(key: string, until: string): Promise<boolean> {
try {
const entry = await this.store.get(this.stored(key));
if (!entry || entry.value.until !== until || Date.parse(until) > this.now().getTime()) return false;
await this.store.delete(this.stored(key));
const a = answered(key, entry.value);
await this.say({ node: this.node, change: "expired", allowances: [a], at: this.now().toISOString() });
this.journal(`the allowance ${key} (${a.protocol} ${a.port}${a.interface ? ` on ${a.interface}` : ""}${a.source ? ` from ${a.source}` : ""}, ` +
`set by ${a.by} at ${a.at}: ${a.why}) expired at ${until} and is removed`);
return true;
} catch (err) {
this.journal(`the allowance ${key} expired at ${until} and could not be removed: ${err instanceof Error ? err.message : String(err)}`);
return false;
}
}
/** Open a port (§2): one key per named interface, or the default set's one key; a key there already is replaced
* whole and answered. */
async allow(args: Record<string, unknown>, by: string): Promise<{ allowances: Kept[]; said: boolean }> {
this.proven("allow", by);
only(args, ["port", "protocol", "interfaces", "source", "until", "why"]);
const convergence = await this.askConverged("allow");
this.refuseAdopted("allow", convergence);
// The outward links as the node-engine classifies them for its render, from the converged answer of this call; the
// loaded table's shape is never read to tell one from the tunnel (§4, H-A-325).
const outward = new Set(convergence.outward);
const now = this.now();
const port = portOf(args.port);
const protocol = protocolOf(args.protocol);
const interfaces = interfacesOf(args.interfaces);
const source = sourceOf(args.source);
const until = untilOf(args.until, now);
const why = whyOf(args.why);
if (!source && privateOnly(await this.asks.meshTable(), protocol, port)) {
refuse("port", `${protocol} ${port} is admitted by a derived rule of the loaded mesh table from given sources only (the private network's, such as the store's ` +
"port or the broker's management port); an allowance from any source would widen it: give a source");
}
const most = await this.asks.most();
const kept = await this.ownKeys();
const after = new Set([...kept, ...interfaces.map((i) => keyOf(protocol, port, i))]).size;
if (after > most) refuse("port", `${after} allowances would be kept on this node, more than allowances-max ${most}`);
const at = now.toISOString();
const keys = interfaces.map((i) => keyOf(protocol, port, i));
await this.recorded({ what: `allowed ${keys.join(", ")} on ${this.node}${source ? ` from ${source}` : ""}${until ? ` until ${until}` : ""}`,
why: `${why} (${by})`, cause: "allowance-set", verb: "allow" });
const out: Kept[] = [];
for (const iface of interfaces) {
const key = keyOf(protocol, port, iface);
const value: AllowanceValue = {
port, protocol,
...(iface ? { interface: iface } : {}),
...(source ? { source } : {}),
...(until ? { until } : {}),
why, by, at,
};
const before = await this.store.get(this.stored(key));
await this.store.put(this.stored(key), value);
out.push({
...answered(key, value),
...(iface && outward.has(iface) ? { outward: iface } : {}),
...(before ? { replaced: answered(key, before.value) } : {}),
});
}
await this.say({ node: this.node, change: "allowed", allowances: out.map(({ replaced: _, outward: __, ...a }) => a), by, why, at });
return { allowances: out, said: true };
}
/** Take back allowances (§3): every key named is deleted, or, when one does not exist, it is refused by name and
* none is deleted. */
async disallow(args: Record<string, unknown>, by: string): Promise<{ allowances: Allowance[]; said: boolean }> {
this.proven("disallow", by);
only(args, ["port", "protocol", "interfaces", "why"]);
this.refuseAdopted("disallow", await this.askConverged("disallow"));
const port = portOf(args.port);
const protocol = protocolOf(args.protocol);
const interfaces = interfacesOf(args.interfaces);
const why = whyOf(args.why);
const found: Allowance[] = [];
for (const iface of interfaces) {
const key = keyOf(protocol, port, iface);
const entry = await this.store.get(this.stored(key));
if (!entry) throw new Error(`no allowance ${key} is kept on this node; nothing is deleted`);
found.push(answered(key, entry.value));
}
await this.recorded({ what: `disallowed ${found.map((a) => a.key).join(", ")} on ${this.node}`, why: `${why} (${by})`, cause: "allowance-taken-back", verb: "disallow" });
for (const a of found) await this.store.delete(this.stored(a.key));
await this.say({ node: this.node, change: "disallowed", allowances: found, by, why, at: this.now().toISOString() });
return { allowances: found, said: true };
}
/** Written in the hand-act log before the state is written; not written, the call is refused (§4, ADR 0565 §10). */
private async recorded(act: HandAct): Promise<void> {
try {
await this.asks.handAct(act);
} catch (err) {
throw new Error(`${act.verb}: the hand act was not recorded, so nothing is written: ${err instanceof Error ? err.message : String(err)}`);
}
}
/** Every allowance kept for this node only, sorted by key (§6); none on an adopted node, and refused while converged
* has never answered and does not answer now (H-A-318). One whose until passed before its timer fired is removed,
* said and left out, so none answered has expired. */
async allowed(args: Record<string, unknown> = {}): Promise<{ allowances: Allowance[] }> {
only(args, []);
const c = this.convergence ?? (await this.askConverged("allowed"));
if (c.state === "adopted") return { allowances: [] };
return { allowances: await this.kept() };
}
/** This node's allowances, every one past its until removed first (§11). */
private async kept(): Promise<Allowance[]> {
const now = this.now().getTime();
const allowances: Allowance[] = [];
for (const key of await this.ownKeys()) {
const entry = await this.store.get(this.stored(key));
if (!entry) continue;
if (entry.value.until && Date.parse(entry.value.until) <= now) {
await this.expire(key, entry.value.until);
continue;
}
allowances.push(answered(key, entry.value));
}
return allowances;
}
}
/** A seat verb held until the holder's start ends (§8, §11): the watch has handed every key over, every expiry it
* started has ended and started is said, so no verb answers before started; a start that failed refuses it with
* the start's error. */
export function held<A extends unknown[], R>(start: Promise<unknown>, run: (...args: A) => Promise<R>): (...args: A) => Promise<R> {
return async (...args) => {
await start;
return run(...args);
};
}
/** Each allowance marked loaded only when a rule of the mesh's table carries its comment `allowance <key> <at>` with
* an equal at, and every such comment whose key is not kept: loaded, not kept (§7, H-A-315 (5)). */
export function markLoaded(allowances: Allowance[], meshTable: string): {
allowances: (Allowance & { loaded: boolean })[];
loadedNotKept: { key: string; at: string }[];
} {
const comments = [...meshTable.matchAll(/comment "allowance (\S+) (\S+)"/g)].map((m) => ({ key: m[1], at: m[2] }));
const kept = new Set(allowances.map((a) => a.key));
const seen = new Set<string>();
const loadedNotKept = comments.filter((c) => !kept.has(c.key) && !seen.has(c.key) && seen.add(c.key));
return {
allowances: allowances.map((a) => ({ ...a, loaded: comments.some((c) => c.key === a.key && c.at === a.at) })),
loadedNotKept,
};
}
/** Whether the loaded mesh table's input chain accepts `protocol` `port` only with a source match (`ip saddr` or `ip6
* saddr`) and never from any source: a port a derived rule admits from the private network only (§3, H-A-315 (7),
* H-A-319). */
export function privateOnly(meshTable: string, protocol: Protocol, port: number): boolean {
const input = /chain input \{([\s\S]*?)\n\t?\}/.exec(meshTable)?.[1] ?? "";
let withSource = false;
for (const line of input.split("\n")) {
if (!/\baccept\b/.test(line) || !line.includes(`${protocol} dport `)) continue;
const ports = /dport (\{[^}]*\}|\S+)/.exec(line)?.[1] ?? "";
const names = ports.replace(/[{}]/g, "").split(",").map((p) => p.trim()).filter(Boolean);
const named = names.some((p) => {
const [lo, hi] = p.split("-").map(Number);
return hi === undefined ? lo === port : lo <= port && port <= hi;
});
if (!named) continue;
if (!/\bip6? saddr\b/.test(line)) return false;
withSource = true;
}
return withSource;
}
+27 -1
View File
@@ -15,10 +15,28 @@
"serves": [
"rules",
"reload",
"remove"
"remove",
"allow",
"disallow",
"allowed"
]
}
],
"requires": [
{
"name": "allowances",
"storage": "key-value",
"template": "state",
"options": {
"history": 1,
"per-machine": true
}
}
],
"invokes": [
"seat:node-engine.converged",
"seat:mesh-hand-acts.hand-act"
],
"filtering": {
"into": "/etc/nftables.conf"
},
@@ -71,6 +89,14 @@
"tools": [
"firewall_rules"
],
"config": {
"allowances-max": {
"type": "number",
"description": "The most allowances this machine keeps, which allow refuses past (design 33 control-verbs §3): it keeps the rendered chain readable and allowed in one answer.",
"trusted": false,
"default": 100
}
},
"build": {
"artifacts": [
{
+5 -3
View File
@@ -1,15 +1,17 @@
{
"name": "@novox/module-nftables",
"version": "0.1.0",
"description": "nftables — loads the mesh's packet filter and holds the node-packet-filter seat: its verbs rules, reload and remove (novox/hq ADR 0045, ADR 0170).",
"description": "nftables — loads the mesh's packet filter and holds the node-packet-filter seat: its verbs rules, reload, remove, allow, disallow and allowed (novox/hq ADR 0045, ADR 0170; design 47 H-A-304).",
"type": "module",
"private": true,
"scripts": {
"build": "tsc client.ts tools/index.ts --module NodeNext --moduleResolution NodeNext --target ES2022 --rootDir . --outDir dist",
"build": "tsc client.ts allowances.ts tools/index.ts --module NodeNext --moduleResolution NodeNext --target ES2022 --rootDir . --outDir dist",
"test": "node --test --experimental-strip-types 'test/*.test.ts'"
},
"dependencies": {
"@novox/mesh-sdk": "^0.1.1"
"@novox/mesh-sdk": "^0.1.1",
"@novox/mesh-connectivity-contracts-node-packet-filter": "^0.1.0",
"@novox/mesh-contracts-node-engine": "^0.1.0"
},
"devDependencies": {
"@types/node": "^22.0.0",
+345
View File
@@ -0,0 +1,345 @@
// The operator's allowances against a state in memory (ADR 0683; design 33 control-verbs §14; design 47 H-A-315,
// H-A-317, H-A-318, H-A-319).
import { test } from "node:test";
import assert from "node:assert/strict";
import { readFileSync } from "node:fs";
import {
Allowances, CONVERGED_BOUND_MS, LONGEST_WAIT_MS, MOST, STATE, held, markLoaded, privateOnly,
type AllowanceValue as Value, type AllowancesChanged as Changed, type Convergence, type HandAct, type Schedule, type Store,
} from "../allowances.ts";
interface Timer { ms: number; fire: () => void; cancelled: boolean }
// The loaded mesh table: the store's port from the private network only, 80 from anywhere and 8443 from both.
const TABLE = 'table inet mesh {\n\tchain input {\n\t\ttype filter hook input priority filter; policy drop;\n' +
'\t\tip saddr { 10.77.0.1, 10.77.0.2 } tcp dport 5432 accept\n\t\tip6 saddr { fd77::1 } tcp dport 5432 accept\n' +
'\t\ttcp dport 80 accept\n\t\tip saddr { 10.77.0.1 } tcp dport { 8443, 9500-9600 } accept\n\t\ttcp dport 8443 accept\n\t}\n}\n';
// gate, when given, holds every delete until it opens, and every get waits a turn of the event loop: a slow store.
function fake(at = "2026-11-21T20:00:00Z", node = "shanks", kept = new Map<string, Value>(), gate?: Promise<void>) {
const said: Changed[] = [];
const journal: string[] = [];
const timers: Timer[] = [];
const clock = { now: new Date(at) };
const schedule: Schedule = (ms, fire) => {
const t = { ms, fire, cancelled: false };
timers.push(t);
return () => { t.cancelled = true; };
};
const store: Store = {
get: async (k) => {
if (gate) await new Promise((r) => setImmediate(r));
return kept.has(k) ? { value: kept.get(k)! } : null;
},
put: async (k, v) => { kept.set(k, v); return kept.size; },
delete: async (k) => { await gate; kept.delete(k); },
keys: async () => [...kept.keys()],
};
const acts: HandAct[] = [];
const asks = {
convergence: { state: "converged", since: "2026-11-01T00:00:00Z" } as Convergence | "silent",
asked: 0,
most: MOST,
table: TABLE,
};
const a = new Allowances(store, node, async (b) => { said.push(b); }, {
converged: () => {
asks.asked++;
return asks.convergence === "silent" ? new Promise<Convergence>(() => {}) : Promise.resolve(asks.convergence);
},
handAct: async (act) => { acts.push(act); },
most: async () => asks.most,
meshTable: async () => asks.table,
}, () => clock.now, schedule, (l) => journal.push(l));
return { a, kept, said, clock, timers, journal, acts, asks };
}
/** The state's key of an allowance's key on shanks (H-A-317). */
const k = (key: string) => `shanks.${key}`;
const settle = () => new Promise((r) => setImmediate(r));
test("allow keeps one key per interface, and with none the default set's key, which names no interface", async () => {
const cases: { name: string; args: Record<string, unknown>; keys: string[] }[] = [
{ name: "defaults", args: { port: 8000, why: "the operator's dev server" }, keys: ["allow.tcp.8000"] },
{ name: "two interfaces", args: { port: 5353, protocol: "udp", interfaces: ["enp6s0", "wlan0"], source: "192.168.1.0/24", why: "w" },
keys: ["allow.udp.5353.enp6s0", "allow.udp.5353.wlan0"] },
{ name: "an interface with a dot", args: { port: 1, interfaces: ["enp6s0.10"], why: "w" }, keys: ["allow.tcp.1.enp6s0.10"] },
];
for (const c of cases) {
const { a, kept, said } = fake();
const got = await a.allow(c.args, "lead@shanks");
assert.deepEqual(got.allowances.map((x) => x.key), c.keys, c.name);
assert.equal(got.said, true, c.name);
assert.deepEqual([...kept.keys()].sort(), c.keys.map(k), c.name);
assert.deepEqual(said.map((b) => [b.change, b.node, b.by, b.why, b.allowances.map((x) => x.key)]),
[["allowed", "shanks", "lead@shanks", c.args.why, c.keys]], c.name);
for (const v of kept.values()) {
assert.equal(v.by, "lead@shanks", c.name);
assert.equal(v.at, "2026-11-21T20:00:00.000Z", c.name);
assert.equal(v.source, c.args.source, c.name);
if (!c.args.interfaces) assert.equal("interface" in v, false, "the private network's interface is never in the defaults");
}
}
});
test("allow naming an outward link answers it as outward, as converged lists them; another interface and the default set do not (H-A-325)", async () => {
const { a, asks, said } = fake();
asks.convergence = { state: "converged", since: "2026-11-01T00:00:00Z", tunnel: "mesh0", outward: ["eth1"] };
const named = await a.allow({ port: 8000, interfaces: ["eth1", "enp6s0"], why: "w" }, "lead@shanks");
assert.deepEqual(named.allowances.map((x) => [x.key, x.outward]), [["allow.tcp.8000.eth1", "eth1"], ["allow.tcp.8000.enp6s0", undefined]]);
assert.equal("outward" in named.allowances[1], false, "an interface that is no outward link answers without the field");
const none = await a.allow({ port: 8001, why: "w" }, "lead@shanks");
assert.equal("outward" in none.allowances[0], false, "the default set opens no outward link");
assert.equal(said.flatMap((b) => b.allowances).some((x) => "outward" in x), false, "the event's allowances carry no outward");
});
test("allow and disallow refuse what is not an allowance, writing nothing", async () => {
const bad: [string, Record<string, unknown>, RegExp][] = [
["port 0", { port: 0, why: "w" }, /port: /],
["port 65536", { port: 65536, why: "w" }, /port: /],
["port 80a", { port: "80a", why: "w" }, /port: /],
["protocol icmp", { port: 80, protocol: "icmp", why: "w" }, /protocol: /],
["interface", { port: 80, interfaces: ["a b"], why: "w" }, /interfaces: /],
["interface too long", { port: 80, interfaces: ["abcdefghijklmnop"], why: "w" }, /interfaces: /],
["source", { port: 80, source: "10.0.0.0/33", why: "w" }, /source: /],
["until in the past", { port: 80, until: "2026-11-21T19:00:00Z", why: "w" }, /until: .*not in the future/],
["until not a time", { port: 80, until: "tomorrow", why: "w" }, /until: /],
["until without seconds", { port: 80, until: "2026-11-22T08:00Z", why: "w" }, /until: .* with seconds/],
["empty why", { port: 80, why: "" }, /why: /],
["long why", { port: 80, why: "x".repeat(501) }, /why: /],
["unknown argument", { port: 80, why: "w", comment: "x" }, /comment: not an argument/],
["the store's port with no source", { port: 5432, why: "w" }, /port: tcp 5432 is admitted .* from given sources only/],
];
for (const [name, args, want] of bad) {
const { a, kept, said, acts } = fake();
await assert.rejects(a.allow(args, "x"), want, name);
assert.equal(kept.size, 0, name);
assert.equal(said.length, 0, name);
assert.equal(acts.length, 0, name);
}
const { a, kept } = fake();
await assert.rejects(a.allow({ port: 80, why: "w" }, "unproven"), /allow: refused: the request carries no caller the tool runner proved/);
await assert.rejects(a.disallow({ port: 80, why: "w" }, "unproven"), /disallow: refused: the request carries no caller/);
await a.allow({ port: 5432, source: "192.168.1.0/24", why: "the store from the home network" }, "x");
assert.deepEqual([...kept.keys()], [k("allow.tcp.5432")]);
await assert.rejects(a.disallow({ port: 80, why: "w", source: "10.0.0.1" }, "x"), /source: not an argument/);
await assert.rejects(a.disallow({ port: 80 }, "x"), /why: /);
await assert.rejects(a.allowed({ port: 80 }), /port: not an argument/);
});
test("a 101st allowance is refused at the default allowances-max; a replacement at the bound is not", async () => {
const { a, kept } = fake();
for (let p = 1; p <= MOST; p++) await a.allow({ port: p, why: "w" }, "x");
await assert.rejects(a.allow({ port: 1000, why: "w" }, "x"), /101 allowances would be kept on this node, more than allowances-max 100/);
assert.equal(kept.size, MOST);
await a.allow({ port: 1, why: "again" }, "x");
assert.equal(kept.get(k("allow.tcp.1"))!.why, "again");
});
test("an allowance kept on one node is absent from another node's allowed, and counts not toward its bound", async () => {
const shared = new Map<string, Value>();
const shanks = fake(undefined, "shanks", shared);
const ace = fake(undefined, "ace", shared);
ace.asks.most = 1;
await shanks.a.allow({ port: 8000, why: "w" }, "x");
assert.deepEqual((await ace.a.allowed()).allowances, []);
await ace.a.allow({ port: 9000, why: "w" }, "x");
assert.deepEqual([...shared.keys()].sort(), ["ace.allow.tcp.9000", "shanks.allow.tcp.8000"]);
assert.deepEqual((await shanks.a.allowed()).allowances.map((x) => x.key), ["allow.tcp.8000"]);
await assert.rejects(ace.a.disallow({ port: 8000, why: "w" }, "x"), /no allowance allow\.tcp\.8000 /);
});
test("on an adopted node allow and disallow are refused naming the adoption and allowed answers none; a silent converged refuses both", async () => {
const { a, kept, asks, timers, said } = fake();
await a.allow({ port: 8000, why: "w" }, "x");
asks.convergence = { state: "adopted", since: "2026-11-20T10:00:00Z", found: "ufw" };
await assert.rejects(a.allow({ port: 9000, why: "w" }, "x"), /allow: refused: this node is adopted \(the found firewall ufw, since 2026-11-20T10:00:00Z\)/);
await assert.rejects(a.disallow({ port: 8000, why: "w" }, "x"), /disallow: refused: this node is adopted/);
assert.deepEqual((await a.allowed()).allowances, []);
assert.deepEqual([...kept.keys()], [k("allow.tcp.8000")]);
asks.convergence = "silent";
const pending = a.allow({ port: 9000, why: "w" }, "x");
const bound = timers.at(-1)!;
assert.equal(bound.ms, CONVERGED_BOUND_MS);
bound.fire();
await assert.rejects(pending, /allow: refused: the node-engine's converged did not answer \(no answer within 10 s\)/);
const gone = a.disallow({ port: 8000, why: "w" }, "x");
timers.at(-1)!.fire();
await assert.rejects(gone, /disallow: refused: the node-engine's converged did not answer/);
assert.equal(said.length, 1);
});
test("allowed asks converged again while none answered, and refuses when it does not answer (H-A-318)", async () => {
const { a, asks, timers, said, journal } = fake();
asks.convergence = "silent";
const start = a.started();
timers.at(-1)!.fire();
await start;
assert.match(journal.join("\n"), /start: refused: the node-engine's converged did not answer/);
assert.equal(said.at(-1)!.change, "started");
const asked = asks.asked;
const pending = a.allowed();
timers.at(-1)!.fire();
await assert.rejects(pending, /allowed: refused: the node-engine's converged did not answer/);
assert.equal(asks.asked, asked + 1);
asks.convergence = { state: "converged" };
assert.deepEqual((await a.allowed()).allowances, []);
await a.allowed();
assert.equal(asks.asked, asked + 2, "an answer held is not asked again");
});
test("privateOnly: a port accepted only with a source match, never from any source", () => {
assert.equal(privateOnly(TABLE, "tcp", 5432), true);
assert.equal(privateOnly(TABLE, "tcp", 9550), true, "a range of a set");
assert.equal(privateOnly(TABLE, "tcp", 80), false);
assert.equal(privateOnly(TABLE, "tcp", 8443), false, "accepted from anywhere too");
assert.equal(privateOnly(TABLE, "udp", 5432), false);
assert.equal(privateOnly(TABLE, "tcp", 8000), false, "no derived rule");
});
test("allow then disallow writes the key, then deletes it, saying allowed and disallowed, with two hand-act log lines", async () => {
const { a, kept, said, acts } = fake();
const got = await a.allow({ port: 8000, interfaces: ["enp6s0"], why: "open" }, "x");
assert.equal(got.said, true);
assert.deepEqual((await a.allowed()).allowances.map((x) => x.key), ["allow.tcp.8000.enp6s0"]);
const gone = await a.disallow({ port: 8000, interfaces: ["enp6s0"], why: "done" }, "y");
assert.deepEqual(gone.allowances.map((x) => x.key), ["allow.tcp.8000.enp6s0"]);
assert.equal(gone.said, true);
assert.equal(kept.size, 0);
assert.deepEqual(said.map((b) => [b.change, b.by, b.why]), [["allowed", "x", "open"], ["disallowed", "y", "done"]]);
assert.deepEqual(acts, [
{ what: "allowed allow.tcp.8000.enp6s0 on shanks", why: "open (x)", cause: "allowance-set", verb: "allow" },
{ what: "disallowed allow.tcp.8000.enp6s0 on shanks", why: "done (y)", cause: "allowance-taken-back", verb: "disallow" },
]);
});
test("disallow is all or nothing: a key not kept is refused by name and none is deleted", async () => {
const { a, kept, said } = fake();
await a.allow({ port: 8000, interfaces: ["enp6s0"], why: "w" }, "x");
await assert.rejects(a.disallow({ port: 8000, interfaces: ["enp6s0", "wlan0"], why: "w" }, "x"), /no allowance allow\.tcp\.8000\.wlan0/);
assert.deepEqual([...kept.keys()], [k("allow.tcp.8000.enp6s0")]);
// no interfaces names the default set's key, never one of an interface
await assert.rejects(a.disallow({ port: 8000, why: "w" }, "x"), /no allowance allow\.tcp\.8000 /);
assert.equal(said.length, 1);
});
test("a second allow of one key replaces it whole and answers what it replaced; the source is no part of the key", async () => {
const { a, kept } = fake();
await a.allow({ port: 8000, source: "192.168.1.0/24", until: "2026-11-22T00:00:00Z", why: "first" }, "x");
const got = await a.allow({ port: 8000, source: "10.0.0.5", why: "second" }, "y");
assert.deepEqual(got.allowances.map((x) => [x.key, x.source, x.replaced?.source, x.replaced?.why]),
[["allow.tcp.8000", "10.0.0.5", "192.168.1.0/24", "first"]]);
assert.deepEqual([...kept.entries()].map(([key, v]) => [key, v.source, v.by, v.until]), [[k("allow.tcp.8000"), "10.0.0.5", "y", undefined]]);
});
test("at its until an allowance is deleted by its timer, says expired, is journalled, and allowed answers without it", async () => {
const { a, kept, said, clock, timers, journal } = fake();
await a.allow({ port: 8000, until: "2026-11-21T21:00:00Z", why: "for an hour" }, "x");
await a.allow({ port: 9000, why: "w" }, "x");
await a.watched({ key: k("allow.tcp.8000"), op: "put", value: kept.get(k("allow.tcp.8000"))! });
const live = timers.filter((t) => !t.cancelled);
assert.equal(live.length, 1, "one expiry timer; every converged bound is cancelled once answered");
assert.equal(live[0].ms, 3600_000);
clock.now = new Date("2026-11-21T21:00:00Z");
live[0].fire();
await settle();
assert.deepEqual([...kept.keys()], [k("allow.tcp.9000")]);
assert.deepEqual([said.at(-1)!.change, said.at(-1)!.allowances.map((x) => x.key)], ["expired", ["allow.tcp.8000"]]);
assert.match(journal.join("\n"), /the allowance allow\.tcp\.8000 .*for an hour\) expired at 2026-11-21T21:00:00Z and is removed/);
assert.deepEqual((await a.allowed()).allowances.map((x) => x.key), ["allow.tcp.9000"]);
});
test("the expiry timers follow the state: a delete cancels, a far until waits in steps, a stale timer removes nothing", async () => {
const { a, kept, clock, timers } = fake();
const v: Value = { port: 1, protocol: "tcp", why: "w", by: "x", at: "2026-11-21T20:00:00Z", until: "2026-11-21T22:00:00Z" };
await a.watched({ key: k("allow.tcp.1"), op: "put", value: v });
await a.watched({ key: k("allow.tcp.1"), op: "delete" });
assert.equal(timers.at(-1)!.cancelled, true);
const armed = timers.length;
await a.watched({ key: "ace.allow.tcp.1", op: "put", value: v });
assert.equal(timers.length, armed, "another node's key arms no timer");
await a.watched({ key: k("allow.tcp.2"), op: "put", value: { ...v, until: "2027-06-01T00:00:00Z" } });
assert.equal(timers.at(-1)!.ms, LONGEST_WAIT_MS);
await a.allow({ port: 9000, until: "2026-11-21T21:30:00Z", why: "w" }, "x");
await a.allow({ port: 9000, until: "2026-11-21T23:00:00Z", why: "w" }, "x");
clock.now = new Date("2026-11-21T21:30:00Z");
assert.equal(await a.expire("allow.tcp.9000", "2026-11-21T21:30:00Z"), false);
assert.ok(kept.has(k("allow.tcp.9000")));
// at a start, a key already past its until is removed at once
clock.now = new Date("2026-11-22T00:00:00Z");
await a.watched({ key: k("allow.tcp.9000"), op: "put", value: kept.get(k("allow.tcp.9000"))! });
await settle();
assert.ok(!kept.has(k("allow.tcp.9000")));
});
test("at a start every expired allowance is removed before started is said (H-A-312 (2))", async () => {
const { a, kept, said, clock } = fake();
await a.allow({ port: 8000, until: "2026-11-21T21:00:00Z", why: "w" }, "x");
await a.allow({ port: 9000, why: "w" }, "x");
clock.now = new Date("2026-11-21T22:00:00Z");
await a.started();
assert.deepEqual([...kept.keys()], [k("allow.tcp.9000")]);
assert.deepEqual(said.slice(-2).map((b) => [b.change, b.allowances.map((x) => x.key)]),
[["expired", ["allow.tcp.8000"]], ["started", ["allow.tcp.9000"]]]);
// §11, §14: a key past its until that the watch hands over while the start runs, through a store whose get and delete
// wait: the start waits for that expiry, so it is removed and said once, and started after it.
let open!: () => void;
const waits = new Promise<void>((r) => { open = r; });
const slow = fake("2026-11-21T22:00:00Z", "shanks", new Map(), waits);
const past: Value = { port: 7000, protocol: "tcp", why: "w", by: "x", at: "2026-11-21T20:00:00Z", until: "2026-11-21T21:00:00Z" };
slow.kept.set(k("allow.tcp.7000"), past);
await slow.a.watched({ key: k("allow.tcp.7000"), op: "put", value: past });
const start = slow.a.started();
for (let i = 0; i < 10; i++) await settle(); // the start runs as far as it goes while the expiry's delete waits
open();
await start;
assert.deepEqual(slow.said.map((b) => [b.change, b.allowances.map((x) => x.key)]),
[["expired", ["allow.tcp.7000"]], ["started", []]]);
});
test("started is said before the first allowed: the verbs are held until the start ends, a failed start refuses them", async () => {
const { a, asks, said, timers } = fake();
asks.convergence = "silent";
const start = a.started();
let answered = false;
const first = held(start, () => a.allowed())().then((r) => { answered = true; return r; });
await settle();
assert.equal(answered, false, "allowed answered while the start still waits on converged");
assert.equal(said.length, 0);
asks.convergence = { state: "converged", tunnel: "mesh0", outward: [] };
timers.find((t) => t.ms === CONVERGED_BOUND_MS && !t.cancelled)!.fire();
await first;
assert.deepEqual(said.map((b) => b.change), ["started"], "started was said before allowed answered");
const failed = Promise.reject(new Error("the allowances did not start: no bus"));
failed.catch(() => {});
await assert.rejects(held(failed, () => a.allowed())(), /did not start: no bus/);
});
test("a start says started with every allowance", async () => {
const { a, said } = fake();
await a.allow({ port: 8000, why: "w" }, "x");
await a.started();
assert.deepEqual([said.at(-1)!.change, said.at(-1)!.node, said.at(-1)!.allowances.map((x) => x.key)], ["started", "shanks", ["allow.tcp.8000"]]);
});
test("rules marks an allowance loaded only on a comment with an equal at, and lists a comment with no kept key as loaded, not kept", () => {
const table = 'table inet mesh {\n\tchain input {\n' +
'\t\tiifname != "mesh0" tcp dport 8000 accept comment "allowance allow.tcp.8000 2026-11-21T20:00:00.000Z"\n' +
'\t\tiifname != "mesh0" tcp dport 8001 accept comment "allowance allow.tcp.8001 2026-11-21T19:00:00.000Z"\n' +
'\t\tiifname != "mesh0" tcp dport 7000 accept comment "allowance allow.tcp.7000 2026-11-20T08:00:00.000Z"\n\t}\n}\n';
const base: Value = { port: 8000, protocol: "tcp", why: "w", by: "x", at: "2026-11-21T20:00:00.000Z" };
const got = markLoaded([{ key: "allow.tcp.8000", ...base }, { key: "allow.tcp.8001", ...base, port: 8001 }, { key: "allow.tcp.9000", ...base, port: 9000 }], table);
assert.deepEqual(got.allowances.map((x) => [x.key, x.loaded]), [["allow.tcp.8000", true], ["allow.tcp.8001", false], ["allow.tcp.9000", false]]);
assert.deepEqual(got.loadedNotKept, [{ key: "allow.tcp.7000", at: "2026-11-20T08:00:00.000Z" }]);
});
test("the manifest serves the six verbs, requires the state per machine, invokes converged and the hand-act log, and sets allowances-max", () => {
const m = JSON.parse(readFileSync(new URL("../module.json", import.meta.url), "utf8"));
assert.deepEqual(m.claims[0].serves, ["rules", "reload", "remove", "allow", "disallow", "allowed"]);
assert.deepEqual(m.requires, [{ name: STATE, storage: "key-value", template: "state", options: { history: 1, "per-machine": true } }]);
assert.deepEqual(m.invokes, ["seat:node-engine.converged", "seat:mesh-hand-acts.hand-act"]);
assert.equal(m.emits, undefined, "the claim says the seat's event (H-A-317)");
assert.equal(m.config["allowances-max"].default, MOST);
});
+131 -10
View File
@@ -1,35 +1,102 @@
// The packet filter's tools: the node-packet-filter seat's three verbs — what the machine enforces,
// reload the mesh's own, remove one thing the mesh did not write — and the module's own reading of
// the mesh's table (novox/hq ADR 0045, ADR 0168, ADR 0170).
// The packet filter's tools: the node-packet-filter seat's six verbs — what the machine enforces, reload the
// mesh's own, remove one thing the mesh did not write, and the operator's allowances (allow, disallow, allowed) —
// and the module's own reading of the mesh's table (novox/hq ADR 0045, ADR 0168, ADR 0170, ADR 0683; design 33
// control-verbs).
import { registerModuleTools, type ToolDefinition } from "@novox/mesh-sdk/tools";
import { callerOf } from "@novox/mesh-sdk/callers";
import { broker } from "@novox/mesh-sdk/messaging";
import { config, seatPublish } from "@novox/mesh-sdk/runtime";
import { keeps, state } from "@novox/mesh-sdk/state";
import {
allowancesChangedSubject, seat as packetFilter, verbs,
type AllowInput, type AllowedInput, type DisallowInput,
} from "@novox/mesh-connectivity-contracts-node-packet-filter";
import { seat as nodeEngine, verbs as nodeEngineVerbs } from "@novox/mesh-contracts-node-engine";
import { FirewallClient } from "../client.js";
import { Allowances, MOST, STATE, held, markLoaded, type AllowanceValue, type Convergence, type Store } from "../allowances.js";
export function getSeatVerbs(firewall: FirewallClient): ToolDefinition[] {
const port = { type: "integer", description: "the port, 1-65535" };
const protocol = { type: "string", description: "tcp (the default) or udp" };
const interfaces = {
type: "array",
items: { type: "string" },
description: "the input interfaces, each 1-15 letters, digits, _ . - (optional; none: every input interface that is neither the private network's tunnel nor an outward link, so an outward link is opened only when named)",
};
const why = { type: "string", description: "why, in words the operator reads (1-500 characters)" };
// The seat's verbs, held until the holder's start ends (§8, §11): started is said before any answers. The inputs' types
// name exactly the arguments of the seat's generated inputs; the holder validates every value itself.
export function getSeatVerbs(firewall: FirewallClient, allowances: Allowances, start: Promise<unknown> = Promise.resolve()): ToolDefinition[] {
const allowed = held(start, () => allowances.allowed());
return [
{
name: "rules",
name: verbs.rules,
description:
"The packet filter as this machine enforces it now: the nftables ruleset and, where the tool exists, the legacy filter's listings. Narrowed to one table or chain when asked.",
"The packet filter as this machine enforces it now: the nftables ruleset and, where the tool exists, the legacy filter's listings. Narrowed to one table or chain when asked. " +
"Beside it every allowance, marked loaded only when a rule of the mesh's table carries its comment allowance <key> <at> with an equal at, " +
"and every loaded allowance comment whose key is not kept, listed as loaded, not kept.",
input: {
table: { type: "string", description: "one nftables table, as `family name` (optional)" },
chain: { type: "string", description: "one chain of that table (optional)" },
},
run: async (args) => firewall.rules(args.table ? String(args.table) : undefined, args.chain ? String(args.chain) : undefined),
run: async (args) => {
const table = args.table ? String(args.table) : undefined;
const seen = await firewall.rules(table, args.chain ? String(args.chain) : undefined);
// The mark reads the mesh's own table whole, whatever the listing was narrowed to.
const mesh = !table && !args.chain ? seen.nftables : await firewall.ruleset();
const marked = markLoaded((await allowed()).allowances, mesh);
return { ...seen, allowances: marked.allowances, "loaded-not-kept": marked.loadedNotKept };
},
},
{
name: "reload",
name: verbs.reload,
description: "Load the mesh's own filter again from the file the mesh writes, and answer with the mesh's table as loaded.",
input: {},
run: async () => firewall.reload(),
},
{
name: "remove",
name: verbs.remove,
description:
"Remove one rule set the mesh did not write, named exactly as `node show` lists it: `chain X (iptables-legacy)` or `table ip6 filter, chain DOCKER-USER`. " +
"Refuses the mesh's tables, the runtime's own chains, a built-in chain and an active found firewall's chains. An operator's act, by name, never a flush.",
input: { where: { type: "string", description: "the rule set, as `node show` lists it" } },
run: async (args) => firewall.remove(String(args.where ?? "")),
},
{
name: verbs.allow,
description:
"Open a port by hand on this machine, beside the ports derived from the modules' listens: one allowance per protocol, port and interface, " +
"kept as this module's state and rendered by the node-engine into the inet mesh table's input chain at its next compose of the filter. " +
"Who set it is the caller the bus proved. A key that exists is replaced whole and answered. Written in the hand-act log. " +
"Refuses, naming the argument and writing nothing, an unproven caller, an adopted node or a converged that does not answer, a bad port, protocol, interface, source or until, " +
"an empty or longer why, an unknown argument, a port the loaded mesh table admits from given sources only unless a source is given, " +
"and more allowances on the node than the setting allowances-max (default 100). Whether the loaded chain carries it is read with rules.",
input: {
port,
protocol,
interfaces,
source: { type: "string", description: "one IPv4 or IPv6 CIDR it is opened to (optional; none: any)" },
until: { type: "string", description: "when it expires, an RFC 3339 time with seconds in the future (optional)" },
why,
} satisfies Record<keyof AllowInput, object>,
run: held(start, async (args: Readonly<Record<string, unknown>>) => allowances.allow({ ...args }, callerOf(args))),
},
{
name: verbs.disallow,
description:
"Take back allowances: the keys the protocol, port and interfaces name (none: the default set's key) are deleted, or, when one does not exist, " +
"it is refused by name and none is deleted. Written in the hand-act log. Refuses as allow does an unproven caller, an adopted node and a converged that does not answer.",
input: { port, protocol, interfaces, why } satisfies Record<keyof DisallowInput, object>,
run: held(start, async (args: Readonly<Record<string, unknown>>) => allowances.disallow({ ...args }, callerOf(args))),
},
{
name: verbs.allowed,
description:
"Every allowance kept for this machine only, with its key, port, protocol, interface, source, until, why, by and at; none on an adopted node. " +
"An allowance is removed at its until, so none answered has expired.",
input: {} satisfies Record<keyof AllowedInput, object>,
run: held(start, async (args: Readonly<Record<string, unknown>>) => allowances.allowed({ ...args })),
},
];
}
@@ -45,7 +112,61 @@ export function getFirewallTools(firewall: FirewallClient): ToolDefinition[] {
}
const firewall = FirewallClient.onThisMachine();
// The allowances are this module's own state, announced as kept (ADR 0495), one bucket per machine whose keys carry
// the node, so this holder watches and lists its own node's keys only (H-A-317); every change is said as the seat's
// event allowances-changed of this node; watching its own state gives every key with an until its timer, at every
// start too; and a start asks the node-engine's converged and says started, so a node-engine that refused a compose
// for want of an answer composes again.
const node = process.env.MESH_NODE ?? "";
const kept = state<AllowanceValue>(STATE);
const own = new Set<string>();
// The node's keys are known once the watch handed every current value over; a start that failed refuses every listing.
let watched!: { done: () => void; failed: (err: unknown) => void };
const watching = new Promise<void>((done, failed) => { watched = { done, failed }; });
watching.catch(() => {});
const store: Store = {
get: (key) => kept.get(key),
put: (key, value) => kept.put(key, value),
delete: (key) => kept.delete(key),
keys: async () => {
await watching;
return [...own];
},
};
const allowances = new Allowances(store, node, (body) => seatPublish(allowancesChangedSubject(node), body), {
converged: async () => (await broker().request<object, Convergence>(`seat:${nodeEngine}.${nodeEngineVerbs.converged}@${node}`, {})),
handAct: async (act) => { await broker().request("seat:mesh-hand-acts.hand-act", act); },
most: async () => {
const v = await config("allowances-max");
if (v === undefined || v === null || v === "") return MOST;
const n = Number(v);
if (!Number.isInteger(n) || n < 1) throw new Error(`the setting allowances-max is ${JSON.stringify(v)}, not a whole number of allowances`);
return n;
},
meshTable: () => firewall.ruleset(),
});
const start = (async () => {
// A node-engine package older than the verb converged names none: refused here, loudly, never asked as
// seat:node-engine.undefined (H-A-312).
if (!nodeEngineVerbs.converged) {
throw new Error("the node-engine seat's package names no verb converged (a package older than design 33 control-verbs §10)");
}
await keeps([{ name: STATE }]);
await kept.watch(async (change) => {
if (change.op === "put") own.add(change.key);
else own.delete(change.key);
await allowances.watched(change);
}, { key: `${node}.**` });
watched.done();
await allowances.started();
})().catch((err) => {
const failed = new Error(`the allowances did not start: ${err}`);
watched.failed(failed);
console.error(`[nftables] the allowances did not start (no expiry timers, no started said): ${err}`);
throw failed;
});
start.catch(() => {});
// The seat's verbs under the seat's name: the runtime serves them on the seat's subjects where this
// module holds it (ADR 0159, 0160). The module's own under its own.
registerModuleTools("node-packet-filter", () => getSeatVerbs(firewall));
registerModuleTools(packetFilter, () => getSeatVerbs(firewall, allowances, start));
registerModuleTools("nftables", () => getFirewallTools(firewall));
+2 -1
View File
@@ -10,6 +10,7 @@
},
"include": [
"client.ts",
"allowances.ts",
"tools/index.ts"
]
}
}
+5
View File
@@ -328,6 +328,11 @@ func (c keeperClient) exchange(r KeeperRequest) (a KeeperAnswer, err error, unan
defer conn.Close()
_ = conn.SetDeadline(time.Now().Add(KeeperTimeout))
if err := json.NewEncoder(conn).Encode(r); err != nil {
// A keeper that refuses the peer answers and closes without reading, so the write can meet a closed socket
// (broken pipe) while the refusal already waits to be read: the refusal is the answer (issue 1960).
if derr := json.NewDecoder(conn).Decode(&a); derr == nil && a.Error != "" {
return a, errors.New(a.Error), false
}
return KeeperAnswer{}, err, true
}
if err := json.NewDecoder(conn).Decode(&a); err != nil {
+17 -4
View File
@@ -8,6 +8,7 @@ import (
"os"
"path/filepath"
"strings"
"sync/atomic"
"testing"
)
@@ -91,10 +92,12 @@ func shortTempDir(prefix string) (string, error) {
// keeper answering it acts as with the bus down, with a warning (A-8).
func TestOnlyRootOpensTheKeepersSocketAndANeedlessKeeperIsTheBusDown(t *testing.T) {
st := memStates{}
uid := 0
// The peer's uid is read by the keeper's serving goroutines and switched by the test: an atomic, so no connection
// sees it half-written (issue 1960: a data race failed the test under -race).
var uid atomic.Int32
k := &Keeper{Emit: func(string, any) error { return nil }, States: st,
Call: func(string, any) (json.RawMessage, error) { return nil, errors.New("down") },
PeerOf: func(net.Conn) (int, error) { return uid, nil }}
PeerOf: func(net.Conn) (int, error) { return int(uid.Load()), nil }}
dir, err := shortTempDir("zk")
if err != nil {
t.Fatal(err)
@@ -121,13 +124,21 @@ func TestOnlyRootOpensTheKeepersSocketAndANeedlessKeeperIsTheBusDown(t *testing.
if _, err := c.call(ToolSchedule, map[string]any{}); err == nil || !strings.Contains(err.Error(), "down") {
t.Errorf("an invoke that failed: %v", err)
}
uid = 1000
uid.Store(1000)
if err := c.Put(StateSchedule, "x", 1); err == nil || !strings.Contains(err.Error(), "only root") {
t.Errorf("a peer that is not root: %v", err)
}
// A request larger than the socket's buffer: the keeper refuses and closes before it is written whole, so the write
// meets a closed socket every time; the peer is still told it is not root (issue 1960).
if err := c.Put(StateSchedule, "y", strings.Repeat("x", 1<<20)); err == nil || !strings.Contains(err.Error(), "only root") {
t.Errorf("a peer that is not root, its write cut: %v", err)
}
if _, left := st["schedule/x"]; left {
t.Errorf("a refused peer's call was read")
}
if _, left := st["schedule/y"]; left {
t.Errorf("a refused peer's cut call was read")
}
// No socket variable: the machine keeps no bus and says so.
t.Setenv(KeeperSocketEnv, "")
m := newFake(t).machine()
@@ -151,6 +162,8 @@ func TestOnlyRootOpensTheKeepersSocketAndANeedlessKeeperIsTheBusDown(t *testing.
if err != nil {
t.Fatal(err)
}
// The socket is bound before the call below, so the call waits in its backlog until Serve accepts it: the keeper
// is up from the moment Listen returns, whenever its goroutine runs.
lctx, lcancel := context.WithCancel(context.Background())
defer lcancel()
go (&Keeper{Emit: func(string, any) error { return nil }, States: memStates{}, PeerOf: func(net.Conn) (int, error) { return 0, nil },
@@ -159,7 +172,7 @@ func TestOnlyRootOpensTheKeepersSocketAndANeedlessKeeperIsTheBusDown(t *testing.
t.Errorf("a keeper that came up later is not used: %v", err)
}
t.Setenv(KeeperSocketEnv, sock)
uid = 0
uid.Store(0)
m2 := newFake(t).machine()
m2.JoinBus()
if m2.Emit == nil || m2.States == nil || m2.Call == nil || m2.keeper == nil {