Compare commits

...
Author SHA1 Message Date
jschoubben fc66555ecc Keep the client filling secrets and reading the controller's assignments until the lease write; plans for held modules; warrant says yes
Review round 3 (issue 1954): the node-engine of today writes ${secret:} as given and the new one fills only
provision keys, so the client fills secrets this release; plans exist only for modules with a want, so a
module without a seat plan is read from the controller's assignment and never leaves for want of one, and
the holder composes held modules at the build they run; the SDK's provider lets go only on "yes".
2026-10-30 00:25:12 +01:00
jschoubben 9cb177cae6 Pin mesh-sdk 6e4a301 in the modules the release changes, and keep the others on their pins (issue 774) 2026-10-30 00:23:45 +01:00
jschoubben fe52629c28 Keep unchanged modules on their pins, so the delivery moves only what this release changes (issue 774) 2026-10-30 00:15:44 +01:00
jschoubben 1b16720030 Pin mesh-sdk 007b6fd, the release's final head (issue 774) 2026-10-30 00:10:13 +01:00
jschoubben 8ae8e88ddd mesh-self-check: the setting probes is read by the module's code at runtime (ADR 0348 §5) 2026-10-30 00:05:05 +01:00
jschoubben 8432cb844f Merge remote-tracking branch 'origin/feat/774-l3-reland' into merge-leaves 2026-10-30 00:04:42 +01:00
jschoubben 818f8d2616 mesh-self-check: the setting probes lists the probes that run, D13 and D-fill by default; one unlisted runs nothing and the verb says it is off by setting (the operator, 2026-10-30) 2026-10-30 00:04:41 +01:00
jschoubben 475ffdcb33 gitea waits and keeps serving while its keys are unread; pin mesh-sdk 007b6fd
After a restart the forge-account provider returned a plain error, which the helper writes as failed
with its values cleared, so every consumer lost its forge entry until the keys loaded. provider.ErrNotReady
keeps the entry serving, in waits (issue 1954).
2026-10-30 00:04:14 +01:00
jschoubben 5a1916cb46 Merge remote-tracking branch 'origin/feat/774-l3-reland' into merge-leaves 2026-10-30 00:02:54 +01:00
jschoubben c1eb7783e8 Pin the newest mesh-sdk in the modules of the leaves (issue 1955) 2026-10-30 00:02:39 +01:00
jschoubben cb7a5c9939 mesh-provisioner composes through fromruntime's reader; pin mesh-sdk fa5a6ae across A3's modules
The SDK now reads a module's `reads` through the runtime, so busread.go's own copy goes, and the
manifest is held to frombus.StatesReads (issue 1954).
2026-10-30 00:02:38 +01:00
jschoubben 0e32f95492 Merge branch 'feat/774-l3-reland-leaves' into merge-leaves 2026-10-30 00:01:12 +01:00
jschoubben a783ffa497 Carry the kinds of D-root, D11 and D13's data held twice, which the operator dropped too, until the lease write (issue 1952) 2026-10-30 00:00:53 +01:00
jschoubben 0c0e8960d8 Carry the kinds of the probes the operator dropped (D2, D3, D4, D12, D-heard) until the lease write, so compare names them as ending with the controller and never as missing (issue 1952) 2026-10-30 00:00:09 +01:00
jschoubben 4a329f0443 Providers read the seat's plans and retire only on the operator's yes, removing the credential
ADR 0570 clauses 2 and 5: gitea, keycloak, postgres and route-adapter read mesh-provisioner.plans and
.provisions instead of the controller's assignments. postgres hands the database to a keeper and drops the
login; keycloak replaces the client's secret and keeps the client, so the data stays (issue 1954, V4).
2026-10-30 00:00:00 +01:00
jschoubben 6e5fd647c0 mesh-self-check: D13 raises empty-replacement and cleanup-waiting for retired readings and counts a missing backup from the first measurement; sdk 31e90aa (issue 1952, ADR 0233) 2026-10-29 23:58:11 +01:00
jschoubben 60231234a5 disk-load: keep a reading of an item no longer declared, retired with when and why, and clear it when declared again; sdk 31e90aa (issue 1955, ADR 0233) 2026-10-29 23:57:18 +01:00
jschoubben 936e7308bf mesh-self-check: give D-fill's written-finding fixture its severity, so the row keeping a fill between the thresholds reads the finding it wrote (issue 1952) 2026-10-29 23:56:20 +01:00
jschoubben 62b53c2eda docker: ask the seat mesh-artifact-sweep for the images a machine's declarations name, as ADR 0121 names the seat (issue 1955) 2026-10-29 23:55:58 +01:00
jschoubben c2209864ee mesh-self-check: D-fill judges every module's bucket and log against its cap from the bus module's nats_streams, with the controller's kind, key and words (issue 1952, ADR 0297 §6)
Its registry row and test rows are already in f7f02abb; this is the probe they call. Each KV_<module>_<name> and
LOG_<module>_<name> stream is said filling at three quarters of its max_bytes, and kept while a finding is written
until under seven tenths. One read of every stream, so the bus module not answering is the probe failing to run, which
clears nothing; bucket-or-log-unread is not raised.
2026-10-29 23:55:50 +01:00
jschoubben 50d46f6e94 Hold the seat by its name mesh-artifact-sweep, as ADR 0121 names a mesh seat for its scope (issue 1955, ADR 0571 clause 2) 2026-10-29 23:55:15 +01:00
jschoubben f7f02abbb3 mesh-self-check: D13 judges the data from disk-load's readings, restic's last good nights and the indexer's manifests, with the controller's kinds, keys and words (issue 1952, ADR 0233) 2026-10-29 23:54:54 +01:00
jschoubben 89f63b3ab8 mesh-self-check: DA judges each machine's agent account on its engine's own report, so the probe moves off the controller with its kind, key and words (issue 1952, ADR 0266)
The agent account a machine names comes from placement; the engine's verdicts on it from its health report
(mesh-host.health, NodeHealth.Root and Contract); whether that verdict is on now from the watchman's presence, since
the report is written on a change only. The judgement is the controller's: no report, not heard, an engine older than
the judging, no verdict, or any verdict not healthy is not confined; a verdict waiting for the setuid search is quiet
for twelve hours from when this self-check first saw it waiting. machine.<node>.agent-root, urgent, the controller's
words. Rows in TestTheBusProbes; pins mesh-sdk f3677eb.
2026-10-29 23:52:17 +01:00
jschoubben 542c3214c6 mesh-provisioner-client: read plans from the seat, record no needs, leave ${secret:} for the node-engine
ADR 0570 clauses 1, 2 and 6 move the plan to mesh-provisioner_plans, fold needs into the seat and give
the node-engine the secret's fill, so the client no longer holds a value (issue 1954, V3).
2026-10-29 23:51:33 +01:00
jschoubben 63023ca5fe Judge the store's mirror copies by what the builder's log says each build stood on, and the images a manifest pins by itself, so mirrors are recorded and image-missing covers what the controller judged (issue 1955, ADR 0571 clause 2) 2026-10-29 23:49:58 +01:00
jschoubben cab031bf10 nats: nats_streams answers each stream's max_bytes beside its size (issue 1952) 2026-10-29 23:48:41 +01:00
jschoubben f73f04b1e9 Remove the compiled programs of the build modules from the tree and ignore them (issue 1577) 2026-10-29 23:47:46 +01:00
jschoubben 86e1082fa0 restic: write each item's last good night when a night's files change, with no timer, on the sdk's layout; the night is judged by its unit and the awake process by the health tool (issue 1955) 2026-10-29 23:46:57 +01:00
jschoubben de86f8123d mesh-catalog: take builds in from the builder's log through builder_builds_since and its done keys, not the controller's built-before (issue 1955, ADR 0571 clause 9) 2026-10-29 23:46:57 +01:00
jschoubben b1fa4855e6 restic: judge the night by its health tool again beside an awake process, and keep each item's last good night as its own state good (issue 1955, ADR 0571) 2026-10-29 23:46:57 +01:00
jschoubben 9d0ebe556e disk-load: compose the node's data items through the sdk's runtime reader, not a copy of it (issue 1955) 2026-10-29 23:46:57 +01:00
jschoubben ce8ea7bbe2 restic: the night process is judged by its unit; the module has no runtime account for a tool check to stand beside (issue 1955) 2026-10-29 23:46:56 +01:00
jschoubben 03938f8244 mounts: learn where the mesh keeps data from disk-load's readings, not the controller's data (issue 1955, ADR 0571 clause 3) 2026-10-29 23:46:56 +01:00
jschoubben 4ddf702145 nfs-server: learn the backup holder's data paths from disk-load's readings, not the controller's data (issue 1955, ADR 0571 clause 3) 2026-10-29 23:46:56 +01:00
jschoubben 44a3f24456 restic: read each item's reading and peak from disk-load's state and the item list from the sdk; the measuring leaves the backup holder (issue 1955, ADR 0571 clause 3) 2026-10-29 23:46:56 +01:00
jschoubben f149768f5d disk-load: take the node's data items from the bus and the readings layout from the sdk, with no composed file (issue 1955, ADR 0571 clause 3) 2026-10-29 23:46:56 +01:00
jschoubben 45902cd728 minio: read who holds a seat from the placement seat's state, not the controller's seats (issue 1955, ADR 0571 clause 4) 2026-10-29 23:46:55 +01:00
jschoubben 34a6932288 nfs-server: read the backup seat's holder from the placement seat's state, not the controller's seats (issue 1955, ADR 0571 clause 4) 2026-10-29 23:46:55 +01:00
jschoubben 7b5728b1f3 docker: ask the artifact-store seat for the images a machine's declarations name, not the controller (issue 1955, ADR 0571 clause 2) 2026-10-29 23:46:55 +01:00
jschoubben 3d3524dd69 mesh-watcher: read who holds the controller and the bus from the placement seat's state, not the controller's seats (issue 1955, ADR 0571 clause 4) 2026-10-29 23:46:55 +01:00
jschoubben 2802fdf783 disk-load: measure the node's data items and keep each reading and seven-day peak as its own state readings (issue 1955, ADR 0571 clause 3) 2026-10-29 23:46:55 +01:00
jschoubben c2a6a6f6dd Read the seat artifact-store's findings in the condition keeper, so S25's image-missing comes from its holder and not the controller (issue 1955, ADR 0571 clause 2) 2026-10-29 23:45:09 +01:00
jschoubben 1b1019972d Hold the seat artifact-store in artifact-store-tools: the keep set from the store's listing, the wants and each module's five newest builds, the sweep at a build's end once the lease names no controller with each deletion recorded, and S25's image-missing (issue 1955, ADR 0571 clause 2) 2026-10-29 23:44:39 +01:00
jschoubben 1ba6c0de27 mesh-self-check: D8 reads placement's addresses and endpoints and asks each heard holder's ban list, so the probe moves off the controller with its kind, key and words (issue 1952, ADR 0564 §6)
The mesh's own addresses come from placement (each node's private address and its endpoint); who holds
node-intrusion-prevention, from seat-holders; who is heard, from the watchman's presence. Each heard holder is asked
seat:node-intrusion-prevention.banned@<node>, all at once. The finding is the controller's: machine.<node>.bans-the-mesh,
urgent, the same summary and words. A heard holder that does not answer makes the probe fail to run, never pass.
Rows in TestTheBusProbes; the manifest reads and invokes what the code does.
2026-10-29 23:44:35 +01:00
jschoubben bb18541469 The vault reads each plan from mesh-provisioner, says each credential's outcome in its own bound state, mints nothing for form none, and answers rotate and issued for any proven caller (issue 1954)
Each outcome names the value by the vault's own name and generation, never the value; held is provided, issued or given waits, an entry goes only on a retiring key's Approve. The vault no longer reads the controller's assignments, so it leaves the list of their readers. Pins mesh-sdk 6b8c694.
2026-10-29 22:58:53 +01:00
jschoubben a95ad78a9a Carry plan-failed and stalled until the lease write as kinds that end with the controller, as the lead decided, so compare names no kind without a home (issue 1952) 2026-10-29 22:53:53 +01:00
jschoubben 1b1a94f35b Give every alert kind the controller raises its home in the keeper's table, as the lead settled them under the operator's yes, so compare names no kind without one (issue 1952) 2026-10-29 22:52:54 +01:00
jschoubben 313aae7440 Declare the credential form of seven offers, read from each provider's code, so the plan says what the software makes (issue 1954)
A login per consumer is pair (postgres, keycloak's client, gitea's npm registry, the vault's secret); an offer whose consumers present none is none (gitea's git and forge account, route-adapter's route).
2026-10-29 22:51:19 +01:00
jschoubben 4cc94f06d2 Merge the leaves branch at dd9b6a04 into the release branch, with its check of the providers' read outcomes (issue 1952) 2026-10-29 22:50:21 +01:00
jschoubben d276452d46 Merge the leaves branch into the release branch, so it carries a5's work (issue 1952) 2026-10-29 22:50:09 +01:00
jschoubben dd9b6a0485 checks/credentialform: a module that provides and keeps a bound state must be read by mesh-provisioner (issue 1952) 2026-10-29 22:50:06 +01:00
jschoubben 1f0ff6ed98 Every offer of twenty providers says its credential form; a check refuses one that does not; re-pin sdk 6b8c694 (issue 1952) 2026-10-29 22:49:41 +01:00
jschoubben 451d790edd Judge each provider by what its manifest declares and fill what a provision depends on, so a templated pair provider is provided only after its credential (issue 1954)
A provider declaring bound is judged by its entries, any other by its own have, and only once every provision it depends on, its secret-<provision> first, is provided. The credential is the vault's own provision. One integration test runs the holder on a real bus over mesh-sdk's in-memory mesh: a want composes the key, the outcomes provide it, leaving the node asks. Pins mesh-sdk 6b8c694.
2026-10-29 22:49:26 +01:00
jschoubben bc2efe658c Merge the release branch; checks/health takes its newer sdk pin (issue 1952) 2026-10-29 22:45:43 +01:00
jschoubben bc2e8b1fd2 Read the outcome of a provider that makes no credential per consumer from its own have, so its consumers' keys turn provided once it runs healthy at its wanted build (issue 1954)
Failed is failed with the have's words; anything else stays asked, saying what it waits for. Providers that make a credential per consumer keep their bound entries.
2026-10-29 22:44:59 +01:00
jschoubben 6bee0b6789 Never set a provision provided without its provider's outcome, read from the entry's state and why, and pin mesh-sdk 692dbf9 (issue 1954)
A provider that writes no outcome leaves its keys asked (ADR 0570 clause 3), so a provider that fails is never hidden.
2026-10-29 22:43:16 +01:00
jschoubben 0a2334e2cd Compose every provision and plan in mesh-provisioner from the bus on change events, its state from the providers' outcomes, and ask before retiring (issue 1954)
The seat holds provisions and plans alone (ADR 0570 §1-§3, §5): a node is composed at its wanted builds and its newest, so a key ahead of its want is never retiring; a key no build needs is retiring with one ask, and the warrant goes into the key for its provider. The registry's placement kinds name plans; needs is emptied at start. Pins mesh-sdk 66ff33d.
2026-10-29 22:41:29 +01:00
jschoubben 799f03fc2a Refuse a manifest that records into the controller's hand-act log, naming the recorders' fallbacks until the lease write (issue 1953)
ADR 0565 §11's catalog check: every recorder records through mesh-hand-acts. The fallbacks are listed with why, and the list only shrinks.
2026-10-29 22:41:29 +01:00
jschoubben 077135a7f3 Keep mesh-config's held values in mesh-sdk's KeptValue layout, which the checks read, and pin mesh-sdk 66ff33d (issue 1953)
The layout and its keys are unchanged, so no value moves.
2026-10-29 22:41:29 +01:00
jschoubben 4117ec3a57 Serve compare, take-over and hand-back as the mesh-config seat's verbs, beside the module's own tools; pin mesh-sdk 4204afd (issue 1953) 2026-10-29 22:29:48 +01:00
jschoubben 45db0e650e Record a drill in mesh-hand-acts and count S15 over both logs, so the healers no longer refuse a drill (issue 1953) 2026-10-29 22:27:23 +01:00
jschoubben d34645361d Build config's switch from the controller's records: declared, compare, take-over and hand-back, so the operator approves the last move only once nothing differs (issue 1953)
compare also resolves every placed module's values on every node both ways, as ADR 0565's insight of 2026-10-29 asks after #558. After the take-over the predefined list and pull requests' config come from the indexer and pull-declares is refused; a hand-back returns to the mirrors until the lease write. From the lease write a trusted value is set only on an Approve. Acts are recorded through mesh-hand-acts. Pins mesh-sdk 2e5cc4f.
2026-10-29 22:27:23 +01:00
jschoubben 0842361f85 Keep the hand-act log's acts 400 days, as ADR 0565 §10 has it, now that a module log may say its age; pin mesh-sdk 2e5cc4f (issue 1953) 2026-10-29 22:18:59 +01:00
jschoubben 371abdaad1 Wait on keys that expire on the bus instead of in-process timers, so the healers' settles and S15's fortnight survive a restart (issue 1952)
A settle and the fortnight end are keys of wakes whose time to live is the wait, heard on expiry; attempts and would keep each key a week by its own time to live. Rearm stays only for a settle that expired while no instance watched. Pins mesh-sdk bc6c8a6.
2026-10-29 22:17:29 +01:00
jschoubben 4cbdd0fc74 Give each delivery's hold bound its own key's time to live and map a seat's finding to the module the placement says holds it, so the keeper holds the right findings for the right time (issue 1952 C7) 2026-10-29 22:16:23 +01:00
jschoubben b5fdcb88c8 Say checked, healer-acted and act-recorded on their seats' subjects, where their consumers hear them, and pin mesh-sdk fbd00c8 (issues 1952, 1953)
stdio.Emit put them on the module's subject, which no consumer of a seat's event hears. The manifests no longer list a seat's events as the module's own, which the SDK now refuses. mesh-hand-acts takes its act layout and names from mesh-sdk's handacts, and counts a repeated cause by its Repair rule.
2026-10-29 22:12:08 +01:00
jschoubben d632461eb9 Add the condition keeper, mesh-conditions, so conditions are raised and cleared from the observers' findings without the controller's lease, held while their module is delivered, and switched only on the operator's take-over (issue 1952 C1-C8, ADR 0564) 2026-10-29 22:05:45 +01:00
jschoubben 8fac0f89f7 Re-pin the release sdk; the nats tools' secret directory is /var/lib/mesh/nats-tools (issue 1952) 2026-10-29 22:01:47 +01:00
jschoubben 9ccb6f4060 Re-pin the release sdk; the nats tools' credential gets a directory of its own the tools can reach (issue 1952) 2026-10-29 22:00:20 +01:00
jschoubben 43c44bcbee checks/health: take the release sdk, which reads the build.artifacts bus word of the nats tools (issue 1952) 2026-10-29 21:57:14 +01:00
jschoubben 5b2b70cd6c Keep the hand-act log as mesh-hand-acts' module log, the stream ADR 0565 §10 names, so every act is held in its order (issue 1953)
The warrants stay a bucket and no longer age, so a warrant is recorded once; one whose act was lost is written again from the router's record. hand-acts says the log's fill against its cap.
2026-10-29 21:56:42 +01:00
jschoubben 48502c4ade nats: the tools read their credential from a word of their own, since the runtime strips MESH_BROKER_FILE (issue 1952 N1-N3) 2026-10-29 21:56:08 +01:00
jschoubben 2669fc29d8 Re-pin the release sdk; the bus module's tools declare the two removals they hold and leave the credential to the runtime (issue 1952 N1-N3) 2026-10-29 21:54:42 +01:00
jschoubben e1cdeebcdf Add mesh-hand-acts, holder of the seat mesh-hand-acts, so the hand-act log has a home the controller's lease does not hold (issue 1953)
Acts are kept 400 days under the runtime-proven caller and said as act-recorded; a warrant is read from the router's own record and recorded once per ask; hand-acts answers over both logs. The act layout is a copy of mesh-sdk's until L5-S2 lands.
2026-10-29 21:54:21 +01:00
jschoubben 9dd27d2edb Re-arm every running settle from the attempts kept when the healers start, so a restart loses no wake (issue 1952)
A settle that ended while they were down is judged at once. The README says the week-long bucket age is the deleter of would and attempts for this release.
2026-10-29 21:49:46 +01:00
jschoubben 885b7a3a8e Pin checks/health to the group's mesh-sdk 69fcb6c, so it knows the health seats the new modules claim (issue 1952) 2026-10-29 21:49:46 +01:00
jschoubben 4c11c2c0c0 Move the healers out of the controller into mesh-healers, acting on the keeper's events and only once the lease names no controller (issue 1952)
Until the lease write each act is written to would, kept a week by its bucket's age, so a would-act counts for no budget or brake. Acts begin by a create in attempts; H3 asks the grants part's assert; the brake and S15 are its own findings.
2026-10-29 21:48:09 +01:00
jschoubben babe2f194a Write the self-check's findings and checked in mesh-sdk's health layout, so the condition keeper reads them as it reads every observer's (issue 1952)
Pins the group's mesh-sdk head 69fcb6c. A seat verb's run="true" is taken as the seat schema gives it.
2026-10-29 21:48:09 +01:00
jschoubben 17588fee10 Pin the release sdk in the build modules so a change's own failing test ends its level (issue 1289 R4c-2) 2026-10-29 21:46:16 +01:00
jschoubben 77f6b4292c mesh-watcher: hear the self-check run the take-over key names; take the release sdk and go 1.26 (issue 1952 W1) 2026-10-29 21:46:15 +01:00
jschoubben 19bfbd82d4 messenger: hear the seat the take-over key names, so the keeper's conditions are not held beside the controller's (issue 1952 R1) 2026-10-29 21:46:15 +01:00
jschoubben f7d821b023 nats: the tools run on the process account nats-tools with its own secret, and take the release sdk, so the bus grants them only the four removals (issue 1952 N1-N3) 2026-10-29 21:46:15 +01:00
jschoubben d7247eae19 Move the self-check out of the controller into mesh-self-check, so its runs and findings no longer need the controller's lease (issue 1952)
The probes whose facts are on the bus run with their assertions unchanged (D6, D7, D15, DW, H-bus); the rest answer deferred, naming what they lack, never a pass. Findings go to its own bucket for the condition keeper; each run is kept a week and said as checked. The finding layout is a copy of mesh-sdk's S3 until that lands.
2026-10-29 21:43:55 +01:00
jschoubben aee377765e nats: tools that remove one consumer and one node's keys from a bucket, with why and an audit event (issue 1952 N1-N3) 2026-10-29 21:26:49 +01:00
jschoubben 16378e0850 Take mesh-config off the assignments readers: since #558 (issue 1953) it reads placement and its modules' manifests alone (issue 774) 2026-10-29 21:08:02 +01:00
jschoubben 071429c3be Pin mesh-sdk 22c9ddd in mesh-vault, the L3 re-land group's sdk head (issue 774) 2026-10-29 21:08:02 +01:00
jschoubben 3dd7b734aa Revert "Merge pull request 'Revert L3: its node-engine removed every held module on novox, the bus server among them (issue 774)' (#557) from revert/774-l3 into main"
This reverts commit 030f9ee4e7, reversing
changes made to 1555241393.
2026-10-29 21:05:14 +01:00
mesh-tools 96b129e544 Merge pull request 'mesh-config: read config declarations from each placed module's own manifests on the bus, never from the controller (issue 1953)' (#558) from fix/1953-config-from-builds into main 2026-10-29 19:47:55 +00:00
jschoubben 0d71d8b56c Move the head so the checker takes #558 again: a reopened pull request is not taken back as a candidate (recovery of de-spiegel, operator's yes 2026-10-29 17:56 CET)
mesh/check being carried to 0d71d8b56c55 on 96b129e54441 | from 030f9ee4e780
mesh/delivery merged: source.main moved to novox/mesh-catalog@96b129e54441
2026-10-29 18:10:27 +01:00
jschoubben 96a37f58ca mesh-config: read declarations from each placed module's own manifests as built, and placement alone, so a module from its own repository resolves (issue 1953)
mesh/delivery closed: the pull request closed unmerged
mesh/check pass 96a37f58ca76 on 030f9ee4e780 | unit 1173 run, 3272 kept, 13 skipped | integration 1212 run, 3274 kept, 22 skipped | over bound: unit, …
The controller's declared-config mirror never held de-spiegel's keys, and Resolve skipped
an assigned module it held no declarations of, silently: de-spiegel stayed held on novox
with its values set. Declarations now come from the indexer's trunk manifest with the
wanted build's laid over it per node; which module runs where from the seat
mesh-placement alone. A placed module with no readable manifest is listed as unresolved
with why, logged, and keeps its last resolved values.
2026-10-29 15:19:24 +01:00
mesh-tools 030f9ee4e7 Merge pull request 'Revert L3: its node-engine removed every held module on novox, the bus server among them (issue 774)' (#557) from revert/774-l3 into main 2026-10-29 08:59:53 +00:00
jschoubben c8240c38bb Revert L3 (mesh-catalog, merge 15552413): its node-engine removed every held module on novox, the bus server among them, at 08:54 CET on 2026-10-29 (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-catalog@030f9ee4e780
mesh/delivery-group group revert/774-l3 merged: every member is merged
mesh/check being carried to c8240c38bb89 on 030f9ee4e780 | from 1555241393fe | novox/mesh-catalog#557 in group revert/774-l3
L3 lands again with two fixes: a held module is never removed, and the grants cover L3's new subjects.
2026-10-29 09:42:30 +01:00
mesh-tools 1555241393 Merge pull request 'L3: mesh-vault mints, retires and signs; nats makes its certificate; messenger asks each engine (issue 774, ADR 0561), rebased onto L2' (#556) from feat/774-vault-mints-l2 into main 2026-10-29 07:44:23 +00:00
jschoubben add6f143ca mesh-delivery-gate: an answer stands only for the ask it answers, and a merge never handed says why
mesh/delivery merged: source.main moved to novox/mesh-catalog@1555241393fe
mesh/delivery-group group feat/774-vault-mints-l2 merged: every member is merged
mesh/check pass add6f143ca23 on 158348ede789 | unit 1601 run, 5122 kept, 47 skipped | integration 1637 run, 5215 kept, 49 skipped | over bound: unit, …
On 2026-10-29 the group feat/774-vault-mints-l2 was approved and never handed: hq#495 kept a Decline of an earlier ask beside the Approve of the next, readyAsks skipped it and handTurn returned nil for every member, with no log line. The Approve and a new ask now clear an earlier Decline, a run's answer names its ask (answered_ask), an approved merge that waits for nothing and is not handed logs why once and is a stalled line, and delivery_turns lists no unwritten turn while the check gates no merge.
2026-10-29 02:15:34 +01:00
jschoubben b46089b698 Merge remote-tracking branch 'origin/main' into l3-mc-main 2026-10-29 02:15:33 +01:00
mesh-tools 158348ede7 Merge pull request 'mesh-delivery-gate asks the operator's Approve on a verdict only: the one-hour fallback goes (issue 1733)' (#555) from fix/1733-no-hour into main 2026-10-28 21:29:52 +00:00
mesh-tools ff4c6c0435 Merge pull request 'The Go, Node and image build modules watch their settings file and say a capacity of 0 in their health' (#554) from fix/park-wakes into main 2026-10-28 21:03:17 +00:00
jschoubben c193d2b389 Pin mesh-sdk 8eafc0c, the L3 group's head on mesh-sdk main with #120 and #121 (issue 774)
mesh/delivery-group group feat/774-vault-mints-l2 ready: every member ready, and composed together they pass
mesh/delivery closed: a newer head of the same pull request
mesh/check pass c193d2b38970 on 158348ede789 | unit 1319 run, 5401 kept, 47 skipped | integration 1359 run, 5490 kept, 49 skipped | over bound: unit, …
2026-10-28 21:56:06 +01:00
jschoubben 10447d8c1e mesh-config reads the placement beside the controller's assignments while both stand, as L2 has it and the vault does, and is listed as a reader of them (issue 774, ADR 0549 §5; re-review) 2026-10-28 21:54:36 +01:00
jschoubben 5be6576117 Follow L2 under L3: the vault and mesh-config read the seat mesh-placement's placement, and the check that only listed modules read the controller's assignments moves into noplacement, as L2 deleted the placement check (issue 774, ADR 0567, ADR 0549 §3, §6) 2026-10-28 21:54:36 +01:00
jschoubben 24fad2b92a mesh-vault: pin mesh-sdk 4e6e5cc, the L3 group's head rebased on L2 (issue 774) 2026-10-28 21:54:36 +01:00
jschoubben d3eb5cae80 mesh-vault: secret-ask answers the callers its manifest names, each named as the runtime proved it, within the same bounds; the controller's relay as before (issue 774, ADR 0563 §8) 2026-10-28 21:54:36 +01:00
jschoubben 2043200201 messenger: ask each machine's own engine whether it is root-free, the node seat node-engine's root-free, not the controller's (issue 774, ADR 0563 §6) 2026-10-28 21:54:36 +01:00
jschoubben ef618e8199 nats: the bus's certificate command moves here from mesh-controller with the same output, made once and checked, so the bus module makes its own certificate when a mesh is raised from nothing (issue 774, design 47 L3 s11) 2026-10-28 21:54:36 +01:00
jschoubben c00005c141 mesh-vault signs: its signing key and authority made once with their operator copies, the verb sign for a node's own certificate from bus state, the event signed, the state signers and the tools that read and reassert them (issue 774, ADR 0550 §2) 2026-10-28 21:54:36 +01:00
jschoubben bd5f170af0 The placement check's list of modules still reading the controller's assignments only shrinks against the trunk's copy (issue 774, ADR 0529 clause 2) 2026-10-28 21:54:36 +01:00
jschoubben 6bbb8329e4 mesh-config resolves for the modules the catalogue's placement puts on a node alone, and the placement check refuses any other module that reads the controller's assignments unless listed with why (issue 774, ADR 0549 §3, §6) 2026-10-28 21:54:36 +01:00
jschoubben e6b913cac7 mesh-vault: account_retire refuses while the controller holds the lease or before the wants are read whole, as the passes do (issue 774, review round 1) 2026-10-28 21:54:36 +01:00
jschoubben 411c1b8de8 mesh-vault: prove the vault whole on a real bus mints only from the lease write, and try a failed mint again at the pass; adopt the operator key before the watches open (issue 774, ADR 0561 §1-§4) 2026-10-28 21:54:36 +01:00
jschoubben 64e6688ec8 mesh-vault: tell where modules stand and which node holds the bus from the catalogue's placement first, so the vault does not depend on the controller's assignments for them (issue 774, ADR 0549 §3, §5) 2026-10-28 21:54:36 +01:00
jschoubben 2d57c9663a mesh-vault: serve issue-account, the hand path that mints one wanted account now, and record a rotation at most once per loaded list's revision; a covered account still never rotates in place (issue 774, ADR 0561 §4, §9) 2026-10-28 21:54:36 +01:00
jschoubben 5c1729523b mesh-vault: retire an account the grants part no longer wants only within ADR 0230's bounds, keeping its value, and re-enable it when wanted again; account_retire for a person past the bound (issue 774, ADR 0561 §2) 2026-10-28 21:54:36 +01:00
jschoubben 1628db6c80 mesh-vault: mint only while the controller's lease is handed away, read afresh before each mint, and say in vault_status what is held back and why (issue 774, ADR 0561 §3, §9) 2026-10-28 21:54:36 +01:00
jschoubben 0b9f19b612 mesh-vault: mint each account the grants part wants and the vault holds no value or adopted hash for, in the controller's shape, hashed before account-issued and secret-changed (issue 774, ADR 0561 §1, §2, §4) 2026-10-28 21:54:36 +01:00
mesh-tools 39ce149f74 Merge pull request 'mesh-build-go's compose integration test gives its module one slot (issue 1126)' (#553) from fix/1126-capacity-transition into main 2026-10-28 20:24:49 +00:00
jschoubben f37f93e812 Drop the unreachable 'before its check ends' caveat (issue 1733)
mesh/delivery merged: source.main moved to novox/mesh-catalog@158348ede789
mesh/delivery-group group fix/1733-no-hour merged: every member is merged
mesh/check pass f37f93e812df on ff4c6c04358a | carried from mesh-delivery 1f6f4eda3208 | unit 470 run, 4760 kept, 13 skipped | integration 503 run, 48…
2026-10-28 21:00:09 +01:00
jschoubben cf4879eaa1 Ask the Approve when the check's verdict arrives, not after an hour (issue 1733) 2026-10-28 20:59:46 +01:00
jschoubben 1e377fe862 Go build module: the compose test gives its module a slot, since a node with no capacity set now pulls nothing (issue 1126, ADR 0567 §15)
mesh/delivery merged: source.main moved to novox/mesh-catalog@39ce149f7435
mesh/check checking 1e377fe8628c on 39ce149f7435
The test built its module with no settings file and no Capacity; the new
buildstep.NodeCapacity answers 0 for that (no default names a node), so the
puller took no ask and the test timed out after 90 s in check 205f634c30df.
Failed first locally at origin/main af89893d (90.1 s), passes with the slot (2.5 s).
2026-10-28 20:23:42 +01:00
jschoubben 5c6899e641 Build modules: pin mesh-sdk 9e5666b (fix/park-wakes)
mesh/delivery merged: source.main moved to novox/mesh-catalog@ff4c6c04358a
mesh/delivery-group group fix/park-wakes merged: every member is merged
mesh/check pass 5c6899e64166 on 39ce149f7435 | judged by the candidate's mesh-build-go at 01347a5ab27d
2026-10-28 20:08:55 +01:00
jschoubben 848c0cbd03 Build modules: give the puller the settings file and say a capacity of 0 in health (needs the mesh-sdk fix/park-wakes pin) 2026-10-28 20:02:30 +01:00
mesh-tools af89893d42 Merge pull request 'No repository names a node: placement files, particulars and per-node build defaults go (issue 1126, delivery 2 of 2)' (#552) from feat/1126-placement-seat into main 2026-10-28 18:33:00 +00:00
jschoubben 105c4d628c Particulars check: skip the real-catalogue test when the checker hands no node names yet (ADR 0567 §16)
mesh/delivery merged: source.main moved to novox/mesh-catalog@af89893d4202
mesh/delivery-group group feat/1126-placement-seat merged: every member is merged
mesh/check failed: integration: meshbuildgo/cmd/mesh-build-go… | 105c4d628cbc on d836a31aa721 | judged by the candidate's mesh-build-go at 565c52f06c02
2026-10-28 18:53:44 +01:00
jschoubben 59cad7154a Pin mesh-sdk to the group's head 38dbe1d in the build modules, mesh-config, the gate and the vault; the vault's test answers by proposal (issue 1126) 2026-10-28 18:52:35 +01:00
jschoubben 3b35e78733 build-hours takes one window and names no node, node names enter digests and checks by their clauses, and a leftover particulars file is refused 2026-10-28 18:35:06 +01:00
jschoubben e072abb35e Go build module: a package whose tests read MESH_NODE_NAMES has the node list in its digest, so a new node misses a kept verdict 2026-10-28 18:16:34 +01:00
jschoubben e389e770d9 Build modules: capacity and build hours are this node's own values, no default names a node; the Go test tool hands the check's node names to the tests 2026-10-28 18:15:33 +01:00
jschoubben 00613b438c gitea commit trees: drop the listed directories, which only the bridge's placement files used 2026-10-28 18:12:13 +01:00
jschoubben d1f42564e3 Node names come from the run, not a file of the repository: the particulars check reads MESH_NODE_NAMES and fails without it 2026-10-28 18:11:23 +01:00
jschoubben 5548e1030c mesh-config reads placement from the seat's state (held by the assigner), not the indexer's 2026-10-28 18:10:58 +01:00
jschoubben 7bb07ce349 New check: no placement directory and no manifest reading the indexer's placement, so the repository path cannot come back 2026-10-28 17:58:53 +01:00
jschoubben 4e3b140d43 Delivery gate: a placement change is no pull request path, so the ask drops its placement wording and case 2026-10-28 17:58:22 +01:00
jschoubben b5aa226649 Build capacity is an integer per node layer, default 0: no manifest names a node's share (manifest side; the reader follows mesh-sdk S6) 2026-10-28 17:57:57 +01:00
jschoubben 77b511a79a Delete the placement check: placement is bus state, no repository file holds it 2026-10-28 17:57:15 +01:00
mesh-tools d836a31aa7 Merge pull request 'power: its loop and boot tests run on a fake clock, so a busy build node cannot fail them (issue 1126)' (#551) from fix/power-loop-flake into main 2026-10-28 16:26:05 +00:00
mesh-tools 683ce7e37f Merge pull request 'dnsmasq starts at boot: its drop-in no longer orders after a target the package orders it before (issue 1126)' (#550) from fix/dnsmasq-cycle into main 2026-10-28 16:26:01 +00:00
jschoubben 3494548c88 power: the loop and run tests drive their own clock, so a loaded build node cannot starve their timing (flake in #549's check)
mesh/delivery merged: source.main moved to novox/mesh-catalog@d836a31aa721
mesh/check checking 3494548c8830 on d836a31aa721
2026-10-28 17:12:11 +01:00
mesh-tools 3feeefc70f Merge pull request 'gitea: set any field of a branch's protection or a repository's settings, saying whether each trunk stays protected (issue 1126, ADR 0566)' (#549) from feat/protect-as-trunk into main 2026-10-28 16:02:32 +00:00
jschoubben 3f552a17f6 dnsmasq: the drop-in no longer orders after network-online.target and its address wait is 2 s at most (cycle dropped the start at boot, issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-catalog@683ce7e37f29
mesh/check being carried to 3f552a17f697 on 683ce7e37f29 | from 3feeefc70f7c
2026-10-28 17:00:14 +01:00
jschoubben 16ea3824b5 gitea: gate and judge every trunk the gate's source names, not main alone
mesh/delivery merged: source.main moved to novox/mesh-catalog@3feeefc70f7c
mesh/check checking 16ea3824b5ea on 3feeefc70f7c
The operator, 2026-10-28: the gates and trunk_rule apply to every trunk,
the branches the mesh installs from, and later there may be more than
main. The trunks are read at the call from mesh-delivery-gate's source,
in one function (trunksOf), and fall back to main, said, only when it
names none. A force-push onto any trunk is refused, and trunk_rule says
whether the branch asked is a trunk.
2026-10-28 16:52:50 +01:00
jschoubben fabb62da00 gitea: refuse a force-push onto a rule covering main until the module can ask the operator's Approve for it
The operator, 2026-10-28: a call waits for their Approve only if it removes
branch protection or allows a force-push to main. No TypeScript tool can
ask yet (no asker in the TypeScript SDK; ADR 0286's mark is not built in
the runner), so such a call is refused with nothing sent, and every other
call runs at once. No tool removes a rule.
2026-10-28 16:48:55 +01:00
jschoubben 8068cf74a7 gitea: read a repository's settings back in the forge's own form, and take the settings it was missing
enable_prune is never in the forge's answer and mirror_interval comes back
as a Go duration, so a write that worked failed its read-back. The first is
now said as not read back, the second compared as a duration.
allow_merge_update, default_update_style and has_code are taken,
projects_mode needs has_projects true, and the description names what is
left out: the name, the tracker and wiki objects, and the mirror's
credentials.
2026-10-28 16:44:11 +01:00
jschoubben 6e97ba3867 gitea: fail a protection or settings write the forge answered and did not apply, and take every field gitea 1.27.3 edits
Gitea reads the push whitelist and force-push allowlist flags only beside
their parent set true and passes over a missing default branch, both with
a 200, so the tool said set for what did not change. It now refuses a
dependent field without its parent, reads the rule back and names each
field not applied, and says what the forge changed beside the call. The
bypass allowlist fields complete the 32. gitea_repo_settings_set_full sets
a repository's settings, its default branch among them, as the forge's
administrator: gitea_default_branch_set is refused as mesh-tools.
2026-10-28 16:39:54 +01:00
jschoubben 448310908c gitea: set any field of a branch protection rule through a tool, so a trunk's merge whitelist and override need no hand at the forge
gitea_branch_protection_set_full takes every field of gitea's branch
protection API by its own name, changes only those given, writes as the
forge's administrator, logs the required why, answers the rule before and
after, and says whether the controller's trunk rule (judgedRule in
mesh-control build_source.go) now holds. The operator, 2026-10-28: a
generic tool for any configuration; design 47 L2 needs de-spiegel's main
set as a trunk.
2026-10-28 16:25:16 +01:00
mesh-tools 58309a9729 Merge pull request 'power: every reboot, poweroff and halt is bounded at 10 minutes (group fix/shutdown-bound, issue 1126)' (#547) from fix/shutdown-bound into main 2026-10-28 15:05:06 +00:00
jschoubben 6724dab0cf power: bound every shutdown at ten minutes, not systemd's thirty (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-catalog@58309a972945
mesh/delivery-group group fix/shutdown-bound merged: every member is merged
mesh/check pass 6724dab0cf96 on 387ee59140c4 | unit 2192 run, 3794 kept, 57 skipped | integration 2208 run, 3829 kept, 57 skipped | over bound: unit, …
On ace's reboot of 2026-10-28 every unit had stopped within 17 s, yet the bus and
polkit were started again during the shutdown, nothing stopped them a second
time, and the reboot waited on reboot.target's 30-minute job timeout, silent,
until the operator forced it. node-power owns the shutdown on every machine, so
its drop-ins on reboot, poweroff and halt shorten the bound; each target keeps
its own forcing action, so the last phase still ends every process, unmounts
and syncs. The cause, node-tools starting bundles again while it stops, is
fixed in mesh-tools.
2026-10-28 15:30:54 +01:00
mesh-tools 387ee59140 Merge pull request 'mesh-build-image: a node that cannot build here now stays healthy and says not eligible, so it never holds the delivery (issue 1126)' (#546) from fix/1126-ineligible-healthy into main 2026-10-28 14:20:40 +00:00
mesh-tools 85bbeb37e3 Merge pull request 'mesh-vault: own answers a node within its bound — assignments from a watched mirror, one read per module, one synced record per answer (issue 1126)' (#545) from fix/1126-vault-own into main 2026-10-28 14:20:35 +00:00
jschoubben 363039f337 mesh-build-image README: a node without tun is healthy and not eligible, as the module now says (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-catalog@387ee59140c4
mesh/check checking 363039f337ad on 387ee59140c4
2026-10-28 15:01:42 +01:00
jschoubben a9d96b30be mesh-vault: a watch of the assignments begun again keeps nothing an earlier attempt took, and a module the mirror lacks is read directly before a credential is refused (issue 1126, review round 1)
mesh/delivery merged: source.main moved to novox/mesh-catalog@85bbeb37e3e0
mesh/delivery-group group fix/1126-vault-own merged: every member is merged
mesh/check pass a9d96b30be8d on 69056948bfa8 | unit 1639 run, 4201 kept, 46 skipped | integration 1674 run, 4215 kept, 47 skipped | over bound: unit, …
A watch that failed partway kept what it had taken; begun again, the runtime does not hand a key deleted meanwhile, so
a removed assignment read as standing and credentialRefusal could hand a consumer credential to a node no longer
running its module. Each attempt now begins from nothing held, and a late delivery to an older attempt changes nothing.
A 'not on the node' refusal is confirmed by a direct read first, since a watched mirror can be behind.
2026-10-28 14:55:36 +01:00
jschoubben 644a34cf49 mesh-build-image stays healthy on a node that cannot build here now (no tun) and says it is not eligible: the assigner read unhealthy as a failed delivery and held the build (issue 1126) 2026-10-28 14:50:28 +01:00
jschoubben 277a4d0060 mesh-vault answers own in one pass: assignments from its watch, each module's once, one synced record per answer (issue 1126)
Seed-9's probe (13:31-13:36Z): novox's own of 57 secrets and 146 credentials took 26 s, past the node-engine's 15 s,
and ace and novox refused their whole shadow answer. own read the controller's assignment over the bus once per secret
and per credential (203 reads), read every record of every node, and synced the hand-out record once per value, under
the one lock every node's answers share.

Now the vault watches the assignments mirror (a watch, never a poll) and reads it from memory once caught up, over the
bus until then; one answer reads each module's assignment once; only own/<node>/ records are read; and every hand-out
of one answer is recorded in one synced write, none handed when it cannot be.

Quick fix weighed: a longer bound in the node-engine (VaultWithin). Thorough fix chosen: the cost grows with every
module and every node, and the lock is shared by every shadow answer and every apply.
2026-10-28 14:47:20 +01:00
mesh-tools 69056948bf Merge pull request 'The kernel module loads tun at boot and keeps the running kernel's modules across an upgrade; mesh-build-image takes no ask where it cannot build (group fix/ace-tun)' (#543) from fix/ace-tun into main 2026-10-28 13:07:42 +00:00
mesh-tools 9cdac09ad9 Merge pull request 'knowledge-base.progress takes since: what changed in the last hour; issue-tracker.changes says each issue's moves from what to what' (#541) from feat/progress-since into main 2026-10-28 11:56:32 +00:00
jschoubben 86d37a14e7 mesh-build-image: pin mesh-sdk 1618837, the group's head (fix/ace-tun)
mesh/delivery merged: source.main moved to novox/mesh-catalog@69056948bfa8
mesh/delivery-group group fix/ace-tun merged: every member is merged
mesh/check pass 86d37a14e7ab on 9cdac09ad962 | carried from 25f615456423 | unit 2293 run, 3984 kept, 16 skipped | integration 2331 run, 4031 kept, 24 …
2026-10-28 12:50:47 +01:00
jschoubben d13f74df40 A node that cannot open /dev/net/tun takes no image build, and every node loads tun at boot and keeps its running kernel's modules
An image build node up 24 days had lost its running kernel's modules to a linux-lts upgrade, so tun
could not load and every RUN line failed in pasta. mesh-build-image now proves the tun device and,
whatever its look refuses, judges its node not eligible (capacity 0 with the reason), so the asks go
to a node that can build them. The kernel module loads tun at boot and declares kernel-modules-hook,
which keeps the running kernel's modules across an upgrade. Needs mesh-sdk fix/ace-tun (Puller.Eligible).
2026-10-28 12:44:58 +01:00
jschoubben dfe490d9e2 issue-tracker.changes reads each issue's moves over a window from the tracker's own log, so progress since says from what each issue moved, not only where it is
mesh/delivery merged: source.main moved to novox/mesh-catalog@9cdac09ad962
mesh/check being carried to dfe490d9e245 on 9cdac09ad962 | from 25f615456423
2026-10-28 12:33:01 +01:00
jschoubben 4f317201c9 knowledge-base.progress takes since: only what changed in a window, read from the tracker's and mesh-delivery's history, so a five-minute report says what moved (the operator, 2026-10-28) 2026-10-28 12:33:01 +01:00
mesh-tools 25f6154564 Merge pull request 'Retire the spelling catalogue in the words check, and refuse a compiled program anywhere in the repository (group words/catalog)' (#540) from words/catalog into main 2026-10-28 11:31:37 +00:00
jschoubben b8ba9dd966 Widen the no-binaries check to every tracked file: a program under checks/ is refused too
mesh/delivery merged: source.main moved to novox/mesh-catalog@25f615456423
mesh/delivery-group group words/catalog merged: every member is merged
mesh/check being carried to b8ba9dd9669c on 25f615456423 | from 774fcdfa93ad, hq 8f8393fa9343 | novox/mesh-catalog#540 in group words/catalog
2026-10-28 12:17:21 +01:00
jschoubben 002a0e1a12 Keep the claude-code fixture equal to the renderer's catalog wording 2026-10-28 12:17:21 +01:00
jschoubben 9bb85a37f2 Retire the spelling catalogue: the words check lists it and skips module paths in imports 2026-10-28 12:17:21 +01:00
mesh-tools 774fcdfa93 Merge pull request 'gitea announces from the forge's events, not a poll: an edited base and a close are announced at once, one pass at start (issue 1422)' (#539) from fix/1422-gitea-events into main 2026-10-28 11:16:46 +00:00
jschoubben 470687ea4c Gitea announces from the forge's events and one pass at the start, not by polling (issue 1422)
mesh/delivery merged: source.main moved to novox/mesh-catalog@774fcdfa93ad
mesh/check being carried to 470687ea4cce on 774fcdfa93ad | from 2a03c106f4b9
An edited pull request is announced into its new base at once; a close without a merge is announced by its event; the 30 s/5 min polls and the 10 min sweep are gone.
2026-10-28 12:05:03 +01:00
mesh-tools 2a03c106f4 Merge pull request 'Gate seeds a never-merged repository's trunk head and follows a retargeted base; gitea announces a retarget and sets a default branch and a pull request's base (issue 1422; group fix/1422-trunk-unknown)' (#537) from fix/1422-trunk-unknown into main 2026-10-28 10:34:46 +00:00
jschoubben f5d4b2f86e Gate: take a retargeted pull request's new base, so a retarget to main is seeded and asked and one away from main is not (issue 1422)
mesh/delivery merged: source.main moved to novox/mesh-catalog@2a03c106f4b9
mesh/delivery-group group fix/1422-trunk-unknown merged: every member is merged
mesh/check being carried to f5d4b2f86ed4 on 2a03c106f4b9 | from 9fd95693db54, mesh-delivery ae7ec702a5e9
A re-announcement updated the base head but never the base: a pull request retargeted from nox-mesh to main stayed 'into
nox-mesh' and was never asked or seeded, and one retargeted away from main kept its open ask.
2026-10-28 11:12:21 +01:00
jschoubben 7c2aa0a739 gitea: set a default branch and a pull request's base by named tools; announce a retargeted pull request again (issue 1422) 2026-10-28 11:07:31 +01:00
jschoubben e057670bc4 Gate: seed source.main from the latest announcement's base head, not the newest delivery's (issue 1422)
A delivery made earlier can be announced again after a newer one, with main's head moved on: its base_sha is the freshest.
The gate now keeps when each base_sha was announced and seeds from the latest.
2026-10-28 11:05:38 +01:00
jschoubben 0c6b7b5d64 Gate: seed source.main with the trunk head of a repository it names none of (issue 1422)
source.main moved only at a merge, so a repository never merged through the gate had no trunk head and the checker could
never judge its pull requests. The gate already hears main's head on each gitea.pull.updated (base_sha); it now adds it for
a repository source.main lacks, from the newest open pull request into main, in one compare-and-set write that never
replaces a head. A pull request into another base seeds nothing (ADR 0484 §11).
2026-10-28 11:03:40 +01:00
mesh-tools 9fd95693db Merge pull request 'mesh-vault takes what runs of a node's own secret through move-in, on the node's ask (issue 1126 C4, group fix/1126-seed-same)' (#535) from fix/1126-seed-same into main 2026-10-28 09:46:28 +00:00
mesh-tools 1a9570f5e9 Merge pull request 'messenger: say what waits in its burst window, and stop reporting a state failure that has passed (issue 1507)' (#536) from fix/1507-config-ask-silent into main 2026-10-28 09:17:59 +00:00
jschoubben d871258f8b Hold that the state error clears once a write succeeds again (issue 1507, review)
mesh/delivery merged: source.main moved to novox/mesh-catalog@1a9570f5e92f
mesh/check being carried to d871258f8b00 on 1a9570f5e92f | from 3fb7951524e6
2026-10-28 10:08:13 +01:00
jschoubben e7abe3e3e4 messenger: clear the state problem once a write succeeds again, so the status stops saying a failure that passed (issue 1507) 2026-10-28 10:05:09 +01:00
jschoubben 0974d1a57d mesh-vault's own_drift row carries the id as a node runs it, its module's name first, and the vault pins mesh-sdk 3028c7e (issue 1126, cause C4; review round 2)
mesh/delivery merged: source.main moved to novox/mesh-catalog@9fd95693db54
mesh/delivery-group group fix/1126-seed-same merged: every member is merged
mesh/check being carried to 0974d1a57d99 on 2a03c106f4b9 | from 9fd95693db54
2026-10-28 10:02:46 +01:00
jschoubben 6cea12a023 messenger: say what waits in the outbox in the history, so an ask held for its burst window does not read as lost (issue 1507) 2026-10-28 10:02:27 +01:00
jschoubben a17356a162 Fill go.sum for mesh-sdk 1062b33, the group's head (issue 1126) 2026-10-28 09:53:34 +01:00
jschoubben 35ad58e6e2 mesh-vault pins the group's mesh-sdk head 1062b33 (issue 1126; go.sum is filled once that head is pushed) 2026-10-28 09:51:48 +01:00
jschoubben da5da4f6f7 The vault takes what runs through move-in's replace_running, not a verb of its own, so no seat promises anything new (issue 1126, cause C4) 2026-10-28 09:48:14 +01:00
jschoubben 671a6a1f47 own-take-running has a person's switch, own_take_switch, kept in the vault's store (issue 1126, cause C4) 2026-10-28 09:39:16 +01:00
jschoubben 47ed384d97 The vault takes the value that runs on a machine over the one it was first handed, from that machine alone (issue 1126, cause C4)
move-in never replaces a value the vault holds, so where the controller made an own secret anew after the move, the
new value runs while the vault keeps the first. The verb own-take-running takes it from the node-engine of the machine
it is for, keeps the replaced value as the previous generation, raises the generation and says own-replaced without a
value; own_drift lists every own secret a node's shadow answer says another value of, mesh-wide. The module needs the
mesh-sdk head of fix/1126-seed-same-own-resync (delivery.OwnValuesDiffer, the seat's verb).
2026-10-28 09:37:54 +01:00
mesh-tools 3fb7951524 Merge pull request 'claude-code: render a 660 s timeout for the mesh console, so mesh_wait is not cut at a minute (issue 1871)' (#533) from fix/1871-wait-returns-early into main 2026-10-28 07:46:26 +00:00
jschoubben ae96afcaa8 Render a 660 s timeout for the mesh console and the MCP idle bound, so mesh_wait is not cut at a minute (issue 1871)
mesh/delivery merged: source.main moved to novox/mesh-catalog@3fb7951524e6
mesh/check being carried to ae96afcaa8da on 3fb7951524e6 | from 0c63b4c906c6
2026-10-28 03:52:32 +01:00
mesh-tools 0c63b4c906 Merge pull request 'mesh-delivery-gate: ask the operator's Approve only once the check has a verdict (issue 1733) — wait for its check before approving this one' (#530) from fix/ask-after-check into main 2026-10-27 23:25:26 +00:00
jschoubben 354d62340d mesh-delivery-gate: the group re-push row holds no verdict at the new heads, so only the missing-run rule keeps it unasked (issue 1733, review round 2)
mesh/delivery merged: source.main moved to novox/mesh-catalog@0c63b4c906c6
mesh/check being carried to 354d62340d4d on 0c63b4c906c6 | from 00d155da60f5
2026-10-28 00:07:39 +01:00
jschoubben dea390ac97 mesh-delivery-gate: the Approve's verdict bound counts from the latest head, restarts with its check, and speaks the operator's time (issue 1733, review round 1)
A group member made ready in an event handler has no ready run when its sibling
is looked at; the bound counted from the older sibling alone, so a re-push past
60 minutes asked at once as unanswered. Such a member now counts as ready now.

The bound also counts from a check starting again at the same heads, or its
verdict being taken away, so a pass that turns back into running never asks at
once as unanswered. Times in the wait and the ask are the operator's zone
(Europe/Brussels, as the messenger says them), and a placement change whose pass
holds no composition says that no node answered its placement.
2026-10-28 00:05:36 +01:00
mesh-tools 00d155da60 Merge pull request 'Take the placement files out again: every node composing its own fails (issue 1126, revert of #528) — approve now' (#529) from revert/1126-seed-4 into main 2026-10-27 22:59:36 +00:00
jschoubben 89c30c44be mesh-delivery-gate: while the check gates no merge, ask the Approve only once the check has a verdict (issue 1733)
The operator, 2026-10-27 23:54: "if you don't want me to approve it, don't send
the approval request". The gate asked the Approve the moment a pull request
opened, before its check answered, and five changes were approved unchecked.

Now the ask waits for the checker's verdict at the heads it covers (a group's
verdict of all its heads together; a placement change's verdict ends only once
its nodes answered, ADR 0559 §5). A pass or a failure is asked, the failure in
its words, since check-gates-merge false lets the operator merge it anyway. A
check with no verdict 60 minutes after the heads became ready is asked anyway,
saying so, and asked again in new words when its verdict comes.

Quick fix (only the ask's wording) against thorough (the one place that decides
when to ask, ungatedWaits): thorough, which is one place here.
2026-10-27 23:59:09 +01:00
jschoubben bd65c839a8 Take the placement files out again: every node composing its own fails (novox: nats' ${dir:mesh-state} not declared; ace, g14, shanks: the build index unread at a bus pause), so each returns to the controller's compositions (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-catalog@00d155da60f5
mesh/check checking bd65c839a881 on 00d155da60f5
2026-10-27 23:56:08 +01:00
mesh-tools b7a036e186 Merge pull request 'Seed the placement files a third time (issue 1126) — Approve ONLY after the lead says all four nodes answer 'same'' (#528) from feat/1126-seed-3 into main 2026-10-27 22:53:41 +00:00
jschoubben 3abf7cd50b Seed the placement files a third time, now that every node agent reads its placement only on a successful read and its shadow names every difference (issue 1126, ADR 0549 §4)
mesh/delivery merged: source.main moved to novox/mesh-catalog@b7a036e18693
mesh/check checking 3abf7cd50bdf on b7a036e18693
2026-10-27 23:52:35 +01:00
mesh-tools d7e8a8824e Merge pull request 'Take the placement files out again: since the seed no delivery reaches any node (issue 1126, revert of #525)' (#527) from revert/1126-seed-3 into main 2026-10-27 22:48:46 +00:00
jschoubben 83e9ce5b19 Take the placement files out again: since the seed no delivery reaches any node, because every node agent runs the controller's compositions while the wants ask them to compose their own (issue 1126, revert of #525)
mesh/delivery merged: source.main moved to novox/mesh-catalog@d7e8a8824e9c
mesh/check checking 83e9ce5b1957 on d7e8a8824e9c
2026-10-27 23:47:52 +01:00
mesh-tools 653cead338 Merge pull request 'Seed the placement files again from what each node runs now (issue 1126, ADR 0549 §4) — Approve only after every node answers 'same'' (#525) from feat/1126-seed-2 into main 2026-10-27 22:02:13 +00:00
jschoubben d1ca199265 Seed the placement files again from what each node runs now, so placement lives in the catalogue (issue 1126, ADR 0549 §4)
mesh/delivery merged: source.main moved to novox/mesh-catalog@653cead33815
mesh/check checking d1ca19926529 on 653cead33815
2026-10-27 22:04:46 +01:00
467 changed files with 29703 additions and 2573 deletions
+1 -1
View File
@@ -36,8 +36,8 @@ modules/mesh-build-image/cmd/mesh-build-image/mesh-build-image
/modules/mesh-build-go/mesh-build-go
/modules/mesh-delivery-gate/mesh-delivery-gate
modules/mesh-config/cmd/mesh-config/mesh-config
checks/placement/placement
# Built by their modules' build step (issue 1577), never committed.
modules/forticlient/cmd/forticlient-tools/forticlient-tools
modules/networkmanager/uplink-tools
checks/words/words
+5
View File
@@ -0,0 +1,5 @@
module checks/credentialform
go 1.26.0
require git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029231453-6e4a3014cbde
+2
View File
@@ -0,0 +1,2 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029231453-6e4a3014cbde h1:ddf/jKI9c87sQKDmZLoG6PMnvXwsNH16ByVveyuFME4=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029231453-6e4a3014cbde/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
+70
View File
@@ -0,0 +1,70 @@
// Package credentialform refuses an offer that does not say the form of its credential: an own secret, an issuer, or
// `form` pair or none (the lead, 2026-10-29). It reads every modules/*/module.json through mesh-sdk's
// CredentialFormProblems.
package credentialform
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"sort"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
// Problems is every offer of the manifests under root/modules that says no credential form, one line each.
func Problems(root string) ([]string, error) {
files, err := filepath.Glob(filepath.Join(root, "modules", "*", "module.json"))
if err != nil {
return nil, err
}
sort.Strings(files)
var out []string
reads := map[string]bool{}
var binders []string
for _, f := range files {
raw, err := os.ReadFile(f)
if err != nil {
return nil, err
}
var m catalogue.Manifest
if err := json.Unmarshal(raw, &m); err != nil {
return nil, fmt.Errorf("%s: %w", f, err)
}
out = append(out, catalogue.CredentialFormProblems(m)...)
var kept struct {
Reads []string `json:"reads"`
State []json.RawMessage `json:"state"`
}
_ = json.Unmarshal(raw, &kept) // reads may hold objects in another module; only the provisioner's are read
if m.Module == "mesh-provisioner" {
for _, r := range kept.Reads {
reads[r] = true
}
}
if len(m.Provides) > 0 && keepsBound(kept.State) {
binders = append(binders, m.Module)
}
}
for _, module := range binders {
if !reads[module+".bound"] {
out = append(out, fmt.Sprintf("%s provides and keeps a `bound` state, and mesh-provisioner's manifest does not read %s.bound", module, module))
}
}
return out, nil
}
// keepsBound says whether a manifest's `state` names `bound`, as a plain name or an object with a name.
func keepsBound(state []json.RawMessage) bool {
for _, s := range state {
var name string
var obj struct {
Name string `json:"name"`
}
if json.Unmarshal(s, &name) == nil && name == "bound" || json.Unmarshal(s, &obj) == nil && obj.Name == "bound" {
return true
}
}
return false
}
+49
View File
@@ -0,0 +1,49 @@
package credentialform
import (
"os"
"path/filepath"
"strings"
"testing"
)
func aCatalogue(t *testing.T, manifests map[string]string) string {
t.Helper()
root := t.TempDir()
for module, body := range manifests {
dir := filepath.Join(root, "modules", module)
if err := os.MkdirAll(dir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "module.json"), []byte(body), 0o644); err != nil {
t.Fatal(err)
}
}
return root
}
// An offer that says no form is refused and named; one that says pair, none or an own secret passes; and the real
// catalogue is clean.
func TestEveryOfferSaysTheFormOfItsCredential(t *testing.T) {
root := aCatalogue(t, map[string]string{
"silent": `{"module":"silent","provides":[{"name":"thing","scope":"mesh"}]}`,
"binder": `{"module":"binder","provides":[{"name":"d","credential":{"form":"pair"}}],"state":["bound"]}`,
"mesh-provisioner": `{"module":"mesh-provisioner","reads":["other.bound"]}`,
"says": `{"module":"says","provides":[{"name":"a","credential":{"form":"pair"}},{"name":"b","credential":{"form":"none"}},{"name":"c","credential":{"own":"jwt"}}]}`,
})
got, err := Problems(root)
if err != nil {
t.Fatal(err)
}
if len(got) != 2 || !strings.Contains(got[0], `silent provides "thing"`) ||
!strings.Contains(got[1], "binder") || !strings.Contains(got[1], "mesh-provisioner") {
t.Fatalf("refused %v, wanted the silent offer and the binder the provisioner does not read", got)
}
real, err := Problems(filepath.Join("..", ".."))
if err != nil {
t.Fatal(err)
}
if len(real) != 0 {
t.Fatalf("the catalogue has offers with no credential form:\n%s", strings.Join(real, "\n"))
}
}
+3
View File
@@ -0,0 +1,3 @@
# What these tests read outside their package, as paths from the repository's root (issue 1936).
# every manifest, for what each offer says of its credential's form.
modules/*/module.json
+8
View File
@@ -0,0 +1,8 @@
# The calls of the controller's hand-act log a manifest of this catalogue may still invoke, each `<module> <verb> <why>`:
# the fallback of a recorder (mesh-sdk handacts.Recorder) while the seat mesh-hand-acts has no holder that answers, until
# the controller's lease is written to a holder that is no controller. The check of this directory (handacts) refuses any
# other, and an entry whose module no longer invokes it: the list only shrinks, and it is empty before the lease write
# (novox/hq ADR 0565 §11; issue 1953, issue 774).
mesh-config hand-act fallback until the lease write: its sets, clears and held values record through mesh-hand-acts first
mesh-config warranted fallback until the lease write: its warrants record through mesh-hand-acts first
mesh-healers drill fallback until the lease write: a drill records through mesh-hand-acts first
+3
View File
@@ -0,0 +1,3 @@
module git.novox.be/novox/mesh-catalog/checks/handacts
go 1.22
+109
View File
@@ -0,0 +1,109 @@
// Command handacts refuses a manifest that records into the controller's hand-act log (novox/hq ADR 0565 §11): every
// recorder records through the seat mesh-hand-acts from its delivery, so no manifest invokes
// `seat:mesh-controller.hand-act`, `.warranted` or `.drill`. The recorders' fallbacks to the controller's log while the
// seat has no holder are named in checks/handacts/fallbacks, each module and verb with why; that list only shrinks, and
// an entry whose module no longer invokes the verb is refused too.
//
// go run . <catalog root>
//
// Exits 1 on a finding.
package main
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
)
// Verbs are the controller's hand-act verbs, as a manifest invokes them, and Allowed the list of named fallbacks.
var Verbs = []string{"hand-act", "warranted", "drill"}
const (
invokePrefix = "seat:mesh-controller."
Allowed = "checks/handacts/fallbacks"
)
// Findings is every refusal in the catalog at root, sorted.
func Findings(root string) ([]string, error) {
allowed := map[string]bool{}
raw, err := os.ReadFile(filepath.Join(root, Allowed))
if err != nil && !os.IsNotExist(err) {
return nil, err
}
for _, line := range strings.Split(string(raw), "\n") {
if f := strings.Fields(line); len(f) >= 3 && !strings.HasPrefix(f[0], "#") {
allowed[f[0]+" "+f[1]] = true
} else if len(f) > 0 && !strings.HasPrefix(f[0], "#") {
return nil, fmt.Errorf("%s: %q is not `<module> <verb> <why>`", Allowed, line)
}
}
manifests, err := filepath.Glob(filepath.Join(root, "modules", "*", "module.json"))
if err != nil {
return nil, err
}
var out []string
invoked := map[string]bool{}
for _, path := range manifests {
raw, err := os.ReadFile(path)
if err != nil {
return nil, err
}
var m struct {
Module string `json:"module"`
Invokes []string `json:"invokes"`
}
if json.Unmarshal(raw, &m) != nil || m.Module == "mesh-controller" {
continue
}
for _, inv := range m.Invokes {
verb, ok := strings.CutPrefix(strings.TrimSpace(inv), invokePrefix)
if !ok || !contains(Verbs, verb) {
continue
}
invoked[m.Module+" "+verb] = true
if !allowed[m.Module+" "+verb] {
out = append(out, fmt.Sprintf("%s invokes %s%s: record through seat:mesh-hand-acts.%s (novox/hq ADR 0565 §11), "+
"or name the fallback in %s", m.Module, invokePrefix, verb, verb, Allowed))
}
}
}
for entry := range allowed {
if !invoked[entry] {
out = append(out, fmt.Sprintf("%s names %q, which is no longer invoked: take it off the list, which only shrinks",
Allowed, entry))
}
}
sort.Strings(out)
return out, nil
}
func contains(list []string, s string) bool {
for _, x := range list {
if x == s {
return true
}
}
return false
}
func main() {
if len(os.Args) != 2 {
fmt.Fprintln(os.Stderr, "usage: handacts <catalog root>")
os.Exit(2)
}
found, err := Findings(os.Args[1])
if err != nil {
fmt.Println(err)
os.Exit(1)
}
for _, f := range found {
fmt.Println(f)
}
if len(found) > 0 {
os.Exit(1)
}
fmt.Println("no manifest records into the controller's hand-act log but the fallbacks named")
}
+59
View File
@@ -0,0 +1,59 @@
package main
import (
"os"
"path/filepath"
"strings"
"testing"
)
// No manifest invokes the controller's hand-act verbs but a named fallback; a named fallback no longer invoked is
// refused (novox/hq ADR 0565 §11's catalog test).
func TestFindings(t *testing.T) {
for _, row := range []struct {
name string
files map[string]string
want string // empty: clean
}{
{"clean", map[string]string{"modules/a/module.json": `{"module":"a","invokes":["seat:mesh-hand-acts.hand-act"]}`}, ""},
{"an unnamed call", map[string]string{"modules/a/module.json": `{"module":"a","invokes":["seat:mesh-controller.warranted"]}`},
"a invokes seat:mesh-controller.warranted"},
{"a named fallback, and the controller's own", map[string]string{
"modules/a/module.json": `{"module":"a","invokes":["seat:mesh-controller.drill"]}`,
"modules/mesh-controller/module.json": `{"module":"mesh-controller","invokes":["seat:mesh-controller.hand-act"]}`,
Allowed: "# why\na drill fallback until the lease write\n"}, ""},
{"a fallback no longer invoked", map[string]string{Allowed: "gone hand-act fallback\n"}, `names "gone hand-act"`},
{"another verb of the controller", map[string]string{"modules/a/module.json": `{"module":"a","invokes":["seat:mesh-controller.conditions"]}`}, ""},
} {
t.Run(row.name, func(t *testing.T) {
root := t.TempDir()
for path, text := range row.files {
p := filepath.Join(root, path)
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(p, []byte(text), 0o644); err != nil {
t.Fatal(err)
}
}
got, err := Findings(root)
if err != nil {
t.Fatal(err)
}
if all := strings.Join(got, "\n"); (row.want == "") != (all == "") || !strings.Contains(all, row.want) {
t.Fatalf("findings = %q, want one containing %q", all, row.want)
}
})
}
}
// The run of this check over the real tree.
func TestTheCatalogRecordsOnlyThroughTheSeat(t *testing.T) {
got, err := Findings(filepath.Join("..", ".."))
if err != nil {
t.Fatal(err)
}
for _, f := range got {
t.Error(f)
}
}
+3
View File
@@ -0,0 +1,3 @@
# What these tests read outside their package, as paths from the repository's root (issue 1936).
# every manifest, for what it invokes.
modules/*/module.json
+1 -1
View File
@@ -2,4 +2,4 @@ module checks/health
go 1.26.0
require git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023150321-a2c813a6b468
require git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029231453-6e4a3014cbde
+2 -2
View File
@@ -1,2 +1,2 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023150321-a2c813a6b468 h1:UcMee+IV2ii5E4dRZdQUaSaeh9IedTFYumuQbBiB0k0=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023150321-a2c813a6b468/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029231453-6e4a3014cbde h1:ddf/jKI9c87sQKDmZLoG6PMnvXwsNH16ByVveyuFME4=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029231453-6e4a3014cbde/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
+12 -6
View File
@@ -1,4 +1,4 @@
// Package nobinaries refuses a compiled program under modules/ (issue 1577): it is build output, made by the module's build step,
// Package nobinaries refuses a compiled program anywhere in the repository (issue 1577; first a checks/ binary slipped past a modules/-only rule): it is build output, made by the module's build step,
// and `go build` in a module's folder writes one beside go.mod where `git add -A` takes it.
package nobinaries
@@ -27,10 +27,10 @@ func Kind(head []byte) string {
return ""
}
// Files are the files of the catalogue's root that the change holds under modules/: the tracked ones when the root is a git
// checkout (an ignored binary a local build left is not the change's), else every file there.
// Files are the files of the catalogue's root that the change holds, anywhere: the tracked ones when the root is a git
// checkout (an ignored binary a local build left is not the change's), else every file there but .git's.
func Files(root string) ([]string, error) {
cmd := exec.Command("git", "-C", root, "ls-files", "-z", "--", "modules")
cmd := exec.Command("git", "-C", root, "ls-files", "-z")
if out, err := cmd.Output(); err == nil && len(out) > 0 {
var files []string
for _, f := range strings.Split(strings.TrimRight(string(out), "\x00"), "\x00") {
@@ -41,10 +41,16 @@ func Files(root string) ([]string, error) {
return files, nil
}
var files []string
err := filepath.WalkDir(filepath.Join(root, "modules"), func(p string, d os.DirEntry, err error) error {
if err != nil || d.IsDir() || !d.Type().IsRegular() {
err := filepath.WalkDir(root, func(p string, d os.DirEntry, err error) error {
if err != nil {
return err
}
if d.IsDir() && d.Name() == ".git" {
return filepath.SkipDir
}
if d.IsDir() || !d.Type().IsRegular() {
return nil
}
rel, _ := filepath.Rel(root, p)
files = append(files, rel)
return nil
+37 -2
View File
@@ -2,6 +2,7 @@ package nobinaries
import (
"os"
"os/exec"
"path/filepath"
"testing"
)
@@ -16,6 +17,7 @@ func TestFindNamesEachCompiledProgramAndNothingElse(t *testing.T) {
"modules/e/short": []byte("\x7f"),
"modules/f/empty": {},
"modules/g/Foo.class": {0xca, 0xfe, 0xba, 0xbe, 0, 0, 0, 52}, // Java, not Mach-O
"checks/words/words": []byte("\x7fELF\x02\x01\x01\x00rest"), // outside modules/ too
}
var names []string
for f, b := range files {
@@ -32,8 +34,8 @@ func TestFindNamesEachCompiledProgramAndNothingElse(t *testing.T) {
if err != nil {
t.Fatal(err)
}
if len(got) != 3 {
t.Fatalf("found %v; wanted modules/a/a, modules/b/cmd/b/b.exe and modules/c/c only", got)
if len(got) != 4 {
t.Fatalf("found %v; wanted modules/a/a, modules/b/cmd/b/b.exe and modules/c/c and checks/words/words only", got)
}
}
@@ -52,3 +54,36 @@ func TestTheCatalogueHoldsNoCompiledProgram(t *testing.T) {
t.Errorf("%s: a compiled program is build output: remove it and add it to .gitignore (issue 1577)", f)
}
}
// A compiled program outside modules/ is refused too (a binary under checks/ slipped past the modules/-only rule): Files holds
// every tracked file of a git checkout and every file of a plain folder, and Find names the binary in both.
func TestAProgramUnderChecksIsRefused(t *testing.T) {
for _, git := range []bool{true, false} {
root := t.TempDir()
p := filepath.Join(root, "checks", "words", "words")
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(p, []byte("\x7fELF\x02\x01\x01\x00rest"), 0o644); err != nil {
t.Fatal(err)
}
if git {
for _, args := range [][]string{{"init", "-q"}, {"add", "-A"}} {
if out, err := exec.Command("git", append([]string{"-C", root}, args...)...).CombinedOutput(); err != nil {
t.Fatalf("git %v: %v: %s", args, err, out)
}
}
}
files, err := Files(root)
if err != nil {
t.Fatal(err)
}
found, err := Find(root, files)
if err != nil {
t.Fatal(err)
}
if len(found) != 1 {
t.Errorf("git=%v: found %v; wanted checks/words/words (ELF)", git, found)
}
}
}
+1 -5
View File
@@ -1,5 +1 @@
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# every file under modules/ (git ls-files and a walk).
modules
# every tracked file of the repository (git ls-files, no path limit): the whole tree, so this package's verdict is never kept.
+4
View File
@@ -0,0 +1,4 @@
# The modules of this catalogue that still read the controller's assignments (mesh-controller.assignments), each with
# why. The check of this directory (noplacement) refuses any other module that reads them, and an entry here whose module no longer
# does: the list only shrinks, and it is empty before the controller's lease is written to a holder that is no
# controller (novox/hq ADR 0549 §3, §6; issue 774).
+7
View File
@@ -0,0 +1,7 @@
module git.novox.be/novox/mesh-catalog/checks/noplacement
go 1.22
require git.novox.be/novox/mesh-catalog/checks/trunk v0.0.0
replace git.novox.be/novox/mesh-catalog/checks/trunk => ../trunk
+170
View File
@@ -0,0 +1,170 @@
// Command noplacement refuses what the repository path of placement left behind (novox/hq ADR 0567 §10, §12).
//
// Placement, which module runs on which node, is bus state held by the seat mesh-placement; no file of the
// catalogue says it, and no manifest reads it from the delivery indexer, which no longer keeps it. It refuses:
//
// 1. a `placement` directory (or file) at the root of the catalogue;
//
// 2. a `particulars` file at the root (the mesh's node names come from the checker's run, §16);
//
// 3. a manifest naming a state, stream, event or bucket of the indexer's placement (`mesh-delivery-indexer.placement…`
// or `mesh-delivery-indexer_placement…`) in any string value, such as `reads`;
//
// 4. a manifest, other than mesh-controller's, that reads the controller's assignments (`mesh-controller.assignments`):
// every reader takes placement from the seat mesh-placement (novox/hq ADR 0549 §3, ADR 0567), and the lease write is
// refused while one still reads them (ADR 0549 §6; issue 774). The modules that still read them for what the
// placement does not carry are named in checks/noplacement/assignments-readers, each with why; that list only
// shrinks: an entry whose module no longer reads them is refused too.
//
// go run . <catalogue root>
//
// Exits 1 on a finding.
package main
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
)
// Prefixes are the names of the indexer's placement parts; a manifest string starting with one names them.
var Prefixes = []string{"mesh-delivery-indexer.placement", "mesh-delivery-indexer_placement"}
// Findings is every refusal in the catalogue at root, sorted.
func Findings(root string) ([]string, error) {
var out []string
if _, err := os.Lstat(filepath.Join(root, "placement")); err == nil {
out = append(out, "placement: the repository holds no placement; it is bus state of the seat mesh-placement (novox/hq ADR 0567 §10)")
}
if _, err := os.Lstat(filepath.Join(root, "particulars")); err == nil {
out = append(out, "particulars: the repository lists no node names; the checker hands them to the check (novox/hq ADR 0567 §16)")
}
manifests, err := filepath.Glob(filepath.Join(root, "modules", "*", "module.json"))
if err != nil {
return nil, err
}
sort.Strings(manifests)
for _, file := range manifests {
raw, err := os.ReadFile(file)
if err != nil {
return nil, err
}
var doc any
if err := json.Unmarshal(raw, &doc); err != nil {
return nil, fmt.Errorf("%s: %w", file, err)
}
module := filepath.Base(filepath.Dir(file))
for _, s := range named(doc, nil) {
out = append(out, fmt.Sprintf("%s: names %q, which the indexer no longer keeps; placement is the seat mesh-placement's (novox/hq ADR 0567 §12)", module, s))
}
}
readers, err := assignmentsReaders(root)
if err != nil {
return nil, err
}
out = append(out, readers...)
sort.Strings(out)
return out, nil
}
// AssignmentsRead is the controller's assignments as a manifest reads them, and Allowed the file, relative to the
// catalogue's root, that names the modules still reading them: `<module> <why>` per line, `#` a comment.
const (
AssignmentsRead = "mesh-controller.assignments"
Allowed = "checks/noplacement/assignments-readers"
)
// assignmentsReaders is every finding on the reads of the controller's assignments: a module other than the controller
// reading them that Allowed does not name, and a module Allowed names that no longer reads them (issue 774).
func assignmentsReaders(root string) ([]string, error) {
allowed := map[string]bool{}
raw, err := os.ReadFile(filepath.Join(root, Allowed))
if err != nil && !os.IsNotExist(err) {
return nil, err
}
for _, line := range strings.Split(string(raw), "\n") {
if f := strings.Fields(line); len(f) > 0 && !strings.HasPrefix(f[0], "#") {
allowed[f[0]] = true
}
}
manifests, err := filepath.Glob(filepath.Join(root, "modules", "*", "module.json"))
if err != nil {
return nil, err
}
var found []string
reads := map[string]bool{}
for _, path := range manifests {
raw, err := os.ReadFile(path)
if err != nil {
return nil, err
}
var m struct {
Module string `json:"module"`
Reads []string `json:"reads"`
}
if json.Unmarshal(raw, &m) != nil || m.Module == "mesh-controller" {
continue // a manifest that does not read is the manifest checks' to refuse
}
for _, r := range m.Reads {
if strings.TrimSpace(r) != AssignmentsRead {
continue
}
reads[m.Module] = true
if !allowed[m.Module] {
found = append(found, fmt.Sprintf("%s reads %s: read where modules run from mesh-delivery-assigner.placement, "+
"the seat mesh-placement's (novox/hq ADR 0549 §3, ADR 0567); the lease write is refused while a module reads "+
"the assignments (ADR 0549 §6)", m.Module, AssignmentsRead))
}
}
}
for module := range allowed {
if !reads[module] {
found = append(found, fmt.Sprintf("%s names %s, which no longer reads %s: take it off the list, which only shrinks",
Allowed, module, AssignmentsRead))
}
}
return found, nil
}
// named collects the strings of a decoded manifest that start with one of the Prefixes.
func named(v any, into []string) []string {
switch v := v.(type) {
case string:
for _, p := range Prefixes {
if strings.HasPrefix(v, p) {
return append(into, v)
}
}
case []any:
for _, e := range v {
into = named(e, into)
}
case map[string]any:
for _, e := range v {
into = named(e, into)
}
}
return into
}
func main() {
if len(os.Args) != 2 {
fmt.Fprintln(os.Stderr, "usage: noplacement <catalogue root>")
os.Exit(2)
}
found, err := Findings(os.Args[1])
if err != nil {
fmt.Println(err)
os.Exit(1)
}
for _, f := range found {
fmt.Println(f)
}
if len(found) > 0 {
os.Exit(1)
}
fmt.Println("no placement directory, no manifest reads the indexer's placement, and only the modules listed read the controller's assignments")
}
+88
View File
@@ -0,0 +1,88 @@
package main
import (
"os"
"path/filepath"
"strings"
"testing"
"git.novox.be/novox/mesh-catalog/checks/trunk"
)
func TestFindingsRefuseWhatThePlacementPathLeft(t *testing.T) {
for _, row := range []struct {
name string
files map[string]string
want string // empty: clean
}{
{"clean", map[string]string{"modules/a/module.json": `{"module":"a","reads":["mesh-delivery-assigner.placement"]}`}, ""},
{"a placement directory", map[string]string{"placement/ace.json": `{}`}, "placement: the repository holds no placement"},
{"a particulars file", map[string]string{"particulars": "ace\n"}, "particulars: the repository lists no node names"},
{"a read of the indexer's placement", map[string]string{"modules/a/module.json": `{"module":"a","reads":["mesh-delivery-indexer.placement"]}`}, `a: names "mesh-delivery-indexer.placement"`},
{"a read of its candidates", map[string]string{"modules/b/module.json": `{"module":"b","reads":["mesh-delivery-indexer.placement-candidates"]}`}, "placement-candidates"},
{"its event", map[string]string{"modules/c/module.json": `{"module":"c","on":[{"event":"mesh-delivery-indexer.placement-changed"}]}`}, "placement-changed"},
// Issue 774 (ADR 0549 §3, §6): a reader of the controller's assignments is refused unless the list names it, the
// controller's own manifest is not judged, and a name on the list whose module no longer reads them is refused.
{"an unlisted reader of the assignments", map[string]string{"modules/r/module.json": `{"module":"r","reads":["mesh-controller.assignments"]}`}, "r reads mesh-controller.assignments"},
{"a listed reader and the controller", map[string]string{"modules/l/module.json": `{"module":"l","reads":["mesh-controller.assignments"]}`,
"modules/mesh-controller/module.json": `{"module":"mesh-controller","reads":["mesh-controller.assignments"]}`, Allowed: "# why\nl its plan\n"}, ""},
{"a listed module that reads them no more", map[string]string{Allowed: "gone no longer reads\n"}, "names gone, which no longer reads"},
{"the seat's own state is allowed", map[string]string{"modules/d/module.json": `{"module":"d","reads":["mesh-placement.placement"]}`}, ""},
} {
t.Run(row.name, func(t *testing.T) {
root := t.TempDir()
for path, text := range row.files {
p := filepath.Join(root, path)
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(p, []byte(text), 0o644); err != nil {
t.Fatal(err)
}
}
got, err := Findings(root)
if err != nil {
t.Fatal(err)
}
if all := strings.Join(got, "\n"); (row.want == "") != (all == "") || !strings.Contains(all, row.want) {
t.Fatalf("findings = %q, want one containing %q", all, row.want)
}
})
}
}
// The script's run of this check over the real tree (novox/hq ADR 0529 clause 2).
func TestTheCatalogueHoldsNoPlacementOfTheRepositoryPath(t *testing.T) {
got, err := Findings(filepath.Join("..", ".."))
if err != nil {
t.Fatal(err)
}
for _, f := range got {
t.Error(f)
}
// Issue 774 (ADR 0549 §3, §6; the comparison of novox/hq ADR 0529 clause 2): the list of modules still reading the
// controller's assignments gains none the trunk's copy at the merge base does not hold. Its first copy gains nothing.
path, ok := trunk.Copy(t, Allowed)
if !ok {
return
}
names := func(path string) map[string]bool {
raw, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
out := map[string]bool{}
for _, line := range strings.Split(string(raw), "\n") {
if f := strings.Fields(line); len(f) > 0 && !strings.HasPrefix(f[0], "#") {
out[f[0]] = true
}
}
return out
}
base := names(path)
for module := range names(filepath.Join("..", "..", Allowed)) {
if !base[module] {
t.Errorf("%s gains %s, which the trunk's copy does not hold: the list only shrinks", Allowed, module)
}
}
}
+6
View File
@@ -0,0 +1,6 @@
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# the root, for a placement directory, and every manifest, for a read of the indexer's placement states.
?placement
modules/*/module.json
+22 -7
View File
@@ -2,8 +2,9 @@
// issue 1838): a module is for any mesh, so no manifest, file, code string, identifier, comment, test name or
// fixture of a covered module names a node of this one.
//
// The names are in the file `particulars` at the catalogue's root, one a line, `#` comments — the one place
// they may be written. The modules covered are in `covered` at checks/particulars/covered, one a line. A name is matched
// The names are the mesh's own, not the repository's: the checker hands them in the environment value
// MESH_NODE_NAMES, comma-separated, from the node records it reads (novox/hq ADR 0567 §16). A run with none fails
// loudly, since it would pass on anything. The modules covered are in `covered` at checks/particulars/covered, one a line. A name is matched
// whole and in any case; a hyphen, an underscore or a CamelCase boundary parts one, so a longer word holding a name is another word.
// The forge is not a node name: its host (git.novox.be) and the organisation path (novox/<repo>) are blanked
// before a line is read.
@@ -34,14 +35,28 @@ func main() {
if len(os.Args) > 1 {
root = os.Args[1]
}
os.Exit(check(root))
os.Exit(check(root, namesFromEnv()))
}
// NamesEnv is the environment value that carries the mesh's node names (mesh-sdk delivery.NodeNamesEnv).
const NamesEnv = "MESH_NODE_NAMES"
// namesFromEnv is the names NamesEnv holds, comma-separated, trimmed, empty entries left out.
func namesFromEnv() []string {
var out []string
for _, n := range strings.Split(os.Getenv(NamesEnv), ",") {
if n = strings.TrimSpace(n); n != "" {
out = append(out, n)
}
}
return out
}
// check runs the check over a catalogue and returns the exit code: 0 clean, 1 a finding, 2 a list unreadable.
func check(root string) int {
names, err := readList(filepath.Join(root, "particulars"))
if err == nil && len(names) == 0 {
err = fmt.Errorf("particulars lists no name — the check would pass on anything")
func check(root string, names []string) int {
var err error
if len(names) == 0 {
err = fmt.Errorf("no node names (%s is empty) — the check would pass on anything", NamesEnv)
}
var covered []string
if err == nil {
+9 -8
View File
@@ -74,26 +74,24 @@ func TestANodeNamedPathIsRefusedThroughRun(t *testing.T) {
}
}
func TestCheckReturnsOneOnAFindingAndTwoOnAnUnreadableList(t *testing.T) {
func TestCheckReturnsOneOnAFindingAndTwoOnNoNames(t *testing.T) {
root := t.TempDir()
write := func(p, s string) {
p = filepath.Join(root, p)
os.MkdirAll(filepath.Dir(p), 0o755)
os.WriteFile(p, []byte(s), 0o644)
}
write("particulars", "ace\n")
write("checks/particulars/covered", "kernel\n")
write("modules/kernel/a", "clean\n")
if got := check(root); got != 0 {
if got := check(root, []string{"ace"}); got != 0 {
t.Errorf("clean = %d, want 0", got)
}
write("modules/kernel/a", "on ace\n")
if got := check(root); got != 1 {
if got := check(root, []string{"ace"}); got != 1 {
t.Errorf("finding = %d, want 1", got)
}
os.Remove(filepath.Join(root, "particulars"))
if got := check(root); got != 2 {
t.Errorf("no list = %d, want 2", got)
if got := check(root, nil); got != 2 {
t.Errorf("no names = %d, want 2", got)
}
}
@@ -117,7 +115,10 @@ func TestOnlyCoveredModulesAreRead(t *testing.T) {
// The script's run of this check over the real tree (novox/hq ADR 0529 clause 2): no node name of this mesh in a covered module.
func TestTheCoveredModulesNameNoNode(t *testing.T) {
if code := check("../.."); code != 0 {
if _, set := os.LookupEnv(NamesEnv); !set {
t.Skip("ADR 0567 §16: this checker hands no node names (before D14 is delivered); the skip goes once it is")
}
if code := check("../..", namesFromEnv()); code != 0 {
t.Fatalf("the check of the real catalogue exits %d (see what it printed above)", code)
}
}
+1 -2
View File
@@ -1,6 +1,5 @@
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# check("../.."): the root's list and every file of the modules it names.
particulars
# check("../.."): every file of the modules the covered list names. The node names come from the run (MESH_NODE_NAMES).
modules
-5
View File
@@ -1,5 +0,0 @@
module checks/placement
go 1.26.0
require git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261027014310-cadd7b1b2f8f
-18
View File
@@ -1,18 +0,0 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026210326-979644420117 h1:88CNZOoQgu7YPYKW16qkWT8Gw9Ucg07aTrIImA2nYUk=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026210326-979644420117/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261027012704-062c29c3a22f h1:AKzV973UTWe0OYFoIjgGu0f+B26xh8QsAHzCpsTIjM8=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261027012704-062c29c3a22f/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261027014310-cadd7b1b2f8f h1:mimAGERguQmCGYVNdRTrFcZ1dhpU6GWo+Q3EDhHOrVk=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261027014310-cadd7b1b2f8f/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/klauspost/compress v1.20.0 h1:a3C1ke2ohxFymNlb2HWAHjDeKCI90scRskErZkR0ezA=
github.com/klauspost/compress v1.20.0/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/nats-io/nats.go v1.54.0 h1:vsXoOxjHp/GmPUN+EcI7uOf/uB+iAP+kEsAFNQN0yzA=
github.com/nats-io/nats.go v1.54.0/go.mod h1:y+DZoD1oBOYfZTU681eTUiUjI0vbqYGixNVFHcjHJ0k=
github.com/nats-io/nkeys v0.4.16 h1:rd5oAuLOb8mnAycB0xleuEBNS1pVVnN0fv/FF34Eypg=
github.com/nats-io/nkeys v0.4.16/go.mod h1:llLgWoI0o4z/Q57q2R1kHfmocyhGV6VG/U18Glg1Afs=
github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw=
github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
-80
View File
@@ -1,80 +0,0 @@
// Command placement is the catalogue's check of its placement files (novox/hq ADR 0549 §1, ADR 0555; issue 1126): every
// `placement/<node>.json` reads as a placement of the node its name says, and the files together are refused, every
// finding named at once, for two modules of one node on one host port, a pin, binding or holding naming a provider not
// placed where it says, a seat held once for the mesh held twice, two nodes with one address on the private network, or
// more than one hub (mesh-sdk delivery.CheckPlacements).
//
// A module the catalogue does not hold and a node not enrolled are the composition check's to refuse: placement names
// modules of every repository the indexer reads and the nodes the mesh enrolled, neither of which this tree holds.
//
// go run . <catalogue root>
//
// Exits 1 on a finding.
package main
import (
"fmt"
"os"
"path/filepath"
"sort"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// Check reads every placement file under root and answers each finding; none is a catalogue whose placement holds. A
// catalogue with no placement directory has none to check.
func Check(root string) ([]string, error) {
dir := filepath.Join(root, delivery.PlacementDir)
entries, err := os.ReadDir(dir)
if os.IsNotExist(err) {
return nil, nil
}
if err != nil {
return nil, err
}
var found []string
placements := map[string]delivery.Placement{}
for _, e := range entries {
rel := delivery.PlacementDir + "/" + e.Name()
node, ok := delivery.PlacementFileNode(rel)
if e.IsDir() || !ok {
found = append(found, fmt.Sprintf("%s is not a placement file: the directory holds only <node>.json, one per node", rel))
continue
}
raw, err := os.ReadFile(filepath.Join(dir, e.Name()))
if err != nil {
return nil, err
}
p, err := delivery.DecodePlacement(raw)
if err != nil {
found = append(found, fmt.Sprintf("%s does not read as a placement: %v", rel, err))
continue
}
if p.Revision != "" {
found = append(found, fmt.Sprintf("%s names a revision, which only the indexer writes into the key", rel))
}
placements[node] = p
}
all := func(string) bool { return true }
found = append(found, delivery.CheckPlacements(placements, all, all)...)
sort.Strings(found)
return found, nil
}
func main() {
if len(os.Args) != 2 {
fmt.Fprintln(os.Stderr, "usage: placement <catalogue root>")
os.Exit(2)
}
found, err := Check(os.Args[1])
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(2)
}
for _, f := range found {
fmt.Println(f)
}
if len(found) > 0 {
os.Exit(1)
}
}
-55
View File
@@ -1,55 +0,0 @@
package main
import (
"os"
"path/filepath"
"strings"
"testing"
)
// The catalogue's placement files as the merge check reads them (ADR 0549 §1, ADR 0555): the real tree holds every
// rule, and a tree that breaks them is refused with every finding named at once.
func TestPlacementFiles(t *testing.T) {
found, err := Check("../..")
if err != nil {
t.Fatal(err)
}
for _, f := range found {
t.Errorf("%s (novox/hq ADR 0549 §1)", f)
}
root := t.TempDir()
write := func(name, body string) {
t.Helper()
if err := os.MkdirAll(filepath.Join(root, "placement"), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(root, "placement", name), []byte(body), 0o644); err != nil {
t.Fatal(err)
}
}
if found, err := Check(root); err != nil || len(found) != 0 {
t.Fatalf("a catalogue with no placement directory: %v, %v", found, err)
}
write("ace.json", `{"v":1,"node":"ace","decided":{"hub":true,"address":"10.77.0.1"},"modules":[{"module":"a","ports":{"80":8080}},{"module":"b","ports":{"81":8080}}]}`)
write("g14.json", `{"v":1,"node":"g14","decided":{"hub":true,"address":"10.77.0.1"},"modules":[{"module":"c","bindings":{"pg":{"node":"ace","module":"pg"}}}]}`)
write("novox.json", `{"v":1,"node":"shanks","modules":[]}`)
write("bad.json", `{"v":1,"node":"bad","modules":{}}`)
write("dated.json", `{"v":1,"node":"dated","revision":"abc","modules":[]}`)
write("notes.md", "x")
found, err = Check(root)
if err != nil {
t.Fatal(err)
}
for _, want := range []string{"a and b are both published on host port 8080", "bound to ace/pg", "both have the address 10.77.0.1",
"ace, g14 are each placed as it", "placement file of novox names the node", "bad.json does not read", "dated.json names a revision",
"notes.md is not a placement file"} {
hit := false
for _, f := range found {
hit = hit || strings.Contains(f, want)
}
if !hit {
t.Errorf("no finding says %q; found %v", want, found)
}
}
}
-1
View File
@@ -1 +0,0 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
+8
View File
@@ -206,6 +206,9 @@ func excerpt(s string, from, to int) string {
return strings.Join(strings.Fields(s[a:b]), " ")
}
// modulePath matches a string that is only a module or import path ("git.example.org/org/repo/go/pkg").
var modulePath = regexp.MustCompile(`^[a-z0-9.-]+\.[a-z]+/[A-Za-z0-9_./-]+$`)
// goStrings returns a Go file's string literals, adjacent ones joined across `+` as the program joins them.
func goStrings(path, rel string) ([]text, error) {
src, err := os.ReadFile(path)
@@ -231,6 +234,11 @@ func goStrings(path, rel string) ([]text, error) {
if err != nil {
v = lit
}
if modulePath.MatchString(v) {
// An import path or another module path names code, not words an agent reads.
cur, joining = nil, false
continue
}
if cur != nil && joining {
cur.s += v
} else {
@@ -23,7 +23,7 @@ import (
// manifest's.
const NamesFileEnv = "MESH_ARTIFACT_STORE_NAMES_FILE"
// CatalogueModules and CatalogueModule are the catalogue's tools these definitions are read through.
// CatalogueModules and CatalogueModule are the catalog's tools these definitions are read through.
const (
CatalogueModules = "mesh-catalog.catalog_modules"
CatalogueModule = "mesh-catalog.catalog_module"
@@ -143,21 +143,21 @@ func PinsIn(manifest json.RawMessage, names StoreNames) []pinned {
return out
}
// Definitions is what the catalogue's definitions pin in the store, and what of them could not be read.
// Definitions is what the catalog's definitions pin in the store, and what of them could not be read.
type Definitions struct {
// Pins are by Key(repository, digest).
Pins map[string][]Pin
// Read is how many definitions were read; Unread says, per module, why its definition was not.
Read int
Unread map[string]string
// NotAsked says why the catalogue could not be asked at all, when it could not.
// NotAsked says why the catalog could not be asked at all, when it could not.
NotAsked string
// NamesNotRead says why the store's older names could not be read, when they could not.
NamesNotRead string
Names StoreNames
}
// Catalogue reads the definitions through the catalogue's tools, and keeps each definition it read by its
// Catalogue reads the definitions through the catalog's tools, and keeps each definition it read by its
// module and commit, which never change together.
type Catalogue struct {
Ask Asking
@@ -171,21 +171,21 @@ type catalogueEntry struct {
Commit string `json:"commit"`
}
// Read asks the catalogue for every module and reads each definition, bounded by DefinitionsTimeout.
// Read asks the catalog for every module and reads each definition, bounded by DefinitionsTimeout.
func (c *Catalogue) Read(ctx context.Context, names StoreNames) *Definitions {
defs := &Definitions{Pins: map[string][]Pin{}, Unread: map[string]string{}, Names: names}
if c == nil || c.Ask == nil {
defs.NotAsked = "this bundle has no way to ask the catalogue"
defs.NotAsked = "this bundle has no way to ask the catalog"
return defs
}
raw, err := c.Ask(CatalogueModules, map[string]any{})
if err != nil {
defs.NotAsked = "the catalogue's modules could not be asked: " + err.Error()
defs.NotAsked = "the catalog's modules could not be asked: " + err.Error()
return defs
}
answer, output, ok := answerOf(raw)
if !ok {
defs.NotAsked = "the catalogue refused its modules: " + lastLine(output)
defs.NotAsked = "the catalog refused its modules: " + lastLine(output)
return defs
}
var list struct {
@@ -193,7 +193,7 @@ func (c *Catalogue) Read(ctx context.Context, names StoreNames) *Definitions {
Error string `json:"error"`
}
if err := json.Unmarshal(answer, &list); err != nil || list.Error != "" || list.Modules == nil {
defs.NotAsked = "the catalogue's modules answer is not readable: " + firstOf(list.Error, errText(err), lastLine(output))
defs.NotAsked = "the catalog's modules answer is not readable: " + firstOf(list.Error, errText(err), lastLine(output))
return defs
}
c.mu.Lock()
@@ -0,0 +1,129 @@
package main
// The artifact store's door and the bus, as the holder reaches them through this module's runtime.
import (
"context"
"encoding/json"
"fmt"
"os"
"strings"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"artifactstoretools/internal/registry"
)
// door is registry.Store at the store's address as this machine reaches it.
type door struct{ s registry.Store }
func (d *door) Address() string { return d.s.Address }
func (d *door) Hold(ctx context.Context, r string) (bool, error) { return d.s.Hold(ctx, r) }
func (d *door) Holds(ctx context.Context, r string) error { return d.s.Holds(ctx, r) }
func (d *door) IsIndex(ctx context.Context, r string) (bool, error) {
return d.s.IsIndex(ctx, r)
}
func (d *door) LetGo(ctx context.Context, r string) error { return d.s.LetGo(ctx, r) }
func (d *door) SpareKept(ctx context.Context, keptImages, touching []string) error {
spare, err := d.s.SpareKeptIndexes(ctx, keptImages, touching)
if err == nil {
d.s.Spare = spare
}
return err
}
// DoorAt reads the store's address (host:port) from the file the manifest fills from the seat mesh-artifact-store's
// reach, each time, so a store that moved is reached where it is now.
func DoorAt(file string) func() (Door, error) {
return func() (Door, error) {
if file == "" {
return nil, fmt.Errorf("this bundle was given no file naming the artifact store's address")
}
raw, err := os.ReadFile(file)
if err != nil {
return nil, fmt.Errorf("the artifact store's address: %w", err)
}
address := strings.TrimSpace(string(raw))
if address == "" {
return nil, fmt.Errorf("the artifact store's address is empty in %s", file)
}
return &door{s: registry.Store{Address: address}}, nil
}
}
// runtimeBus is the holder's bus through the runtime: its reads, its findings and its seat's event.
type runtimeBus struct{}
func (runtimeBus) Keys(state string) ([]string, error) { return stdio.State(state).Keys() }
func (runtimeBus) Get(state, key string) ([]byte, bool, error) {
e, err := stdio.State(state).Get(key)
if err != nil || e == nil {
return nil, false, err
}
return e.Value, true, nil
}
func (runtimeBus) History(state, key string) ([]stdio.StateRevision, error) {
return stdio.State(state).History(key)
}
func (runtimeBus) LogSince(read string, at time.Time) ([]stdio.LogEntry, error) {
var out []stdio.LogEntry
page, err := stdio.Log(read).Since(at, "", 0)
for err == nil {
out = append(out, page.Entries...)
if page.Next == 0 {
return out, nil
}
page, err = stdio.Log(read).Read(page.Next, "", 0)
}
return nil, err
}
func (runtimeBus) Put(state, key string, value any) error {
_, err := stdio.State(state).Put(key, value)
return err
}
func (runtimeBus) Delete(state, key string) error { return stdio.State(state).Delete(key) }
func (runtimeBus) Publish(subject string, body any) error {
return stdio.SeatPublish(subject, body, "")
}
// follow watches the builder's done keys and the wants: a build's end sweeps (once the lease names no controller) and
// observes again; a want changed observes again. Event-driven: nothing reads them on a timer.
func follow(h *Holder) {
<-stdio.Served()
watch := func(state string, handle func(stdio.StateChange) error) {
for {
if err := stdio.State(state).Watch("", handle); err == nil {
return
} else {
h.logf("watching %s: %v; asked again in 10s", state, err)
}
time.Sleep(10 * time.Second)
}
}
observe := func() {
if err := h.Observe(context.Background()); err != nil {
h.logf("image-missing could not be observed: %v", err)
}
}
watch(BuildsRead, func(c stdio.StateChange) error {
if c.Current || c.Op != stdio.OpPut {
return nil
}
var b struct {
State string `json:"state"`
}
if json.Unmarshal(c.Value, &b) == nil && b.State == "done" {
go func() { h.AfterBuild(c.Key); observe() }()
}
return nil
})
watch(WantsRead, func(c stdio.StateChange) error {
if !c.Current {
go observe()
}
return nil
})
observe()
}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,260 @@
//go:build integration
package main
// The seat mesh-artifact-sweep's holder whole against a real bus (novox/hq ADR 0571 clause 2, issue 1955): the builder's done
// keys, the wants and the controller's lease in a real nats-server's key-value buckets, a build's end heard by a watch
// of the builds as the runtime hands it over, each deletion recorded by a request to the seat mesh-hand-acts on the bus,
// artifacts-gone said on the seat's subject, and S25's finding written to the holder's own bucket. Only the store's
// door is in memory: its HTTP is the copied registry package's, tested there.
import (
"context"
"encoding/json"
"strings"
"sync"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/handacts"
"git.novox.be/novox/mesh-sdk/go/testbus"
)
// realBus is the holder's Bus over a real nats-server: each read state a bucket named as the runtime names it.
type realBus struct {
ctx context.Context
nc *nats.Conn
js jetstream.JetStream
// log is the builder's log as the runtime's log verbs would read it (mesh-tools adds them; the sdk's own read of a
// log is tested in mesh-sdk's process package).
log []stdio.LogEntry
}
func (b realBus) LogSince(_ string, at time.Time) ([]stdio.LogEntry, error) {
var out []stdio.LogEntry
for _, e := range b.log {
if !e.At.Before(at) {
out = append(out, e)
}
}
return out, nil
}
func bucketOf(state string) string { return strings.Replace(state, ".", "_", 1) }
func (b realBus) kv(state string) (jetstream.KeyValue, error) {
return b.js.KeyValue(b.ctx, bucketOf(state))
}
func (b realBus) Keys(state string) ([]string, error) {
kv, err := b.kv(state)
if err != nil {
return nil, err
}
keys, err := kv.Keys(b.ctx)
if err == jetstream.ErrNoKeysFound {
return nil, nil
}
return keys, err
}
func (b realBus) Get(state, key string) ([]byte, bool, error) {
kv, err := b.kv(state)
if err != nil {
return nil, false, err
}
e, err := kv.Get(b.ctx, key)
if err == jetstream.ErrKeyNotFound {
return nil, false, nil
}
if err != nil {
return nil, false, err
}
return e.Value(), true, nil
}
func (b realBus) History(state, key string) ([]stdio.StateRevision, error) {
kv, err := b.kv(state)
if err != nil {
return nil, err
}
h, err := kv.History(b.ctx, key)
if err != nil {
return nil, err
}
var out []stdio.StateRevision
for _, e := range h {
out = append(out, stdio.StateRevision{Key: key, Value: e.Value(), Revision: e.Revision(), At: e.Created()})
}
return out, nil
}
func (b realBus) Put(state, key string, value any) error {
kv, err := b.kv(state)
if err != nil {
return err
}
raw, _ := json.Marshal(value)
_, err = kv.Put(b.ctx, key, raw)
return err
}
func (b realBus) Delete(state, key string) error {
kv, err := b.kv(state)
if err != nil {
return err
}
return kv.Delete(b.ctx, key)
}
func (b realBus) Publish(subject string, body any) error {
raw, _ := json.Marshal(body)
return b.nc.Publish(subject, raw)
}
func TestTheSeatSweepsAtABuildsEndOnARealBusOnceTheLeaseIsAway(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
nc, err := nats.Connect(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
defer nc.Close()
js, err := jetstream.New(nc)
if err != nil {
t.Fatal(err)
}
for _, s := range []string{BuildsRead, WantsRead, HavesRead, LeaseRead, ManifestsRead, "artifact-store-tools." + FindingsState} {
if _, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: bucketOf(s), History: 5}); err != nil {
t.Fatal(err)
}
}
bus := realBus{ctx: ctx, nc: nc, js: js}
// The mesh as aMesh lays it, copied onto the real buckets: letta's seven builds, a want, the store's listing.
mem, memb, door, _ := aMesh(t, &buscompose.LeaseHolder{Instance: "c1", Host: "novox", Epoch: 3})
bus.log = memb.log
for _, s := range []string{BuildsRead, WantsRead, LeaseRead, ManifestsRead} {
keys, _ := memb.Keys(s)
for _, k := range keys[:min(len(keys), 6)] { // the seventh build ends later, on the bus
raw, _, _ := memb.Get(s, k)
if err := bus.Put(s, k, json.RawMessage(raw)); err != nil {
t.Fatal(err)
}
}
}
time.Sleep(20 * time.Millisecond) // each build's end is its key's time on the bus: the seventh the newest
// The seat mesh-hand-acts, answering every act it records; artifacts-gone heard on the seat's subject.
var mu sync.Mutex
var acts, gone []string
if _, err := nc.Subscribe("mesh.seat."+handacts.Seat+".tool.hand-act", func(m *nats.Msg) {
mu.Lock()
acts = append(acts, string(m.Data))
mu.Unlock()
_ = m.Respond([]byte(`{"ok":true}`))
}); err != nil {
t.Fatal(err)
}
if _, err := nc.Subscribe(GoneSubject, func(m *nats.Msg) {
mu.Lock()
gone = append(gone, string(m.Data))
mu.Unlock()
}); err != nil {
t.Fatal(err)
}
h := &Holder{Bus: findingsAt{realBus: bus}, Door: mem.Door, Listed: mem.Listed, Now: time.Now, Logf: t.Logf}
h.Acts = handacts.Recorder{LeaseAway: h.LeaseAway, Ask: func(key string, body any) (json.RawMessage, error) {
raw, _ := json.Marshal(body)
m, err := nc.Request("mesh.seat."+strings.TrimPrefix(strings.Replace(key, ".", ".tool.", 1), "seat:"), raw, 5*time.Second)
if err != nil {
return nil, err
}
return m.Data, nil
}}
// A build's end heard from the builds bucket, as follow hands it over.
kv, _ := bus.kv(BuildsRead)
w, err := kv.WatchAll(ctx, jetstream.UpdatesOnly())
if err != nil {
t.Fatal(err)
}
ended := func() {
keys, _ := memb.Keys(BuildsRead)
raw, _, _ := memb.Get(BuildsRead, keys[6])
if err := bus.Put(BuildsRead, keys[6], json.RawMessage(raw)); err != nil {
t.Fatal(err)
}
select {
case e := <-w.Updates():
h.AfterBuild(e.Key())
case <-ctx.Done():
t.Fatal("the build's end was not heard")
}
}
// While the lease names the controller, a build's end lets nothing go.
ended()
if len(door.asked) != 0 {
t.Fatalf("before the lease write the store was asked %v", door.asked)
}
// The lease handed away: the next build's end holds the kept archives and lets go of letta's 2nd, which the 7th moved
// past the five newest and no want names (the want keeps the 1st), recording each deletion.
if err := bus.Put(LeaseRead, buscompose.ControllerLeaseKey, buscompose.LeaseHolder{HandedTo: "mesh-grants", Why: "the lease write"}); err != nil {
t.Fatal(err)
}
ended()
deadline := time.Now().Add(5 * time.Second)
for {
mu.Lock()
n := len(gone)
mu.Unlock()
if n > 0 || time.Now().After(deadline) {
break
}
time.Sleep(20 * time.Millisecond)
}
mu.Lock()
defer mu.Unlock()
if len(acts) != 3 || !strings.Contains(acts[0], archiveRef("letta", 2)) || !strings.Contains(acts[1], imageRef("letta", 2)) ||
!strings.Contains(acts[2], mirror(1)) {
t.Fatalf("the hand acts recorded are %v; the store was asked %v", acts, door.asked)
}
if len(gone) != 1 || !strings.Contains(gone[0], `"fingerprint":"`+fp(2)+`"`) {
t.Fatalf("artifacts-gone said %v", gone)
}
// S25 through the real findings bucket: letta's wanted image missing from the store is a finding there.
view, _ := mem.Listed(ctx)
repo, digest, _ := splitReference(imageRef("letta", 1))
delete(view.L.Repos[repo].Revisions, digest)
if err := h.Observe(ctx); err != nil {
t.Fatal(err)
}
if keys, _ := bus.Keys("artifact-store-tools." + FindingsState); len(keys) != 1 || keys[0] != "module.letta.ace.image-missing" {
t.Fatalf("the findings are %v", keys)
}
}
// findingsAt reads and writes the holder's own findings in its bucket, artifact-store-tools_findings.
type findingsAt struct{ realBus }
func (f findingsAt) own(state string) string {
if state == FindingsState {
return Module + "." + state
}
return state
}
func (f findingsAt) Keys(state string) ([]string, error) { return f.realBus.Keys(f.own(state)) }
func (f findingsAt) Get(state, key string) ([]byte, bool, error) {
return f.realBus.Get(f.own(state), key)
}
func (f findingsAt) Put(state, key string, value any) error {
return f.realBus.Put(f.own(state), key, value)
}
func (f findingsAt) Delete(state, key string) error { return f.realBus.Delete(f.own(state), key) }
@@ -0,0 +1,351 @@
package main
import (
"context"
"encoding/json"
"fmt"
"sort"
"strings"
"sync"
"testing"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/handacts"
"git.novox.be/novox/mesh-sdk/go/health"
"artifactstoretools/internal/registry"
)
// memBus is the holder's bus in memory: states by name, each key's history, and what was published.
type memBus struct {
mu sync.Mutex
states map[string]map[string][]byte
history map[string][]stdio.StateRevision
log []stdio.LogEntry
published []string
}
func (b *memBus) LogSince(read string, at time.Time) ([]stdio.LogEntry, error) {
var out []stdio.LogEntry
for _, e := range b.log {
if read == delivery.BuildLogRead && !e.At.Before(at) {
out = append(out, e)
}
}
return out, nil
}
func (b *memBus) Keys(state string) ([]string, error) {
b.mu.Lock()
defer b.mu.Unlock()
var out []string
for k := range b.states[state] {
out = append(out, k)
}
sort.Strings(out)
return out, nil
}
func (b *memBus) Get(state, key string) ([]byte, bool, error) {
b.mu.Lock()
defer b.mu.Unlock()
v, ok := b.states[state][key]
return v, ok, nil
}
func (b *memBus) History(state, key string) ([]stdio.StateRevision, error) {
return b.history[state+" "+key], nil
}
func (b *memBus) Put(state, key string, value any) error {
raw, _ := json.Marshal(value)
b.mu.Lock()
defer b.mu.Unlock()
if b.states[state] == nil {
b.states[state] = map[string][]byte{}
}
b.states[state][key] = raw
return nil
}
func (b *memBus) Delete(state, key string) error {
b.mu.Lock()
defer b.mu.Unlock()
delete(b.states[state], key)
return nil
}
func (b *memBus) Publish(subject string, body any) error {
raw, _ := json.Marshal(body)
b.mu.Lock()
defer b.mu.Unlock()
b.published = append(b.published, subject+" "+string(raw))
return nil
}
// memDoor is the store's door in memory: what it holds, which references are indexes, and what was asked.
type memDoor struct {
holds map[string]bool
indexes map[string]bool
asked []string
}
func (d *memDoor) Address() string { return "10.0.0.5:5000" }
func (d *memDoor) Hold(_ context.Context, r string) (bool, error) {
d.asked = append(d.asked, "hold "+r)
if !d.holds[r] {
return false, registry.Gone
}
return false, nil
}
func (d *memDoor) Holds(_ context.Context, r string) error {
if !d.holds[r] {
return registry.Gone
}
return nil
}
func (d *memDoor) IsIndex(_ context.Context, r string) (bool, error) { return d.indexes[r], nil }
func (d *memDoor) LetGo(_ context.Context, r string) error {
d.asked = append(d.asked, "let go "+r)
delete(d.holds, r)
return nil
}
func (d *memDoor) SpareKept(context.Context, []string, []string) error {
d.asked = append(d.asked, "spare")
return nil
}
// mirror is a copy of an upstream base in the store, the n-th.
func mirror(n int) string {
return "artifact-store://upstream/docker.io/library/alpine@sha256:" + strings.Repeat(fmt.Sprintf("%x", n%16), 64)
}
// pinnedImage is the image letta's manifest names by itself, at the store's address.
const pinnedImage = "10.0.0.5:5000/letta/pinned@sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc"
func digestOf(ref string) string { _, d, _ := splitReference(ref); return d }
func fp(n int) string { return "src2:" + strings.Repeat(fmt.Sprintf("%x", n%16), 64) }
func imageRef(module string, n int) string {
return "artifact-store://" + module + "/image@sha256:" + strings.Repeat(fmt.Sprintf("%x", n%16), 64)
}
func archiveRef(module string, n int) string {
return "artifact-store://" + module + "/files/blobs/sha256:" + strings.Repeat(fmt.Sprintf("%x", (n+8)%16), 64)
}
// A mesh with seven done builds of letta, the n-th ending n hours after a day began, each with an image and an archive;
// a want names the first on ace; the store lists them all and one image no build names; the lease as given.
func aMesh(t *testing.T, lease *buscompose.LeaseHolder) (*Holder, *memBus, *memDoor, *[]string) {
t.Helper()
day := time.Date(2026, 10, 1, 0, 0, 0, 0, time.UTC)
b := &memBus{states: map[string]map[string][]byte{}, history: map[string][]stdio.StateRevision{}}
door := &memDoor{holds: map[string]bool{}, indexes: map[string]bool{}}
listing := &Listing{Blobs: map[string]int64{}, Repos: map[string]*Repo{}}
for n := 1; n <= 7; n++ {
key, _ := delivery.BuilderBuildKey("letta", fp(n))
raw, err := delivery.EncodeBuilderBuild(delivery.BuilderBuild{State: delivery.BuildDone, Manifest: strings.Repeat("d", 64), Artifacts: []delivery.BuiltArtifact{
{Name: "image", Kind: "image", Digest: digestOf(imageRef("letta", n)), Reference: imageRef("letta", n)},
{Name: "files", Kind: "archive", Digest: digestOf(archiveRef("letta", n)), Reference: archiveRef("letta", n)},
{Name: "pkg", Kind: "package", Digest: "sha256:" + strings.Repeat("f", 64), Reference: "letta@1.0." + fmt.Sprint(n)}}})
if err != nil {
t.Fatal(err)
}
_ = b.Put(BuildsRead, key, json.RawMessage(raw))
b.history[BuildsRead+" "+key] = []stdio.StateRevision{{Key: key, Value: raw, At: day.Add(time.Duration(n) * time.Hour)}}
for _, r := range []string{imageRef("letta", n), archiveRef("letta", n)} {
door.holds[r] = true
repo, digest, kind := splitReference(r)
if kind == "archive" {
listing.Blobs[digest] = 10
} else {
listing.repo(repo).Revisions[digest] = true
}
}
}
listing.repo("letta/image").Revisions["sha256:"+strings.Repeat("e", 64)] = true // no build names it
listing.repo("letta/pinned").Revisions["sha256:"+strings.Repeat("c", 64)] = true
// Three copies of a base: letta's 2nd stood on the first, its 7th on the second, and no build on the third.
for n := 1; n <= 3; n++ {
repo, digest, _ := splitReference(mirror(n))
listing.repo(repo).Revisions[digest] = true
door.holds[mirror(n)] = true
}
for n := 1; n <= 7; n++ {
e := delivery.BuildLogEntry{Event: delivery.BuildEndedEvent, Module: "letta", FP: fp(n), Repository: "novox/x", Commit: "c",
Node: "ace", At: day.Add(time.Duration(n) * time.Hour), Outcome: "done"}
switch n {
case 2:
e.Against = []string{mirror(1)}
case 7:
e.Against = []string{mirror(2)}
}
raw, _ := delivery.EncodeBuildLogEntry(e)
b.log = append(b.log, stdio.LogEntry{Sequence: uint64(n), Key: "letta", Value: raw, At: e.At})
}
manifest, _ := delivery.EncodeManifest(delivery.Manifest{Module: "letta",
JSON: `{"module":"letta","resources":[{"id":"app","type":"container","image":"` + pinnedImage + `"},` +
`{"id":"built","type":"container","image":"${artifact:image}"}]}`})
mk, _ := delivery.ManifestsKey(strings.Repeat("d", 64))
_ = b.Put(ManifestsRead, mk, json.RawMessage(manifest))
want, _ := json.Marshal(delivery.Want{V: delivery.Version, FP: fp(1), Self: true})
_ = b.Put(WantsRead, "want.ace.letta", json.RawMessage(want))
if lease != nil {
_ = b.Put(LeaseRead, buscompose.ControllerLeaseKey, lease)
}
var acts []string
h := &Holder{Bus: b, Door: func() (Door, error) { return door, nil },
Listed: func(context.Context) (*View, error) { return &View{L: listing}, nil },
Now: func() time.Time { return day.Add(48 * time.Hour) },
Acts: handacts.Recorder{Ask: func(key string, body any) (json.RawMessage, error) {
raw, _ := json.Marshal(body)
acts = append(acts, key+" "+string(raw))
return json.RawMessage(`{}`), nil
}}}
h.Acts.LeaseAway = h.LeaseAway
return h, b, door, &acts
}
// ADR 0571 clause 2: the keep set from the store's listing, the wants and the five newest builds; nothing let go of while
// the lease names a controller; once it names another holder, a build's end lets go of the eligible within the bound,
// leaves an index for a confirmed collect, records each deletion and says artifacts-gone.
func TestTheSeatKeepsTheFiveNewestAndTheWantedAndSweepsOnlyOnceTheLeaseIsAway(t *testing.T) {
controller := &buscompose.LeaseHolder{Instance: "c1", Host: "novox", Epoch: 3}
handed := &buscompose.LeaseHolder{HandedTo: "mesh-grants", HandedBy: "person.jochen", Why: "the lease write"}
h, _, _, _ := aMesh(t, controller)
r, err := h.Artifacts("")
if err != nil {
t.Fatal(err)
}
states := map[string]string{}
for _, x := range r.References {
states[x.Reference] = x.State
}
// letta's 3rd to 7th are the five newest; its 1st a want names on ace, however old; its 2nd is eligible; the package
// and the manifest no build names are no reference of the seat's.
for n, want := range map[int]string{1: "kept", 2: "eligible", 3: "kept", 7: "kept"} {
if states[imageRef("letta", n)] != want || states[archiveRef("letta", n)] != want {
t.Errorf("letta's build %d is %s and %s, not %s", n, states[imageRef("letta", n)], states[archiveRef("letta", n)], want)
}
}
// And the copies: the one letta's 2nd stood on is eligible with it, the 7th's kept; the third no build names.
if states[mirror(1)] != "eligible" || states[mirror(2)] != "kept" || states[mirror(3)] != "" {
t.Errorf("the copies are %s, %s and %q", states[mirror(1)], states[mirror(2)], states[mirror(3)])
}
if m, err := h.MirrorsNow(); err != nil || len(m.Copies) != 3 || m.Copies[2].State != "unrecorded" || m.Held != "" {
t.Errorf("mirrors answered %+v: %v", m, err)
}
if len(r.References) != 16 || r.Counts["kept"] != 13 || r.Counts["eligible"] != 3 || r.Store != "10.0.0.5:5000" {
t.Errorf("the references are %d, counted %v, at %q", len(r.References), r.Counts, r.Store)
}
for _, c := range []struct {
name string
lease *buscompose.LeaseHolder
index bool
most int
letGo []string
left int
skipped int
unlog int
}{
{name: "the lease names a controller", lease: controller},
{name: "no lease key", lease: nil},
{name: "handed away", lease: handed, most: MostPerSweep, letGo: []string{archiveRef("letta", 2), imageRef("letta", 2), mirror(1)}},
{name: "an index waits", lease: handed, index: true, most: MostPerSweep, letGo: []string{archiveRef("letta", 2), mirror(1)}, skipped: 1},
{name: "bounded", lease: handed, most: 1, letGo: []string{archiveRef("letta", 2)}, left: 2},
// A kept build the log no longer holds: what it stood on is unknown, so no copy goes.
{name: "a kept build's bases unknown", lease: handed, most: MostPerSweep, letGo: []string{archiveRef("letta", 2), imageRef("letta", 2)}, unlog: 3},
} {
t.Run(c.name, func(t *testing.T) {
h, b, door, acts := aMesh(t, c.lease)
door.indexes[imageRef("letta", 2)] = c.index
if c.unlog > 0 {
b.log = append(b.log[:c.unlog-1:c.unlog-1], b.log[c.unlog:]...)
}
if c.most == 0 {
h.AfterBuild("letta.x")
if _, err := h.Collect("tidy", true, 0); err == nil {
t.Error("a confirmed collect was let through while the controller sweeps")
}
if len(door.asked) != 0 || len(*acts) != 0 || len(b.published) != 0 {
t.Fatalf("before the lease write the store was asked %v, acts %v, said %v", door.asked, *acts, b.published)
}
return
}
a, err := h.Sweep(context.Background(), Sweeping{Most: c.most, By: "after the build letta.x"})
if err != nil {
t.Fatal(err)
}
if fmt.Sprint(a.LetGo) != fmt.Sprint(c.letGo) || a.Left != c.left || a.Skipped != c.skipped || a.Held != 6 || a.Missing != 0 {
t.Fatalf("let go of %v (left %d, skipped %d, held %d, missing %d), want %v", a.LetGo, a.Left, a.Skipped, a.Held, a.Missing, c.letGo)
}
// Every kept archive held before the first deletion.
first := -1
for i, x := range door.asked {
if strings.HasPrefix(x, "let go ") && first < 0 {
first = i
}
}
if first != 6 {
t.Errorf("the store was asked %v: the six kept archives are held first", door.asked)
}
if len(*acts) != len(c.letGo) || !strings.HasPrefix((*acts)[0], "seat:mesh-hand-acts.hand-act ") ||
!strings.Contains((*acts)[0], c.letGo[0]) {
t.Errorf("the hand acts recorded are %v", *acts)
}
if len(b.published) != 1 || !strings.HasPrefix(b.published[0], GoneSubject+" ") ||
!strings.Contains(b.published[0], `"fingerprint":"`+fp(2)+`"`) {
t.Errorf("said %v", b.published)
}
})
}
}
// S25 (image-missing): a want whose build names an image the store does not hold is a finding in the seat's own state,
// written once and deleted once the store holds it again; images names what a machine's wants and haves name.
func TestAWantedImageTheStoreLacksIsAFindingAndImagesNamesWantsAndHaves(t *testing.T) {
h, b, _, _ := aMesh(t, nil)
view, _ := h.Listed(context.Background())
repo, digest, _ := splitReference(imageRef("letta", 1))
delete(view.L.Repos[repo].Revisions, digest)
h.Listed = func(context.Context) (*View, error) { return view, nil }
if err := h.Observe(context.Background()); err != nil {
t.Fatal(err)
}
key := health.FindingKey(health.ScopeModule, "letta.ace", ImageMissing)
raw, found, _ := b.Get(FindingsState, key)
f, err := health.DecodeFinding(raw)
if !found || err != nil || f.Subject.Machine != "ace" || f.Source != "S25" || !strings.Contains(f.Evidence[0].Said, imageRef("letta", 1)) {
t.Fatalf("the finding is %+v (%v, %v)", f, found, err)
}
view.L.Repos[repo].Revisions[digest] = true
if err := h.Observe(context.Background()); err != nil {
t.Fatal(err)
}
if _, found, _ := b.Get(FindingsState, key); found {
t.Fatal("the finding stayed once the store held the image again")
}
// An image the manifest pins by itself, under the store's address, is judged as the controller judged it.
delete(view.L.Repos["letta/pinned"].Revisions, "sha256:"+strings.Repeat("c", 64))
if err := h.Observe(context.Background()); err != nil {
t.Fatal(err)
}
if raw, found, _ := b.Get(FindingsState, key); !found || !strings.Contains(string(raw), "letta/pinned@sha256:") {
t.Fatalf("the pinnedImage image missing is %s", raw)
}
have, _ := json.Marshal(delivery.Have{V: delivery.Version, FP: fp(7), Phase: "running"})
_ = b.Put(HavesRead, "ace.letta", json.RawMessage(have))
images, err := h.ImagesOf("ace", "10.0.0.5:5000")
if err != nil || len(images.Images) != 3 || !images.SentKnown || images.Images[0].Image != "10.0.0.5:5000/letta/image@sha256:"+strings.Repeat("1", 64) ||
fmt.Sprint(images.Images[0].In) != "[declared]" || fmt.Sprint(images.Images[1].In) != "[sent]" ||
images.Images[2].Image != pinnedImage || fmt.Sprint(images.Images[2].In) != "[declared sent]" {
t.Fatalf("the images of ace are %+v: %v", images, err)
}
}
@@ -1,30 +1,38 @@
// The artifact-store-tools module's Go bundle (novox/hq ADR 0251 §1–3, to-be 51): a process the node's
// runtime launches and speaks MCP over stdio to. It says what the artifact store holds — its
// repositories, its size, and what the controller's records say of each manifest — read from the
// store's own files through the store's own container, and writing nothing. A collection is asked of
// the controller, which decides and records what it lets go of (ADR 0189).
// The artifact-store-tools module's Go bundle (novox/hq ADR 0251 §1–3, ADR 0571 clause 2, to-be 51): a process the
// node's runtime launches and speaks MCP over stdio to. It says what the artifact store holds — its repositories, its
// size, and what the mesh keeps of each manifest — read from the store's own files through the store's own container,
// and it holds the seat mesh-artifact-sweep: the keep set, the sweep at each build's end once the controller's lease names
// no controller, and S25's image-missing.
//
// **A module beside the store, not the store's own.** A module that provides the artifact store cannot
// also build an artifact: building publishes to the store, so the store would be needed to create
// itself, and the controller refuses that manifest. So these tools are a second module, assigned to the
// machine that holds the store. stdout is the protocol; what this bundle says, it says on stderr.
// **A module beside the store, not the store's own.** A module that provides the artifact store cannot also build an
// artifact: building publishes to the store, so the store would be needed to create itself. So these tools are a
// second module, assigned to the machine that holds the store. stdout is the protocol; what this bundle says, it says
// on stderr.
package main
import (
"fmt"
"os"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/handacts"
)
func logf(format string, a ...any) { fmt.Fprintf(os.Stderr, format+"\n", a...) }
func main() {
s := &Store{
Container: os.Getenv("MESH_ARTIFACT_STORE_CONTAINER"),
Run: ExecRunner,
UID: os.Getuid(),
}
h := &Holder{Bus: runtimeBus{}, Door: DoorAt(os.Getenv("MESH_ARTIFACT_STORE_ADDRESS_FILE")), Listed: s.View,
Now: time.Now, Logf: logf}
h.Acts = handacts.Recorder{Ask: stdio.Ask, LeaseAway: h.LeaseAway}
go follow(h)
// An empty name serves as the module the runtime names (MESH_SERVED_MODULE): artifact-store-tools.
if err := stdio.Serve("", Tools(s, Controller{Ask: stdio.Ask}, &Catalogue{Ask: stdio.Ask})); err != nil {
if err := stdio.Serve("", append(Tools(s, h, &Catalogue{Ask: stdio.Ask}), SeatTools(h, s)...)); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
@@ -0,0 +1,122 @@
package main
// The seat mesh-artifact-sweep's verbs (novox/hq ADR 0571 clause 2; catalogue.ArtifactSweepSeat), each a tool named
// `artifact-store.<verb>` as the runtime serves a seat's verb, answering as the controller's verb answered: {output,
// ok, answer}.
import (
"fmt"
"strings"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
// verbAnswer is the controller's verbs' answer shape, which the builder and docker read (answerOf).
type verbAnswer struct {
Output string `json:"output"`
OK bool `json:"ok"`
Answer any `json:"answer"`
}
// SeatTools are the seat's verbs over h; s reads the store's listing for mirrors.
func SeatTools(h *Holder, s *Store) []stdio.Tool {
seat := catalogue.ArtifactSweepSeat + "."
return []stdio.Tool{
{Name: seat + "artifacts", Description: "Every artifact the mesh made that the store holds: kept and why, eligible, " +
"or with collected those it no longer holds.",
Run: func(args map[string]any) (any, error) {
r, err := h.Artifacts(text(args, "repository"))
if err != nil {
return nil, err
}
if text(args, "collected") != "true" {
refs := r.References[:0:0]
for _, x := range r.References {
if x.State != "collected" {
refs = append(refs, x)
}
}
r.References = refs
}
return verbAnswer{Output: fmt.Sprintf("%d kept, %d eligible, %d collected\n", r.Counts["kept"],
r.Counts["eligible"], r.Counts["collected"]), OK: true, Answer: r}, nil
}},
{Name: seat + "held", Description: "Whether the store holds each reference given: held, gone or unknown with why.",
Run: func(args map[string]any) (any, error) {
a, err := h.Held(listOf(args["references"]))
if err != nil {
return nil, err
}
return verbAnswer{Output: fmt.Sprintf("the artifact store at %s holds %d of the %d reference(s) asked\n",
a.Store, len(a.Held), len(a.Held)+len(a.Gone)+len(a.Unknown)), OK: true, Answer: a}, nil
}},
{Name: seat + "collect", Description: "The sweep now: a dry run without confirm; with confirm and why, every kept " +
"archive held, then each eligible artifact let go of, each recorded in the hand-act log.",
Run: func(args map[string]any) (any, error) {
most, err := count(args, "most", MostPerSweep, 1, MostAtAll)
if err != nil {
return nil, err
}
a, err := h.Collect(text(args, "why"), text(args, "confirm") == "true", most)
if err != nil {
return nil, err
}
return verbAnswer{Output: fmt.Sprintf("let go of %d, would let go of %d, %d left\n", len(a.LetGo),
len(a.WouldLetGo), a.Left), OK: true, Answer: a}, nil
}},
{Name: seat + "mirrors", Description: "The bases the mesh copied into its store, each kept (a kept build stood on " +
"it), eligible (only builds no longer kept stood on it) or unrecorded (no build the builder's log holds stood " +
"on it: never removed).",
Run: func(args map[string]any) (any, error) {
if text(args, "record") != "" {
return nil, fmt.Errorf("mirrors records nothing by hand on this seat: a copy is recorded by the builds the " +
"builder's log says stood on it (novox/hq ADR 0571 clause 1)")
}
m, err := h.MirrorsNow()
if err != nil {
return nil, err
}
return verbAnswer{Output: fmt.Sprintf("%d copies\n", len(m.Copies)), OK: true, Answer: m}, nil
}},
{Name: seat + "images", Description: "Every container image one machine's wants and haves name.",
Run: func(args map[string]any) (any, error) {
node := text(args, "node")
if node == "" {
return nil, fmt.Errorf("images names the machine")
}
address := ""
if d, err := h.Door(); err == nil {
address = d.Address()
}
a, err := h.ImagesOf(node, address)
if err != nil {
return nil, err
}
return verbAnswer{Output: fmt.Sprintf("%s: %d images\n", node, len(a.Images)), OK: true, Answer: a}, nil
}},
}
}
// listOf is a list argument: a list, or one text separated by commas.
func listOf(v any) []string {
var out []string
add := func(s string) {
for _, p := range strings.Split(s, ",") {
if p = strings.TrimSpace(p); p != "" && !contains(out, p) {
out = append(out, p)
}
}
}
switch x := v.(type) {
case string:
add(x)
case []any:
for _, e := range x {
if s, ok := e.(string); ok {
add(s)
}
}
}
return out
}
@@ -1,6 +1,8 @@
package main
import (
"git.novox.be/novox/mesh-sdk/go/catalogue"
"context"
"encoding/json"
"fmt"
@@ -411,6 +413,11 @@ func TestTheToolsServedAreTheToolsTheManifestNames(t *testing.T) {
var m struct {
Tools []string `json:"tools"`
Invokes []string `json:"invokes"`
Reads []string `json:"reads"`
Claims []struct {
Name string `json:"name"`
Serves []string `json:"serves"`
} `json:"claims"`
}
if err := json.Unmarshal(raw, &m); err != nil {
t.Fatal(err)
@@ -428,10 +435,30 @@ func TestTheToolsServedAreTheToolsTheManifestNames(t *testing.T) {
if !reflect.DeepEqual(served, listed) {
t.Fatalf("served %v, manifest %v", served, listed)
}
want := []string{"seat:mesh-controller.artifacts", "seat:mesh-controller.collect", CatalogueModules, CatalogueModule}
// ADR 0571 clause 2: it holds the seat mesh-artifact-sweep and serves its verbs, and calls no verb of the controller's;
// the lease is the one controller state it reads (clause 10's exemption).
want := []string{"seat:mesh-hand-acts.hand-act", CatalogueModules, CatalogueModule}
if !reflect.DeepEqual(m.Invokes, want) {
t.Errorf("invokes %v", m.Invokes)
}
var verbs []string
for _, tl := range SeatTools(&Holder{}, &Store{}) {
verbs = append(verbs, strings.TrimPrefix(tl.Name, catalogue.ArtifactSweepSeat+"."))
}
seat, _ := catalogue.SeatNamed(catalogue.ArtifactSweepSeat)
var contract []string
for _, v := range seat.Serves {
contract = append(contract, v.Name)
}
if len(m.Claims) != 1 || m.Claims[0].Name != catalogue.ArtifactSweepSeat || !reflect.DeepEqual(m.Claims[0].Serves, verbs) ||
!reflect.DeepEqual(verbs, contract) {
t.Errorf("claims %+v, serves %v, the seat's verbs %v", m.Claims, verbs, contract)
}
for _, r := range m.Reads {
if strings.HasPrefix(r, "mesh-controller.") && r != LeaseRead {
t.Errorf("reads the controller's %s", r)
}
}
}
func TestASocketRefusalIsAskedAgainThroughSudo(t *testing.T) {
@@ -11,9 +11,16 @@ import (
stdio "git.novox.be/novox/mesh-sdk/go"
)
// Recorded is what says which references the mesh keeps and lets go of them: the seat's holder (Holder, novox/hq ADR
// 0571 clause 2), or before it the controller (Controller), each in the controller's answer's shape.
type Recorded interface {
Artifacts(repository string) (*Records, error)
Collect(why string, confirm bool, most int) (*CollectAnswer, error)
}
// Tools are the artifact store's tools (novox/hq ADR 0251 §1). The first three only read; artifact_store_collect
// only reads too, unless it is a real run, and then it asks the controller, which decides and deletes.
func Tools(s *Store, c Controller, cat *Catalogue) []stdio.Tool {
// only reads too, unless it is a real run, and then it asks the seat's sweep, which decides and deletes.
func Tools(s *Store, c Recorded, cat *Catalogue) []stdio.Tool {
ctx := context.Background()
repoArg := map[string]any{"type": "string", "description": "one repository, as the store names it (<module>/<artifact>)"}
return []stdio.Tool{
@@ -54,7 +61,7 @@ func Tools(s *Store, c Controller, cat *Catalogue) []stdio.Tool {
Name: "artifact_store_references",
Description: "What the controller's records say of each manifest the artifact store holds: kept (a definition " +
"names it, or one of the five most recent builds of a module the mesh holds), named by a definition (no " +
"record names it, and a definition in the catalogue pins it by digest, under the store's current name or " +
"record names it, and a definition in the catalog pins it by digest, under the store's current name or " +
"an older one: the module is said), the holder of a kept archive, " +
"eligible (the mesh made it and keeps it for no reason), the holder of an eligible archive, let go yet present, " +
"a named document the controller keeps under a tag, or unrecorded — no record names it. A platform manifest " +
@@ -0,0 +1 @@
${seat:mesh-artifact-store:reach}
+19 -2
View File
@@ -1,5 +1,22 @@
module artifactstoretools
go 1.22
go 1.26.0
require git.novox.be/novox/mesh-sdk/go v0.1.7
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029231453-6e4a3014cbde
github.com/nats-io/nats.go v1.54.0
)
require (
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op // indirect
github.com/google/go-tpm v0.9.8 // indirect
github.com/klauspost/compress v1.20.0 // indirect
github.com/minio/highwayhash v1.0.4 // indirect
github.com/nats-io/jwt/v2 v2.8.1 // indirect
github.com/nats-io/nats-server/v2 v2.11.17 // indirect
github.com/nats-io/nkeys v0.4.16 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
golang.org/x/crypto v0.57.0 // indirect
golang.org/x/sys v0.48.0 // indirect
golang.org/x/time v0.15.0 // indirect
)
+27 -2
View File
@@ -1,2 +1,27 @@
git.novox.be/novox/mesh-sdk/go v0.1.7 h1:C0sTQmtTiyYH7bnqZb7PusXnqA37gKuT7Nqjn9gG47w=
git.novox.be/novox/mesh-sdk/go v0.1.7/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029231453-6e4a3014cbde h1:ddf/jKI9c87sQKDmZLoG6PMnvXwsNH16ByVveyuFME4=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029231453-6e4a3014cbde/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
github.com/google/go-tpm v0.9.8/go.mod h1:h9jEsEECg7gtLis0upRBQU+GhYVH6jMjrFxI8u6bVUY=
github.com/klauspost/compress v1.20.0 h1:a3C1ke2ohxFymNlb2HWAHjDeKCI90scRskErZkR0ezA=
github.com/klauspost/compress v1.20.0/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/minio/highwayhash v1.0.4 h1:asJizugGgchQod2ja9NJlGOWq4s7KsAWr5XUc9Clgl4=
github.com/minio/highwayhash v1.0.4/go.mod h1:GGYsuwP/fPD6Y9hMiXuapVvlIUEhFhMTh0rxU3ik1LQ=
github.com/nats-io/jwt/v2 v2.8.1 h1:V0xpGuD/N8Mi+fQNDynXohVvp7ZztevW5io8CUWlPmU=
github.com/nats-io/jwt/v2 v2.8.1/go.mod h1:nWnOEEiVMiKHQpnAy4eXlizVEtSfzacZ1Q43LIRavZg=
github.com/nats-io/nats-server/v2 v2.11.17 h1:GKEghcFK6A+aFx11Yf1LjgLC3txAwvyhnYzhBIQZA8I=
github.com/nats-io/nats-server/v2 v2.11.17/go.mod h1:B1sFVz4StNosQ903ak4N1G01Fl/9f8e06mXpFIE2K24=
github.com/nats-io/nats.go v1.54.0 h1:vsXoOxjHp/GmPUN+EcI7uOf/uB+iAP+kEsAFNQN0yzA=
github.com/nats-io/nats.go v1.54.0/go.mod h1:y+DZoD1oBOYfZTU681eTUiUjI0vbqYGixNVFHcjHJ0k=
github.com/nats-io/nkeys v0.4.16 h1:rd5oAuLOb8mnAycB0xleuEBNS1pVVnN0fv/FF34Eypg=
github.com/nats-io/nkeys v0.4.16/go.mod h1:llLgWoI0o4z/Q57q2R1kHfmocyhGV6VG/U18Glg1Afs=
github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw=
github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
@@ -0,0 +1,57 @@
package registry
import (
"context"
"crypto/sha256"
"encoding/hex"
"fmt"
"io"
"net/http"
)
// blobBound is the most GetBlob reads: a templates archive is a few kilobytes; anything near this is no archive a
// take-in reads text from.
const blobBound = 64 << 20
// GetBlob is the archive a recorded reference names (`artifact-store://<repository>/blobs/sha256:<hex>`), read whole
// and checked against the digest it names: bytes that hash to another digest are refused, never read (R4e: the
// controller reads a templates archive by the reference of a builder's done key). Gone when the store does not hold
// it; ErrNotOurs for a reference outside the store or one that is not an archive.
func (s Store) GetBlob(ctx context.Context, reference string) ([]byte, error) {
repository, digest, archive, err := s.archive(reference)
if err != nil {
return nil, err
}
if !archive {
return nil, fmt.Errorf("%w: %s is not an archive", ErrNotOurs, reference)
}
request, err := http.NewRequestWithContext(ctx, http.MethodGet, s.url(repository, "blobs", digest), nil)
if err != nil {
return nil, err
}
response, err := s.client().Do(request)
if err != nil {
return nil, fmt.Errorf("cannot reach the artifact store at %s: %w", s.Address, err)
}
defer response.Body.Close()
switch response.StatusCode {
case http.StatusOK:
case http.StatusNotFound:
return nil, Gone
default:
return nil, fmt.Errorf("the artifact store answered %s for %s/blobs/%s", response.Status, repository, digest)
}
body, err := io.ReadAll(io.LimitReader(response.Body, blobBound+1))
if err != nil {
return nil, fmt.Errorf("reading %s/blobs/%s: %w", repository, digest, err)
}
if len(body) > blobBound {
return nil, fmt.Errorf("%s/blobs/%s is larger than %d bytes, which no archive read here is", repository, digest, blobBound)
}
sum := sha256.Sum256(body)
if got := "sha256:" + hex.EncodeToString(sum[:]); got != digest {
return nil, fmt.Errorf("the artifact store answered %s/blobs/%s with bytes of the digest %s: refused", repository,
digest, got)
}
return body, nil
}
@@ -0,0 +1,44 @@
package registry
import (
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"net/http"
"net/http/httptest"
"strings"
"testing"
)
// A take-in reads a templates archive by its reference, and only bytes of the digest the reference names.
func TestGetBlobReadsOnlyTheBytesOfItsDigest(t *testing.T) {
good := []byte("the archive")
sum := sha256.Sum256(good)
digest := "sha256:" + hex.EncodeToString(sum[:])
liar := "sha256:" + strings.Repeat("0", 64)
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch {
case strings.HasSuffix(r.URL.Path, "/x/files/blobs/"+digest), strings.HasSuffix(r.URL.Path, "/x/files/blobs/"+liar):
_, _ = w.Write(good)
default:
w.WriteHeader(http.StatusNotFound)
}
}))
defer server.Close()
s := Store{Address: strings.TrimPrefix(server.URL, "http://")}
ctx := context.Background()
body, err := s.GetBlob(ctx, "artifact-store://x/files/blobs/"+digest)
if err != nil || string(body) != string(good) {
t.Fatalf("the archive: %q %v", body, err)
}
if _, err := s.GetBlob(ctx, "artifact-store://x/files/blobs/"+liar); err == nil || !strings.Contains(err.Error(), "refused") {
t.Fatalf("bytes of another digest are read: %v", err)
}
if _, err := s.GetBlob(ctx, "artifact-store://x/other/blobs/"+digest); !errors.Is(err, Gone) {
t.Fatalf("a blob the store does not hold: %v", err)
}
if _, err := s.GetBlob(ctx, "registry.example/x@"+digest); !errors.Is(err, ErrNotOurs) {
t.Fatalf("a reference outside the store: %v", err)
}
}
@@ -0,0 +1,345 @@
package registry
import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"io"
"net/http"
"strconv"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
// Every archive the store keeps is held by a manifest (novox/hq issue 253, ADR 0189).
//
// **The store's collector marks only from manifests.** The mesh's store is a stock registry, and
// its nightly `registry garbage-collect` walks every manifest in every repository, marks the blobs
// those manifests name, and deletes every blob it did not mark. An image is a manifest, so what
// the mesh keeps of an image survives. An archive was not: the builder put it in the store as a
// bare blob — upload, then `PUT ?digest=` — and nothing in the store names it. To the collector a
// bare blob is unreferenced, so the first real collection would have deleted every archive the
// mesh holds, kept or not, and every machine pinning a bundle would have found it gone. The
// collector runs `--dry-run` until this is true.
//
// **So each archive gets a holder**: the smallest OCI image manifest that names it — the empty
// config, one layer, nothing else — put in the archive's own repository, by digest, untagged. The
// collector marks it and so keeps the archive; the sweep lets go of an archive by deleting its
// holder first, which is what lets the bytes go at the next collection.
//
// **Nothing a machine reads changes.** The recorded reference stays
// `artifact-store://<module>/<artifact>/blobs/sha256:…`, and machines fetch the blob exactly as
// before. The holder is the store's bookkeeping, not a second way to reach anything.
//
// **Deterministic, so it never needs recording.** The holder is composed from the archive's digest
// and size alone, in a fixed field order with no timestamps or annotations, so the sweep can
// compute which manifest holds any archive from the reference it already has plus one HEAD for the
// size. No schema change, no second record that could disagree with the store.
const (
// mediaManifest is the type a holder is put and asked for as.
mediaManifest = "application/vnd.oci.image.manifest.v1+json"
// mediaEmpty is the OCI empty descriptor's type: a config that says nothing, for a manifest
// whose only purpose is to name its layer.
mediaEmpty = "application/vnd.oci.empty.v1+json"
// emptyDigest is the digest of `{}`, the empty config's content, fixed by the OCI spec.
emptyDigest = "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a"
// mediaArchive is the layer type an archive is held as. Every archive the builder publishes is
// `pack`'s gzipped tar, so this is the true type and not a placeholder — and it is a constant,
// not read from anywhere, because the holder must be recomputable from the reference alone.
mediaArchive = "application/vnd.oci.image.layer.v1.tar+gzip"
)
// emptyConfig is the content emptyDigest names.
var emptyConfig = []byte("{}")
// manifestAccept is what a manifest is asked for as. A registry answers a manifest HEAD only in a
// type the caller named, and answers 404 to a bare one for a manifest it holds perfectly well
// (measured 2026-09-28; internal/builder/registry.go says how that was found).
var manifestAccept = []string{
mediaManifest,
"application/vnd.docker.distribution.manifest.v2+json",
}
type descriptor struct {
MediaType string `json:"mediaType"`
Digest string `json:"digest"`
Size int64 `json:"size"`
}
type holderManifest struct {
SchemaVersion int `json:"schemaVersion"`
MediaType string `json:"mediaType"`
Config descriptor `json:"config"`
Layers []descriptor `json:"layers"`
}
// Holder is the manifest that holds an archive in the store, and its digest.
//
// A pure function of the archive's digest and size: the same two in give the same bytes out,
// always, because `encoding/json` writes a struct's fields in their declared order and there is
// nothing here that varies by when or where it was composed.
func Holder(digest string, size int64) (body []byte, holder string) {
body, err := json.Marshal(holderManifest{
SchemaVersion: 2,
MediaType: mediaManifest,
Config: descriptor{MediaType: mediaEmpty, Digest: emptyDigest, Size: int64(len(emptyConfig))},
Layers: []descriptor{{MediaType: mediaArchive, Digest: digest, Size: size}},
})
if err != nil {
// Marshalling a struct of strings and integers cannot fail.
panic(err)
}
sum := sha256.Sum256(body)
return body, "sha256:" + hex.EncodeToString(sum[:])
}
// Hold makes sure the store holds this archive by a manifest, and says whether it had to write one.
//
// Takes a reference as the mesh records it. An image is its own manifest and needs no holder, so
// it answers false and nothing is asked. Idempotent: a holder already there is left alone, which
// is what lets the sweep run it over every kept archive on every build and so backfill the bare
// blobs published before holders existed (novox/hq issue 253).
//
// Gone when the store does not hold the archive at all: there is nothing to hold, and that is a
// fact the caller reports rather than one this invents a remedy for.
func (s Store) Hold(ctx context.Context, reference string) (bool, error) {
repository, digest, archive, err := s.archive(reference)
if err != nil || !archive {
return false, err
}
size, err := s.blobSize(ctx, repository, digest)
if err != nil {
return false, err
}
return s.HoldBlob(ctx, repository, digest, size)
}
// Held is whether the store holds this archive by its manifest. Asks and changes nothing — the
// question an operator needs answered with "none unheld" before the collector is let loose.
//
// An image answers true: it is its own manifest. An archive the store does not have answers Gone.
func (s Store) Held(ctx context.Context, reference string) (bool, error) {
repository, digest, archive, err := s.archive(reference)
if err != nil {
return false, err
}
if !archive {
return true, nil
}
size, err := s.blobSize(ctx, repository, digest)
if err != nil {
return false, err
}
_, holder := Holder(digest, size)
return s.has(ctx, s.url(repository, "manifests", holder), manifestAccept...)
}
// HoldBlob puts the holder for a blob of this digest and size into its repository, unless it is
// there already. Answers whether it wrote one.
//
// The builder calls this with the size it has just uploaded; the sweep, through Hold, with the size
// the store reports. Both arrive at the same holder, which is the point of composing it.
func (s Store) HoldBlob(ctx context.Context, repository, digest string, size int64) (bool, error) {
if s.Address == "" {
return false, fmt.Errorf("this mesh has no artifact store on its network to hold %s/%s in", repository, digest)
}
body, holder := Holder(digest, size)
there, err := s.has(ctx, s.url(repository, "manifests", holder), manifestAccept...)
if err != nil {
return false, err
}
if there {
return false, nil
}
// The config must be in the repository before a manifest naming it is accepted: a registry
// refuses a manifest whose blobs it cannot find there, which is the property that makes a
// holder mean something.
if err := s.putBlob(ctx, repository, emptyDigest, emptyConfig); err != nil {
return false, err
}
// **By digest, never by tag.** A tag would be one more name to move and one more thing the
// collector's `--delete-untagged` would read as meaningful; the mesh names nothing by tag that
// it pins by digest, and an untagged manifest is kept by plain collection.
request, err := http.NewRequestWithContext(ctx, http.MethodPut,
s.url(repository, "manifests", holder), bytes.NewReader(body))
if err != nil {
return false, err
}
request.Header.Set("Content-Type", mediaManifest)
response, err := s.client().Do(request)
if err != nil {
return false, err
}
defer response.Body.Close()
if response.StatusCode != http.StatusCreated {
said, _ := io.ReadAll(io.LimitReader(response.Body, 4096))
return false, fmt.Errorf("the artifact store refused to hold %s/%s: %s %s",
repository, digest, response.Status, strings.TrimSpace(string(said)))
}
return true, nil
}
// letGoOfHolder deletes the manifest holding an archive, before the archive's own link goes.
//
// **Holder first.** Deleting the blob link alone leaves a manifest still naming the blob, and the
// collector would keep its bytes for ever on the strength of it — the sweep would record the
// archive collected while the disk said otherwise. Deleting the holder first and failing before
// the link goes leaves an unheld archive that the next sweep still offers, which is safe.
//
// A store that no longer has the blob answers Gone: without its size the holder cannot be named,
// and without the blob there is nothing left for a holder to keep. A store that never had a holder
// for it — an archive published before holders, never backfilled — answers 404 to the delete, and
// that is the outcome wanted.
func (s Store) letGoOfHolder(ctx context.Context, repository, digest string) error {
size, err := s.blobSize(ctx, repository, digest)
if err != nil {
return err
}
_, holder := Holder(digest, size)
err = s.remove(ctx, s.url(repository, "manifests", holder), repository+"/manifests/"+holder)
if err == Gone {
return nil
}
return err
}
// archive reads a recorded reference into its repository and digest, and whether it is an archive
// at all. Refuses as ErrNotOurs anything the mesh did not put in its own store.
func (s Store) archive(reference string) (repository, digest string, archive bool, err error) {
path, kept := catalogue.InArtifactStore(reference)
if !kept {
return "", "", false, fmt.Errorf("%w: %s", ErrNotOurs, reference)
}
if s.Address == "" {
return "", "", false, fmt.Errorf("this mesh has no artifact store on its network to ask about %s", reference)
}
repository, kind, digest, err := split(path)
if err != nil {
return "", "", false, err
}
return repository, digest, kind == "blobs", nil
}
// blobSize is how large the store says a blob is; Gone when it does not have it.
func (s Store) blobSize(ctx context.Context, repository, digest string) (int64, error) {
request, err := http.NewRequestWithContext(ctx, http.MethodHead, s.url(repository, "blobs", digest), nil)
if err != nil {
return 0, err
}
response, err := s.client().Do(request)
if err != nil {
return 0, fmt.Errorf("cannot reach the artifact store at %s: %w", s.Address, err)
}
defer response.Body.Close()
switch response.StatusCode {
case http.StatusOK:
case http.StatusNotFound:
return 0, Gone
default:
return 0, fmt.Errorf("the artifact store answered %s for %s/blobs/%s", response.Status, repository, digest)
}
// Read from the header rather than ContentLength: a HEAD's ContentLength is what the response
// says it would have sent, which Go reports faithfully, but a proxy in between is free to drop
// it, and the header is what the registry itself wrote.
if length := response.Header.Get("Content-Length"); length != "" {
if n, err := strconv.ParseInt(length, 10, 64); err == nil && n >= 0 {
return n, nil
}
}
if response.ContentLength >= 0 {
return response.ContentLength, nil
}
return 0, fmt.Errorf("the artifact store holds %s/blobs/%s and will not say how large it is", repository, digest)
}
// putBlob uploads a small blob unless the repository already has it: ask where, then put it there
// naming the digest — the registry's own two steps, the same the builder takes for an archive.
func (s Store) putBlob(ctx context.Context, repository, digest string, body []byte) error {
if there, err := s.has(ctx, s.url(repository, "blobs", digest)); err != nil {
return err
} else if there {
return nil
}
start, err := http.NewRequestWithContext(ctx, http.MethodPost,
"http://"+s.Address+"/v2/"+repository+"/blobs/uploads/", nil)
if err != nil {
return err
}
begun, err := s.client().Do(start)
if err != nil {
return fmt.Errorf("cannot start an upload to %s: %w", repository, err)
}
begun.Body.Close()
if begun.StatusCode != http.StatusAccepted {
return fmt.Errorf("the artifact store answered %s when asked where to put a blob in %s", begun.Status, repository)
}
where := begun.Header.Get("Location")
if where == "" {
return fmt.Errorf("the artifact store accepted an upload to %s and said nowhere to put it", repository)
}
if strings.HasPrefix(where, "/") {
where = "http://" + s.Address + where
}
separator := "?"
if strings.Contains(where, "?") {
separator = "&"
}
put, err := http.NewRequestWithContext(ctx, http.MethodPut, where+separator+"digest="+digest, bytes.NewReader(body))
if err != nil {
return err
}
put.Header.Set("Content-Type", "application/octet-stream")
done, err := s.client().Do(put)
if err != nil {
return err
}
defer done.Body.Close()
if done.StatusCode != http.StatusCreated {
said, _ := io.ReadAll(io.LimitReader(done.Body, 4096))
return fmt.Errorf("the artifact store refused a blob in %s: %s %s", repository, done.Status, strings.TrimSpace(string(said)))
}
return nil
}
// has is whether the store answers 200 for a HEAD at that URL.
func (s Store) has(ctx context.Context, url string, accept ...string) (bool, error) {
request, err := http.NewRequestWithContext(ctx, http.MethodHead, url, nil)
if err != nil {
return false, err
}
for _, media := range accept {
request.Header.Add("Accept", media)
}
response, err := s.client().Do(request)
if err != nil {
return false, fmt.Errorf("cannot reach the artifact store at %s: %w", s.Address, err)
}
defer response.Body.Close()
switch response.StatusCode {
case http.StatusOK:
return true, nil
case http.StatusNotFound:
return false, nil
default:
return false, fmt.Errorf("the artifact store answered %s for %s", response.Status, url)
}
}
func (s Store) url(repository, kind, digest string) string {
return "http://" + s.Address + "/v2/" + repository + "/" + kind + "/" + digest
}
func (s Store) client() *http.Client {
if s.HTTP != nil {
return s.HTTP
}
return &http.Client{Timeout: 30 * time.Second}
}
@@ -0,0 +1,268 @@
package registry
import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"net/http/httptest"
"strconv"
"strings"
"sync"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
// Every kept archive is held by a manifest (novox/hq issue 253, ADR 0189).
//
// Against an in-memory registry that keeps blobs and manifests per repository and refuses what a
// registry refuses — a blob whose digest does not match, a manifest whose digest does not match
// or whose blobs the repository does not have, a manifest asked for without an Accept naming its
// type. What is asserted is this side's decisions; the live test below asserts the registry's.
type memRegistry struct {
mu sync.Mutex
blobs map[string][]byte // repository + "@" + digest
manifests map[string][]byte // repository + "@" + digest
writes []string // every PUT and DELETE, as "METHOD path"
}
func digestOf(body []byte) string {
sum := sha256.Sum256(body)
return "sha256:" + hex.EncodeToString(sum[:])
}
func (m *memRegistry) serve(t *testing.T) Store {
t.Helper()
m.blobs = map[string][]byte{}
m.manifests = map[string][]byte{}
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
m.mu.Lock()
defer m.mu.Unlock()
path := strings.TrimPrefix(r.URL.Path, "/v2/")
if r.Method == http.MethodPut || r.Method == http.MethodDelete {
m.writes = append(m.writes, r.Method+" "+r.URL.Path)
}
switch {
case r.Method == http.MethodPost && strings.HasSuffix(path, "/blobs/uploads/"):
repository := strings.TrimSuffix(path, "/blobs/uploads/")
w.Header().Set("Location", "/upload/"+repository+"?state=x")
w.WriteHeader(http.StatusAccepted)
case r.Method == http.MethodPut && strings.HasPrefix(r.URL.Path, "/upload/"):
repository := strings.TrimPrefix(r.URL.Path, "/upload/")
body, _ := io.ReadAll(r.Body)
digest := r.URL.Query().Get("digest")
if digest != digestOf(body) {
w.WriteHeader(http.StatusBadRequest)
return
}
m.blobs[repository+"@"+digest] = body
w.WriteHeader(http.StatusCreated)
case strings.Contains(path, "/blobs/"):
repository, digest, _ := strings.Cut(path, "/blobs/")
key := repository + "@" + digest
body, ok := m.blobs[key]
if !ok {
w.WriteHeader(http.StatusNotFound)
return
}
switch r.Method {
case http.MethodHead:
w.Header().Set("Content-Length", strconv.Itoa(len(body)))
w.WriteHeader(http.StatusOK)
case http.MethodDelete:
delete(m.blobs, key)
w.WriteHeader(http.StatusAccepted)
default:
w.WriteHeader(http.StatusMethodNotAllowed)
}
case strings.Contains(path, "/manifests/"):
repository, digest, _ := strings.Cut(path, "/manifests/")
key := repository + "@" + digest
switch r.Method {
case http.MethodHead:
if _, ok := m.manifests[key]; !ok || !strings.Contains(r.Header.Get("Accept"), mediaManifest) {
w.WriteHeader(http.StatusNotFound)
return
}
w.WriteHeader(http.StatusOK)
case http.MethodPut:
body, _ := io.ReadAll(r.Body)
if digest != digestOf(body) {
w.WriteHeader(http.StatusBadRequest)
return
}
var named holderManifest
if err := json.Unmarshal(body, &named); err != nil {
w.WriteHeader(http.StatusBadRequest)
return
}
for _, d := range append([]descriptor{named.Config}, named.Layers...) {
if _, ok := m.blobs[repository+"@"+d.Digest]; !ok {
w.WriteHeader(http.StatusBadRequest)
fmt.Fprintf(w, "MANIFEST_BLOB_UNKNOWN %s", d.Digest)
return
}
}
m.manifests[key] = body
w.WriteHeader(http.StatusCreated)
case http.MethodDelete:
if _, ok := m.manifests[key]; !ok {
w.WriteHeader(http.StatusNotFound)
return
}
delete(m.manifests, key)
w.WriteHeader(http.StatusAccepted)
}
default:
w.WriteHeader(http.StatusNotFound)
}
}))
t.Cleanup(server.Close)
return Store{Address: strings.TrimPrefix(server.URL, "http://")}
}
// bare puts an archive in the store the way the builder did before holders: a blob, nothing more.
func (m *memRegistry) bare(repository string, body []byte) string {
m.mu.Lock()
defer m.mu.Unlock()
digest := digestOf(body)
m.blobs[repository+"@"+digest] = body
return catalogue.ArtifactStoreScheme + repository + "/blobs/" + digest
}
func TestTheHolderIsComposedFromTheDigestAndSizeAlone(t *testing.T) {
// The sweep must arrive at the very manifest the builder wrote, with nothing recorded between
// them. Same inputs, same bytes — and a different size is a different holder, so a holder can
// never be mistaken for one of a different blob.
digest := "sha256:" + strings.Repeat("a", 64)
one, first := Holder(digest, 42)
two, second := Holder(digest, 42)
if !bytes.Equal(one, two) || first != second {
t.Fatalf("the same archive composed two holders:\n%s\n%s", one, two)
}
if _, other := Holder(digest, 43); other == first {
t.Fatal("a different size composed the same holder")
}
want := `{"schemaVersion":2,"mediaType":"application/vnd.oci.image.manifest.v1+json",` +
`"config":{"mediaType":"application/vnd.oci.empty.v1+json",` +
`"digest":"sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a","size":2},` +
`"layers":[{"mediaType":"application/vnd.oci.image.layer.v1.tar+gzip","digest":"` + digest + `","size":42}]}`
if string(one) != want {
t.Fatalf("the holder is\n%s\nwant\n%s", one, want)
}
if digestOf(emptyConfig) != emptyDigest {
t.Fatalf("the empty config's digest is %s, not %s", digestOf(emptyConfig), emptyDigest)
}
}
func TestHoldBackfillsABareArchiveAndIsIdempotent(t *testing.T) {
// The archives published before this have no holder. Hold, run over every kept archive on
// every sweep, writes one the first time and nothing after.
m := &memRegistry{}
store := m.serve(t)
ctx := context.Background()
body := []byte("a theme")
reference := m.bare("shell/config", body)
if held, err := store.Held(ctx, reference); err != nil || held {
t.Fatalf("a bare blob reads as held=%v (%v)", held, err)
}
wrote, err := store.Hold(ctx, reference)
if err != nil {
t.Fatal(err)
}
if !wrote {
t.Fatal("holding a bare archive wrote nothing")
}
_, holder := Holder(digestOf(body), int64(len(body)))
if _, ok := m.manifests["shell/config@"+holder]; !ok {
t.Fatalf("the store holds manifests %v; want %s", m.manifests, holder)
}
if held, err := store.Held(ctx, reference); err != nil || !held {
t.Fatalf("after holding, held=%v (%v)", held, err)
}
writes := len(m.writes)
wrote, err = store.Hold(ctx, reference)
if err != nil {
t.Fatal(err)
}
if wrote || len(m.writes) != writes {
t.Fatalf("holding again wrote %v", m.writes[writes:])
}
}
func TestAnImageNeedsNoHolderAndAMissingArchiveIsGone(t *testing.T) {
m := &memRegistry{}
store := m.serve(t)
ctx := context.Background()
// An image is its own manifest: nothing is asked.
wrote, err := store.Hold(ctx, catalogue.ArtifactStoreScheme+"web/app@sha256:"+strings.Repeat("b", 64))
if err != nil || wrote || len(m.writes) != 0 {
t.Fatalf("holding an image wrote=%v err=%v writes=%v", wrote, err, m.writes)
}
// An archive the store does not have is a fact to report, not something to invent a holder for.
_, err = store.Hold(ctx, catalogue.ArtifactStoreScheme+"web/config/blobs/sha256:"+strings.Repeat("c", 64))
if !errors.Is(err, Gone) {
t.Fatalf("holding a missing archive answered %v, want Gone", err)
}
// And a reference that is not the mesh's is refused as such.
if _, err := store.Hold(ctx, "docker.io/library/registry@sha256:abc"); !errors.Is(err, ErrNotOurs) {
t.Fatalf("holding a vendor's image answered %v, want ErrNotOurs", err)
}
}
func TestLettingGoOfAnArchiveDeletesItsHolderFirst(t *testing.T) {
// A holder left behind would keep the bytes through every collection while the record said
// collected; the link deleted first and the holder failing after would be that exactly.
m := &memRegistry{}
store := m.serve(t)
ctx := context.Background()
body := []byte("an old theme")
reference := m.bare("shell/config", body)
if _, err := store.Hold(ctx, reference); err != nil {
t.Fatal(err)
}
m.writes = nil
if err := store.LetGo(ctx, reference); err != nil {
t.Fatal(err)
}
_, holder := Holder(digestOf(body), int64(len(body)))
want := []string{
"DELETE /v2/shell/config/manifests/" + holder,
"DELETE /v2/shell/config/blobs/" + digestOf(body),
}
if strings.Join(m.writes, "\n") != strings.Join(want, "\n") {
t.Fatalf("the store was asked\n%s\nwant\n%s", strings.Join(m.writes, "\n"), strings.Join(want, "\n"))
}
if len(m.manifests) != 0 {
t.Fatalf("a holder survived: %v", m.manifests)
}
// Asked again, the archive is already gone, which is the outcome wanted.
if err := store.LetGo(ctx, reference); !errors.Is(err, Gone) {
t.Fatalf("letting go twice answered %v, want Gone", err)
}
}
func TestLettingGoOfAnUnheldArchiveStillDeletesIt(t *testing.T) {
// An archive published before holders and let go of before any sweep held it: the holder's
// delete answers 404, which is the outcome wanted, and the blob still goes.
m := &memRegistry{}
store := m.serve(t)
reference := m.bare("shell/config", []byte("never held"))
if err := store.LetGo(context.Background(), reference); err != nil {
t.Fatal(err)
}
if len(m.blobs) != 0 {
t.Fatalf("the blob survived: %v", m.blobs)
}
}
@@ -0,0 +1,176 @@
package registry
import (
"context"
"errors"
"fmt"
"net/http"
"net/http/httptest"
"slices"
"strings"
"sync"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
// An index is let go of with the platform manifests it names, except those a kept index of the same
// repository names too (novox/hq ADR 0257). Each platform manifest is a manifest of the repository in
// its own right, and the store's collector keeps every manifest a repository holds — so an index let go
// of alone frees nothing of the images it names.
func digestN(n int) string { return fmt.Sprintf("sha256:%064x", n) }
// indexStore holds indexes and images in one repository, answers GETs with their documents, and records
// every delete.
type indexStore struct {
mu sync.Mutex
indexes map[string][]string // digest → the platform manifests it names
images map[string]bool
deleted []string
}
func (s *indexStore) serve(t *testing.T) Store {
t.Helper()
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
s.mu.Lock()
defer s.mu.Unlock()
digest := r.URL.Path[strings.LastIndex(r.URL.Path, "/")+1:]
switch r.Method {
case http.MethodGet:
if children, ok := s.indexes[digest]; ok {
var named []string
for _, c := range children {
named = append(named, `{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"`+c+`","size":1}`)
}
w.Header().Set("Content-Type", "application/vnd.oci.image.index.v1+json")
_, _ = w.Write([]byte(`{"schemaVersion":2,"mediaType":"application/vnd.oci.image.index.v1+json","manifests":[` +
strings.Join(named, ",") + `]}`))
return
}
if s.images[digest] {
w.Header().Set("Content-Type", "application/vnd.oci.image.manifest.v1+json")
_, _ = w.Write([]byte(`{"schemaVersion":2,"mediaType":"application/vnd.oci.image.manifest.v1+json","layers":[]}`))
return
}
w.WriteHeader(http.StatusNotFound)
case http.MethodHead:
if _, ok := s.indexes[digest]; ok || s.images[digest] {
w.WriteHeader(http.StatusOK)
return
}
w.WriteHeader(http.StatusNotFound)
case http.MethodDelete:
if _, ok := s.indexes[digest]; !ok && !s.images[digest] {
w.WriteHeader(http.StatusNotFound)
return
}
s.deleted = append(s.deleted, digest)
delete(s.indexes, digest)
delete(s.images, digest)
w.WriteHeader(http.StatusAccepted)
default:
w.WriteHeader(http.StatusBadRequest)
}
}))
t.Cleanup(server.Close)
return Store{Address: strings.TrimPrefix(server.URL, "http://")}
}
func TestAnIndexGoesWithItsPlatformsAndAKeptIndexKeepsItsOwn(t *testing.T) {
// Two indexes of one image's repository: the old one names platforms 11 and 12, the kept one 12 and 13.
s := &indexStore{
indexes: map[string][]string{digestN(1): {digestN(11), digestN(12)}, digestN(2): {digestN(12), digestN(13)}},
images: map[string]bool{digestN(11): true, digestN(12): true, digestN(13): true},
}
store := s.serve(t)
repository := "upstream/docker.io/library/golang"
old := catalogue.ArtifactStoreScheme + repository + "@" + digestN(1)
kept := catalogue.ArtifactStoreScheme + repository + "@" + digestN(2)
spare, err := store.SpareKeptIndexes(context.Background(), []string{kept}, []string{old})
if err != nil {
t.Fatal(err)
}
store.Spare = spare
if err := store.LetGo(context.Background(), old); err != nil {
t.Fatal(err)
}
// Its own platform first, the index last; the platform the kept index names is left in place.
if !slices.Equal(s.deleted, []string{digestN(11), digestN(1)}) {
t.Fatalf("deleted %v; want the old index's own platform, then the index", s.deleted)
}
if !s.images[digestN(12)] || !s.images[digestN(13)] {
t.Fatal("a platform a kept index names was let go of")
}
}
func TestWithoutSparingAnIndexGoesAloneAndAnImageNamesNoPlatforms(t *testing.T) {
s := &indexStore{indexes: map[string][]string{digestN(1): {digestN(11)}}, images: map[string]bool{digestN(11): true, digestN(5): true}}
store := s.serve(t)
// No Spare: as before ADR 0257, the index alone.
if err := store.LetGo(context.Background(), catalogue.ArtifactStoreScheme+"web/base@"+digestN(1)); err != nil {
t.Fatal(err)
}
if !slices.Equal(s.deleted, []string{digestN(1)}) {
t.Fatalf("without a Spare, deleted %v", s.deleted)
}
// An image is one delete, with a Spare or without.
s.deleted = nil
spare, err := store.SpareKeptIndexes(context.Background(), nil, []string{catalogue.ArtifactStoreScheme + "web/app@" + digestN(5)})
if err != nil {
t.Fatal(err)
}
store.Spare = spare
if err := store.LetGo(context.Background(), catalogue.ArtifactStoreScheme+"web/app@"+digestN(5)); err != nil {
t.Fatal(err)
}
if !slices.Equal(s.deleted, []string{digestN(5)}) {
t.Fatalf("an image was let go of as %v", s.deleted)
}
// An index the store no longer holds is Gone, and nothing is deleted on its account.
s.deleted = nil
err = store.LetGo(context.Background(), catalogue.ArtifactStoreScheme+"web/base@"+digestN(9))
if !errors.Is(err, Gone) || len(s.deleted) != 0 {
t.Fatalf("an index the store does not hold: %v, deleted %v", err, s.deleted)
}
}
func TestASpareThatCannotBeReadKeepsTheIndex(t *testing.T) {
s := &indexStore{indexes: map[string][]string{digestN(1): {digestN(11)}}, images: map[string]bool{digestN(11): true}}
store := s.serve(t)
store.Spare = func(context.Context, string) (map[string]bool, error) { return nil, fmt.Errorf("the store went away") }
err := store.LetGo(context.Background(), catalogue.ArtifactStoreScheme+"web/base@"+digestN(1))
if err == nil || !strings.Contains(err.Error(), "was kept") {
t.Fatalf("a spare that could not be read: %v", err)
}
if len(s.deleted) != 0 {
t.Fatalf("deleted %v without knowing what a kept index names", s.deleted)
}
}
// The spare list is read whole before the sweep: a kept index that cannot be read is an error before
// anything is deleted, and a repository not read before keeps its index.
func TestASpareListIsReadWholeBeforeAnythingGoes(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusInternalServerError)
}))
t.Cleanup(server.Close)
store := Store{Address: strings.TrimPrefix(server.URL, "http://")}
kept := catalogue.ArtifactStoreScheme + "web/base@" + digestN(2)
old := catalogue.ArtifactStoreScheme + "web/base@" + digestN(1)
if _, err := store.SpareKeptIndexes(context.Background(), []string{kept}, []string{old}); err == nil {
t.Fatal("a kept index the store would not answer for was taken as naming nothing")
}
s := &indexStore{indexes: map[string][]string{digestN(1): {digestN(11)}}, images: map[string]bool{digestN(11): true}}
good := s.serve(t)
spare, err := good.SpareKeptIndexes(context.Background(), nil, []string{catalogue.ArtifactStoreScheme + "other/repo@" + digestN(3)})
if err != nil {
t.Fatal(err)
}
good.Spare = spare
if err := good.LetGo(context.Background(), old); err == nil || len(s.deleted) != 0 {
t.Fatalf("an index in a repository not read before the sweep was let go of: %v, deleted %v", err, s.deleted)
}
}
@@ -0,0 +1,325 @@
// Package registry speaks to the mesh's artifact store over its own door.
//
// Only what the mesh needs that nothing else does: letting go of something it put there
// (novox/hq ADR 0189, issue 108), and holding every archive it keeps by a manifest so the store's
// own collector does not take it (novox/hq issue 253). Pushing is the builder's, through the container runtime; reading
// is every machine's, through its runtime. This is the one operation that belongs to the thing
// holding the records, because it is the only one that is a decision rather than a transfer.
//
// Copied from mesh-controller internal/artifacts at e04846e9 (novox/hq ADR 0571 clause 2): the seat mesh-artifact-sweep lets
// go of what it no longer keeps exactly as the controller's sweep did; the controller's copy goes with its lease.
package registry
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"strings"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
// Store is the artifact store at an address, as this machine reaches it.
type Store struct {
// Address is `host:port` — the store as the caller reaches it now, composed and never
// recorded (novox/hq 04-ISSUES/102).
Address string
// HTTP is the client used; nil is a client with a modest timeout.
HTTP *http.Client
// Spare says, for one repository, which manifests a kept index there names: the platform
// manifests an index being let go of must leave in place (novox/hq ADR 0257). Nil spares every
// platform manifest — an index is then let go of alone, as it was before.
Spare func(ctx context.Context, repository string) (map[string]bool, error)
}
// Gone is the answer when the store does not hold it: the outcome wanted, already true.
var Gone = errors.New("the store does not hold it")
// ErrNotOurs is a reference this sweep will not address: not the mesh's own, or naming nothing
// the store holds by digest.
//
// **A fact about the record, not about the store** (novox/hq issue 226). The two deserve opposite
// responses — skip one and go on, abandon the sweep for the other — and collapsing them into "an
// error" is how a cautious loop became one that did nothing while reporting the right number.
var ErrNotOurs = errors.New("not a reference into the mesh's artifact store")
// LetGo asks the store to drop one artifact the mesh recorded making.
//
// Takes a reference as the mesh records it — `artifact-store://<module>/<artifact>@sha256:…` for
// an image, `…/blobs/sha256:…` for an archive — because that is the identity every record uses,
// and composes the address here at the moment of use.
//
// An archive is let go of in two deletes, its holder manifest and then the blob's link (novox/hq
// issue 253); an image in one.
//
// Returns Gone when the store answers that it does not have it. That is not a failure: the sweep
// wants the artifact absent, and it is. It is distinguished from success only so a caller can say
// which of the two happened.
func (s Store) LetGo(ctx context.Context, reference string) error {
// **Strict, and deliberately** (novox/hq issue 226). Only a reference the mesh keeps in its
// own vocabulary is addressed here. `Recorded` would read `docker.io/library/registry@sha256:…`
// as the mesh's too — it cannot tell one registry host from another — so normalising belongs
// where the provenance is known, which is the sweep reading its own build records, not here
// where the only job is to refuse anything that is not plainly ours.
path, kept := catalogue.InArtifactStore(reference)
if !kept {
// Nothing the mesh put in its own store. Refused rather than attempted: composing a
// delete for a reference of unknown shape is how a sweep reaches something that is not
// the mesh's. Distinguished from a store that refuses, so a sweep skips this and goes on.
return fmt.Errorf("%w: %s", ErrNotOurs, reference)
}
if s.Address == "" {
return fmt.Errorf("this mesh has no artifact store on its network to ask about %s", reference)
}
repository, kind, digest, err := split(path)
if err != nil {
return err
}
if kind == "blobs" {
// **An archive's holder goes before the archive** (novox/hq issue 253): a manifest left
// naming the blob would keep its bytes through every collection while the record said
// collected. Gone here means the store has no such blob, so there is nothing to let go.
if err := s.letGoOfHolder(ctx, repository, digest); err != nil {
return err
}
}
if kind == "manifests" && s.Spare != nil {
// **An index's platform manifests go before the index** (novox/hq ADR 0257). Each was put
// under its own digest, so each is a manifest of the repository in its own right, and the
// store's collector keeps every manifest a repository holds: an index let go of alone frees
// no byte of the images it names. Before, not after, so a sweep cut short leaves the index
// to be asked about again, and with it the list of what is still to go.
if err := s.letGoOfPlatforms(ctx, repository, digest, reference); err != nil {
return err
}
}
return s.remove(ctx, s.url(repository, kind, digest), reference)
}
// letGoOfPlatforms deletes the manifests an index names, except those a kept index of the same
// repository also names. A manifest that is not an index names none.
func (s Store) letGoOfPlatforms(ctx context.Context, repository, digest, reference string) error {
children, err := s.Platforms(ctx, repository, digest)
if err != nil || len(children) == 0 {
// Gone: there is no index to read, and the delete that follows says so.
if errors.Is(err, Gone) {
return nil
}
return err
}
spared, err := s.Spare(ctx, repository)
if err != nil {
return fmt.Errorf("cannot say which platform manifests of %s a kept index names, so %s was kept: %w",
repository, reference, err)
}
for _, child := range children {
if spared[child] {
continue
}
if err := s.remove(ctx, s.url(repository, "manifests", child), reference+" (its "+child+")"); err != nil && !errors.Is(err, Gone) {
return err
}
}
return nil
}
// anyManifest is every manifest the store may hold, indexes included, so it answers with the
// document as it is.
var anyManifest = []string{
"application/vnd.oci.image.index.v1+json",
"application/vnd.docker.distribution.manifest.list.v2+json",
"application/vnd.oci.image.manifest.v1+json",
"application/vnd.docker.distribution.manifest.v2+json",
}
// Platforms is the manifests an index names, by digest; none for a manifest that is not an index.
// Gone when the store does not hold it.
func (s Store) Platforms(ctx context.Context, repository, digest string) ([]string, error) {
request, err := http.NewRequestWithContext(ctx, http.MethodGet, s.url(repository, "manifests", digest), nil)
if err != nil {
return nil, err
}
request.Header.Set("Accept", strings.Join(anyManifest, ", "))
response, err := s.client().Do(request)
if err != nil {
return nil, err
}
defer response.Body.Close()
switch response.StatusCode {
case http.StatusOK:
case http.StatusNotFound:
return nil, Gone
default:
return nil, fmt.Errorf("the artifact store answered %s when asked for %s@%s", response.Status, repository, digest)
}
var document struct {
Manifests []struct {
Digest string `json:"digest"`
} `json:"manifests"`
}
if err := json.NewDecoder(io.LimitReader(response.Body, 4<<20)).Decode(&document); err != nil {
return nil, fmt.Errorf("%s@%s is not a manifest: %w", repository, digest, err)
}
var out []string
for _, m := range document.Manifests {
if strings.HasPrefix(m.Digest, "sha256:") {
out = append(out, m.Digest)
}
}
return out, nil
}
// SpareKeptIndexes is a Spare for a sweep, read whole before it starts: for every repository an
// image reference of `touching` is in, each kept manifest there and every platform manifest a kept
// index there names. Any read that fails is an error, and the sweep lets nothing go. A repository not
// read here is answered with an error, so an index in it is kept. A kept index the store no longer
// holds names nothing.
func (s Store) SpareKeptIndexes(ctx context.Context, kept, touching []string) (func(context.Context, string) (map[string]bool, error), error) {
imageIn := func(reference string) (repository, digest string, ok bool) {
path, ours := catalogue.InArtifactStore(reference)
if !ours {
return "", "", false
}
repository, kind, digest, err := split(path)
if err != nil || kind != "manifests" {
return "", "", false
}
return repository, digest, true
}
read := map[string]map[string]bool{}
for _, reference := range touching {
if repository, _, ok := imageIn(reference); ok {
read[repository] = map[string]bool{}
}
}
for _, reference := range kept {
repository, digest, ok := imageIn(reference)
spared, touched := read[repository]
if !ok || !touched {
continue
}
spared[digest] = true
children, err := s.Platforms(ctx, repository, digest)
if errors.Is(err, Gone) {
continue
}
if err != nil {
return nil, fmt.Errorf("reading the kept %s@%s: %w", repository, digest, err)
}
for _, c := range children {
spared[c] = true
}
}
return func(_ context.Context, repository string) (map[string]bool, error) {
spared, done := read[repository]
if !done {
return nil, fmt.Errorf("%s was not read before the sweep began", repository)
}
return spared, nil
}, nil
}
// remove asks the store to delete what is at url. Gone when it has no such thing.
func (s Store) remove(ctx context.Context, url, what string) error {
request, err := http.NewRequestWithContext(ctx, http.MethodDelete, url, nil)
if err != nil {
return err
}
response, err := s.client().Do(request)
if err != nil {
return err
}
defer response.Body.Close()
switch response.StatusCode {
case http.StatusAccepted, http.StatusOK, http.StatusNoContent:
return nil
case http.StatusNotFound:
return Gone
case http.StatusMethodNotAllowed:
// The registry was started without deletion enabled. Said plainly, because the remedy is
// a setting on the store's module and not anything about this artifact.
return fmt.Errorf(
"the artifact store refuses deletion: its server was started without it enabled "+
"(REGISTRY_STORAGE_DELETE_ENABLED), so nothing can be collected until the store "+
"module is applied again (novox/hq ADR 0189). Asking about %s", what)
default:
return fmt.Errorf("the artifact store answered %s for %s", response.Status, what)
}
}
// split reads a recorded path into the repository, which endpoint names the thing, and the digest.
//
// Two shapes, which are the two the mesh records: `<repository>@sha256:<hex>` is a manifest, and
// `<repository>/blobs/sha256:<hex>` is a blob.
func split(path string) (repository, kind, digest string, err error) {
if before, after, ok := strings.Cut(path, "@sha256:"); ok {
return before, "manifests", "sha256:" + after, nil
}
if before, after, ok := strings.Cut(path, "/blobs/sha256:"); ok {
return before, "blobs", "sha256:" + after, nil
}
return "", "", "", fmt.Errorf("%w: %q names nothing the store holds by digest", ErrNotOurs, path)
}
// IsIndex is whether a recorded reference is an index the store holds: one that names manifests. A
// blob, or a manifest that names none, is not. Gone when the store does not hold it.
func (s Store) IsIndex(ctx context.Context, reference string) (bool, error) {
path, ours := catalogue.InArtifactStore(reference)
if !ours {
return false, fmt.Errorf("%w: %s", ErrNotOurs, reference)
}
repository, kind, digest, err := split(path)
if err != nil || kind != "manifests" {
return false, err
}
children, err := s.Platforms(ctx, repository, digest)
return len(children) > 0, err
}
// HoldsManifest is whether the store holds the manifest a recorded image reference names.
func (s Store) HoldsManifest(ctx context.Context, reference string) (bool, error) {
path, ours := catalogue.InArtifactStore(reference)
if !ours {
return false, fmt.Errorf("%w: %s", ErrNotOurs, reference)
}
repository, kind, digest, err := split(path)
if err != nil {
return false, err
}
if kind != "manifests" {
return false, fmt.Errorf("%w: %s names a blob, not a manifest", ErrNotOurs, reference)
}
return s.has(ctx, s.url(repository, kind, digest), anyManifest...)
}
// Holds is whether the store holds what a recorded reference names: the blob of an archive, the manifest of an image.
// Gone when it does not; asks and changes nothing (ADR 0526 clause 3: a registration by hand is refused at bytes that
// are not there).
func (s Store) Holds(ctx context.Context, reference string) error {
path, ours := catalogue.InArtifactStore(reference)
if !ours {
return fmt.Errorf("%w: %s", ErrNotOurs, reference)
}
repository, kind, digest, err := split(path)
if err != nil {
return err
}
if s.Address == "" {
return fmt.Errorf("this mesh has no artifact store on its network to ask about %s", reference)
}
if kind == "blobs" {
_, err := s.blobSize(ctx, repository, digest)
return err
}
held, err := s.has(ctx, s.url(repository, kind, digest), anyManifest...)
if err != nil {
return err
}
if !held {
return Gone
}
return nil
}
@@ -0,0 +1,134 @@
package registry
import (
"context"
"errors"
"net/http"
"net/http/httptest"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
// Asking the store to let go of what the mesh no longer keeps (novox/hq ADR 0189, issue 108).
//
// A fake store records what it was asked to delete, so what is asserted is the mesh's decision
// and the shape of the request — not the registry's behaviour, which is the registry's to test.
func fakeStore(t *testing.T, answer int) (Store, *[]string) {
t.Helper()
var asked []string
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method == http.MethodHead && strings.Contains(r.URL.Path, "/blobs/") {
// An archive's size, asked so its holder can be named (novox/hq issue 253). A store
// that does not have the thing does not have its blob either.
if answer == http.StatusNotFound {
w.WriteHeader(http.StatusNotFound)
return
}
w.Header().Set("Content-Length", "7")
w.WriteHeader(http.StatusOK)
return
}
if r.Method != http.MethodDelete {
t.Errorf("the store was asked %s %s; collecting is a delete", r.Method, r.URL.Path)
}
asked = append(asked, r.URL.Path)
w.WriteHeader(answer)
}))
t.Cleanup(server.Close)
return Store{Address: strings.TrimPrefix(server.URL, "http://")}, &asked
}
func TestAnImageAndAnArchiveAreAskedForAtTheirOwnEndpoints(t *testing.T) {
// The two shapes the mesh records: a manifest by digest, and a blob by digest. They are
// different endpoints, and asking at the wrong one answers 404 — which this would then
// record as collected, leaving the bytes on disk for ever while the record says otherwise.
store, asked := fakeStore(t, http.StatusAccepted)
ctx := context.Background()
image := catalogue.ArtifactStoreScheme + "web/app@sha256:abc123"
archive := catalogue.ArtifactStoreScheme + "web/config/blobs/sha256:def456"
if err := store.LetGo(ctx, image); err != nil {
t.Fatal(err)
}
if err := store.LetGo(ctx, archive); err != nil {
t.Fatal(err)
}
// The archive's holder goes first, then the archive (novox/hq issue 253).
_, holder := Holder("sha256:def456", 7)
want := []string{
"/v2/web/app/manifests/sha256:abc123",
"/v2/web/config/manifests/" + holder,
"/v2/web/config/blobs/sha256:def456",
}
if strings.Join(*asked, " ") != strings.Join(want, " ") {
t.Fatalf("the store was asked %v; want %v", *asked, want)
}
}
func TestAStoreThatDoesNotHaveItAnswersGone(t *testing.T) {
// The outcome wanted, already true. Told apart from success only so the sweep can say which
// happened; both are recorded, because retrying for ever is the thing to avoid.
store, _ := fakeStore(t, http.StatusNotFound)
err := store.LetGo(context.Background(), catalogue.ArtifactStoreScheme+"web/app@sha256:abc123")
if !errors.Is(err, Gone) {
t.Fatalf("a store that does not hold it answered %v, want Gone", err)
}
}
func TestAStoreWithDeletionOffSaysSoAndNamesTheRemedy(t *testing.T) {
// The registry answers 405 when it was started without deletion enabled. The remedy is a
// setting on the store's module, and saying "405" would send somebody to the wrong place.
store, _ := fakeStore(t, http.StatusMethodNotAllowed)
err := store.LetGo(context.Background(), catalogue.ArtifactStoreScheme+"web/app@sha256:abc123")
if err == nil {
t.Fatal("a store that refuses deletion was read as success")
}
if !strings.Contains(err.Error(), "REGISTRY_STORAGE_DELETE_ENABLED") {
t.Fatalf("the refusal does not name the remedy: %v", err)
}
}
func TestAReferenceThatIsNotTheMeshsOwnIsNeverAsked(t *testing.T) {
// The whole safety of the sweep is that it names only what the mesh recorded putting there.
// A reference of another shape — a vendor's image, a package version — is refused rather
// than composed into a delete somewhere that is not the mesh's store.
store, asked := fakeStore(t, http.StatusAccepted)
for _, reference := range []string{
"docker.io/library/registry@sha256:abc123",
"registry@sha256:abc123",
"1.4.2",
} {
if err := store.LetGo(context.Background(), reference); err == nil {
t.Errorf("%s was asked about; it is not a reference into the mesh's store", reference)
}
}
if len(*asked) != 0 {
t.Fatalf("the store was asked about %v", *asked)
}
}
// A reference this sweep will not address says so as ErrNotOurs, which is a fact about the
// record and not about the store (novox/hq issue 226).
//
// The sweep skips one and abandons itself for the other, so they cannot be the same error. The
// first live run met a reference recorded with the store's old address, read the refusal as "the
// store refuses everything", and collected none of the 1681 it had found.
func TestAReferenceThisSweepWillNotAddressIsToldApartFromAStoreRefusing(t *testing.T) {
store, asked := fakeStore(t, http.StatusAccepted)
for _, reference := range []string{
"docker.io/library/registry@sha256:abc123",
"127.0.0.1:5100/mesh-tools/build@sha256:abc123",
"1.4.2",
} {
err := store.LetGo(context.Background(), reference)
if !errors.Is(err, ErrNotOurs) {
t.Errorf("%s answered %v; a sweep must be able to skip it and go on", reference, err)
}
}
if len(*asked) != 0 {
t.Fatalf("the store was asked about %v", *asked)
}
}
+46 -11
View File
@@ -1,26 +1,52 @@
{
"module": "artifact-store-tools",
"version": "1",
"claims": [
{
"name": "mesh-artifact-sweep",
"scope": "mesh",
"serves": [
"artifacts",
"held",
"collect",
"mirrors",
"images"
]
}
],
"requires": [
"artifact-store"
],
"invokes": [
"seat:mesh-controller.artifacts",
"seat:mesh-controller.collect",
"seat:mesh-hand-acts.hand-act",
"mesh-catalog.catalog_modules",
"mesh-catalog.catalog_module"
],
"reads": [
"mesh-delivery-assigner.wants",
"mesh-delivery-builder.builds",
"mesh-delivery-builder.build-log",
"mesh-delivery-indexer.manifests",
"node-engine.have",
"mesh-controller.lease"
],
"state": [
"findings"
],
"tools": [
"artifact_store_repositories",
"artifact_store_usage",
"artifact_store_references",
"artifact_store_collect"
],
"config": {
"older-store-names": {
"type": "string",
"description": "The names the artifact store had before, as host names or host:port separated by spaces, or none when it had no other. artifact_store_references reads every definition under these as well as under the store's current name, so an image a definition pins under one is said to be named by a definition (novox/hq issue 323).",
"pattern": "none|[A-Za-z0-9.:-]+( [A-Za-z0-9.:-]+)*",
"default": "none"
}
},
"config": {
"older-store-names": {
"type": "string",
"description": "The names the artifact store had before, as host names or host:port separated by spaces, or none when it had no other. artifact_store_references reads every definition under these as well as under the store's current name, so an image a definition pins under one is said to be named by a definition (novox/hq issue 323).",
"pattern": "none|[A-Za-z0-9.:-]+( [A-Za-z0-9.:-]+)*",
"default": "none"
}
},
"resources": [
{
"id": "state",
@@ -36,6 +62,14 @@
"mode": "0600",
"trusted": false,
"template": "files/store-names"
},
{
"id": "store-address",
"type": "file",
"path": "${dir:state}/store-address",
"mode": "0600",
"trusted": false,
"template": "files/store-address"
}
],
"build": {
@@ -52,7 +86,8 @@
],
"env": {
"MESH_ARTIFACT_STORE_CONTAINER": "mesh-registry",
"MESH_ARTIFACT_STORE_NAMES_FILE": "${dir:state}/store-names"
"MESH_ARTIFACT_STORE_NAMES_FILE": "${dir:state}/store-names",
"MESH_ARTIFACT_STORE_ADDRESS_FILE": "${dir:state}/store-address"
}
},
{
+4 -1
View File
@@ -8,7 +8,10 @@
"provides": [
{
"name": "rootless-image-build",
"reach": "machine"
"reach": "machine",
"credential": {
"form": "none"
}
}
],
"resources": [
+2 -2
View File
@@ -45,8 +45,8 @@ whenever the node's tool runtime collects the module's tools:
| file | holds |
|---|---|
| `managed-mcp.json` | the tool servers every session loads: the mesh's console as `mesh`, and the servers set in this module's `mcp_servers` setting. **Exclusive**: a server not listed here does not load — not one added with `claude mcp add`, not a project's `.mcp.json`, not a plugin's |
| `managed-settings.json` | the keys set in this module's `managed_settings` setting — at the controller's terminal alone (hq issue 340) — under the mesh's own keys: the repositories' attribution convention, the claude.ai connectors kept beside the managed servers, the key-helper while the node holds an API-key licence, git's credential file while the node holds the forge's read credential (below), and the two that name the `nox-mesh` marketplace and enable its plugin |
| `managed-mcp.json` | the tool servers every session loads: the mesh's console as `mesh` (with a `timeout` of 660000 ms, mesh_wait's longest wait and a minute, so the agent's one-minute default does not cut a wait, issue 1871), and the servers set in this module's `mcp_servers` setting. **Exclusive**: a server not listed here does not load — not one added with `claude mcp add`, not a project's `.mcp.json`, not a plugin's |
| `managed-settings.json` | the keys set in this module's `managed_settings` setting — at the controller's terminal alone (hq issue 340) — under the mesh's own keys: the repositories' attribution convention, the claude.ai connectors kept beside the managed servers, the key-helper while the node holds an API-key licence, git's credential file while the node holds the forge's read credential (below), and the two that name the `nox-mesh` marketplace and enable its plugin, and `CLAUDE_CODE_MCP_TOOL_IDLE_TIMEOUT` (660000) in `env` beside the operator's own, the mesh's key winning |
| `CLAUDE.md` | how a session on this mesh works, this node's name and role, the conventions, the **"instead of" table** — then the instruction sections registered for every node and for this one |
| `marketplace/` | the `nox-mesh` plugin (hq ADR 0216): the skills, subagents, commands and output styles registered for every node and for this one, offered in a session as `nox-mesh:<name>` — and the **guard on the agent's shell** (`guard/`), the mesh's own hook and the plugin's only one (hq issue 340). Replaced whole, staged beside and swapped in |
@@ -83,6 +83,20 @@ func TestTheRendererWritesWhatTheTypeScriptOneWrote(t *testing.T) {
for k := range MarketplaceKeys() {
delete(m, k)
}
// The wait bound is new since the TypeScript (issue 1871), and is held below.
if env, ok := m["env"].(map[string]any); ok {
delete(env, "CLAUDE_CODE_MCP_TOOL_IDLE_TIMEOUT")
if len(env) == 0 {
delete(m, "env")
}
}
}
if m, ok := a.(map[string]any); ok && file == "managed-mcp.json" {
if servers, ok := m["mcpServers"].(map[string]any); ok {
if entry, ok := servers["mesh"].(map[string]any); ok {
delete(entry, "timeout")
}
}
}
_ = json.Unmarshal([]byte(want[file]), &b)
if !reflect.DeepEqual(a, b) {
@@ -104,6 +118,21 @@ func TestASettingCannotReplaceTheMeshsOwnEntryAndABadNameIsLeftOut(t *testing.T)
if mcp.MCPServers["mesh"]["url"] != "http://127.0.0.1:4270/mcp" || mcp.MCPServers["bad name"] != nil {
t.Fatalf("%v", mcp.MCPServers)
}
// The entry bounds a call by mesh_wait's longest wait and the idle bound is laid over the operator's env
// (issue 1871): a session's wait is not cut at the agent's one-minute default.
if mcp.MCPServers["mesh"]["timeout"] != float64(660000) {
t.Errorf("the mesh entry's timeout is %v", mcp.MCPServers["mesh"]["timeout"])
}
for _, held := range []map[string]any{nil, {"CLAUDE_CODE_MCP_TOOL_IDLE_TIMEOUT": "1", "FOO": "bar"}} {
var m struct {
Env map[string]any `json:"env"`
}
got := Render(Facts{Console: "x"}, Settings{ManagedSettings: map[string]any{"env": held}}, nil, "/h", nil, Config{})
_ = json.Unmarshal([]byte(got["managed-settings.json"]), &m)
if m.Env["CLAUDE_CODE_MCP_TOOL_IDLE_TIMEOUT"] != "660000" || (held != nil && m.Env["FOO"] != "bar") {
t.Errorf("env %v rendered as %v", held, m.Env)
}
}
if !reflect.DeepEqual(Render(Facts{Console: "x"}, Settings{}, nil, "/h", nil, Config{}), Render(Facts{Console: "x"}, Settings{}, nil, "/h", nil, Config{})) {
t.Fatal("rendering is not deterministic")
}
@@ -4,9 +4,9 @@ package main
// the file under the agent's managed directory did not change by a byte when the module moved to Go;
// a test holds it to that renderer's own output (testdata/rendered-by-typescript.json).
func instructionsText(node, role string) string {
return "# This machine is a node of a Novox mesh\n\nWritten by the mesh's `claude-code` module. Edit the module's settings or the catalogue, never this file:\nit is rewritten whenever the module renders.\n\n## Who this node is\n\n- **Node:** `" +
return "# This machine is a node of a Novox mesh\n\nWritten by the mesh's `claude-code` module. Edit the module's settings or the catalog, never this file:\nit is rewritten whenever the module renders.\n\n## Who this node is\n\n- **Node:** `" +
node +
"`\n- **Role:** " +
role +
"\n- The other nodes, their roles and what runs where: ask the controller (`mesh-controller.nodes`,\n `mesh-controller.node`). Nothing here lists them, because a copy drifts.\n\n## How a session on this mesh works\n\nThe console is the only way to the mesh: the MCP server named `mesh`. It offers five tools, and\neverything else is an address you find and call through them:\n\n- `mesh_search` — words in, matching addresses out. `mesh_describe` — one address's arguments.\n- `mesh_call` — call an address. A seat the mesh holds once is `<seat>.<verb>` (the mesh's own verbs\n are `mesh-controller.<verb>`: `status`, `plan`, `node`, `assign`, `push`, `settings`);\n a module on a machine is `<node>/<module>.<tool>`.\n- `mesh_overview` and `mesh_machine` — the mesh's seats and machines, and what one machine runs.\n\n- **Symptom first.** For an error, a failing service or anything unexpected, search the record with the\n literal text before forming a hypothesis: the records module's `records_search`, then\n `records_read`.\n- **Ask the mesh before changing it**, and change it through the controller's verbs or the catalogue.\n- **A licence** through the `anthropic-licence-manager` seat's verbs. Never edit the agent's credentials\n file by hand, never print or ask for a token.\n\n## Hard rules\n\n- A file the mesh manages is changed through the verb or the catalogue that owns it, never on disk. If\n unsure, `mesh-controller.plan` for the node says what the mesh writes there.\n- Never write to a store's database by hand; schema changes are numbered migrations.\n- Never push to a main branch: a branch, a pull request, and a human approval for every merge.\n- The mesh creates no symlinks, and nobody else does either.\n- A package is declared in a module, never installed by hand.\n\n## Conventions\n\n- Commit messages are concise, in the imperative, about why.\n- Test before pushing: nodes update unattended.\n- The playbooks in the record say how research, decisions, designs, issues and hand-offs are done.\n"
"\n- The other nodes, their roles and what runs where: ask the controller (`mesh-controller.nodes`,\n `mesh-controller.node`). Nothing here lists them, because a copy drifts.\n\n## How a session on this mesh works\n\nThe console is the only way to the mesh: the MCP server named `mesh`. It offers five tools, and\neverything else is an address you find and call through them:\n\n- `mesh_search` — words in, matching addresses out. `mesh_describe` — one address's arguments.\n- `mesh_call` — call an address. A seat the mesh holds once is `<seat>.<verb>` (the mesh's own verbs\n are `mesh-controller.<verb>`: `status`, `plan`, `node`, `assign`, `push`, `settings`);\n a module on a machine is `<node>/<module>.<tool>`.\n- `mesh_overview` and `mesh_machine` — the mesh's seats and machines, and what one machine runs.\n\n- **Symptom first.** For an error, a failing service or anything unexpected, search the record with the\n literal text before forming a hypothesis: the records module's `records_search`, then\n `records_read`.\n- **Ask the mesh before changing it**, and change it through the controller's verbs or the catalog.\n- **A licence** through the `anthropic-licence-manager` seat's verbs. Never edit the agent's credentials\n file by hand, never print or ask for a token.\n\n## Hard rules\n\n- A file the mesh manages is changed through the verb or the catalog that owns it, never on disk. If\n unsure, `mesh-controller.plan` for the node says what the mesh writes there.\n- Never write to a store's database by hand; schema changes are numbered migrations.\n- Never push to a main branch: a branch, a pull request, and a human approval for every merge.\n- The mesh creates no symlinks, and nobody else does either.\n- A package is declared in a module, never installed by hand.\n\n## Conventions\n\n- Commit messages are concise, in the imperative, about why.\n- Test before pushing: nodes update unattended.\n- The playbooks in the record say how research, decisions, designs, issues and hand-offs are done.\n"
}
+24 -1
View File
@@ -22,6 +22,7 @@ import (
"encoding/json"
"fmt"
"regexp"
"strconv"
"strings"
)
@@ -98,6 +99,27 @@ func jsonFile(v any) string {
return b.String()
}
// MeshWaitBoundMillis is how long the agent may wait on the mesh's console for one call: mesh_wait's longest
// wait (600 s, mesh-tools node-tools console.WaitLongest) plus a minute's margin. The agent bounds an HTTP
// server's request by the largest of 60 s, the server entry's "timeout" and MCP_TIMEOUT, and a call that
// receives nothing for CLAUDE_CODE_MCP_TOOL_IDLE_TIMEOUT (5 minutes by default) is aborted
// (code.claude.com/docs/en/mcp.md). Without these every mesh_wait ended "The operation timed out." at a
// minute, whatever its timeout (issue 1871).
const MeshWaitBoundMillis = 660000
// meshWaitEnv lays the idle bound into the environment the settings give every session, the mesh's key winning
// over the operator's, like the forge's keys.
func meshWaitEnv(managed map[string]any) {
env := map[string]any{}
if held, ok := managed["env"].(map[string]any); ok {
for k, v := range held {
env[k] = v
}
}
env["CLAUDE_CODE_MCP_TOOL_IDLE_TIMEOUT"] = strconv.Itoa(MeshWaitBoundMillis)
managed["env"] = env
}
// Render composes the three files. registered — what was registered through this module and applies
// here — is laid over the servers the operator set in its settings; config is the rest of what was
// registered (ADR 0216): its settings laid over the operator's `managed_settings`, its instruction sections
@@ -119,7 +141,7 @@ func RenderWithMesh(facts Facts, settings Settings, binding *Binding, helperPath
servers[name] = entry
}
}
servers[meshEntry] = map[string]any{"type": "http", "url": facts.Console}
servers[meshEntry] = map[string]any{"type": "http", "url": facts.Console, "timeout": MeshWaitBoundMillis}
// The operator's `managed_settings` (ADR 0213), set at the controller's terminal alone: nothing registered
// through this module's tools is laid over it (novox/hq issue 340).
@@ -143,6 +165,7 @@ func RenderWithMesh(facts Facts, settings Settings, binding *Binding, helperPath
managed[key] = entries
}
forgeEnv(managed, facts.ForgeCredentials)
meshWaitEnv(managed)
if binding != nil && binding.Kind == "api-key" {
managed["apiKeyHelper"] = helperPath
}
@@ -32,11 +32,11 @@
"withKey": {
"managed-mcp.json": "{\n \"mcpServers\": {\n \"anton\": {\n \"type\": \"stdio\",\n \"command\": \"node\",\n \"args\": [\n \"/a/b.js\"\n ],\n \"env\": {}\n },\n \"docs\": {\n \"type\": \"stdio\",\n \"command\": \"docs-mcp\",\n \"args\": [\n \"--x\"\n ]\n },\n \"mesh\": {\n \"type\": \"http\",\n \"url\": \"http://127.0.0.1:4270/mcp\"\n },\n \"search\": {\n \"type\": \"http\",\n \"url\": \"https://s.example/mcp\"\n }\n }\n}\n",
"managed-settings.json": "{\n \"attribution\": {\n \"commit\": \"\",\n \"pr\": \"\"\n },\n \"allowAllClaudeAiMcps\": true,\n \"apiKeyHelper\": \"/state/api-key-helper\"\n}\n",
"CLAUDE.md": "# This machine is a node of a Novox mesh\n\nWritten by the mesh's `claude-code` module. Edit the module's settings or the catalogue, never this file:\nit is rewritten whenever the module renders.\n\n## Who this node is\n\n- **Node:** `workstation`\n- **Role:** the laptop\n- The other nodes, their roles and what runs where: ask the controller (`mesh-controller.nodes`,\n `mesh-controller.node`). Nothing here lists them, because a copy drifts.\n\n## How a session on this mesh works\n\nThe console is the only way to the mesh: the MCP server named `mesh`. It offers five tools, and\neverything else is an address you find and call through them:\n\n- `mesh_search` — words in, matching addresses out. `mesh_describe` — one address's arguments.\n- `mesh_call` — call an address. A seat the mesh holds once is `<seat>.<verb>` (the mesh's own verbs\n are `mesh-controller.<verb>`: `status`, `plan`, `node`, `assign`, `push`, `settings`);\n a module on a machine is `<node>/<module>.<tool>`.\n- `mesh_overview` and `mesh_machine` — the mesh's seats and machines, and what one machine runs.\n\n- **Symptom first.** For an error, a failing service or anything unexpected, search the record with the\n literal text before forming a hypothesis: the records module's `records_search`, then\n `records_read`.\n- **Ask the mesh before changing it**, and change it through the controller's verbs or the catalogue.\n- **A licence** through the `anthropic-licence-manager` seat's verbs. Never edit the agent's credentials\n file by hand, never print or ask for a token.\n\n## Hard rules\n\n- A file the mesh manages is changed through the verb or the catalogue that owns it, never on disk. If\n unsure, `mesh-controller.plan` for the node says what the mesh writes there.\n- Never write to a store's database by hand; schema changes are numbered migrations.\n- Never push to a main branch: a branch, a pull request, and a human approval for every merge.\n- The mesh creates no symlinks, and nobody else does either.\n- A package is declared in a module, never installed by hand.\n\n## Conventions\n\n- Commit messages are concise, in the imperative, about why.\n- Test before pushing: nodes update unattended.\n- The playbooks in the record say how research, decisions, designs, issues and hand-offs are done.\n"
"CLAUDE.md": "# This machine is a node of a Novox mesh\n\nWritten by the mesh's `claude-code` module. Edit the module's settings or the catalog, never this file:\nit is rewritten whenever the module renders.\n\n## Who this node is\n\n- **Node:** `workstation`\n- **Role:** the laptop\n- The other nodes, their roles and what runs where: ask the controller (`mesh-controller.nodes`,\n `mesh-controller.node`). Nothing here lists them, because a copy drifts.\n\n## How a session on this mesh works\n\nThe console is the only way to the mesh: the MCP server named `mesh`. It offers five tools, and\neverything else is an address you find and call through them:\n\n- `mesh_search` — words in, matching addresses out. `mesh_describe` — one address's arguments.\n- `mesh_call` — call an address. A seat the mesh holds once is `<seat>.<verb>` (the mesh's own verbs\n are `mesh-controller.<verb>`: `status`, `plan`, `node`, `assign`, `push`, `settings`);\n a module on a machine is `<node>/<module>.<tool>`.\n- `mesh_overview` and `mesh_machine` — the mesh's seats and machines, and what one machine runs.\n\n- **Symptom first.** For an error, a failing service or anything unexpected, search the record with the\n literal text before forming a hypothesis: the records module's `records_search`, then\n `records_read`.\n- **Ask the mesh before changing it**, and change it through the controller's verbs or the catalog.\n- **A licence** through the `anthropic-licence-manager` seat's verbs. Never edit the agent's credentials\n file by hand, never print or ask for a token.\n\n## Hard rules\n\n- A file the mesh manages is changed through the verb or the catalog that owns it, never on disk. If\n unsure, `mesh-controller.plan` for the node says what the mesh writes there.\n- Never write to a store's database by hand; schema changes are numbered migrations.\n- Never push to a main branch: a branch, a pull request, and a human approval for every merge.\n- The mesh creates no symlinks, and nobody else does either.\n- A package is declared in a module, never installed by hand.\n\n## Conventions\n\n- Commit messages are concise, in the imperative, about why.\n- Test before pushing: nodes update unattended.\n- The playbooks in the record say how research, decisions, designs, issues and hand-offs are done.\n"
},
"plain": {
"managed-mcp.json": "{\n \"mcpServers\": {\n \"mesh\": {\n \"type\": \"http\",\n \"url\": \"http://127.0.0.1:4270/mcp\"\n }\n }\n}\n",
"managed-settings.json": "{\n \"attribution\": {\n \"commit\": \"\",\n \"pr\": \"\"\n },\n \"allowAllClaudeAiMcps\": true\n}\n",
"CLAUDE.md": "# This machine is a node of a Novox mesh\n\nWritten by the mesh's `claude-code` module. Edit the module's settings or the catalogue, never this file:\nit is rewritten whenever the module renders.\n\n## Who this node is\n\n- **Node:** `workstation`\n- **Role:** not stated — set it in this module's settings for the node\n- The other nodes, their roles and what runs where: ask the controller (`mesh-controller.nodes`,\n `mesh-controller.node`). Nothing here lists them, because a copy drifts.\n\n## How a session on this mesh works\n\nThe console is the only way to the mesh: the MCP server named `mesh`. It offers five tools, and\neverything else is an address you find and call through them:\n\n- `mesh_search` — words in, matching addresses out. `mesh_describe` — one address's arguments.\n- `mesh_call` — call an address. A seat the mesh holds once is `<seat>.<verb>` (the mesh's own verbs\n are `mesh-controller.<verb>`: `status`, `plan`, `node`, `assign`, `push`, `settings`);\n a module on a machine is `<node>/<module>.<tool>`.\n- `mesh_overview` and `mesh_machine` — the mesh's seats and machines, and what one machine runs.\n\n- **Symptom first.** For an error, a failing service or anything unexpected, search the record with the\n literal text before forming a hypothesis: the records module's `records_search`, then\n `records_read`.\n- **Ask the mesh before changing it**, and change it through the controller's verbs or the catalogue.\n- **A licence** through the `anthropic-licence-manager` seat's verbs. Never edit the agent's credentials\n file by hand, never print or ask for a token.\n\n## Hard rules\n\n- A file the mesh manages is changed through the verb or the catalogue that owns it, never on disk. If\n unsure, `mesh-controller.plan` for the node says what the mesh writes there.\n- Never write to a store's database by hand; schema changes are numbered migrations.\n- Never push to a main branch: a branch, a pull request, and a human approval for every merge.\n- The mesh creates no symlinks, and nobody else does either.\n- A package is declared in a module, never installed by hand.\n\n## Conventions\n\n- Commit messages are concise, in the imperative, about why.\n- Test before pushing: nodes update unattended.\n- The playbooks in the record say how research, decisions, designs, issues and hand-offs are done.\n"
"CLAUDE.md": "# This machine is a node of a Novox mesh\n\nWritten by the mesh's `claude-code` module. Edit the module's settings or the catalog, never this file:\nit is rewritten whenever the module renders.\n\n## Who this node is\n\n- **Node:** `workstation`\n- **Role:** not stated — set it in this module's settings for the node\n- The other nodes, their roles and what runs where: ask the controller (`mesh-controller.nodes`,\n `mesh-controller.node`). Nothing here lists them, because a copy drifts.\n\n## How a session on this mesh works\n\nThe console is the only way to the mesh: the MCP server named `mesh`. It offers five tools, and\neverything else is an address you find and call through them:\n\n- `mesh_search` — words in, matching addresses out. `mesh_describe` — one address's arguments.\n- `mesh_call` — call an address. A seat the mesh holds once is `<seat>.<verb>` (the mesh's own verbs\n are `mesh-controller.<verb>`: `status`, `plan`, `node`, `assign`, `push`, `settings`);\n a module on a machine is `<node>/<module>.<tool>`.\n- `mesh_overview` and `mesh_machine` — the mesh's seats and machines, and what one machine runs.\n\n- **Symptom first.** For an error, a failing service or anything unexpected, search the record with the\n literal text before forming a hypothesis: the records module's `records_search`, then\n `records_read`.\n- **Ask the mesh before changing it**, and change it through the controller's verbs or the catalog.\n- **A licence** through the `anthropic-licence-manager` seat's verbs. Never edit the agent's credentials\n file by hand, never print or ask for a token.\n\n## Hard rules\n\n- A file the mesh manages is changed through the verb or the catalog that owns it, never on disk. If\n unsure, `mesh-controller.plan` for the node says what the mesh writes there.\n- Never write to a store's database by hand; schema changes are numbered migrations.\n- Never push to a main branch: a branch, a pull request, and a human approval for every merge.\n- The mesh creates no symlinks, and nobody else does either.\n- A package is declared in a module, never installed by hand.\n\n## Conventions\n\n- Commit messages are concise, in the imperative, about why.\n- Test before pushing: nodes update unattended.\n- The playbooks in the record say how research, decisions, designs, issues and hand-offs are done.\n"
}
}
+1
View File
@@ -0,0 +1 @@
disk-load
+248
View File
@@ -0,0 +1,248 @@
package main
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/health"
"os"
"os/exec"
"sort"
"strings"
"sync"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// Store is the state `readings` as the keeper uses it.
type Store interface {
Get(key string) (json.RawMessage, error)
Put(key string, value any) error
Keys() ([]string, error)
}
// busStore is the state through the runtime.
type busStore struct{}
func (busStore) Get(key string) (json.RawMessage, error) {
e, err := stdio.State(StateReadings).Get(key)
if err != nil || e == nil {
return nil, err
}
return e.Value, nil
}
func (busStore) Put(key string, v any) error {
_, err := stdio.State(StateReadings).Put(key, v)
return err
}
func (busStore) Keys() ([]string, error) { return stdio.State(StateReadings).Keys() }
// Keeper measures the items of its node and keeps the readings.
type Keeper struct {
Node string
// Items is every data item the modules placed on this node declare, composed from the bus (frombus.DataItems).
Items func(ctx context.Context) (map[string][]catalogue.PlacedDataItem, error)
Measurer Measurer
Store Store
Say func(string, ...any)
mu sync.Mutex // one round at a time
}
// Round measures every item that is not a cache, one at a time, and writes the readings that changed. An item
// measured by a walk is walked only when walks is true or its last walk is older than WalkEvery: the hourly round
// reads counters, and a walk runs at most once a day (ADR 0233 §22).
func (k *Keeper) Round(ctx context.Context, walks bool) (written, kept int, err error) {
k.mu.Lock()
defer k.mu.Unlock()
items, err := k.Items(ctx)
if err != nil {
return 0, 0, err
}
modules := make([]string, 0, len(items))
for m := range items {
modules = append(modules, m)
}
sort.Strings(modules)
var failed []string
for _, module := range modules {
for _, it := range items[module] {
if it.Class == ClassCache {
continue
}
w, err := k.measureOne(ctx, it, walks)
if err != nil {
failed = append(failed, fmt.Sprintf("%s.%s: %v", module, it.ID, err))
continue
}
if w {
written++
} else {
kept++
}
}
}
retired, err := k.retireGone(items)
written += retired
if err != nil {
failed = append(failed, err.Error())
}
if len(failed) > 0 {
return written, kept, errors.New("readings not kept: " + strings.Join(failed, "; "))
}
return written, kept, nil
}
// retireGone marks retired every reading of this node whose item the node no longer declares — its module was unassigned
// there, or is no longer pulled in — with when and why, and keeps it: the record of what the module left behind, until a
// person deletes it (ADR 0233). It is cleared when the item is declared again (measureOne).
func (k *Keeper) retireGone(declared map[string][]catalogue.PlacedDataItem) (int, error) {
keys, err := k.Store.Keys()
if err != nil {
return 0, err
}
declaredNow := map[string]bool{}
for module, its := range declared {
for _, it := range its {
declaredNow[health.ReadingKey(k.Node, module, it.ID)] = true
}
}
now := k.Measurer.Now().UTC()
n := 0
for _, key := range keys {
if !strings.HasPrefix(key, k.Node+".") || declaredNow[key] {
continue
}
raw, err := k.Store.Get(key)
if err != nil || len(raw) == 0 {
continue
}
r, err := health.DecodeReading(raw)
if err != nil || r.Node != k.Node || !r.RetiredAt.IsZero() {
continue
}
r.RetiredAt = now
r.RetiredWhy = fmt.Sprintf("no longer declared on %s: its module was unassigned there, or is no longer pulled in", k.Node)
if err := k.Store.Put(key, r); err != nil {
return n, err
}
n++
}
return n, nil
}
func (k *Keeper) measureOne(ctx context.Context, it catalogue.PlacedDataItem, walks bool) (bool, error) {
key := health.ReadingKey(k.Node, it.Module, it.ID)
var prev *health.Reading
raw, err := k.Store.Get(key)
if err != nil {
return false, err
}
if len(raw) > 0 {
if p, err := health.DecodeReading(raw); err == nil {
prev = &p
}
}
now := k.Measurer.Now()
if prev != nil && prev.RetiredAt.IsZero() && it.Measure == catalogue.MeasureWalk && !walks && prev.MeasuredAt != nil && now.Sub(*prev.MeasuredAt) < WalkEvery {
return false, nil
}
m := k.Measurer.Measure(ctx, it)
next, changed := health.NextReading(prev, k.Node, it, m, now)
if prev != nil && !prev.RetiredAt.IsZero() {
// Declared again: it is no longer retired.
next.RetiredAt, next.RetiredWhy, changed = time.Time{}, "", true
}
if !changed {
return false, nil
}
return true, k.Store.Put(key, next)
}
// MeasureNow measures one item now, walk or not, and writes its reading when it changed.
func (k *Keeper) MeasureNow(ctx context.Context, module, item string) (health.Reading, bool, error) {
k.mu.Lock()
defer k.mu.Unlock()
items, err := k.Items(ctx)
if err != nil {
return health.Reading{}, false, err
}
for _, it := range items[module] {
if it.ID != item {
continue
}
if it.Class == ClassCache {
return health.Reading{}, false, fmt.Errorf("%s.%s is a cache, which is listed and never measured", module, item)
}
w, err := k.measureOne(ctx, it, true)
if err != nil {
return health.Reading{}, false, err
}
raw, err := k.Store.Get(health.ReadingKey(k.Node, module, item))
if err != nil {
return health.Reading{}, w, err
}
r, err := health.DecodeReading(raw)
return r, w, err
}
return health.Reading{}, false, fmt.Errorf("%s declares no data item %q on this machine", module, item)
}
// Loop runs a round after a few minutes and then every `every`, with the walks of the night's round when it is the
// hour after the night; ctx ends it.
func (k *Keeper) Loop(ctx context.Context, every time.Duration, first time.Duration) {
select {
case <-ctx.Done():
return
case <-time.After(first):
}
for {
w, kept, err := k.Round(ctx, false)
if err != nil {
k.Say("measuring the data declared here: %v", err)
} else {
k.Say("measured the data declared here: %d readings changed, %d unchanged", w, kept)
}
select {
case <-ctx.Done():
return
case <-time.After(every):
}
}
}
// RootRunner runs a command as root, through sudo without a prompt when this process is not root. A walk is bounded
// by WalkFor itself; the runner's own bound is a little longer.
func RootRunner(ctx context.Context, name string, args ...string) (string, error) {
ctx, cancel := context.WithTimeout(ctx, WalkFor+time.Minute)
defer cancel()
program, argv := name, args
if os.Getuid() != 0 {
program, argv = "sudo", append([]string{"-n", name}, args...)
}
var stdout, stderr bytes.Buffer
cmd := exec.CommandContext(ctx, program, argv...)
cmd.Stdout, cmd.Stderr = &stdout, &stderr
if err := cmd.Run(); err != nil {
said := strings.TrimSpace(stderr.String())
if said == "" {
said = strings.TrimSpace(stdout.String())
}
if program == "sudo" && strings.HasPrefix(said, "sudo:") {
return "", fmt.Errorf("%s needs root and the runtime's account may not run it without a prompt: %s", name, said)
}
lines := strings.Split(said, "\n")
if len(lines) > 3 {
lines = lines[len(lines)-3:]
}
if said == "" {
return "", fmt.Errorf("%s: %w", name, err)
}
return "", errors.New(strings.Join(lines, " / "))
}
return stdout.String(), nil
}
+43 -2
View File
@@ -1,20 +1,61 @@
// The disk-load module's Go bundle (novox/hq ADR 0188, ADR 0193, issue 312): a process the node's
// runtime launches and speaks MCP over stdio to. It reads the machine's storage load from /proc and
// /sys — iowait, pressure stall, each disk's busy share, throughput, IOPS, wait and queue depth, the
// filesystems' space and the processes doing the I/O — and changes nothing.
// filesystems' space and the processes doing the I/O — and changes nothing of the machine.
//
// It also measures the data items the modules of its node declare (ADR 0233) and keeps each reading and the item's
// seven-day peak as its own state `readings` (ADR 0571 clause 3), composed from the bus (no file).
package main
import (
"context"
"fmt"
"os"
"strings"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/announce"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose/frombus"
"git.novox.be/novox/mesh-sdk/go/compose/fromruntime"
)
func logf(format string, args ...any) { fmt.Fprintf(os.Stderr, "[disk-load] "+format+"\n", args...) }
func main() {
if err := stdio.Keeps(announce.State{Name: StateReadings}); err != nil {
logf("%v", err)
os.Exit(1)
}
node := os.Getenv("MESH_NODE")
k := &Keeper{Node: node, Store: busStore{}, Say: logf, Measurer: Measurer{Run: RootRunner, Now: time.Now},
Items: func(ctx context.Context) (map[string][]catalogue.PlacedDataItem, error) {
return placedItems(ctx, node)
}}
if node != "" {
every := time.Hour
if d, err := time.ParseDuration(os.Getenv("MESH_DISK_MEASURE_EVERY")); err == nil && d >= time.Minute {
every = d
}
go k.Loop(context.Background(), every, 3*time.Minute)
}
tools := append(Tools(Here()), ReadingTools(k)...)
// An empty name serves as the module the runtime names (MESH_SERVED_MODULE): disk-load.
if err := stdio.Serve("", Tools(Here())); err != nil {
if err := stdio.Serve("", tools); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
// placedItems is every data item the modules placed on node declare, composed from the bus as the node composes itself.
func placedItems(ctx context.Context, node string) (map[string][]catalogue.PlacedDataItem, error) {
items, unplaced, err := frombus.DataItems(ctx, fromruntime.Reader(), frombus.Mesh{}, frombus.Scope{Node: node})
if err != nil {
return nil, err
}
if len(unplaced) > 0 {
logf("data items left out as not placeable: %s", strings.Join(unplaced, "; "))
}
return items, nil
}
+215
View File
@@ -0,0 +1,215 @@
package main
// How an item is measured (novox/hq ADR 0233 §22 to §25), moved here from the backup holder, which only reads the
// readings now (ADR 0571 clause 3). Never an unbounded walk of something large.
import (
"context"
"fmt"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/health"
"regexp"
"strconv"
"strings"
"time"
)
// The bounds on a walk: a walk is for small items, and one that meets a large one stops and says so.
var (
WalkFor = 10 * time.Minute
WalkFiles = 2_000_000
WalkEvery = 23 * time.Hour
)
// walkCommand sums the files under a path and finds its newest change, in one walk, bounded by time and by files.
// One line out: "<bytes> <epoch seconds> <files>", then the status of find.
func walkCommand(path string) string {
return fmt.Sprintf("timeout %d find '%s' -xdev -printf '%%y %%s %%T@\\n' 2>/dev/null | head -n %d | "+
"awk 'BEGIN{s=0;m=0;n=0} {n++; if ($1==\"f\") s+=$2; if ($3>m) m=$3} END {printf \"%%d %%.0f %%d\\n\", s, m, n}'; "+
"echo \"${PIPESTATUS[0]:-0}\"", int(WalkFor.Seconds()), path, WalkFiles)
}
// topCommand is the newest change among a path and its top-level entries, and how many there are.
func topCommand(path string) string {
return "timeout 60 find '" + path + "' -maxdepth 1 -printf '%T@\\n' 2>/dev/null | " +
"awk 'BEGIN{m=0;n=0} {n++; if ($1>m) m=$1} END {printf \"%.0f %d\\n\", m, n-1}'"
}
func epochTime(s string) *time.Time {
epoch, err := strconv.ParseInt(s, 10, 64)
if err != nil || epoch <= 0 {
return nil
}
t := time.Unix(epoch, 0).UTC()
return &t
}
// Measurer measures items through a runner that has the root the walks need.
type Measurer struct {
Run Runner
Now func() time.Time
}
func (b Measurer) exists(ctx context.Context, path string) bool {
_, err := b.Run(ctx, "test", "-e", path)
return err == nil
}
// Measure is one item, measured now, by its own method.
func (b Measurer) Measure(ctx context.Context, it catalogue.PlacedDataItem) health.Measured {
at := b.Now().UTC()
m := health.Measured{MeasuredAt: &at}
if !b.exists(ctx, it.Path) {
m.Error = it.Path + " does not exist"
zero := int64(0)
m.SizeBytes, m.Precision = &zero, "exact"
return m
}
m.Redundancy = b.redundancyOf(ctx, it.Path)
switch it.Measure {
case catalogue.MeasureDataset, catalogue.MeasureShallow:
out, err := b.Run(ctx, "bash", "-c", topCommand(it.Path))
if f := strings.Fields(out); err == nil && len(f) == 2 {
m.LastWrite = epochTime(f[0])
} else if err != nil {
m.Error = err.Error()
}
m.Precision = "no size: the newest change among its top-level entries only"
if it.Measure == catalogue.MeasureShallow {
return m
}
out, err = b.Run(ctx, "zfs", "list", "-Hp", "-o", "name,used", it.Path)
f := strings.Fields(out)
if err != nil || len(f) != 2 {
m.Precision = "no size: it is on no ZFS dataset to read one from; the newest change among its top-level entries only"
return m
}
used, err := strconv.ParseInt(f[1], 10, 64)
if err != nil {
return m
}
m.SizeBytes = &used
m.Precision = "dataset " + f[0] + ": its whole size, from the filesystem's counters; the newest change among the top-level entries"
return m
}
out, err := b.Run(ctx, "bash", "-c", walkCommand(it.Path))
if err != nil {
m.Error = err.Error()
return m
}
lines := strings.Split(strings.TrimSpace(out), "\n")
f := strings.Fields(lines[0])
if len(f) != 3 || len(lines) < 2 {
m.Error = "the measurement said " + strings.TrimSpace(out)
return m
}
size, err1 := strconv.ParseInt(f[0], 10, 64)
files, err2 := strconv.Atoi(f[2])
if err1 != nil || err2 != nil {
m.Error = "the measurement said " + strings.TrimSpace(out)
return m
}
m.SizeBytes, m.LastWrite = &size, epochTime(f[1])
m.Precision = "exact"
if status := strings.TrimSpace(lines[len(lines)-1]); status == "124" || files >= WalkFiles {
m.Precision = fmt.Sprintf("partial: measured partially — stopped after %s or %d files; the size is a lower bound, "+
"and this item wants measure: dataset or shallow", WalkFor, WalkFiles)
}
return m
}
// redundancyOf is the redundant storage a path is on: a ZFS pool's health, an md array's members, a btrfs
// filesystem's error counters. Nil for storage with no redundancy it knows how to read.
func (b Measurer) redundancyOf(ctx context.Context, path string) *health.Redundancy {
out, err := b.Run(ctx, "findmnt", "-no", "SOURCE,FSTYPE", "--target", path)
if err != nil {
return nil
}
f := strings.Fields(out)
if len(f) < 2 {
return nil
}
source, fstype := f[0], f[1]
yes, no := true, false
switch {
case fstype == "zfs":
pool, _, _ := strings.Cut(source, "/")
r := &health.Redundancy{Kind: "zfs", Where: pool}
health, err := b.Run(ctx, "zpool", "list", "-H", "-o", "health", pool)
if err != nil {
r.Said = "zpool list: " + err.Error()
return r
}
status, err := b.Run(ctx, "zpool", "status", "-x", pool)
if err != nil {
r.Said = "zpool status: " + err.Error()
return r
}
health, status = strings.TrimSpace(health), strings.TrimSpace(status)
r.Said = "health " + health + "; " + firstLineOf(status)
if health == "ONLINE" && strings.Contains(status, "is healthy") {
r.Healthy = &yes
} else {
r.Healthy = &no
r.Said = "health " + health + "; " + oneLineOf(status)
}
return r
case strings.HasPrefix(source, "/dev/md"):
device := strings.TrimPrefix(source, "/dev/")
r := &health.Redundancy{Kind: "md", Where: device}
mdstat, err := b.Run(ctx, "cat", "/proc/mdstat")
if err != nil {
r.Said = err.Error()
return r
}
healthy, said, found := mdHealth(mdstat, device)
if !found {
r.Said = device + " is not in /proc/mdstat"
return r
}
r.Said = said
if healthy {
r.Healthy = &yes
} else {
r.Healthy = &no
}
return r
case fstype == "btrfs":
r := &health.Redundancy{Kind: "btrfs", Where: source}
stats, err := b.Run(ctx, "btrfs", "device", "stats", "--check", path)
if err != nil {
r.Healthy, r.Said = &no, "device errors: "+oneLineOf(err.Error())
return r
}
r.Healthy, r.Said = &yes, firstLineOf(stats)
return r
}
return nil
}
var mdMembers = regexp.MustCompile(`\[([U_]+)\]`)
// mdHealth reads one array's block of /proc/mdstat: healthy when every member is up and it is not recovering.
func mdHealth(mdstat, device string) (bool, string, bool) {
lines := strings.Split(mdstat, "\n")
for i, l := range lines {
if !strings.HasPrefix(l, device+" ") {
continue
}
block := l
for j := i + 1; j < len(lines) && strings.HasPrefix(lines[j], " "); j++ {
block += " " + strings.TrimSpace(lines[j])
}
members := mdMembers.FindStringSubmatch(block)
healthy := members != nil && !strings.Contains(members[1], "_") && !strings.Contains(block, "recovery")
return healthy, oneLineOf(block), true
}
return false, "", false
}
func firstLineOf(s string) string {
line, _, _ := strings.Cut(strings.TrimSpace(s), "\n")
return line
}
func oneLineOf(s string) string { return strings.Join(strings.Fields(s), " ") }
@@ -0,0 +1,12 @@
package main
// The readings layout, their key and the peak are mesh-sdk's (health.Reading, health.NextReading; novox/hq ADR 0571
// clause 3): disk-load measures each placed data item of its node and keeps the readings as its own state.
import "git.novox.be/novox/mesh-sdk/go/health"
// StateReadings is the state disk-load keeps.
const StateReadings = health.ReadingsState
// ClassCache is the class of an item that is listed and never measured.
const ClassCache = "cache"
@@ -0,0 +1,144 @@
package main
import (
"context"
"encoding/json"
"os"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/health"
)
type memStore struct{ m map[string]json.RawMessage }
func (s *memStore) Get(k string) (json.RawMessage, error) { return s.m[k], nil }
func (s *memStore) Put(k string, v any) error {
raw, _ := json.Marshal(v)
s.m[k] = raw
return nil
}
func (s *memStore) Keys() ([]string, error) {
var out []string
for k := range s.m {
out = append(out, k)
}
return out, nil
}
// A round measures what the data file declares but caches, keeps the readings, writes nothing the second time, and a
// walk is not repeated within a day unless asked.
func TestARoundMeasuresTheDeclaredItemsAndWritesWhatChanged(t *testing.T) {
text := "# pg\nitem data valuable /srv/pg /srv/pg backup walk\nitem tmp cache /srv/tmp - none\n# bad-line-module\nitem logs rebuildable /srv/logs - none shallow\n"
placed, err := catalogue.ParseDataItems(text)
if err != nil {
t.Fatal(err)
}
walks := 0
run := func(_ context.Context, name string, args ...string) (string, error) {
switch {
case name == "test":
return "", nil
case name == "findmnt":
return "", os.ErrNotExist
case name == "bash" && strings.Contains(args[1], "-maxdepth 1"):
return "1760000000 2\n", nil
case name == "bash":
walks++
return "4096 1760000000 3\n0\n", nil
}
return "", os.ErrNotExist
}
now := time.Date(2026, 10, 29, 12, 0, 0, 0, time.UTC)
store := &memStore{m: map[string]json.RawMessage{}}
k := &Keeper{Node: "ace", Items: func(context.Context) (map[string][]catalogue.PlacedDataItem, error) { return placed, nil }, Store: store, Say: func(string, ...any) {},
Measurer: Measurer{Run: run, Now: func() time.Time { return now }}}
w, kept, err := k.Round(context.Background(), false)
if err != nil || w != 2 || kept != 0 || len(store.m) != 2 {
t.Fatalf("first round: %d written %d kept %v, keys %d", w, kept, err, len(store.m))
}
now = now.Add(time.Hour)
if w, kept, _ := k.Round(context.Background(), false); w != 0 || kept != 2 || walks != 1 {
t.Fatalf("second round: %d written, %d kept, %d walks; wanted none written and the walk not repeated", w, kept, walks)
}
if _, _, err := k.MeasureNow(context.Background(), "pg", "data"); err != nil || walks != 2 {
t.Fatalf("measure now: %v, %d walks", err, walks)
}
// An item no longer declared is kept, retired with when and why; declared again, it is not retired any more.
delete(placed, "pg")
if _, _, err := k.Round(context.Background(), false); err != nil {
t.Fatal(err)
}
gone, _ := health.DecodeReading(store.m["ace.pg.data"])
if gone.RetiredAt.IsZero() || !strings.Contains(gone.RetiredWhy, "no longer declared on ace") || gone.FirstSeen.IsZero() {
t.Fatalf("a reading of an item no longer declared: %+v", gone)
}
placed["pg"] = []catalogue.PlacedDataItem{{Module: "pg", ID: "data", Class: "valuable", Path: "/srv/pg", Measure: catalogue.MeasureWalk}}
if _, _, err := k.Round(context.Background(), false); err != nil {
t.Fatal(err)
}
if back, _ := health.DecodeReading(store.m["ace.pg.data"]); !back.RetiredAt.IsZero() {
t.Fatalf("a reading declared again is still retired: %+v", back)
}
if _, _, err := k.MeasureNow(context.Background(), "pg", "tmp"); err == nil {
t.Fatal("a cache was measured")
}
}
// The redundant storage under an item is read: a ZFS pool healthy, then degraded; an md array with a member missing; and
// plain storage, which is no redundancy at all. A dataset is measured from the filesystem's counters and a shallow item
// from its top-level entries only; a walk that meets more than its bound stops and says so.
func TestTheStorageUnderAnItemAndTheLargeMethodsAreRead(t *testing.T) {
poolHealth, statusX, fs := "ONLINE\n", "pool 'storage' is healthy\n", "storage/media zfs\n"
run := func(_ context.Context, name string, args ...string) (string, error) {
line := name + " " + strings.Join(args, " ")
switch {
case name == "test":
return "", nil
case name == "findmnt":
return fs, nil
case name == "zpool" && args[0] == "list":
return poolHealth, nil
case name == "zpool":
return statusX, nil
case name == "cat":
return "Personalities : [raid1]\nmd127 : active raid1 sdb1[1] sda1[0]\n 976630464 blocks super 1.2 [2/1] [U_]\n\nunused devices: <none>\n", nil
case name == "zfs" && args[len(args)-1] == "/storage/media/movies":
return "storage/media\t97067690722560\n", nil
case name == "zfs":
return "", os.ErrNotExist
case name == "bash" && strings.Contains(line, "-maxdepth 1"):
return "1759744800 12673\n", nil
case name == "bash":
return "123 1759700000 2000000\n141\n", nil
}
return "", nil
}
m := Measurer{Run: run, Now: time.Now}
ctx := context.Background()
got := m.Measure(ctx, catalogue.PlacedDataItem{Module: "media", ID: "movies", Path: "/storage/media/movies", Measure: catalogue.MeasureDataset})
if got.SizeBytes == nil || *got.SizeBytes != 97067690722560 || !strings.HasPrefix(got.Precision, "dataset storage/media") ||
got.LastWrite == nil || got.Redundancy == nil || got.Redundancy.Kind != "zfs" || got.Redundancy.Healthy == nil || !*got.Redundancy.Healthy {
t.Fatalf("the library from its dataset: %+v", got)
}
poolHealth, statusX = "DEGRADED\n", " pool: storage\n state: DEGRADED\nstatus: One or more devices has been removed\n"
if r := m.redundancyOf(ctx, "/storage/media/movies"); r.Healthy == nil || *r.Healthy || !strings.Contains(r.Said, "DEGRADED") {
t.Fatalf("a degraded pool: %+v", r)
}
fs = "/dev/md127 ext4\n"
if r := m.redundancyOf(ctx, "/srv/x"); r == nil || r.Kind != "md" || r.Healthy == nil || *r.Healthy {
t.Fatalf("an array missing a member: %+v", r)
}
fs = "/dev/nvme0n1p3 ext4\n"
if r := m.redundancyOf(ctx, "/var/lib/x"); r != nil {
t.Fatalf("plain storage read as redundant: %+v", r)
}
if s := m.Measure(ctx, catalogue.PlacedDataItem{Path: "/mnt/plex/config", Measure: catalogue.MeasureShallow}); s.SizeBytes != nil || !strings.HasPrefix(s.Precision, "no size") {
t.Fatalf("a shallow item: %+v", s)
}
if w := m.Measure(ctx, catalogue.PlacedDataItem{Path: "/srv/big", Measure: catalogue.MeasureWalk}); !strings.HasPrefix(w.Precision, "partial") {
t.Fatalf("a walk stopped at its bound: %+v", w)
}
}
@@ -0,0 +1,93 @@
package main
import (
"context"
"fmt"
"sort"
"strings"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/health"
)
// ReadingTools are the tools over the state `readings`: every key, one key, how full the state is, and one item
// measured now (a module's tools are its command line).
func ReadingTools(k *Keeper) []stdio.Tool {
return []stdio.Tool{
{
Name: "disk_readings",
Description: "The data readings disk-load keeps for this machine's data items (ADR 0233, ADR 0571 clause 3): every key " +
"`<node>.<module>.<item>` with its size, newest write, measured-at, precision and seven-day peak, or one key with " +
"all it holds when `key` is given; and how full the state is: keys kept, their bytes, and the cap the bus puts on " +
"a state (none declared). Filter by `module`. (r)",
Input: map[string]any{
"key": map[string]any{"type": "string", "description": "one key, <node>.<module>.<item>, with everything it holds"},
"module": map[string]any{"type": "string", "description": "only this module's items"},
},
Run: func(args map[string]any) (any, error) {
if key := text(args, "key"); key != "" {
raw, err := k.Store.Get(key)
if err != nil {
return nil, err
}
if len(raw) == 0 {
return nil, fmt.Errorf("no reading is kept at %q", key)
}
r, err := health.DecodeReading(raw)
return r, err
}
keys, err := k.Store.Keys()
if err != nil {
return nil, err
}
sort.Strings(keys)
module := text(args, "module")
type line struct {
Key string `json:"key"`
SizeBytes *int64 `json:"size_bytes,omitempty"`
PeakBytes *int64 `json:"peak_bytes,omitempty"`
Precision string `json:"precision,omitempty"`
Error string `json:"error,omitempty"`
Reading health.Reading `json:"-"`
}
var lines []line
total := 0
for _, key := range keys {
raw, err := k.Store.Get(key)
if err != nil {
return nil, err
}
total += len(raw)
r, err := health.DecodeReading(raw)
if err != nil || (module != "" && r.Module != module) {
continue
}
lines = append(lines, line{Key: key, SizeBytes: r.SizeBytes, PeakBytes: r.Peak, Precision: r.Precision, Error: r.Error})
}
return map[string]any{"readings": lines, "fill": map[string]any{"keys": len(keys), "bytes": total,
"cap": "none declared: the bus's own limit on a state applies"}}, nil
},
},
{
Name: "disk_measure",
Description: "Measure one data item of this machine now, whatever its method (a walk too), and write its reading when it " +
"changed. Reads the disks as root: a walk is bounded by time and files and says when it was partial.",
Input: map[string]any{
"module": map[string]any{"type": "string", "description": "the module that declares the item"},
"item": map[string]any{"type": "string", "description": "the item's id"},
},
Run: func(args map[string]any) (any, error) {
module, item := text(args, "module"), text(args, "item")
if module == "" || item == "" {
return nil, fmt.Errorf("module and item are required")
}
r, written, err := k.MeasureNow(context.Background(), module, item)
if err != nil {
return nil, err
}
return map[string]any{"written": written, "reading": r, "key": health.ReadingKey(k.Node, module, item),
"note": strings.TrimSpace(map[bool]string{true: "the reading changed and was written", false: "the reading had not changed"}[written])}, nil
},
},
}
}
@@ -3,6 +3,7 @@ package main
import (
"encoding/json"
"os"
"regexp"
"sort"
"strings"
"testing"
@@ -35,9 +36,11 @@ func TestTheManifestNamesExactlyTheToolsTheBundleServesAndNoMachine(t *testing.T
t.Fatal(err)
}
var served []string
for _, tl := range Tools(newFake(t).machine()) {
k := &Keeper{Store: &memStore{m: map[string]json.RawMessage{}}}
for _, tl := range append(Tools(newFake(t).machine()), ReadingTools(k)...) {
served = append(served, tl.Name)
if !strings.HasSuffix(tl.Description, "(r)") {
// disk_measure writes the module's own state of readings and nothing else of the machine.
if !strings.HasSuffix(tl.Description, "(r)") && tl.Name != "disk_measure" {
t.Errorf("%s only reads and does not say so", tl.Name)
}
}
@@ -50,7 +53,7 @@ func TestTheManifestNamesExactlyTheToolsTheBundleServesAndNoMachine(t *testing.T
t.Errorf("the module only reads, and declares %d resources", len(m.Resources))
}
for _, banned := range []string{"g14", "novox", "ace", "shanks", "jochen", "/home/"} {
if strings.Contains(string(raw), banned) {
if regexp.MustCompile(`(^|[^a-z0-9-])` + regexp.QuoteMeta(banned) + `($|[^a-z0-9-])`).Match(raw) {
t.Errorf("the manifest says %q", banned)
}
}
+2 -2
View File
@@ -1,5 +1,5 @@
module diskload
go 1.22
go 1.26.0
require git.novox.be/novox/mesh-sdk/go v0.1.6
require git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029231453-6e4a3014cbde
+14 -2
View File
@@ -1,2 +1,14 @@
git.novox.be/novox/mesh-sdk/go v0.1.6 h1:9qzdYONYbJdWcu6sxQcq9v1LI0JxcfkiKYkMUzJSkVQ=
git.novox.be/novox/mesh-sdk/go v0.1.6/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029231453-6e4a3014cbde h1:ddf/jKI9c87sQKDmZLoG6PMnvXwsNH16ByVveyuFME4=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029231453-6e4a3014cbde/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/klauspost/compress v1.20.0 h1:a3C1ke2ohxFymNlb2HWAHjDeKCI90scRskErZkR0ezA=
github.com/klauspost/compress v1.20.0/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/nats-io/nats.go v1.54.0 h1:vsXoOxjHp/GmPUN+EcI7uOf/uB+iAP+kEsAFNQN0yzA=
github.com/nats-io/nats.go v1.54.0/go.mod h1:y+DZoD1oBOYfZTU681eTUiUjI0vbqYGixNVFHcjHJ0k=
github.com/nats-io/nkeys v0.4.16 h1:rd5oAuLOb8mnAycB0xleuEBNS1pVVnN0fv/FF34Eypg=
github.com/nats-io/nkeys v0.4.16/go.mod h1:llLgWoI0o4z/Q57q2R1kHfmocyhGV6VG/U18Glg1Afs=
github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw=
github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
+15 -2
View File
@@ -7,7 +7,9 @@
"disk_filesystems",
"disk_folders",
"disk_devices",
"cpu_top"
"cpu_top",
"disk_readings",
"disk_measure"
],
"build": {
"artifacts": [
@@ -23,5 +25,16 @@
]
}
]
}
},
"state": [
"readings"
],
"reads": [
"mesh-config.resolved",
"mesh-delivery-assigner.placement",
"mesh-delivery-builder.builds",
"mesh-delivery-indexer.manifests",
"mesh-delivery-indexer.modules",
"node-engine.have"
]
}
+4 -1
View File
@@ -4,7 +4,10 @@
"provides": [
{
"name": "artifact-store",
"scope": "mesh"
"scope": "mesh",
"credential": {
"form": "none"
}
}
],
"claims": [
@@ -4,6 +4,9 @@ import (
"encoding/json"
"os"
"path/filepath"
"regexp"
"strconv"
"strings"
"testing"
)
@@ -62,3 +65,60 @@ func TestEveryFileTakesItsTextFromATemplate(t *testing.T) {
t.Errorf("the build packs files/ into %d archives, not one", packs)
}
}
// The packaged dnsmasq.service orders itself Before= network-online.target and nss-lookup.target. A drop-in
// that orders it After= either of them makes an ordering cycle, and systemd drops the start job: the mesh
// resolver stayed down for 90 minutes at a boot (novox/hq issue 1126). Wants=, Requires= and BindsTo= are
// refused too: they pull the target in, and with the packaged ordering they are the same mistake. Because the
// tunnel is ordered after those targets, the drop-in cannot wait for the address either: a start that finds it
// missing fails at once and Restart=on-failure brings it up (see the next test).
func TestDropInDoesNotOrderAfterATargetThePackagedUnitOrdersBefore(t *testing.T) {
raw, err := os.ReadFile(filepath.Join(moduleDir, "files", "mesh.conf"))
if err != nil {
t.Fatal(err)
}
forbidden := []string{"network-online.target", "nss-lookup.target"}
for n, line := range strings.Split(string(raw), "\n") {
line = strings.TrimSpace(line)
if strings.HasPrefix(line, "#") {
continue
}
key, val, ok := strings.Cut(line, "=")
key, val = strings.TrimSpace(key), strings.TrimSpace(val)
if !ok || (key != "After" && key != "Requires" && key != "Wants" && key != "BindsTo") {
continue
}
for _, target := range forbidden {
if strings.Contains(val, target) {
t.Errorf("mesh.conf line %d: %s=%s orders dnsmasq after a target its packaged unit orders itself before: a cycle", n+1, key, val)
}
}
}
}
// The wait for the address in ExecStartPre is at most two seconds: the tunnel that brings the address is ordered
// after network-online.target, which the resolver is ordered before, so a longer wait would hold the tunnel back.
func TestAddressWaitIsAtMostTwoSeconds(t *testing.T) {
raw, err := os.ReadFile(filepath.Join(moduleDir, "files", "mesh.conf"))
if err != nil {
t.Fatal(err)
}
found := false
for _, line := range strings.Split(string(raw), "\n") {
line = strings.TrimSpace(line)
if !strings.HasPrefix(line, "ExecStartPre=") {
continue
}
m := regexp.MustCompile(`seq 1 (\d+)`).FindStringSubmatch(line)
if m == nil {
t.Fatalf("ExecStartPre has no bounded wait (seq 1 N): %s", line)
}
found = true
if n, _ := strconv.Atoi(m[1]); n > 2 {
t.Errorf("ExecStartPre waits up to %d s for the address, more than 2 s", n)
}
}
if !found {
t.Fatal("mesh.conf has no ExecStartPre")
}
}
+6 -5
View File
@@ -1,14 +1,15 @@
# Written by the mesh (module dnsmasq, novox/hq issue 348). Replaced on every push.
#
# The resolver binds its addresses once, at start (bind-interfaces), so it starts only once this
# machine's private address is there: it waits up to 80 seconds for it, and a start that found it
# missing is tried again every ten seconds rather than left failed.
# machine's private address is there: a start that finds it missing fails after at most two seconds and
# is tried again every ten seconds (Restart=on-failure, no start limit) until the tunnel has brought it.
# It does not order itself After= or Wants= network-online.target, and its wait is short: the packaged
# unit is Before= network-online.target and nss-lookup.target, and the tunnel is ordered after them, so
# a long wait would hold the tunnel back and a drop-in ordering would be a cycle (issue 1126).
[Unit]
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=0
[Service]
ExecStartPre=/usr/bin/sh -c 'for i in $(seq 1 80); do ip -4 -o addr show to ${machine:address} | grep -q . && exit 0; sleep 1; done; echo "${machine:address} is not on this machine" >&2; exit 1'
ExecStartPre=/usr/bin/sh -c 'for i in $(seq 1 2); do ip -4 -o addr show to ${machine:address} | grep -q . && exit 0; sleep 1; done; echo "${machine:address} is not on this machine" >&2; exit 1'
Restart=on-failure
RestartSec=10s
+2 -2
View File
@@ -1,5 +1,5 @@
module dnsmasq
go 1.25.0
go 1.26.0
require git.novox.be/novox/mesh-sdk/go v0.1.7
require git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029231453-6e4a3014cbde
+2 -2
View File
@@ -1,2 +1,2 @@
git.novox.be/novox/mesh-sdk/go v0.1.7 h1:C0sTQmtTiyYH7bnqZb7PusXnqA37gKuT7Nqjn9gG47w=
git.novox.be/novox/mesh-sdk/go v0.1.7/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029231453-6e4a3014cbde h1:ddf/jKI9c87sQKDmZLoG6PMnvXwsNH16ByVveyuFME4=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029231453-6e4a3014cbde/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
+4 -1
View File
@@ -9,7 +9,10 @@
{
"name": "wildcard-resolution",
"scope": "mesh",
"identity": false
"identity": false,
"credential": {
"form": "none"
}
}
],
"requires": [
@@ -9,7 +9,7 @@ package main
//
// - every image a container on this machine uses, in any state;
// - every image this machine's declaration names — what the mesh would send it now and what it was
// last sent — as the controller's `images` verb answers. **No answer, no removal**: what the
// last sent — as the mesh-artifact-sweep seat's `images` verb answers. **No answer, no removal**: what the
// declaration names is the one thing that cannot be guessed;
// - every image younger than the floor (seven days, the weekly prune's week);
// - in each line of images holding one of those, the newest image besides them: the previous
@@ -43,7 +43,7 @@ type PruneImagesAsk struct {
Limit int
}
// declaredImages is the controller's `images` answer (novox/hq ADR 0251 §5).
// declaredImages is the mesh-artifact-sweep seat's `images` answer (novox/hq ADR 0251 §5).
type declaredImages struct {
Node string `json:"node"`
Images []struct {
@@ -129,7 +129,7 @@ func digestOf(ref string) string {
return ""
}
// askDeclared asks the controller what this machine's declarations name. An error means unknown.
// askDeclared asks the mesh-artifact-sweep seat what this machine's declarations name. An error means unknown.
func (c *Client) askDeclared() (*declaredImages, error) {
if c.Node == "" {
return nil, errors.New("the runtime did not say which machine this is (MESH_NODE is empty)")
@@ -137,20 +137,20 @@ func (c *Client) askDeclared() (*declaredImages, error) {
if c.Ask == nil {
return nil, errors.New("this bundle cannot ask the mesh")
}
raw, err := c.Ask("seat:mesh-controller.images", map[string]any{"node": c.Node})
raw, err := c.Ask("seat:mesh-artifact-sweep.images", map[string]any{"node": c.Node})
if err != nil {
return nil, fmt.Errorf("the controller's images verb did not answer: %w", err)
return nil, fmt.Errorf("the mesh-artifact-sweep seat's images verb did not answer: %w", err)
}
answer, output, ok := answerOf(raw)
if !ok {
return nil, fmt.Errorf("the controller refused images: %s", firstLine(output))
return nil, fmt.Errorf("the mesh-artifact-sweep seat refused images: %s", firstLine(output))
}
var d declaredImages
if err := json.Unmarshal(answer, &d); err != nil {
return nil, fmt.Errorf("the controller's images answer is not readable: %v", err)
return nil, fmt.Errorf("the mesh-artifact-sweep seat's images answer is not readable: %v", err)
}
if d.Node != "" && d.Node != c.Node {
return nil, fmt.Errorf("the controller answered for %q, not for this machine %q", d.Node, c.Node)
return nil, fmt.Errorf("the mesh-artifact-sweep seat answered for %q, not for this machine %q", d.Node, c.Node)
}
return &d, nil
}
@@ -43,7 +43,7 @@ func imagesMachine() *fake {
func declaring(t *testing.T, answer string) Asker {
return func(key string, body any) (json.RawMessage, error) {
if key != "seat:mesh-controller.images" {
if key != "seat:mesh-artifact-sweep.images" {
t.Errorf("asked %s", key)
}
if b, _ := body.(map[string]any); b["node"] != "laptop" {
+2 -2
View File
@@ -1,5 +1,5 @@
module docker
go 1.22
go 1.26.0
require git.novox.be/novox/mesh-sdk/go v0.1.6
require git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029231453-6e4a3014cbde
+2 -2
View File
@@ -1,2 +1,2 @@
git.novox.be/novox/mesh-sdk/go v0.1.6 h1:9qzdYONYbJdWcu6sxQcq9v1LI0JxcfkiKYkMUzJSkVQ=
git.novox.be/novox/mesh-sdk/go v0.1.6/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029231453-6e4a3014cbde h1:ddf/jKI9c87sQKDmZLoG6PMnvXwsNH16ByVveyuFME4=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029231453-6e4a3014cbde/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
+1 -1
View File
@@ -7,7 +7,7 @@
"privileged"
],
"invokes": [
"seat:mesh-controller.images"
"seat:mesh-artifact-sweep.images"
],
"claims": [
{
+9
View File
@@ -312,6 +312,15 @@ read-only. With a `user` it adds the teams that user is in and its effective rig
listing those above read: the check that a build key's user, `nox-builder` in `builders`, reads and pushes nothing. The
tools' token lacks `read:organization` (the forge answers 403), so the read goes with the administrator's login, GET only.
### Two settings have named tools, and a retargeted pull request is judged again
`gitea_api` refuses a repository's settings and a pull request's base (guard.ts). Issue 1422 note 7 gave each its tool:
`gitea_default_branch_set` (owner, repo, branch, why) and `gitea_pull_base_set` (owner, repo, number, base, why). Both
act as the tools' account, need a reason, refuse a branch the repository does not have, and answer the value before and
after; a default must also be a branch a protection rule guards, and only an open pull request has its base changed.
A pull request's announced head is kept under `owner/repo#number@base`, so a retarget (same head, new base) is
announced again into the new base by the next poll instead of being skipped as already announced.
### The merging account writes the repositories the mesh delivers
`mesh-tools` sets every commit status and merges; outside the organisation it only reads (above), so a pull request of
+61 -19
View File
@@ -560,7 +560,7 @@ export class GiteaClient {
* **A listing the forge cut is no listing** (it says `truncated` on a page with nothing more to read, or the pages
* run past a bound): the call fails naming the repository and the commit, never answering the directories it did
* see, since a directory left out would be taken for one without a module. Reads only. */
async commitTrees(owner: string, repo: string, sha: string, paths?: Record<string, string[]>, listed?: string[]): Promise<{ root: string; trees: Record<string, string>; paths?: Record<string, Record<string, string>>; listed?: Record<string, Record<string, string>> }> {
async commitTrees(owner: string, repo: string, sha: string, paths?: Record<string, string[]>): Promise<{ root: string; trees: Record<string, string>; paths?: Record<string, Record<string, string>> }> {
const base = `${repoPath(owner, repo)}/git/trees/${commitSegment(sha)}?recursive=true`;
const cut = (why: string) => new Error(`the forge's tree listing of ${owner}/${repo} at ${sha} was cut (${why}); no trees are answered from a part`);
const entries: any[] = [];
@@ -587,23 +587,7 @@ export class GiteaClient {
if (!tree) throw new Error(`the forge's tree listing of ${owner}/${repo} at ${sha} holds ${e.path} but not the tree of ${dir}`);
trees[dir] = tree;
}
// **Listed directories** (novox/hq ADR 0549 §2, issue 1126): each asked directory's files directly under it, by name
// to blob id, from the same listing: how the forge bridge finds the catalogue's placement files, one per node, at a
// merge. A directory the commit lacks lists nothing (the catalogue before its first placement file).
let files: Record<string, Record<string, string>> | undefined;
if (listed !== undefined) {
files = {};
for (const dir of listed) {
files[dir] = {};
const prefix = dir === "" ? "" : `${dir}/`;
for (const e of entries) {
if (e?.type !== "blob" || typeof e.path !== "string" || !e.path.startsWith(prefix)) continue;
const name = e.path.slice(prefix.length);
if (name !== "" && !name.includes("/")) files[dir][name] = String(e.sha);
}
}
}
if (paths === undefined) return files === undefined ? { root, trees } : { root, trees, listed: files };
if (paths === undefined) return { root, trees };
// **Named paths** (novox/hq ADR 0499 §3, issue 1758): each asked path's git object id at the commit, a file's or a
// directory's, from the same listing, by directory and by path relative to it. A path the listing lacks fails
// naming both, and nothing is answered in part.
@@ -618,7 +602,7 @@ export class GiteaClient {
named[dir][p] = id;
}
}
return files === undefined ? { root, trees, paths: named } : { root, trees, paths: named, listed: files };
return { root, trees, paths: named };
}
/** Whether a repository holds a file at a commit: true, false for a 404, thrown when the forge cannot say. */
@@ -758,6 +742,15 @@ export class GiteaClient {
return { created: true, rule };
}
/** Write a branch's rule with exactly the fields given (gitea_branch_protection_set_full): an edit of the rule named
* for the branch, or a new one. As this client's account, which the forge refuses where it is no administrator. */
async writeBranchProtection(owner: string, repo: string, branch: string, body: Record<string, unknown>, creating: boolean): Promise<BranchProtection> {
return creating
? this.request<BranchProtection>(`${repoPath(owner, repo)}/branch_protections`, { method: "POST", body: JSON.stringify({ rule_name: branch, ...body }) })
: this.request<BranchProtection>(`${repoPath(owner, repo)}/branch_protections/${pathSegments("branch", branch)}`,
{ method: "PATCH", body: JSON.stringify(body) });
}
async createPullRequest(
owner: string,
repo: string,
@@ -834,6 +827,30 @@ export class GiteaClient {
return (raw ?? []).map((b) => ({ name: String(b?.name ?? ""), commit: String(b?.commit?.id ?? ""), protected: Boolean(b?.protected) }));
}
/** One branch as the forge keeps it: whether a protection rule guards it; null when the repository has no such branch. */
async branchState(owner: string, repo: string, branch: string): Promise<{ protected: boolean } | null> {
try {
const b = await this.request<any>(`${repoPath(owner, repo)}/branches/${pathSegments("branch", branch)}`);
return { protected: Boolean(b?.protected) };
} catch (err) {
if (/: 404 /.test(err instanceof Error ? err.message : "")) return null;
throw err;
}
}
/** Make a branch the repository's default (issue 1422): what the forge shows, clones and opens pull requests against. */
async setDefaultBranch(owner: string, repo: string, branch: string): Promise<GiteaRepo> {
return GiteaClient.mapRepo(await this.request<any>(repoPath(owner, repo), { method: "PATCH", body: JSON.stringify({ default_branch: branch }) }));
}
/** Change the branch an open pull request merges into (issue 1422). The head stays; the forge recomputes the diff. */
async setPullBase(owner: string, repo: string, index: number, base: string): Promise<GiteaPull> {
indexOf(index);
return GiteaClient.mapPull(
await this.request<any>(`${repoPath(owner, repo)}/pulls/${index}`, { method: "PATCH", body: JSON.stringify({ base }) }),
);
}
async deleteBranch(owner: string, repo: string, branch: string): Promise<void> {
await this.request(`${repoPath(owner, repo)}/branches/${pathSegments("branch", branch)}`, { method: "DELETE" });
}
@@ -1028,6 +1045,31 @@ export class GiteaAdmin {
return Array.isArray(res.body) ? res.body : [];
}
/** Write a branch's rule with exactly the fields given, as the forge's administrator (gitea_branch_protection_set_full;
* the operator, 2026-10-28): an edit of the rule named for the branch, or a new one. */
async writeBranchProtection(owner: string, repo: string, branch: string, body: Record<string, unknown>, creating: boolean): Promise<BranchProtection> {
const path = creating ? `${repoPath(owner, repo)}/branch_protections` : `${repoPath(owner, repo)}/branch_protections/${pathSegments("branch", branch)}`;
const res = await this.request(path, { method: creating ? "POST" : "PATCH", body: JSON.stringify(creating ? { rule_name: branch, ...body } : body) });
if (res.status !== 200 && res.status !== 201) GiteaAdmin.fail(path, res);
return res.body as BranchProtection;
}
/** A repository as the forge answers it to its administrator: every setting (gitea_repo_settings_set_full). */
async repoSettings(owner: string, repo: string): Promise<Record<string, unknown>> {
const path = repoPath(owner, repo);
const res = await this.request(path);
if (res.status !== 200) GiteaAdmin.fail(path, res);
return res.body as Record<string, unknown>;
}
/** Edit a repository's settings with exactly the fields given, as the forge's administrator. */
async editRepoSettings(owner: string, repo: string, body: Record<string, unknown>): Promise<Record<string, unknown>> {
const path = repoPath(owner, repo);
const res = await this.request(path, { method: "PATCH", body: JSON.stringify(body) });
if (res.status !== 200) GiteaAdmin.fail(path, res);
return res.body as Record<string, unknown>;
}
/** Put main's rule right in every repository of an organisation (novox/hq ADR 0398 §2, issue 948): no push, only
* the tools' account merges, and no status required (ADR 0534 §1). The rule there is edited; a repository with no
* rule for main gets one. Answers the repositories it touched. With the administrator's login:
@@ -192,17 +192,14 @@ func (r *teamRig) pass() []provider.Outcome {
return out
}
// bind makes module on node bind gitea's forge-account in the assignments mirror.
// bind makes module on node bind gitea's forge-account in the seat mesh-provisioner's plans.
func (r *teamRig) bind(node, module string) {
r.bus.set(provider.AssignmentsBucket, node+"."+module, []byte(fmt.Sprintf(
`{"node":%q,"module":%q,"provisioning":{"provisions":[{"provision":"forge-account","provider":"novox","provider-module":"gitea","as":"mesh_%s"}]}}`,
node, module, strings.ReplaceAll(node+"_"+module, "-", "_"))))
r.bus.bindPlan(node, module, "mesh_"+strings.ReplaceAll(node+"_"+module, "-", "_"))
}
// unbind is the operator's yes to retiring module's forge-account on node (ADR 0570 clause 5).
func (r *teamRig) unbind(node, module string) {
r.bus.mu.Lock()
delete(r.bus.buckets[provider.AssignmentsBucket], node+"."+module)
r.bus.mu.Unlock()
r.bus.retire(node, module)
}
// key reports a key in module's state keys, of the kind and for the account given.
@@ -30,6 +30,7 @@ import (
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/provider"
"git.novox.be/novox/mesh-sdk/go/provisions"
)
const bundleVar = "MESH_FORGE_ACCOUNT_TEST_BUNDLE"
@@ -80,7 +81,7 @@ func startBus(t *testing.T) (*runtime, context.CancelFunc) {
rt := &runtime{t: t, js: js, kvs: map[string]jetstream.KeyValue{}, ctx: ctx, wait: map[string]chan json.RawMessage{}}
// The controller's job: a bucket per state the module declares (bound) or reads (the others).
for state, bucket := range map[string]string{"bound": "gitea_bound", "ssh-client.keys": "ssh-client_keys",
"mesh-controller.assignments": "mesh-controller_assignments"} {
"mesh-provisioner.plans": "mesh-provisioner_plans", "mesh-provisioner.provisions": "mesh-provisioner_provisions"} {
kv, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: bucket, History: 1})
if err != nil {
t.Fatal(err)
@@ -296,9 +297,11 @@ func TestTheProviderWholeAnswersAMachineThatReportedItsKey(t *testing.T) {
}
rt, cancel := startBus(t)
t.Cleanup(cancel)
rt.put("mesh-controller.assignments", "g14.ssh-client", map[string]any{"node": "g14", "module": "ssh-client",
"provisioning": map[string]any{"provisions": []map[string]any{{"provision": "forge-account", "provider": "novox",
rt.put("mesh-provisioner.plans", "g14.ssh-client", map[string]any{"v": 1, "node": "g14", "module": "ssh-client",
"plan": map[string]any{"provisions": []map[string]any{{"provision": "forge-account", "provider": "novox",
"provider-module": "gitea", "as": "mesh_g14_ssh_client"}}}})
rt.put("mesh-provisioner.provisions", "g14.ssh-client.forge-account",
json.RawMessage(provisionKey("g14", "ssh-client", provisions.Asked, nil)))
rt.put("ssh-client.keys", "g14.nox", map[string]string{"node": "g14", "account": "nox", "kind": "machine",
"public": pubA + " nox@g14 (mesh)", "fingerprint": "SHA256:x", "at": "2026-10-16T10:00:00Z"})
rt.start([]string{"MESH_NODE=novox", "MESH_MODULE=gitea", "MESH_SERVED_MODULE=gitea",
@@ -25,7 +25,7 @@ func TestConfigFromEnvNamesWhatIsMissing(t *testing.T) {
}
// Issue 955: the runtime grants a module only the states its manifest `reads`, and the provider watches the machines'
// keys and, through the SDK's provider, the controller's assignments (where it learns its consumers). Run as the
// keys and, through the SDK's provider, the seat mesh-provisioner's plans and provisions (where it learns its consumers, ADR 0570). Run as the
// bus test does, with every bucket made, a missing grant passes unseen; the manifest is read here instead.
func TestTheManifestReadsEveryStateTheProviderWatches(t *testing.T) {
raw, err := os.ReadFile("../../module.json")
@@ -38,7 +38,7 @@ func TestTheManifestReadsEveryStateTheProviderWatches(t *testing.T) {
if err := json.Unmarshal(raw, &m); err != nil {
t.Fatal(err)
}
for _, state := range []string{"mesh-controller.assignments", "ssh-client.keys"} {
for _, state := range []string{"mesh-provisioner.plans", "mesh-provisioner.provisions", "ssh-client.keys"} {
if !slices.Contains(m.Reads, state) {
t.Errorf("gitea's reads %v lack %q, so the runtime refuses its watch", m.Reads, state)
}
@@ -184,7 +184,9 @@ func (s *Service) Make(ctx context.Context, c provider.Consumer) error {
s.mu.Unlock()
if !loaded {
// After a restart the first pass can come before the keys arrive: nothing is removed or deleted on that.
return fmt.Errorf("the keys of %s are not read yet, so nothing is made, removed or deleted", source)
// Not ready, not failed: the entry waits and keeps serving what it served (provider.ErrNotReady).
return fmt.Errorf("the keys of %s are not read yet, so nothing is made, removed or deleted: %w", source,
provider.ErrNotReady)
}
have := s.keysFrom(source, node, kind)
if len(have) == 0 {
@@ -17,6 +17,7 @@ import (
"time"
"git.novox.be/novox/mesh-sdk/go/provider"
"git.novox.be/novox/mesh-sdk/go/provisions"
"gitea/internal/forgehttp"
)
@@ -78,6 +79,22 @@ func (b *fakeBus) entry(t *testing.T, key string) provider.Entry {
}
return e
}
// serves is whether gitea's entry under key serves its consumer: there, in state (provided when none is given) and with
// values (ADR 0570 clause 3: an entry the provider could not make is kept, failed, serving nothing; one it is not
// ready to make waits and keeps serving).
func (b *fakeBus) serves(key string, state ...string) bool {
want := provisions.Provided
if len(state) > 0 {
want = state[0]
}
b.mu.Lock()
raw, ok := b.buckets[provider.BucketOf("gitea")][key]
b.mu.Unlock()
var e provider.Entry
return ok && json.Unmarshal(raw, &e) == nil && e.State == want && len(e.Values) > 0
}
func (b *fakeBus) has(key string) bool {
b.mu.Lock()
defer b.mu.Unlock()
@@ -85,11 +102,41 @@ func (b *fakeBus) has(key string) bool {
return ok
}
// bindClient makes ssh-client on node bind gitea's forge-account in the assignments mirror.
// bindClient makes ssh-client on node bind gitea's forge-account in the seat mesh-provisioner's plans.
func (b *fakeBus) bindClient(node string) {
b.set(provider.AssignmentsBucket, node+".ssh-client", []byte(fmt.Sprintf(
`{"node":%q,"module":"ssh-client","provisioning":{"provisions":[{"provision":"forge-account","provider":"novox","provider-module":"gitea","as":"mesh_%s_ssh_client"}]}}`,
node, node)))
b.bindPlan(node, "ssh-client", "mesh_"+node+"_ssh_client")
}
// bindPlan makes module on node bind gitea's forge-account as the seat mesh-provisioner keeps it: its plan, and the
// provision key asked of gitea on novox (novox/hq ADR 0570 clauses 1 to 3).
func (b *fakeBus) bindPlan(node, module, as string) {
b.set(provisions.PlansBucket, node+"."+module, []byte(fmt.Sprintf(
`{"v":1,"node":%q,"module":%q,"plan":{"provisions":[{"provision":"forge-account","provider":"novox","provider-module":"gitea","as":%q}]}}`,
node, module, as)))
b.set(provisions.ProvisionsBucket, node+"."+module+".forge-account", provisionKey(node, module, provisions.Asked, nil))
}
// retire is the operator's yes to the retiring of module's forge-account on node: its plan gone, its key retiring
// with the warrant (ADR 0570 clause 5), the only way the provider lets an entry go.
func (b *fakeBus) retire(node, module string) {
b.mu.Lock()
delete(b.buckets[provisions.PlansBucket], node+"."+module)
b.mu.Unlock()
b.set(provisions.ProvisionsBucket, node+"."+module+".forge-account", provisionKey(node, module, provisions.Retiring,
&provisions.Warrant{Ask: "ask-1", Option: "yes", By: "operator", At: time.Date(2026, 10, 29, 12, 0, 0, 0, time.UTC)}))
}
func provisionKey(node, module, state string, w *provisions.Warrant) []byte {
pr := provisions.Provision{Provision: "forge-account", State: state, Warrant: w,
Consumer: provisions.End{Node: node, Module: module}, Provider: provisions.End{Node: "novox", Module: "gitea"}}
if state == provisions.Retiring {
pr.Why = "no want needs it"
}
raw, err := provisions.Encode(pr)
if err != nil {
panic(err)
}
return raw
}
// fakeForge is gitea's API as far as the provider uses it.
@@ -343,9 +390,7 @@ func TestAConsumerThatNoLongerBindsLosesItsMeshKeysAndKeepsItsAccount(t *testing
r.forge.mu.Lock()
r.forge.held["nox-agent"] = append(r.forge.held["nox-agent"], map[string]any{"id": int64(99), "title": "my own", "key": pubB})
r.forge.mu.Unlock()
r.bus.mu.Lock()
delete(r.bus.buckets[provider.AssignmentsBucket], "g14.ssh-client")
r.bus.mu.Unlock()
r.bus.retire("g14", "ssh-client")
r.pass()
if r.bus.has("g14.ssh-client.forge-account") {
t.Fatal("the entry stayed")
@@ -460,7 +505,7 @@ func TestADeletedKeyEntryRemovesTheMeshKeyAndTheBoundEntryButNotTheOperatorsKey(
if got := r.forge.titles("nox-agent"); len(got) != 1 || got[0] != "my own" {
t.Fatalf("keys = %v", got)
}
if r.bus.has("g14.ssh-client.forge-account") {
if r.bus.serves("g14.ssh-client.forge-account") {
t.Fatal("the entry still serves a machine whose key is gone")
}
}
@@ -482,8 +527,8 @@ func TestAPassBeforeTheKeysAreLoadedRemovesAndDeletesNothing(t *testing.T) {
if got := strings.Join(r.forge.titles("nox-agent"), ","); got != "mesh:g14.nox,my own" {
t.Fatalf("keys = %s", got)
}
if !r.bus.has("g14.ssh-client.forge-account") {
t.Fatal("the entry was deleted before the keys were read")
if !r.bus.serves("g14.ssh-client.forge-account", provisions.Waits) {
t.Fatal("the entry stopped serving before the keys were read; it waits and keeps serving")
}
if len(out) != 1 || !strings.Contains(out[0].Why, "not read yet") {
t.Fatalf("the wait should be said: %+v", out)
@@ -493,7 +538,7 @@ func TestAPassBeforeTheKeysAreLoadedRemovesAndDeletesNothing(t *testing.T) {
r.svc.loaded = true
r.svc.mu.Unlock()
r.pass()
if got := strings.Join(r.forge.titles("nox-agent"), ","); got != "my own" || r.bus.has("g14.ssh-client.forge-account") {
if got := strings.Join(r.forge.titles("nox-agent"), ","); got != "my own" || r.bus.serves("g14.ssh-client.forge-account") {
t.Fatalf("keys = %s", got)
}
}
@@ -515,7 +560,7 @@ func TestKeepRunningStartsTheProviderAgainAfterARefusedWatch(t *testing.T) {
r.svc.KeepRunning(ctx, func(ctx context.Context) error {
calls++
if calls <= 2 {
return errors.New(`watching mesh-controller_assignments: gitea reads no state called "mesh-controller.assignments"`)
return errors.New(`watching mesh-provisioner_plans: gitea reads no state called "mesh-provisioner.plans"`)
}
close(serving)
<-ctx.Done()
@@ -535,7 +580,7 @@ func TestKeepRunningStartsTheProviderAgainAfterARefusedWatch(t *testing.T) {
if len(pauses) != 2 || pauses[0] != 5*time.Second || pauses[1] != 10*time.Second {
t.Fatalf("the backoff is %v", pauses)
}
if len(said) == 0 || !strings.Contains(said[0], "mesh-controller.assignments") {
if len(said) == 0 || !strings.Contains(said[0], "mesh-provisioner.plans") {
t.Fatalf("a refusal is said loudly: %v", said)
}
loud := 0
@@ -142,7 +142,7 @@ func TestABuildModulesStateThatNeverAnswersHoldsBackOnlyItsOwnEntries(t *testing
}
// 1482/AC4 (the provider's half): the manifest reads the states of both build modules beside ssh-client's and the
// controller's assignments; the provider watches exactly the modules the code names as builders.
// seat mesh-provisioner's plans; the provider watches exactly the modules the code names as builders.
func TestTheManifestReadsTheBuildModulesKeys(t *testing.T) {
raw, err := os.ReadFile("../../module.json")
if err != nil {
@@ -154,7 +154,7 @@ func TestTheManifestReadsTheBuildModulesKeys(t *testing.T) {
if err := json.Unmarshal(raw, &m); err != nil {
t.Fatal(err)
}
for _, state := range []string{"mesh-controller.assignments", "ssh-client.keys", "mesh-build-go.keys", "mesh-build-node.keys", "mesh-build-image.keys"} {
for _, state := range []string{"mesh-provisioner.plans", "mesh-provisioner.provisions", "ssh-client.keys", "mesh-build-go.keys", "mesh-build-node.keys", "mesh-build-image.keys"} {
if !slices.Contains(m.Reads, state) {
t.Errorf("gitea's reads %v lack %q", m.Reads, state)
}
+1 -1
View File
@@ -3,7 +3,7 @@ module gitea
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023150321-a2c813a6b468
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029231453-6e4a3014cbde
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
golang.org/x/crypto v0.57.0
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023150321-a2c813a6b468 h1:UcMee+IV2ii5E4dRZdQUaSaeh9IedTFYumuQbBiB0k0=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023150321-a2c813a6b468/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029231453-6e4a3014cbde h1:ddf/jKI9c87sQKDmZLoG6PMnvXwsNH16ByVveyuFME4=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029231453-6e4a3014cbde/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
+2
View File
@@ -12,6 +12,8 @@
// forge as themselves;
// - `gitea_branch_protection_set` lets no person push and no administrator merge past a rule, and on main keeps
// the tools' account the only one that merges; it requires no status, as no rule of the mesh does (ADR 0534 §5);
// - `gitea_default_branch_set` and `gitea_pull_base_set` are the two settings gitea_api refuses, as named tools: a branch that
// exists (and, for a default, that a rule guards), a stated reason, the tools' account, the value before and after (issue 1422);
// - `gitea_merge_pull_request` merges only into a branch that refuses its push, only a head whose mesh/check passed
// against the trunk's head, approved by the operator where the branch requires no status (ADR 0534 §4), and only
// that head;
+67 -66
View File
@@ -53,8 +53,9 @@ import { announce, keepWatermark, readWatermark } from "./announce.js";
import { mergeTool } from "./merge.js";
import { GiteaAdmin, GiteaClient, movedSince } from "./client.js";
import { keepMainRules, mainRulesStep } from "./mainrules.js";
import { CHECK_CONTEXT, CHECK_STATUS, REPO_CHECK_CONTEXT, commentFor, composedGate, groupGateStatus, headsToAnnounce, maintained,
requiredOf, statusesFor, writesGate, type Announced, type Checked, repoCheckFacts, reposToLook } from "./pulls.js";
import { CHECK_CONTEXT, CHECK_STATUS, REPO_CHECK_CONTEXT, commentFor, composedGate, groupGateStatus, headKey, maintained,
requiredOf, statusesFor, writesGate, type Announced, type Checked, repoCheckFacts } from "./pulls.js";
import { actOn, announceNewHeads, startPass } from "./watch.js";
import { GateWriter } from "./gate.js";
import { HOOK_PATH, HOOK_PORT, Pace, claimHead, ensureSystemHook, eventsTool, keptSecret, receiver, type ForgeEvent } from "./webhook.js";
import { keepSiblings, noteComposed, noteVerdict, readSiblings, siblingsFile, siblingsOf, weighSiblings } from "./siblings.js";
@@ -138,12 +139,7 @@ async function pollMerged(client: GiteaClient): Promise<void> {
// Closed unmerged (novox/hq ADR 0239): announced once, since the watching began, so its delivery stops.
const closedKey = `closed:${repo.full_name}#${pull.number}`;
if (!pull.merged && pull.state === "closed" && !announced.has(closedKey)) {
if (primedMerges && !!pull.updated_at && !!since && pull.updated_at > since) {
await announce("pull.closed", { owner: repo.owner, repo: repo.name, number: pull.number, title: pull.title,
head: pull.head, head_sha: pull.head_sha, base: pull.base, html_url: pull.html_url });
console.log(`[gitea] announced ${repo.full_name}#${pull.number} closed unmerged`);
await siblingMoves(client, repo.full_name, pull.number);
}
if (primedMerges && !!pull.updated_at && !!since && pull.updated_at > since) await announceClosed(client, repo, pull);
announced.add(closedKey);
continue;
}
@@ -168,6 +164,23 @@ async function pollMerged(client: GiteaClient): Promise<void> {
type RepoOf = { owner: string; name: string; full_name: string; clone_url?: string };
type PullOf = Awaited<ReturnType<GiteaClient["listPullRequests"]>>[number];
/** A pull request closed without a merge, announced once (ADR 0239): by its event, or by the start's pass. */
async function announceClosed(client: GiteaClient, repo: RepoOf, pull: PullOf): Promise<boolean> {
const closedKey = `closed:${repo.full_name}#${pull.number}`;
if (pull.merged || pull.state !== "closed" || announced.has(closedKey)) return false;
announced.add(closedKey);
try {
await announce("pull.closed", { owner: repo.owner, repo: repo.name, number: pull.number, title: pull.title,
head: pull.head, head_sha: pull.head_sha, base: pull.base, html_url: pull.html_url });
} catch (err) {
announced.delete(closedKey);
throw err;
}
console.log(`[gitea] announced ${repo.full_name}#${pull.number} closed unmerged`);
await siblingMoves(client, repo.full_name, pull.number);
return true;
}
/** **A merge, announced once, whichever path saw it first** (novox/hq issue 729): the poll or the forge's event. */
async function announceMerge(client: GiteaClient, repo: RepoOf, pull: PullOf): Promise<boolean> {
if (!pull.merged || !pull.merge_commit_sha || announced.has(pull.merge_commit_sha)) return false;
@@ -243,7 +256,7 @@ let heads: Announced = {};
let primedPulls = false;
/** **A head, announced once, whichever path saw it first** (novox/hq issue 729): the poll or the forge's event. */
async function announceHead(client: GiteaClient, repo: RepoOf, pull: OpenOf): Promise<boolean> {
const key = `${repo.full_name}#${pull.number}`;
const key = headKey(repo.full_name, pull.number, pull.base);
const sha = String(pull.head_sha ?? "");
const before = heads[key];
// Claimed before the first wait, so the other path finds it taken while this one is still announcing.
@@ -281,43 +294,18 @@ async function alreadyChecked(client: GiteaClient, repo: RepoOf, pull: OpenOf):
}
let lastPullLook = "";
// **A sweep now and then looks at every repository** (novox/hq issue 347), whatever moved: the floor is a
// heuristic, and a heuristic that misses is caught by the next sweep rather than never.
const SWEEP_MS = 10 * 60_000;
let lastSweep = 0;
async function pollPulls(client: GiteaClient): Promise<void> {
/** The pass over every open pull request: run at the start and when the receiver listens, never on a timer (issue 1422). */
async function passPulls(client: GiteaClient): Promise<void> {
const began = new Date().toISOString();
const floor = lastPullLook ? new Date(Date.parse(lastPullLook) - MARGIN_MS).toISOString() : "";
const dayAgo = new Date(Date.now() - 24 * 3600_000).toISOString();
// A pull request opened, reopened or retargeted moves its own update time and not its repository's: the
// pull requests updated since the floor are found by their own (issue 347). A search that fails is a sweep.
let sweep = Date.now() - lastSweep >= SWEEP_MS;
let updated: { owner: string; name: string; updated_at?: string }[] = [];
if (floor && !sweep) {
try {
updated = await client.searchOpenPulls(floor);
} catch (err) {
console.error(`[gitea] the pull requests updated lately could not be searched, so every repository is looked at — ${err instanceof Error ? err.message : err}`);
sweep = true;
}
}
if (sweep) lastSweep = Date.now();
for (const repo of reposToLook(await client.listAllRepos(), floor, updated, sweep)) {
// Every open pull request updated since the floor, paged (novox/hq issue 579): the newest 20 alone left a head
// beyond them unannounced. A sweep and the first pass read every page.
const open = await client.listOpenPullsSince(repo.owner, repo.name, sweep ? "" : floor);
for (const pull of headsToAnnounce(repo.full_name, open, heads)) {
const key = `${repo.full_name}#${pull.number}`;
const fresh = primedPulls || (!!pull.updated_at && pull.updated_at > dayAgo);
// The first look after a start remembers nothing: a head the forge already shows a merge check of was announced.
if (fresh && !primedPulls && await alreadyChecked(client, repo, pull)) {
heads[key] = String(pull.head_sha);
continue;
}
if (fresh) {
if (await announceHead(client, repo, pull)) pace.miss(`${key} at ${String(pull.head_sha).slice(0, 8)}`);
} else heads[key] = String(pull.head_sha);
}
for (const repo of await client.listAllRepos()) {
// Every open pull request, paged (novox/hq issue 579).
const open = await client.listOpenPullsSince(repo.owner, repo.name, "");
await announceNewHeads(repo.full_name, open, heads, primedPulls, dayAgo, {
alreadyChecked: (pull) => alreadyChecked(client, repo, pull),
announceHead: (pull) => announceHead(client, repo, pull),
missed: (what) => pace.miss(what),
});
if (await trunksMoved(client, repo.owner, repo.name)) changedTrunks = true;
}
if (changedTrunks) keepTrunks();
@@ -497,6 +485,9 @@ const checks = new CheckStatuses({
/** How fast the polls look: 30 s until the forge's events are heard, five minutes while they are; and what
* gitea_events says of them (webhook.ts). */
const pace = new Pace();
/** Asks the pass over every repository again, shortly and once (set where the pass is made): an event that could not be
* acted on is not lost until the next start. */
let catchUp: () => void = () => {};
if (!gitea) pace.unheard("the forge is not watched: no way to a token", false);
// The states this module keeps and which others may read, said on the bus before it serves (novox/hq ADR 0495 clause 2).
@@ -522,22 +513,15 @@ registerModuleTools("gitea", () => [eventsTool(pace), ...readAllTools(readAll),
gates,
}), mergeTool(gitea, { gates })] : [])]);
/** What the forge's event asks of the announcer at once (issue 729): a new head is announced, and a merge. */
async function actOn(client: GiteaClient, event: ForgeEvent): Promise<void> {
const b = event.body as { owner?: string; repo?: string; number?: number };
if (!b.owner || !b.repo || !b.number) return;
if (event.name === "forge.pull-request.closed") {
// Closed unmerged is a sibling's move; a merge is weighed with its announcement below (ADR 0382 §5).
if (!(event.body as { merged?: boolean }).merged) await siblingMoves(client, `${b.owner}/${b.repo}`, b.number);
return;
}
const head = ["forge.pull-request.opened", "forge.pull-request.synchronized", "forge.pull-request.reopened"].includes(event.name);
const merged = event.name === "forge.pull-request.merged";
if (!head && !merged) return;
const [repo, pull] = await Promise.all([client.getRepo(b.owner, b.repo), client.getPullRequest(b.owner, b.repo, b.number)]);
if (head && pull.state === "open") await announceHead(client, repo, pull);
if (merged) await announceMerge(client, repo, pull);
}
/** What the forge's event asks of the announcer at once (issue 729, 1422): see watch.ts. */
const actOnEvent = (client: GiteaClient, event: ForgeEvent): Promise<void> => actOn<RepoOf, PullOf>(event, {
getRepo: (o, r) => client.getRepo(o, r),
getPull: (o, r, n) => client.getPullRequest(o, r, n) as Promise<PullOf & { state?: string; merged?: boolean }>,
announceHead: (repo, pull) => announceHead(client, repo, pull as OpenOf),
announceMerge: (repo, pull) => announceMerge(client, repo, pull),
announceClosed: (repo, pull) => announceClosed(client, repo, pull),
siblingMoves: (full, n) => siblingMoves(client, full, n),
});
/**
* **The receiver, and the forge's one system webhook pointing at it** (issue 729). The secret is minted and kept beside
@@ -563,7 +547,12 @@ async function listenForForge(client: GiteaClient): Promise<boolean> {
// A repository made, or one not yet seen readable, is granted read before anything acts on it (readall.ts).
await Promise.resolve().then(() => readAll().onEvent(event)).catch((err) =>
console.error(`[gitea] whether the mesh's accounts read the repository of ${event.name} could not be settled — ${err instanceof Error ? err.message : err}`));
await actOn(client, event);
try {
await actOnEvent(client, event);
} catch (err) {
catchUp();
throw err;
}
} });
await new Promise<void>((resolve, reject) => {
server.once("error", reject);
@@ -612,9 +601,9 @@ if (gitea) {
// polls slow to every five minutes and announce only what no event said (Pace); otherwise they keep 30 s.
// **Not awaited** (novox/hq issue 1726): the receiver's set-up asks the forge's admin API, and the runtime kills a bundle
// whose import has not ended in its 10 s handshake and never starts it again; a forge slow to answer while every bundle
// of the node starts left gitea_commit_status, gitea_merge_pull_request and gitea_events served nowhere. Until it is
// listening, Pace keeps the polls at 30 s, so nothing is missed meanwhile; listenForForge says its own failure.
void listenForForge(client);
// of the node starts left gitea_commit_status, gitea_merge_pull_request and gitea_events served nowhere. The pass below
// runs again once it is listening, so nothing said before is missed; listenForForge says its own failure.
const listenOk = listenForForge(client);
// The start's one pass over every repository (readall.ts), not awaited for the same reason. A forge not answering yet
// is asked again, a minute apart, until one pass has run; after that only the forge's events grant.
// An account of the list the forge does not hold yet (the forge-account provider makes nox-builder and nox-agent when
@@ -646,8 +635,20 @@ if (gitea) {
// Only while the setting keep-main-rules is true (mainrules.ts); false, the default, leaves every repository's rule as set.
tick(mainRulesStep(keepMainRules(), () => GiteaAdmin.fromEnv().ensureMainMerges("novox"), (line) => console.log(line)), 600_000);
tick(() => pollRepos(client), 60_000);
tick(() => pollMerged(client), () => pace.every());
tick(() => pollPulls(client), () => pace.every());
// **No timer asks the forge for pull requests or merges** (issue 1422): the forge's events announce them. One pass over
// every repository runs at the start and again once the receiver listens, so what happened while the module was away or
// not yet listening is announced; passes run one at a time, a failed one is asked again a minute later.
let passes: Promise<void> = Promise.resolve();
const pass = (): Promise<void> => (passes = passes.catch(() => {}).then(() =>
startPass(async () => { await pollMerged(client); await passPulls(client); }, { retryMs: 60_000 })));
void pass();
let catchUpAsked = false;
catchUp = () => {
if (catchUpAsked) return;
catchUpAsked = true;
setTimeout(() => { catchUpAsked = false; void pass(); }, 30_000);
};
void listenOk.then((ok) => (ok ? pass() : undefined));
// Bound again until they hold, and not awaited (bind.ts, issue 1793): the runtime kills a bundle that has not answered its
// handshake in 10 s, so the import must end while a refused bind is still asked again.
bindInBackground([
+7 -2
View File
@@ -399,6 +399,7 @@
"provides": [
{
"name": "npm-package-registry",
"credential": {"form": "pair"},
"scope": "mesh",
"identity": {
"max": 40,
@@ -407,6 +408,7 @@
},
{
"name": "git",
"credential": {"form": "none"},
"scope": "mesh"
},
{
@@ -419,6 +421,7 @@
},
{
"name": "forge-account",
"credential": {"form": "none"},
"scope": "mesh"
}
],
@@ -563,12 +566,14 @@
"announced"
],
"reads": [
"mesh-controller.assignments",
"mesh-provisioner.plans",
"mesh-provisioner.provisions",
"ssh-client.keys",
"mesh-build-go.keys",
"mesh-build-node.keys",
"mesh-build-image.keys",
"mesh-delivery-checker.owed",
"mesh-delivery-indexer.modules"
"mesh-delivery-indexer.modules",
"mesh-delivery-gate.source"
]
}
+257
View File
@@ -0,0 +1,257 @@
// Every field of a branch protection rule, set through one tool (gitea_branch_protection_set_full; the operator,
// 2026-10-28: "just make a generic tool so you can setup any configuration"). Pure, so it is tested without a forge:
// which fields the forge's rule takes and of what kind, the body a call sends, and whether the rule the forge then
// applies to the branch is one the controller registers a module from (judgedRule, mesh-control
// cmd/mesh-controller/build_source.go).
import { TOOLS_ACCOUNT } from "./tools-account.js";
import { MAIN_BRANCH, type BranchProtection } from "./protection.js";
import { ruleCovers } from "./guard.js";
type Kind = "boolean" | "logins" | "strings" | "number" | "string";
/** unread: the forge never answers it, so it is not read back; duration: a Go duration, compared as one ("8h" is the
* forge's "8h0m0s"). */
type Field = { kind: Kind; says: string; parent?: string; unread?: boolean; duration?: boolean };
/** Every field gitea's branch protection API takes on an edit at 1.27.3 (EditBranchProtectionOption, 32 fields), by its
* own name, and its kind; rule_name is the branch argument, never a field. A field with a parent is read by the forge
* only when the same call sets that parent true, and setting the parent false clears it (gitea's
* routers/api/v1/repo/branch.go): such a field is refused without its parent true in the call. */
export const PROTECTION_FIELDS: Record<string, Field> = {
enable_push: { kind: "boolean", says: "whether anyone who writes may push to the branch directly (false: no direct push; also clears enable_push_whitelist and push_whitelist_deploy_keys)" },
enable_push_whitelist: { kind: "boolean", says: "only the push whitelist may push; needs enable_push: true in the same call", parent: "enable_push" },
push_whitelist_usernames: { kind: "logins", says: "the logins that may push directly" },
push_whitelist_teams: { kind: "strings", says: "the teams that may push directly" },
push_whitelist_deploy_keys: { kind: "boolean", says: "whether deploy keys may push; needs enable_push: true in the same call", parent: "enable_push" },
enable_force_push: { kind: "boolean", says: "whether a force push is taken (false also clears the force-push allowlist flags)" },
enable_force_push_allowlist: { kind: "boolean", says: "only the force-push allowlist may force-push; needs enable_force_push: true in the same call", parent: "enable_force_push" },
force_push_allowlist_usernames: { kind: "logins", says: "the logins that may force-push" },
force_push_allowlist_teams: { kind: "strings", says: "the teams that may force-push" },
force_push_allowlist_deploy_keys: { kind: "boolean", says: "whether deploy keys may force-push; needs enable_force_push: true in the same call", parent: "enable_force_push" },
enable_merge_whitelist: { kind: "boolean", says: "whether only the merge whitelist may merge" },
merge_whitelist_usernames: { kind: "logins", says: "the logins that may merge (the controller's trunk rule wants mesh-tools, no agent-<node>)" },
merge_whitelist_teams: { kind: "strings", says: "the teams that may merge" },
enable_status_check: { kind: "boolean", says: "whether the statuses in status_check_contexts must pass to merge" },
status_check_contexts: { kind: "strings", says: "the required commit statuses, e.g. mesh/check" },
required_approvals: { kind: "number", says: "how many approving reviews a merge needs" },
enable_approvals_whitelist: { kind: "boolean", says: "whether only the approvals whitelist's reviews count" },
approvals_whitelist_username: { kind: "logins", says: "the logins whose approvals count" },
approvals_whitelist_teams: { kind: "strings", says: "the teams whose approvals count" },
block_on_rejected_reviews: { kind: "boolean", says: "a review asking changes blocks the merge" },
block_on_official_review_requests: { kind: "boolean", says: "an open official review request blocks the merge" },
block_on_outdated_branch: { kind: "boolean", says: "a head behind the branch blocks the merge" },
dismiss_stale_approvals: { kind: "boolean", says: "a new commit dismisses earlier approvals" },
ignore_stale_approvals: { kind: "boolean", says: "approvals of an earlier commit do not count" },
require_signed_commits: { kind: "boolean", says: "every commit must be signed" },
protected_file_patterns: { kind: "string", says: "files no push may change, ';'-separated globs" },
unprotected_file_patterns: { kind: "string", says: "files a push may change even without push rights, ';'-separated globs" },
block_admin_merge_override: { kind: "boolean", says: "true: an administrator cannot merge past the rule (the controller's trunk rule wants true)" },
enable_bypass_allowlist: { kind: "boolean", says: "whether the bypass allowlist may merge past the rule's checks" },
bypass_allowlist_usernames: { kind: "logins", says: "the logins that may merge past the rule's checks" },
bypass_allowlist_teams: { kind: "strings", says: "the teams that may merge past the rule's checks" },
priority: { kind: "number", says: "the rule's order among glob rules" },
};
/** The repository settings gitea_repo_settings_set_full sets (gitea's EditRepoOption at 1.27.3), by their own names. Left
* out: name (a rename moves every module's source address), the internal tracker, external tracker and external wiki
* objects, and mirror_username, mirror_password and mirror_token, which are credentials. The forge ignores a
* default_branch that does not exist; the read-back below says so. */
export const REPO_FIELDS: Record<string, Field> = Object.fromEntries(([
["description", "string"], ["website", "string"], ["private", "boolean"], ["template", "boolean"],
["default_branch", "string"], ["has_code", "boolean"], ["has_issues", "boolean"], ["has_wiki", "boolean"],
["has_pull_requests", "boolean"], ["has_projects", "boolean"], ["projects_mode", "string"], ["has_releases", "boolean"],
["has_packages", "boolean"], ["has_actions", "boolean"], ["ignore_whitespace_conflicts", "boolean"],
["allow_merge_commits", "boolean"], ["allow_rebase", "boolean"], ["allow_rebase_explicit", "boolean"],
["allow_squash_merge", "boolean"], ["allow_fast_forward_only_merge", "boolean"], ["allow_manual_merge", "boolean"],
["autodetect_manual_merge", "boolean"], ["allow_rebase_update", "boolean"], ["allow_merge_update", "boolean"],
["default_update_style", "string"], ["default_delete_branch_after_merge", "boolean"], ["default_merge_style", "string"],
["default_allow_maintainer_edit", "boolean"], ["archived", "boolean"], ["mirror_interval", "string"], ["enable_prune", "boolean"],
] as [string, Kind][]).map(([name, kind]) => [name, {
kind, says: `the repository's ${name.replace(/_/g, " ")}`,
...(name === "projects_mode" ? { parent: "has_projects", says: "the repository's projects mode; needs has_projects: true in the same call" } : {}),
...(name === "enable_prune" ? { unread: true, says: "a mirror's prune setting; the forge never answers it, so it is not read back" } : {}),
...(name === "mirror_interval" ? { duration: true, says: "a mirror's interval, a Go duration such as 8h (the forge answers 8h0m0s)" } : {}),
}]));
/** A Go duration ("8h", "1h30m", "8h0m0s", "90s") in milliseconds; NaN when it is none. */
export function durationMs(v: unknown): number {
if (typeof v !== "string" || !/^(\d+(\.\d+)?(ns|us|µs|ms|s|m|h))+$/.test(v)) return v === "0" ? 0 : NaN;
const unit: Record<string, number> = { ns: 1e-6, us: 1e-3, "µs": 1e-3, ms: 1, s: 1e3, m: 6e4, h: 3.6e6 };
let ms = 0;
for (const m of v.matchAll(/(\d+(?:\.\d+)?)(ns|us|µs|ms|s|m|h)/g)) ms += Number(m[1]) * unit[m[2]];
return ms;
}
/** Fields the forge was sent but does not hold after the write, each with what was sent and what it holds: a write it
* answered 200 and did not apply (gitea drops a field whose parent is not set, or a branch that does not exist). */
export function unapplied(sent: Record<string, unknown>, held: Record<string, unknown>, fields: Record<string, Field> = PROTECTION_FIELDS): string[] {
const norm = (k: string, v: unknown) => fields[k]?.duration ? String(durationMs(v))
: Array.isArray(v) ? JSON.stringify([...v].map(String).sort()) : JSON.stringify(v ?? null);
return Object.keys(sent).filter((k) => !fields[k]?.unread && norm(k, sent[k]) !== norm(k, held[k]))
.map((k) => `${k} (sent ${JSON.stringify(sent[k])}, the forge holds ${JSON.stringify(held[k] ?? null)})`);
}
/** Fields that changed between two reads but were not sent: what the forge changed beside the call (enable_push: false
* clears the push whitelist flags). */
export function alsoChanged(before: Record<string, unknown> | null, after: Record<string, unknown>, sent: Record<string, unknown>): string[] {
if (!before) return [];
const keys = new Set([...Object.keys(before), ...Object.keys(after)]);
return [...keys].filter((k) => !(k in sent) && !["updated_at", "created_at"].includes(k) &&
JSON.stringify(before[k] ?? null) !== JSON.stringify(after[k] ?? null));
}
/** A forge login. */
const LOGIN = /^[A-Za-z0-9_][A-Za-z0-9_.-]{0,39}$/;
/** The forge body for the fields given: each checked against its kind, a field not given left out (the forge keeps it on
* an edit), an unknown field refused before anything is sent. Throws on the first wrong field. */
export function fullProtectionBody(args: Record<string, unknown>, given: string[],
fields: Record<string, Field> = PROTECTION_FIELDS, what = "a branch protection rule"): Record<string, unknown> {
const body: Record<string, unknown> = {};
for (const name of given) {
const field = fields[name];
if (!field) throw new Error(`${name} is no field of ${what}; the fields are ${Object.keys(fields).join(", ")}`);
// The parent false clears the field, so only a field false goes beside a parent false.
if (field.parent && args[field.parent] !== true && !(args[field.parent] === false && args[name] === false)) {
throw new Error(`${name} is read by the forge only when the same call sets ${field.parent}: true (gitea ignores it otherwise). Nothing was sent`);
}
const v = args[name];
switch (field.kind) {
case "boolean":
if (typeof v !== "boolean") throw new Error(`${name} is true or false`);
body[name] = v;
break;
case "number":
if (typeof v !== "number" || !Number.isInteger(v) || v < 0) throw new Error(`${name} is a whole number, 0 or more`);
body[name] = v;
break;
case "string":
if (typeof v !== "string") throw new Error(`${name} is a string`);
if (field.duration && Number.isNaN(durationMs(v))) throw new Error(`${name} is a duration, such as 8h or 1h30m`);
body[name] = v;
break;
case "strings":
case "logins": {
if (!Array.isArray(v) || v.some((s) => typeof s !== "string" || s.trim() === "")) throw new Error(`${name} is a list of names`);
const names = [...new Set((v as string[]).map((s) => s.trim()))];
if (field.kind === "logins") {
const bad = names.find((s) => !LOGIN.test(s) || s.includes(".."));
if (bad !== undefined) throw new Error(`${name}: ${JSON.stringify(bad)} is no forge login`);
}
body[name] = names;
break;
}
}
}
return body;
}
/** The fields of a call that are protection fields: every argument but owner, repo, branch and why. */
export function givenFields(args: Record<string, unknown>): string[] {
return Object.keys(args).filter((k) => !["owner", "repo", "branch", "why"].includes(k) && args[k] !== undefined);
}
/** A rule as the controller reads it from gitea_branch_protection_get (protectionRule in build_source.go). */
interface ControllerRule { rule: string; push: boolean; admin_may_override: boolean; merge_whitelist: string[] | null }
function asControllerRule(p: BranchProtection): ControllerRule {
return {
rule: String(p.rule_name ?? p.branch_name ?? ""),
push: p.enable_push ?? false,
admin_may_override: !(p.block_admin_merge_override ?? false),
merge_whitelist: p.enable_merge_whitelist ? (p.merge_whitelist_usernames ?? []) : null,
};
}
/** ruleCovers of build_source.go, as the controller reads a rule's glob: `**` any, `*` within a segment, `?` one. */
function controllerCovers(rule: string, branch: string): boolean {
if (rule === branch) return true;
if (!/[*?[]/.test(rule)) return false;
let re = "^";
for (let i = 0; i < rule.length; i++) {
const c = rule[i];
if (c === "*" && rule[i + 1] === "*") { re += ".*"; i++; }
else if (c === "*") re += "[^/]*";
else if (c === "?") re += "[^/]";
else re += c.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
}
return new RegExp(re + "$").test(branch);
}
/** Whether the controller registers a module from this branch, judged as judgedRule in mesh-control
* cmd/mesh-controller/build_source.go does (ADR 0534 §6): the rule named for the branch, else the first glob rule that
* covers it; it takes no push, its merge whitelist names mesh-tools and no agent-<node>, and no administrator merges
* past it. A mirror, not the judge: the controller's own verdict is the one that counts. */
export function trunkRule(rules: BranchProtection[], branch: string, trunks: string[] = [MAIN_BRANCH]): { holds: boolean; why: string; is_trunk: boolean } {
const judged = judgedTrunkRule(rules, branch);
return { ...judged, is_trunk: trunks.includes(branch) };
}
function judgedTrunkRule(rules: BranchProtection[], branch: string): { holds: boolean; why: string } {
const all = rules.map(asControllerRule);
const r = all.find((x) => x.rule === branch) ?? all.find((x) => controllerCovers(x.rule, branch));
const no = (why: string) => ({ holds: false, why });
if (!r) return no(`no protection rule covers ${branch}`);
if (r.push) return no(`the rule ${r.rule} lets a person push to ${branch} directly`);
if (!r.merge_whitelist || r.merge_whitelist.length === 0) {
return no(`the rule ${r.rule} keeps no merge whitelist naming ${TOOLS_ACCOUNT} alone for ${branch}`);
}
if (!r.merge_whitelist.includes(TOOLS_ACCOUNT)) {
return no(`the rule ${r.rule} lets ${r.merge_whitelist.join(", ")} merge into ${branch}, and not ${TOOLS_ACCOUNT}`);
}
if (r.merge_whitelist.some((l) => l.startsWith("agent-"))) {
return no(`the rule ${r.rule} lets an agent's account merge into ${branch}: ${r.merge_whitelist.join(", ")}`);
}
if (r.admin_may_override) return no(`the rule ${r.rule} lets an administrator merge into ${branch} past it`);
return { holds: true, why: `the rule ${r.rule} takes no push, only ${r.merge_whitelist.join(", ")} merge, and no administrator merges past it` };
}
/** The gate's source as it is read: one key per trunk, each with the heads of the repositories installed from it. */
export type SourceRead = () => Promise<Record<string, unknown>>;
/** The state the trunks are read from: mesh-delivery-gate's source, whose key `main` is today's one trunk of each
* repository it names (each repository's trunk head the checker judges on); a later trunk is a key of its own. */
export const TRUNK_SOURCE = "mesh-delivery-gate.source";
/** A repository's trunks, the branches the mesh installs from (the operator, 2026-10-28 16:5x): each key of the gate's
* source whose heads name the repository. "main" only when the source names none of its trunks or cannot be read, and
* `from` says which. The one place trunks are read: several trunks later change only this. */
export async function trunksOf(owner: string, repo: string, read: SourceRead): Promise<{ names: string[]; from: string }> {
const full = `${owner}/${repo}`;
let src: Record<string, unknown>;
try {
src = await read();
} catch (err) {
return { names: [MAIN_BRANCH], from: `${TRUNK_SOURCE} could not be read (${err instanceof Error ? err.message : String(err)}): ${MAIN_BRANCH} assumed` };
}
const names = Object.keys(src).filter((k) => {
const heads = (src[k] as { heads?: Record<string, unknown> } | null)?.heads;
return !!heads && typeof heads === "object" && full in heads;
}).sort();
if (names.length) return { names, from: TRUNK_SOURCE };
return { names: [MAIN_BRANCH], from: `${TRUNK_SOURCE} names no trunk of ${full}: ${MAIN_BRANCH} assumed` };
}
/** The gate's source read from the bus, every key and its value (the tools bundle's state channel). */
export function sourceFromBus(get: (name: string) => { keys(): Promise<string[]>; get(k: string): Promise<{ value: unknown } | null> }): SourceRead {
return async () => {
const st = get(TRUNK_SOURCE);
const out: Record<string, unknown> = {};
for (const k of await st.keys()) out[k] = (await st.get(k))?.value ?? null;
return out;
};
}
/** Why a call is the operator's, or null: it lets a force-push onto a rule covering a trunk (the operator, 2026-10-28
* 16:4x: a call waits for their Approve only if it removes branch protection or allows a force-push to a trunk). No
* tool of this module can ask for that Approve yet, so such a call is refused with nothing sent. */
export function forcePushToTrunk(rule: string, body: Record<string, unknown>, trunks: string[]): string | null {
const covered = trunks.filter((t) => ruleCovers(rule, t));
if (!covered.length) return null;
const opens = body.enable_force_push === true || body.force_push_allowlist_deploy_keys === true ||
(["force_push_allowlist_usernames", "force_push_allowlist_teams"] as const).some((k) => Array.isArray(body[k]) && (body[k] as unknown[]).length > 0);
if (!opens) return null;
return `the rule ${rule} covers ${covered.join(", ")}, and this call lets a force-push onto it: a force-push to a trunk is ` +
`the operator's Approve (the operator, 2026-10-28), and no tool of this module can ask it yet, so it is the operator's ` +
`at the forge. Nothing was sent`;
}
+8 -2
View File
@@ -141,13 +141,19 @@ export function planText(plan: ChangePlan): string {
return lines.join("\n");
}
/** The heads already announced, keyed `owner/repo#number`, so a restart announces nothing twice. */
/** The heads already announced, keyed by `headKey`, so a restart announces nothing twice. */
export type Announced = Record<string, string>;
/** The key a pull request's announced head is kept under: `owner/repo#number@base`. The base is in it because a pull
* request retargeted keeps its head, and a head judged into one base is not judged into another (issue 1422, de-spiegel #6). */
export function headKey(full: string, number: number, base?: string): string {
return `${full}#${number}@${base ?? ""}`;
}
/** Which open pull requests have a head not yet announced. A pull request merged or closed is not
* open and is never asked about. */
export function headsToAnnounce(full: string, pulls: GiteaPull[], announced: Announced): GiteaPull[] {
return pulls.filter((p) => p.state === "open" && !!p.head_sha && announced[`${full}#${p.number}`] !== p.head_sha);
return pulls.filter((p) => p.state === "open" && !!p.head_sha && announced[headKey(full, p.number, p.base)] !== p.head_sha);
}
/** A pull request the forge found updated, by its repository: what `searchOpenPulls` answers. */
+1 -15
View File
@@ -183,15 +183,6 @@ test("named paths: each asked path's git object id at the commit, a file's or a
});
assert.deepEqual(got.trees, { "": ROOT, "modules/a": h("4"), "modules/b/c": h("8") });
assert.equal((await client.commitTrees("novox", "hq", COMMIT)).paths, undefined, "no paths asked, none answered");
// Listed directories (ADR 0549 §2, issue 1126): the files directly under each, not deeper; a missing one lists none.
const listed = await client.commitTrees("novox", "hq", COMMIT, undefined, ["modules/a", "modules/b", "", "placement"]);
assert.deepEqual(listed.listed, {
"modules/a": { "main.go": h("6"), "module.json": h("5") },
"modules/b": {},
"": { "README.md": h("1"), "module.json": h("2") },
placement: {},
});
assert.equal((await client.commitTrees("novox", "hq", COMMIT)).listed, undefined, "no directory asked, none listed");
} finally {
server.close();
}
@@ -218,12 +209,7 @@ test("gitea_commit_trees takes paths as {directory: [paths]} and refuses another
const { client, server, asked } = await serve([{ sha: COMMIT, url: "u", tree: LISTING, truncated: false, page: 1, total_count: LISTING.length }]);
try {
const tool = getGiteaTools(client).find((t) => t.name === "gitea_commit_trees")!;
assert.ok("paths" in tool.input && "listed" in tool.input, "the tool names the arguments");
const listedAnswer: any = await tool.run({ owner: "novox", repo: "hq", sha: COMMIT, listed: ["modules/d"] } as any);
assert.deepEqual(listedAnswer.listed, { "modules/d": { "notes.md": h("b") } });
for (const bad of ["modules/d", ["../x"], ["/abs"], [1]]) {
await assert.rejects(tool.run({ owner: "novox", repo: "hq", sha: COMMIT, listed: bad } as any), /listed/, JSON.stringify(bad));
}
assert.ok("paths" in tool.input, "the tool names the arguments");
const got: any = await tool.run({ owner: "novox", repo: "hq", sha: COMMIT, paths: { "modules/a": ["main.go"] } } as any);
assert.deepEqual(got.paths, { "modules/a": { "main.go": h("6") } });
const before = asked.length;
+67 -4
View File
@@ -34,6 +34,12 @@ const state = {
// refuses it the rules' list, as gitea's reqAdmin does, and it reads a branch's protection from the branch.
admin: false,
lastCommit: "2026-01-01T00:00:00Z",
// The repository's default branch and pull request 7's base, as the forge keeps them (issue 1422 note 7).
defaultBranch: "main",
base: "main",
prState: "open",
// Branches the forge answers 404 for; "scratch" exists but no protection rule guards it.
absent: ["nope"],
// The pull request's head; the head the merge commit joined (the same, unless a test says otherwise) — the forge's
// record of the pull request keeps saying `head` either way, as it would in the race issue 381 is about; and the
// merge commit's own sha.
@@ -66,7 +72,10 @@ beforeEach(() => {
state.statuses = judgedAt(TRUNK);
state.trunk = TRUNK;
state.emitted = [];
state.rules = [protectedMain()];
state.rules = [protectedMain(), { rule_name: "release/*", enable_push: false }];
state.defaultBranch = "main";
state.base = "main";
state.prState = "open";
state.admin = false;
state.head = REVIEWED;
state.landed = "";
@@ -98,12 +107,22 @@ const server = createServer(async (req, res) => {
state.asked.push(`${method} ${path}${url.search}`);
if (method !== "GET") state.sent.push({ method, path, body });
const repo = "/repos/novox/mesh-catalog";
if (method === "GET" && path === repo) return json(res, 200, { full_name: "novox/mesh-catalog", default_branch: "main" });
if (method === "GET" && path === repo) return json(res, 200, { full_name: "novox/mesh-catalog", default_branch: state.defaultBranch });
if (method === "PATCH" && path === repo) {
if (body?.default_branch) state.defaultBranch = body.default_branch;
return json(res, 200, { full_name: "novox/mesh-catalog", default_branch: state.defaultBranch });
}
if (method === "PATCH" && path === `${repo}/pulls/7`) {
if (body?.base) state.base = body.base;
return json(res, 201, { number: 7, title: "x", state: state.prState, merged: false, head: { ref: "feat/x", sha: state.head },
base: { ref: state.base }, html_url: "http://fake/7" });
}
if (path.startsWith(`${repo}/branch_protections`) && !state.admin) {
return json(res, 403, { message: "user should be an owner or a collaborator with admin write of a repository" });
}
if (method === "GET" && path === `${repo}/branch_protections`) return json(res, 200, state.rules);
const branch = method === "GET" && path.startsWith(`${repo}/branches/`) ? decodeURIComponent(path.slice(`${repo}/branches/`.length)) : null;
if (branch !== null && state.absent.includes(branch)) return json(res, 404, { message: "branch does not exist" });
if (branch !== null) {
// As the forge says it to any reader of the code: the rule that covers the branch, as the forge matches it.
const rule = (state.rules as any[]).find((r) => ruleCovers(String(r.rule_name ?? r.branch_name), branch));
@@ -125,8 +144,8 @@ const server = createServer(async (req, res) => {
return json(res, 200, [{ commit: { committer: { date: state.lastCommit } } }]);
}
if (method === "GET" && path === `${repo}/pulls/7`) {
return json(res, 200, { number: 7, title: "x", state: state.merged ? "closed" : "open", merged: state.merged,
head: { ref: "feat/x", sha: state.head }, base: { ref: "main" }, html_url: "http://fake/7",
return json(res, 200, { number: 7, title: "x", state: state.merged ? "closed" : state.prState, merged: state.merged,
head: { ref: "feat/x", sha: state.head }, base: { ref: state.base }, html_url: "http://fake/7",
merge_commit_sha: state.merged ? state.mergeCommit : null });
}
if (method === "GET" && path === `${repo}/git/commits/${state.mergeCommit}`) {
@@ -809,3 +828,47 @@ test("gitea_merge_pull_request judges a main requiring mesh/repo-check alone on
state.statuses = [judgedAt(TRUNK)[1]];
assert.equal(((await merge()) as Record<string, unknown>).merged, true);
});
// Issue 1422 note 7: the two settings gitea_api refuses (ADR 0266) have named tools, held to the same guard as
// gitea_branch_protection_set: the tools' account, a stated reason, a branch that exists and that a rule guards.
test("gitea_default_branch_set answers the default before and after, and refuses a branch that is missing, unguarded or unexplained", async () => {
const set = (args: Record<string, unknown>) =>
tool("gitea_default_branch_set").run({ owner: "novox", repo: "mesh-catalog", why: "the trunk was recorded as unknown", ...args });
await assert.rejects(set({ branch: "nope" }), /no branch nope/);
await assert.rejects(set({ branch: "scratch" }), /no protection rule/);
await assert.rejects(set({ branch: "release/1", why: " " }), /why/);
await assert.rejects(set({ branch: ".." }), /dot segment|not a branch/);
assert.deepEqual(writes(), []);
assert.deepEqual(await set({ branch: "release/1" }), { branch: "release/1", before: "main", after: "release/1", changed: true,
why: "the trunk was recorded as unknown" });
assert.deepEqual(writes(), ["PATCH /repos/novox/mesh-catalog"]);
assert.deepEqual(state.sent[0].body, { default_branch: "release/1" });
assert.equal(state.defaultBranch, "release/1");
state.sent = [];
assert.equal(((await set({ branch: "release/1" })) as any).changed, false, "already the default: nothing is sent");
assert.deepEqual(writes(), []);
});
test("gitea_pull_base_set answers the base before and after, and refuses a base that is missing, a closed pull request or no reason", async () => {
const set = (args: Record<string, unknown>) =>
tool("gitea_pull_base_set").run({ owner: "novox", repo: "mesh-catalog", number: 7, why: "it was opened against the wrong trunk", ...args });
await assert.rejects(set({ base: "nope" }), /no branch nope/);
await assert.rejects(set({ base: "release/1", why: "" }), /why/);
await assert.rejects(set({ base: "release/1", number: 0 }), /pull request's number/);
state.prState = "closed";
await assert.rejects(set({ base: "release/1" }), /not open/);
state.prState = "open";
assert.deepEqual(writes(), []);
assert.deepEqual(await set({ base: "release/1" }), { number: 7, before: "main", after: "release/1", changed: true,
why: "it was opened against the wrong trunk" });
assert.deepEqual(writes(), ["PATCH /repos/novox/mesh-catalog/pulls/7"]);
assert.deepEqual(state.sent[0].body, { base: "release/1" });
assert.equal(state.base, "release/1");
});
test("the two new tools are named for what mesh_search is asked, and gitea_api still refuses both settings", () => {
assert.match(tool("gitea_default_branch_set").description, /default branch/i);
assert.match(tool("gitea_pull_base_set").description, /pull request.s base/i);
assert.ok("refused" in judgeApiCall("PATCH", "/repos/a/b", { default_branch: "x" }));
assert.ok("refused" in judgeApiCall("PATCH", "/repos/a/b/pulls/7", { base: "x" }));
});
+2 -2
View File
@@ -45,7 +45,7 @@ test("a new head beyond the newest 20 open pull requests is announced, exactly o
const { headsToAnnounce } = await import("../pulls.ts");
// 25 open, every head announced before; #3 — the 23rd most recently updated — got a new head.
const announced: Record<string, string> = {};
for (let n = 1; n <= 25; n++) announced[`novox/hq#${n}`] = sha(n, "a");
for (let n = 1; n <= 25; n++) announced[`novox/hq#${n}@main`] = sha(n, "a");
const { client, server, asked } = await serve(25, { 3: sha(3, "b") });
try {
// The fault: the newest 20 alone do not hold it.
@@ -58,7 +58,7 @@ test("a new head beyond the newest 20 open pull requests is announced, exactly o
const open = await client.listOpenPullsSince("novox", "hq", "");
for (const p of headsToAnnounce("novox/hq", open, announced)) {
said.push(`#${p.number}@${p.head_sha}`);
announced[`novox/hq#${p.number}`] = String(p.head_sha);
announced[`novox/hq#${p.number}@${p.base}`] = String(p.head_sha);
}
}
assert.deepEqual(said, [`#3@${sha(3, "b")}`], "the new head past the newest 20, announced once over two passes");
+250
View File
@@ -0,0 +1,250 @@
// gitea_branch_protection_set_full (the operator, 2026-10-28: "just make a generic tool so you can setup any
// configuration"): any field of a branch protection rule, only the fields given, as the forge's administrator; it
// answers the rule before and after, and whether the controller's trunk rule (judgedRule, mesh-control
// build_source.go) now holds. Against a fake forge that keeps the rules and answers them to the administrator alone.
import { test, after, beforeEach } from "node:test";
import assert from "node:assert/strict";
import { createServer } from "node:http";
import { GiteaAdmin, GiteaClient } from "../dist/client.js";
import { ConfiguredToken } from "../dist/token.js";
import { getGiteaTools } from "../dist/tools/index.js";
import { trunkRule } from "../dist/protectfull.js";
const adminBasic = "Basic " + Buffer.from("mesh-admin:secret").toString("base64");
let rules: any[] = [];
let repoState: any = {};
const branches = ["master", "main"];
let sent: { method: string; path: string; body: any }[] = [];
beforeEach(() => {
// de-spiegel's main as found: no push, but the operator alone merges and an administrator merges past it.
rules = [{ rule_name: "main", enable_push: false, enable_merge_whitelist: true, merge_whitelist_usernames: ["jschoubben"],
enable_status_check: false, status_check_contexts: [], block_admin_merge_override: false, required_approvals: 0,
enable_push_whitelist: false, push_whitelist_deploy_keys: false, enable_force_push: false, enable_force_push_allowlist: false,
force_push_allowlist_deploy_keys: false, merge_whitelist_teams: [] }];
source = {};
repoState = { full_name: "novox/de-spiegel", default_branch: "master", description: "", has_wiki: true };
sent = [];
});
// What gitea 1.27.3's EditBranchProtection does with a body (routers/api/v1/repo/branch.go): the push whitelist flags are
// read only beside enable_push true and cleared by enable_push false, the force-push allowlist flags likewise with
// enable_force_push; teams are passed over on a repository no organisation owns (de-spiegel here, for the test).
function giteaEdit(r: any, body: any) {
const nested = ["enable_push", "enable_push_whitelist", "push_whitelist_deploy_keys", "enable_force_push",
"enable_force_push_allowlist", "force_push_allowlist_deploy_keys", "merge_whitelist_teams", "push_whitelist_teams"];
for (const [k, v] of Object.entries(body)) if (!nested.includes(k) && k !== "rule_name") r[k] = v;
if (body.enable_push === false) Object.assign(r, { enable_push: false, enable_push_whitelist: false, push_whitelist_deploy_keys: false });
if (body.enable_push === true) {
r.enable_push = true;
if (body.enable_push_whitelist !== undefined) r.enable_push_whitelist = body.enable_push_whitelist;
if (body.push_whitelist_deploy_keys !== undefined) r.push_whitelist_deploy_keys = body.push_whitelist_deploy_keys;
}
if (body.enable_force_push === false) Object.assign(r, { enable_force_push: false, enable_force_push_allowlist: false, force_push_allowlist_deploy_keys: false });
if (body.enable_force_push === true) {
r.enable_force_push = true;
if (body.enable_force_push_allowlist !== undefined) r.enable_force_push_allowlist = body.enable_force_push_allowlist;
if (body.force_push_allowlist_deploy_keys !== undefined) r.force_push_allowlist_deploy_keys = body.force_push_allowlist_deploy_keys;
}
}
const server = createServer((req, res) => {
const path = (req.url ?? "").replace(/^\/api\/v1/, "");
const send = (status: number, body: unknown) => { res.writeHead(status, { "Content-Type": "application/json" }); res.end(JSON.stringify(body)); };
if (req.headers.authorization !== adminBasic) return send(403, { message: "not an administrator of the repository" });
let raw = "";
req.on("data", (c) => (raw += c));
req.on("end", () => {
const body = raw ? JSON.parse(raw) : undefined;
if (req.method !== "GET") sent.push({ method: req.method!, path, body });
if (path === "/repos/novox/de-spiegel") {
if (req.method === "GET") return send(200, repoState);
if (req.method === "PATCH") {
// gitea's updateBasicProperties: a default branch that does not exist is passed over, the call still 200.
// enable_prune is a mirror's, kept beside the repository and never in its answer; mirror_interval is answered in Go's
// duration form.
const { default_branch, enable_prune, mirror_interval, ...rest } = body;
Object.assign(repoState, rest);
if (mirror_interval !== undefined) repoState.mirror_interval = mirror_interval === "8h" ? "8h0m0s" : mirror_interval;
if (default_branch !== undefined && branches.includes(default_branch)) repoState.default_branch = default_branch;
return send(200, repoState);
}
}
const m = path.match(/^\/repos\/novox\/de-spiegel\/branch_protections(?:\/(.+))?$/);
if (!m) return send(404, { message: "not found" });
if (req.method === "GET" && !m[1]) return send(200, rules);
if (req.method === "PATCH" && m[1]) {
const r = rules.find((x) => x.rule_name === decodeURIComponent(m[1]));
if (!r) return send(404, { message: "no rule" });
giteaEdit(r, body);
return send(200, r);
}
if (req.method === "POST" && !m[1]) { const r = { rule_name: body.rule_name }; giteaEdit(r, body); rules.push(r); return send(201, r); }
send(405, { message: "no" });
});
});
await new Promise<void>((r) => server.listen(0, "127.0.0.1", () => r()));
after(() => new Promise<void>((r) => server.close(() => r())));
const { port } = server.address() as { port: number };
const url = `http://127.0.0.1:${port}`;
// The gate's source as the tools read it (mesh-delivery-gate.source): one key per trunk, each naming the repositories
// installed from it. A test sets it; empty is a source that names no trunk of de-spiegel.
let source: Record<string, { heads: Record<string, string> }> = {};
const readSource = async () => source;
const tools = getGiteaTools(new GiteaClient(url, new ConfiguredToken("tools-token")), () => new GiteaAdmin(url, "mesh-admin", "secret"), readSource);
const setRepo = (args: Record<string, unknown>) =>
tools.find((t) => t.name === "gitea_repo_settings_set_full")!.run({ owner: "novox", repo: "de-spiegel", why: "L2", ...args });
const setFull = (args: Record<string, unknown>) =>
tools.find((t) => t.name === "gitea_branch_protection_set_full")!.run({ owner: "novox", repo: "de-spiegel", branch: "main", why: "L2", ...args });
test("it sets the merge whitelist and the administrator's override, only the fields given, and says the trunk rule now holds", async () => {
const out: any = await setFull({ merge_whitelist_usernames: ["mesh-tools", "jschoubben"], block_admin_merge_override: true });
assert.deepEqual(sent, [{ method: "PATCH", path: "/repos/novox/de-spiegel/branch_protections/main",
body: { merge_whitelist_usernames: ["mesh-tools", "jschoubben"], block_admin_merge_override: true } }]);
assert.deepEqual(out.before.merge_whitelist, ["jschoubben"]);
assert.equal(out.before.admin_may_override, true);
assert.deepEqual(out.after.merge_whitelist, ["mesh-tools", "jschoubben"]);
assert.equal(out.after.admin_may_override, false);
assert.equal(out.why, "L2");
assert.equal(out.trunk_rule.holds, true, out.trunk_rule.why);
});
test("it loosens as told too, and says the trunk rule no longer holds and why", async () => {
const out: any = await setFull({ enable_push: true, status_check_contexts: ["mesh/check"], enable_status_check: true, required_approvals: 1 });
assert.equal(out.after.push, true);
assert.deepEqual(out.after.required_statuses, ["mesh/check"]);
assert.equal(out.after.required_approvals, 1);
assert.equal(out.trunk_rule.holds, false);
assert.match(out.trunk_rule.why, /push to main directly/);
});
test("a branch with no rule gets one with the fields given", async () => {
const out: any = await setFull({ branch: "release/*", enable_push: false, protected_file_patterns: "*.lock" });
assert.equal(out.created, true);
assert.deepEqual(sent[0], { method: "POST", path: "/repos/novox/de-spiegel/branch_protections",
body: { rule_name: "release/*", enable_push: false, protected_file_patterns: "*.lock" } });
});
test("an unknown field, a field of the wrong kind, no field, no reason or a login that is none: refused, nothing sent", async () => {
await assert.rejects(setFull({ enable_pushes: true }), /no field of a branch protection rule/);
await assert.rejects(setFull({ enable_push: "yes" }), /true or false/);
await assert.rejects(setFull({ required_approvals: -1 }), /whole number/);
await assert.rejects(setFull({ merge_whitelist_usernames: ["../x"] }), /no forge login/);
await assert.rejects(setFull({}), /at least one field/);
await assert.rejects(setFull({ why: "", enable_push: false }), /say why/);
assert.deepEqual(sent, []);
});
// judgedRule's cases (mesh-control cmd/mesh-controller/build_source.go), each in its own words.
test("the trunk rule is judged as the controller judges it", () => {
const ok = { rule_name: "main", enable_push: false, enable_merge_whitelist: true, merge_whitelist_usernames: ["mesh-tools"], block_admin_merge_override: true };
const cases: [any[], string, boolean, RegExp][] = [
[[], "main", false, /no protection rule covers main/],
[[{ ...ok, enable_push: true }], "main", false, /push to main directly/],
[[{ ...ok, enable_merge_whitelist: false }], "main", false, /no merge whitelist/],
[[{ ...ok, merge_whitelist_usernames: ["jschoubben"] }], "main", false, /and not mesh-tools/],
[[{ ...ok, merge_whitelist_usernames: ["mesh-tools", "agent-g14"] }], "main", false, /agent's account/],
[[{ ...ok, block_admin_merge_override: false }], "main", false, /administrator/],
[[ok], "main", true, /takes no push/],
[[{ ...ok, rule_name: "ma*" }], "main", true, /ma\*/],
[[{ ...ok, rule_name: "ma*", enable_push: true }, { ...ok }], "main", true, /rule main/],
];
for (const [rs, branch, holds, why] of cases) {
const got = trunkRule(rs, branch);
assert.equal(got.holds, holds, `${JSON.stringify(rs)}: ${got.why}`);
assert.match(got.why, why);
}
});
// Review round 1 of feat/protect-as-trunk: gitea passes over a dependent field without its parent set true, and answers 200.
test("a field the forge reads only beside its parent is refused without it; with it, it is applied", async () => {
await assert.rejects(setFull({ enable_push_whitelist: true }), /only when the same call sets enable_push: true/);
await assert.rejects(setFull({ force_push_allowlist_deploy_keys: true, enable_force_push: false }), /enable_force_push: true/);
assert.deepEqual(sent, []);
const out: any = await setFull({ enable_push: true, enable_push_whitelist: true, push_whitelist_usernames: ["jschoubben"] });
assert.equal(out.after.fields.enable_push_whitelist, true);
await setFull({ enable_push: false, enable_push_whitelist: false }); // a false beside a false parent is what the forge does
});
test("a field the forge answered and did not apply fails the call, naming it; what it changed beside the call is said", async () => {
await assert.rejects(setFull({ merge_whitelist_teams: ["owners"] }), /did not apply merge_whitelist_teams \(sent \["owners"\], the forge holds \[\]\)/);
rules[0].enable_push = true; rules[0].enable_push_whitelist = true;
const out: any = await setFull({ enable_push: false });
assert.deepEqual(out.also_changed, ["enable_push_whitelist"]);
});
test("gitea_repo_settings_set_full sets the default branch, and fails loudly on a branch the forge passed over", async () => {
const out: any = await setRepo({ default_branch: "main" });
assert.deepEqual(sent.at(-1), { method: "PATCH", path: "/repos/novox/de-spiegel", body: { default_branch: "main" } });
assert.equal(out.before.default_branch, "master");
assert.equal(out.after.default_branch, "main");
await assert.rejects(setRepo({ default_branch: "trunk" }), /did not apply default_branch/);
await assert.rejects(setRepo({ name: "x" }), /no field of a repository's settings/);
await assert.rejects(setRepo({ why: "", has_wiki: false }), /say why/);
});
test("every field gitea 1.27.3 edits on a rule is taken: 32, the bypass allowlist among them", async () => {
const { PROTECTION_FIELDS } = await import("../dist/protectfull.js");
assert.equal(Object.keys(PROTECTION_FIELDS).length, 32);
for (const f of ["enable_bypass_allowlist", "bypass_allowlist_usernames", "bypass_allowlist_teams"]) assert.ok(f in PROTECTION_FIELDS, f);
});
// Review round 2: the settings tool's read-back meets the forge's own answer.
test("A: enable_prune, which the forge never answers, is said as not read back, never failed", async () => {
const pruned: any = await setRepo({ enable_prune: true });
assert.deepEqual(pruned.not_read_back, ["enable_prune"]);
});
test("B: mirror_interval is compared as a duration: 8h and the forge's 8h0m0s are one", async () => {
const mirror: any = await setRepo({ mirror_interval: "8h" });
assert.equal(mirror.after.mirror_interval, "8h0m0s");
await assert.rejects(setRepo({ mirror_interval: "often" }), /duration/);
});
test("C: allow_merge_update, default_update_style and has_code are taken; projects_mode needs has_projects; what is left out is named", async () => {
const more: any = await setRepo({ allow_merge_update: true, default_update_style: "rebase", has_code: true });
assert.deepEqual(more.set, { allow_merge_update: true, default_update_style: "rebase", has_code: true });
await assert.rejects(setRepo({ projects_mode: "repo" }), /only when the same call sets has_projects: true/);
await setRepo({ has_projects: true, projects_mode: "repo" });
const desc = tools.find((t) => t.name === "gitea_repo_settings_set_full")!.description;
for (const w of ["name", "mirror_username", "mirror_password", "mirror_token", "tracker", "wiki"]) assert.match(desc, new RegExp(w));
});
// The operator, 2026-10-28 16:4x: a call waits for their Approve only if it removes branch protection or allows a
// force-push to main. No tool of this module can ask yet (no TypeScript asker; ADR 0286's mark is unbuilt in the
// runner), so until one can, such a call is refused, with nothing sent, and says it is the operator's; everything else
// runs at once.
test("a force-push onto a rule covering main is refused, nothing sent; on another branch, and tightening, it runs", async () => {
for (const args of [{ enable_force_push: true }, { enable_force_push: true, enable_force_push_allowlist: true },
{ force_push_allowlist_usernames: ["jschoubben"] }, { force_push_allowlist_teams: ["owners"] },
{ branch: "ma*", enable_force_push: true }, { branch: "**", force_push_allowlist_usernames: ["x"] }]) {
await assert.rejects(setFull(args), /force-push to a trunk is the operator's/, JSON.stringify(args));
}
assert.deepEqual(sent, []);
await setFull({ branch: "release/*", enable_force_push: true });
const tight: any = await setFull({ enable_force_push: false, force_push_allowlist_usernames: [] });
assert.equal(tight.trunk_rule.holds, false, "trunk_rule is in the answer");
});
// The operator, 2026-10-28 16:5x: the gates and trunk_rule apply to every trunk, the branches the mesh installs from, as
// the gate's source names them; "main" only when it names none, and said.
test("a trunk the gate's source names, beta, is gated and judged as main is; main is then no trunk of the repository", async () => {
source = { main: { heads: { "novox/okioki": "a1" } }, beta: { heads: { "novox/de-spiegel": "b2" } } };
await assert.rejects(setFull({ branch: "beta", enable_force_push: true }), /force-push to a trunk is the operator's/);
await assert.rejects(setFull({ branch: "b*", force_push_allowlist_usernames: ["x"] }), /covers beta/);
assert.deepEqual(sent, []);
const main: any = await setFull({ enable_force_push: true });
assert.deepEqual(main.trunks, { names: ["beta"], from: "mesh-delivery-gate.source" });
assert.equal(main.trunk_rule.is_trunk, false);
const beta: any = await setFull({ branch: "beta", enable_push: false, enable_merge_whitelist: true, merge_whitelist_usernames: ["mesh-tools"], block_admin_merge_override: true });
assert.equal(beta.trunk_rule.is_trunk, true);
assert.equal(beta.trunk_rule.holds, true, beta.trunk_rule.why);
});
test("a source that names no trunk of the repository falls back to main, and says so", async () => {
const out: any = await setFull({ block_admin_merge_override: true });
assert.equal(out.trunks.names[0], "main");
assert.match(out.trunks.from, /names no trunk of novox\/de-spiegel: main assumed/);
assert.equal(out.trunk_rule.is_trunk, true);
});
+13 -2
View File
@@ -8,16 +8,27 @@ import { createServer } from "node:http";
test("each open pull request's new head is announced once, and nothing closed or merged", async () => {
const { headsToAnnounce } = await import("../pulls.ts");
const pulls = [
{ number: 1, title: "a", state: "open", merged: false, head_sha: "aaa", html_url: "" },
{ number: 1, title: "a", state: "open", merged: false, head_sha: "aaa", base: "main", html_url: "" },
{ number: 2, title: "b", state: "open", merged: false, head_sha: "bbb", html_url: "" },
{ number: 3, title: "c", state: "closed", merged: true, head_sha: "ccc", html_url: "" },
{ number: 4, title: "d", state: "open", merged: false, html_url: "" },
];
const announced = { "novox/mesh-catalog#1": "aaa", "novox/mesh-catalog#2": "old" };
const announced = { "novox/mesh-catalog#1@main": "aaa", "novox/mesh-catalog#2@": "old" };
assert.deepEqual(headsToAnnounce("novox/mesh-catalog", pulls, announced).map((p) => p.number), [2],
"only a head not announced, of a pull request that is open");
});
test("a pull request retargeted keeps its head and is announced again, into its new base (de-spiegel #6)", async () => {
const { headsToAnnounce, headKey } = await import("../pulls.ts");
const was = { number: 6, title: "a", state: "open", merged: false, head_sha: "aaa", base: "old-trunk", html_url: "" };
const announced: Record<string, string> = { [headKey("novox/de-spiegel", 6, "old-trunk")]: "aaa" };
assert.deepEqual(headsToAnnounce("novox/de-spiegel", [was], announced), [], "the same head into the same base is announced once");
const retargeted = { ...was, base: "main" };
assert.deepEqual(headsToAnnounce("novox/de-spiegel", [retargeted], announced).map((p) => p.number), [6],
"the same head into a new base is judged afresh");
assert.notEqual(headKey("novox/de-spiegel", 6, "main"), headKey("novox/de-spiegel", 6, "old-trunk"));
});
test("a verdict is the head commit's status; an error is the forge's error, never a success", async () => {
const { statusFor, commentFor, CHECK_CONTEXT } = await import("../pulls.ts");
const base = { owner: "novox", repo: "mesh-catalog", number: 7, commit: "0123456789abcdef", id: "build-1", on: "laptop" };
+2 -1
View File
@@ -538,7 +538,8 @@ test("the tools register once there is a way to a token, and the first call mint
"gitea_get_file",
"gitea_list_branches",
"gitea_delete_branch",
"gitea_branch_protection_get", "gitea_team_rights", "gitea_branch_protection_set",
"gitea_branch_protection_get", "gitea_team_rights", "gitea_branch_protection_set", "gitea_branch_protection_set_full", "gitea_repo_settings_set_full",
"gitea_default_branch_set", "gitea_pull_base_set",
"gitea_note_append", "gitea_delivery_view", "gitea_commit_statuses", "gitea_open_pulls", "gitea_commit_trees", "gitea_contains",
"gitea_list_labels", "gitea_create_label",
"gitea_api",
+73
View File
@@ -0,0 +1,73 @@
import assert from "node:assert/strict";
import { mock, test } from "node:test";
import type { GiteaPull } from "../client.ts";
// **No poll of the forge** (issue 1422, notes 10 and 11): the forge's events announce heads and merges; a pass runs once at
// the start and when the receiver listens.
const edited = (base: string) => ({ name: "forge.pull-request.edited", body: { owner: "novox", repo: "hq", number: 7, base, merged: false } });
function deps() {
const calls: string[] = [];
return { calls, d: {
getRepo: async () => ({ r: 1 }),
getPull: async () => ({ state: "open", base: "trunk" }) as any,
announceHead: async () => { calls.push("head"); },
announceMerge: async () => { calls.push("merge"); },
announceClosed: async () => { calls.push("closed"); },
siblingMoves: async () => { calls.push("siblings"); },
} };
}
test("an edited event announces the head into the base it now has, without a poll", async () => {
const { actOn } = await import("../watch.ts");
const { calls, d } = deps();
await actOn(edited("trunk"), d);
assert.deepEqual(calls, ["head"]);
});
test("a close without a merge is announced by its event, a merged close is left to the merge event", async () => {
const { actOn } = await import("../watch.ts");
const { calls, d } = deps();
await actOn({ name: "forge.pull-request.closed", body: { owner: "novox", repo: "hq", number: 7, merged: false } }, d);
assert.deepEqual(calls, ["closed", "siblings"]);
calls.length = 0;
await actOn({ name: "forge.pull-request.closed", body: { owner: "novox", repo: "hq", number: 7, merged: true } }, d);
assert.deepEqual(calls, []);
});
test("no timer polls after the start: one pass ends and nothing is scheduled; a failed one is asked again", async () => {
const { startPass } = await import("../watch.ts");
mock.timers.enable({ apis: ["setTimeout", "setInterval"] });
try {
let runs = 0;
await startPass(async () => { runs++; }, { retryMs: 60_000, log: () => {} });
mock.timers.tick(24 * 3600_000);
assert.equal(runs, 1, "a day later the forge was not asked again");
let tries = 0;
const retried = startPass(async () => { if (++tries < 2) throw new Error("forge down"); }, { retryMs: 60_000, log: () => {} });
await Promise.resolve(); await Promise.resolve();
mock.timers.tick(60_000);
await retried;
assert.equal(tries, 2);
} finally {
mock.timers.reset();
}
});
test("a start pass announces a head nobody announced, and only remembers one the forge shows a check of", async () => {
const { announceNewHeads } = await import("../watch.ts");
const pull = (n: number, sha: string): GiteaPull => ({ number: n, state: "open", head_sha: sha, base: "main",
updated_at: new Date().toISOString() } as unknown as GiteaPull);
const announced: number[] = [], said: string[] = [];
const heads: Record<string, string> = {};
await announceNewHeads("novox/hq", [pull(1, "a".repeat(40)), pull(2, "b".repeat(40))], heads, false,
new Date(Date.now() - 24 * 3600_000).toISOString(), {
alreadyChecked: async (p) => p.number === 2,
announceHead: async (p) => { announced.push(p.number); return true; },
missed: (w) => said.push(w),
});
assert.deepEqual(announced, [1]);
assert.equal(heads["novox/hq#2@main"], "b".repeat(40), "the checked head is remembered, not announced");
assert.equal(said.length, 1);
});

Some files were not shown because too many files have changed in this diff Show More