Compare commits

...
Author SHA1 Message Date
jschoubben c584dc57b9 gitea: mesh/check is the mesh's own status and never blocks; signed commits, an outdated branch and a rejected review are said as blocking the mesh's merge (issue 1945, review round 2) 2026-10-26 18:37:02 +01:00
jschoubben e1e5a8ff0d gitea: mesh-tools writes a user's delivered repository and main's rule stays the operator's: only its merge whitelist gets mesh-tools, read back; what else blocks the mesh's merge is said (issue 1945, the operator's answer 2026-10-26 18:30) 2026-10-26 18:34:03 +01:00
jschoubben c85472285d gitea: main's rule read back must say what was written, since the forge silently drops a whitelist login that cannot write (issue 1945, review round 1) 2026-10-26 18:23:42 +01:00
jschoubben f0fbe193fa gitea: mesh-tools administers a user's delivered repository and puts main's rule right, permissions read as the account sees them; knowledge-base.progress: minimal answers leave items to the tree, failed calls warn before parse faults (issues 1945, 1947) 2026-10-26 17:59:34 +01:00
mesh-tools 26908a8950 Merge pull request 'knowledge-base.progress: how far a design, milestone, issue or record has come, in one call (issue 1947)' (#504) from feat/1947-progress into main 2026-10-26 16:42:30 +00:00
mesh-tools 27a7757b97 Merge pull request 'The mesh writes the repositories it delivers, and asks no Approve for a merge it cannot carry out (issue 1945)' (#503) from fix/forge-writes-delivered-repos into main 2026-10-26 16:25:07 +00:00
jschoubben 87fe2c45cf Knowledge base: progress counts an issue done only once its fix is delivered, and says every gap (issue 1947, review round 1)
mesh/delivery merged: source.main moved to novox/mesh-catalog@26908a89506f
mesh/delivery-group group feat/1947-progress merged: every member is merged
mesh/check pass 87fe2c45cfed on 26908a89506f | carried from 27a7757b970e, hq 3a5c4c7c4d9c | unit 2091 run, 2327 kept, 6 skipped | integration 2112 run…
A resolved issue whose delivery is held or failed is stuck, one merged and not delivered
is moving; a row the tracker did not answer, an answer that does not read and linked
issues cut by the list's limit are said as warnings instead of passing silently.
2026-10-26 17:02:47 +01:00
jschoubben 24b244bc88 mesh-delivery-gate: ask no Approve of a merge the forge account cannot carry out, and say why (issue 1945)
mesh/delivery merged: source.main moved to novox/mesh-catalog@27a7757b970e
mesh/check pass 24b244bc88dc on 27a7757b970e | carried from 6b6d5cba7195 | unit 2046 run, 2327 kept, 6 skipped | integration 2066 run, 2346 kept, 10 s…
car-hunter #2 was approved and its commit status then refused 403. Before the ask the
gate reads merge.writes for each repository it covers; where the merging account cannot
write, nothing is asked, the delivery waits on 'the forge account cannot write <repo>:
gitea_grant_write' and a condition is raised. gitea.repo.writable reads it again and
asks an approved, refused merge of it again.
2026-10-26 16:39:56 +01:00
jschoubben 5baf8ad1d7 Knowledge base: progress is proven over the bus with the module started whole (issue 1947)
The verb asks three seats and the records module; only a run over a real bus shows its
grants and subjects line up with what the manifest invokes.
2026-10-26 16:39:19 +01:00
jschoubben 8da33e43d4 Knowledge base: progress says how far a design, milestone, issue or record has come (issue 1947)
A lead's five-minute report and 'is design 47 nearing its end' were joined by hand from
the design file, the tracker and the deliveries. One verb now works the state out
(ADR 0424 §1), minimal for every five minutes, detailed for the report's tree.
Milestones are read from the design's table until the tracker's milestone records
(issue 1781) are delivered.
2026-10-26 16:37:54 +01:00
jschoubben 52de4c320e gitea: the merging account writes every repository that builds a registered module, read from the indexer's modules (issue 1945)
car-hunter #2 was approved and its commit status refused 403: mesh-tools only read the
repositories of a user. It is now a write collaborator on each repository the indexer's
modules state names (watched over the bus: at the start and on each registration), read
back and said as gitea.repo.writable; the operator's other repositories stay read-only.
gitea_grant_write and gitea_unwritable_repos do and show it by hand; gitea_commit_statuses
answers merge.writes.
2026-10-26 16:36:50 +01:00
jschoubben 5e516317e9 Issue tracker: list rows say when each issue last moved, and list takes linked (issue 1947)
The knowledge base's progress verb reads a milestone's issues in one call and must say
when each last moved, and the issues linked to a record or a design; neither was
readable without one get per issue.
2026-10-26 16:29:56 +01:00
mesh-tools 6b6d5cba71 Merge pull request 'Image builds run their RUN steps on pasta, and every build failure keeps its real error (issue 1703)' (#502) from fix/1703-image-error-shown into main 2026-10-26 14:48:17 +00:00
mesh-tools 8393e29621 Merge pull request 'An approved group merges whole or not at all, and two groups never interleave in one repository (issue 1275)' (#501) from fix/1275-group-merge-whole into main 2026-10-26 14:47:34 +00:00
mesh-tools 61563abf2f Merge pull request 'Every forge account the mesh's parts use reads every repository (the operator, 2026-10-26)' (#500) from fix/fetch-reads-all into main 2026-10-26 14:47:29 +00:00
jschoubben 071c6885d9 The mesh's forge accounts all read every repository, and a build's fetch says the forge's own words
mesh/delivery merged: source.main moved to novox/mesh-catalog@61563abf2f93
mesh/delivery-group group fix/fetch-reads-all merged: every member is merged
mesh/check checking 071c6885d923 on 8393e2962193
mesh-catalog #495 gave read on every repository to the tools' account alone. The build modules
fetch over ssh as nox-builder (the forge-account provider's builders account, ADR 0466 §3), so
ace's and g14's fetch of jschoubben/car-hunter and jschoubben/snake still failed; the machines'
agents (nox-agent, ADR 0410 §1) could not read them either.

- gitea: readall.ts grants every account of one list, meshAccounts in tools-account.ts (mesh-tools,
  nox-builder, nox-agent; a Go test holds the two Go names to internal/forgeaccount's constants),
  on the same start pass, forge events and tools. gitea_unreadable_repos answers per account and
  names the accounts the forge does not hold yet; gitea_grant_read takes accounts as well as
  repositories. An absent account is asked nothing; the start pass runs again ten minutes apart
  until every account was there once.
- mesh-build-go, mesh-build-node, mesh-build-image: pin mesh-sdk ff81bdf8 (fix/fetch-reads-all),
  whose forgekey.Said keeps git's words past the bare "error:" line: the mirror's line said only
  "git -C: error:". Their own git reasons (ls-files, ls-remote, rev-parse, repack) go through it too.
2026-10-26 15:27:38 +01:00
jschoubben 8c5f27627b mesh-build-image: give a RUN step pasta's network, since the default netavark bridge cannot be set up rootless (issue 1703)
mesh/delivery merged: source.main moved to novox/mesh-catalog@6b6d5cba7195
mesh/check checking 8c5f27627b67 on 6b6d5cba7195
Every RUN failed before it started: 'setup network: netavark: nftables error ... Could not process rule'. Reproduced on
shanks with buildah 1.45.2, a clean environment and a private XDG_RUNTIME_DIR; --network pasta builds, and reaches the
network from a RUN. The sysrq integration row now fails if the RUN never started.
2026-10-26 15:23:19 +01:00
jschoubben 82bd6f49ff A half-merged group keeps its repositories until whole or declined, and the operator's word does not ask it again while another pass it would wait for is under way (issue 1275, review round 1)
mesh/delivery merged: source.main moved to novox/mesh-catalog@8393e2962193
mesh/check checking 82bd6f49ff9c on 8393e2962193
A held half-merged group did not count as under way, so another group sharing a repository could start; asked
again with delivery_merge_again, each then waited on the other. Passes are now read from every approved,
undeclined Approve not merged whole (approvedMerges leaves out merged and refused members), and
delivery_merge_again is refused while passWaits names a pass, naming it. The forge's mergeable is also false
for a work in progress: said so.
2026-10-26 15:23:01 +01:00
mesh-tools ceb3aac9b3 Merge pull request 'Build modules size their steps from the node's build share once it is given (issue 1806, change A)' (#499) from fix/build-cpu-yields into main 2026-10-26 14:22:50 +00:00
jschoubben 98c66fb1e4 Build modules: a failed command's reason says its error, not its last progress line (issue 1703)
buildah writes STEP lines on stdout and its error on stderr, so the last line of both hid every image build's cause.
mesh-build-image's reason is now stderr's last lines (600 bytes) with the exit status, and the journal gets both tails;
mesh-build-node keeps stderr's end and stdout's start apart; mesh-build-go's cause skips go build's '# package' line.
2026-10-26 15:21:01 +01:00
jschoubben 07ec186031 Merge an approved group only when every member merges now, and let no other merge land between its members (issue 1275)
On 2026-10-26 13:40 group fix/734-lost-ask merged two of its three members: the forge answered mesh-delivery#73's
merge 405 "Please try again later", and in the seconds between, group feat/890-placement merged mesh-delivery#72,
so #73 then conflicted.

- mesh-delivery-gate: before any member is handed, every member is read for its required statuses, its head still
  the approved one, and the forge's own mergeable (in conflict, or still being worked out); the forge's "no" is asked
  again every 5 s for 2 minutes, then the group is held and said urgent with nothing merged.
- mesh-delivery-gate: an approved merge waits while another approved merge's pass is under way into a repository it
  goes into (handed until merged whole, or stopped), whether or not mesh/check gates merges (it waits for a merge,
  never for a check: ADR 0551).
- mesh-delivery-gate: tool delivery_merge_again asks a refused or held merge again on the operator's word.
- gitea: the merge asks again after the forge's 405 "Please try again later" (1 s, 2 s, 4 s), and a pull request
  carries the forge's mergeable, which gitea_commit_statuses answers.
2026-10-26 14:53:22 +01:00
jschoubben 2aecbe8055 Merge main into fix/build-cpu-yields; pin mesh-sdk at its head (ADR 0553, ADR 0554)
mesh/delivery merged: source.main moved to novox/mesh-catalog@ceb3aac9b39a
mesh/delivery-group group fix/build-cpu-yields merged: every member is merged
mesh/check being carried to 2aecbe80553c on ceb3aac9b39a | from 9a8e54d988fe, mesh-host bba9081ca127
2026-10-26 14:47:27 +01:00
jschoubben 3def293d94 Build modules: no thread limit without the composer's share; with it, the share over their own capacity (ADR 0553 round 2)
Change A rebuilds the three build modules, so their runtimes restart once; it
changes nothing else about their builds until change B gives them the share.
2026-10-26 14:41:12 +01:00
mesh-tools 9a8e54d988 Merge pull request 'mesh-build-go says which packages it cannot test here, and mesh-build-node's tests need the Node toolchain (issue 890, ADR 0552)' (#498) from feat/890-placement into main 2026-10-26 13:40:41 +00:00
mesh-tools 2da9329df4 Merge pull request 'pacman: the round and the history tools say in plain words what they answer' (#497) from fix/pacman-findable into main 2026-10-26 13:39:16 +00:00
jschoubben ecbee53cfd Merge main into fix/build-cpu-yields; pin mesh-sdk at its head (ADR 0553) 2026-10-26 14:23:33 +01:00
jschoubben dcdf4238c7 Build modules: no cap of their own; a step's threads are its part of the node's one share (ADR 0553, issue 1806)
Review round 1: a cap per module let three build modules take three shares
of the node. The share is now one node value held by the node-engine's
yielding slice; each step divides it by every build slot of the node, from the
composer's words, or assumes every build seat at its own capacity without them.
2026-10-26 14:22:05 +01:00
jschoubben d3423f54ed mesh-build-go: resolve the go.mod and go.sum the main merge left in conflict; pin mesh-sdk 33395f68 (issue 890)
mesh/delivery merged: source.main moved to novox/mesh-catalog@9a8e54d988fe
mesh/delivery-group group feat/890-placement merged: every member is merged
mesh/check checking d3423f54ed58 on 9a8e54d988fe | novox/mesh-catalog#498 in group feat/890-placement
2026-10-26 14:06:37 +01:00
jschoubben 0e322d087f mesh-build-go: pin mesh-sdk 33395f68, feat/890-placement merged with main (issue 890) 2026-10-26 14:05:38 +01:00
jschoubben 0300a6fd99 pacman: drop the plain-word phrases from replaces (it holds shell commands only, ADR 0245); the descriptions' first lines carry them
mesh/delivery merged: source.main moved to novox/mesh-catalog@2da9329df448
mesh/check being carried to 0300a6fd99b5 on 2da9329df448 | from b1e610b2135a
2026-10-26 14:04:09 +01:00
jschoubben d21b81b00a mesh-build-go: node-build-node also needs bubblewrap and a toolchain installed whole (its mark and lock), as mesh-build-node's integration tests do; pins mesh-sdk 45564e69 (issue 890, review round 1) 2026-10-26 14:02:27 +01:00
mesh-tools b1e610b213 Merge pull request 'Build modules end the steps a previous process left running, and serve build_<lang>_step_end (issue 1890)' (#496) from fix/1890-left-running-steps into main 2026-10-26 12:57:34 +00:00
jschoubben 0134697f97 Build modules: the round ends asks past the step bound; pin the sdk head (issue 1890)
mesh/delivery merged: source.main moved to novox/mesh-catalog@b1e610b2135a
mesh/delivery-group group fix/1890-left-running-steps merged: every member is merged
mesh/check pass 0134697f9739 on b1e610b2135a | carried from a4f5df797b68, mesh-sdk b4cd95e7c4a4 | unit 3132 run, 3022 kept, 7 skipped | integration 31…
2026-10-26 13:52:53 +01:00
jschoubben 21583d14ad Pin mesh-sdk at fix/build-cpu-yields' head (issue 1806) 2026-10-26 13:49:54 +01:00
mesh-tools a4f5df797b Merge pull request 'gitea: the tools' forge account reads every repository (the operator, 2026-10-26)' (#495) from fix/forge-account-reads-all into main 2026-10-26 12:46:37 +00:00
jschoubben ac2c56f2a9 Build modules: steps end by compare-and-set; pin the sdk head with the review's fixes (issue 1890) 2026-10-26 13:44:00 +01:00
jschoubben b4a7bb9dea gitea: say that a repository made private later is granted only at the next start or by hand, and let one unreadable permission refuse its repository, not the whole pass
mesh/delivery merged: source.main moved to novox/mesh-catalog@a4f5df797b68
mesh/check being carried to b4a7bb9dea50 on a4f5df797b68
2026-10-26 13:43:46 +01:00
jschoubben 64b9ded2db Build modules: a per-node processor cap, and each step's threads within it (issue 1806)
The operator's desktop slows while it builds (shanks at 92%, the Go build
runtime at 1650% of 24 processors). The build modules declare cpu-cap
(default shanks=50,g14=50,ace=75), which the composer turns into the runtime's
quota at idle priority; each step's go, node and buildah commands get
GOMAXPROCS and UV_THREADPOOL_SIZE for their part of it, and the status tools
show the limits and what the kernel holds the runtime to.
2026-10-26 13:41:05 +01:00
jschoubben 67d6a5a7c1 mesh-build-go: report the needs a package's check-needs marker names, test it only where the node has them, decline an ask needing what the node lacks (issue 890)
mesh-build-node's integration tests read the Node toolchain installed only where the node holds node-build-node, and
skipped on ace: its package now declares node-build-node beside its tests; the integration digest names it, a node
lacking it leaves the package unmet for the checker to place, and declines a placed ask. Re-pins mesh-sdk feat/890-placement.
2026-10-26 13:37:52 +01:00
jschoubben 5fe5f26e7e Build modules: a restart ends the steps it left running, a step shows what it is, build_*_step_end ends one (issue 1890) 2026-10-26 13:27:46 +01:00
jschoubben b3221b6ab8 pacman: a plain-words search finds pacman_round and pacman_upgrades first (answer on the first line, phrases they replace) 2026-10-26 13:27:09 +01:00
jschoubben 0e7e220d49 gitea: give the tools' account read on every repository, at the start and on the forge's events, so a user's private repository no longer answers its tools 404 2026-10-26 13:22:08 +01:00
mesh-tools 879600202a Merge pull request 'Vault: adopt the hashes of bus accounts it holds no value for, handed sealed by the bus node (issue 1945, ADR 0550 clause 4)' (#494) from feat/1945-user-list-switch into main 2026-10-26 12:21:47 +00:00
mesh-tools 565d0a848a Merge pull request 'Gate: merge turns, a merge waits for a running check of its repository, merges on a carried verdict with no second ask (ADR 0548, issue 568)' (#493) from feat/548-one-check into main 2026-10-26 12:01:28 +00:00
mesh-tools 71e094b2de Merge pull request 'mesh-build-go: count skipped tests per package, so the checker keeps no pass that skipped (issue 890)' (#492) from fix/checker-skips-not-kept into main 2026-10-26 12:01:13 +00:00
mesh-tools f96704fa0f Merge pull request 'mesh-build-node tests: never install a toolchain from the registry in the integration level; skip as NOT RUN when none is installed (issue 890)' (#491) from fix/build-node-tests-offline into main 2026-10-26 11:34:17 +00:00
mesh-tools 86fdf74a0a Merge pull request 'Desk channel: keep what was shown in memory while the store does not answer, so a clearing replaces the urgent notification (issue 384)' (#490) from fix/desk-clear-lost into main 2026-10-26 11:33:47 +00:00
mesh-tools 843281c0e5 Merge pull request 'Knowledge base: withdraw a clause's 'changed by' mark once no accepted record changes it (issue 466)' (#489) from fix/kb-stale-marks into main 2026-10-26 11:33:34 +00:00
jschoubben e9279ce3c2 gate: find an approved merge by its ask, not by which member's run was seen first, so a group's turns are written before its first member is handed at every look (ADR 0548 §3; issue 568, review round 1)
mesh/delivery merged: source.main moved to novox/mesh-catalog@565d0a848a85
mesh/delivery-group group feat/548-one-check merged: every member is merged
mesh/check checking e9279ce3c209 on 565d0a848a85 | novox/mesh-catalog#493 in group feat/548-one-check
2026-10-26 12:32:46 +01:00
jschoubben c9414b1c98 Vault: pin mesh-sdk at the switch group's head c2a410bb, one sdk for the adoption and the switch (issue 1945)
mesh/delivery merged: source.main moved to novox/mesh-catalog@879600202a09
mesh/delivery-group group feat/1945-user-list-switch merged: every member is merged
mesh/check checking c9414b1c9856 on 879600202a09 | novox/mesh-catalog#494 in group feat/1945-user-list-switch
2026-10-26 12:31:56 +01:00
jschoubben ea53b2f920 mesh-build-go: each package counts its skipped tests (issue 890)
mesh/delivery merged: source.main moved to novox/mesh-catalog@71e094b2de59
mesh/delivery-group group fix/checker-skips-not-kept merged: every member is merged
mesh/check failed: ace: the check of novox/mesh-catalog#492 refused: the installed go version could not be read: `go version` with GOTOOLCHAIN=local f…
2026-10-26 12:29:19 +01:00
jschoubben 99cc073e2a gate: pin mesh-sdk at the group's head 9c61918ac36f, where a turn never holds the check of heads waiting in it (ADR 0548 §3; issue 568, review round 1) 2026-10-26 12:15:20 +01:00
jschoubben b94150fcdb gitea: the merge tool reads a carried verdict's line as judged on the new trunk, so it merges a carried verdict (ADR 0548 §8; issue 568, review round 1) 2026-10-26 12:15:20 +01:00
jschoubben a01715ddf5 Skip the Node build module's toolchain tests by name where no Node toolchain is installed, instead of installing one from the registry in a level that has no network; merge three tests that share one fixture
mesh/delivery merged: source.main moved to novox/mesh-catalog@f96704fa0ff4
mesh/check checking a01715ddf508 on 565d0a848a85
On ace, which holds no Node build module, five integration tests each spent 70 s in npm's network retries and failed every mesh-catalog check run there (five verdicts since 00:07). The tests now copy the node's installed toolchain or are skipped as NOT RUN; the registry is never reached. Three tests sharing the fixture are one, so the toolchain is copied three times instead of five (10 integration tests in the module, now 8).
2026-10-26 12:05:09 +01:00
jschoubben e2136ec0c9 The vault takes a handing whole once, refusals and all, and names the refused users in its last adoption (review of the adoption, ADR 0547 clause 9) 2026-10-26 12:03:14 +01:00
jschoubben a30c168512 desk-channel: take the work when the store does not answer, and keep what it holds at the next work and the minute's standing
mesh/delivery merged: source.main moved to novox/mesh-catalog@86fdf74a0a83
mesh/check checking a30c16851246 on 565d0a848a85
Review round 1: offering the work again let a redelivered raise follow its own clearing and
replace it, and a redelivery is bounded where an outage is not. The work is now taken; what
was shown stays in memory, recalled first, and is kept at the start of the next work and at
the minute's standing, stopping at the first write the store does not answer. A desk that
restarts before then loses where those are shown, and its log said how many it held.
2026-10-26 11:46:21 +01:00
jschoubben f885250913 desk-channel: keep where a message is shown when the store does not answer, so its clearing still replaces it
On 2026-10-26 09:18 the bus reloading its accounts made the desk's state store answer
'no responders' just as the urgent 'A new build of mesh-controller cannot be delivered'
was shown. The desk logged that it could not keep the notification ids and acked the work;
the router's clearing at 09:34 then found no record, was shown as a new notification, and
the urgent one stayed on the operator's screen for hours.

Now what was shown and not kept is held in memory, recalled before the store by edits,
reminders, clicks and a notifier restart, kept at the next piece of work, and its work is
offered again until the store keeps it, without being shown twice. An edit of a message
nothing records is said in the journal.
2026-10-26 11:46:09 +01:00
jschoubben 7627503d9d gate: pin mesh-sdk at the group's head f5c233a9288c, which holds the merge turns and the carried verdict (ADR 0548 §10; issue 568) 2026-10-26 11:45:30 +01:00
jschoubben 351c7c4bf4 gate: deliveries and show say an approved merge waits for the check of the named delivery on the named repository, and the turns bucket is read by the checker alone, written by the gate alone (ADR 0548 §3, §9; issue 568 G6) 2026-10-26 11:44:29 +01:00
jschoubben 55a26b22cf gate: delivery_turns reads every merge turn with the check it waits for, and delivery_hand_now hands a held merge at once on the operator's word (ADR 0548 §4; issue 568 G5) 2026-10-26 11:43:40 +01:00
jschoubben 100e74f283 gate: an approved merge whose trunk moved waits for its carried verdict and merges on it with no second ask, never on the old one (ADR 0548 §8; issue 568 G4) 2026-10-26 11:43:18 +01:00
jschoubben c8e1e2b2fa gate: a merge held past 30 minutes by another delivery's check is a stalled line naming that check and since when, said as delivery.<id>.stalled (ADR 0548 §4; issue 568 G3) 2026-10-26 11:41:42 +01:00
jschoubben d0d00f922a gate: an approved merge writes its turn on every repository it goes into before it is handed, kept for a group until its last member merged, read by the checker (ADR 0548 §3; issue 568 G2) 2026-10-26 11:41:26 +01:00
jschoubben c2bd7760b2 gate: hand no approved merge into a repository a running check of other heads judges, and hand it on that check's end (ADR 0548 §2; issue 568 G1) 2026-10-26 11:40:35 +01:00
jschoubben f0fd53281b The vault says once why a look of the bus module's have adopted nothing, and again when it changes (ADR 0547 clause 9) 2026-10-26 11:37:14 +01:00
jschoubben fb039250f6 The vault adopts the hashes of the bus accounts it holds no value for from the bus node's handing (ADR 0547 clause 9, issue 1945)
take-over waits on a hash for every user of the loaded list, and the vault holds no value for the controller's own
account, the node accounts and the enrolled devices. The bus node's engine now hands every loaded user's hash in the
bus module's have, sealed to the vault's key (mesh-host and mesh-sdk on the same branch). The vault answers its key in
node-key, reads the haves (manifest reads node-engine.have, granted from the manifest, no controller change), watches
the bus module's have, opens the handing with its own key, and writes each user it holds no value for into
bus-accounts marked adopted, with the list and node it came from, saying account-hashed. It never touches a user whose
covered account it holds or whose key is its own hash, writes an adopted key again only for another hash, and refuses
a handing from any node the assignments do not name as holding the bus module.

Tools: account_adopt adopts now from the have as the watch does; secret_own, unnarrowed by module, lists the adopted
accounts and the last adoption.
2026-10-26 11:36:33 +01:00
jschoubben 447e6a13cd Knowledge base: withdraw a clause's changed mark when its changing record's accepted links drop it, on every judgement and by a tool (stale marks)
mesh/delivery merged: source.main moved to novox/mesh-catalog@843281c0e5f0
mesh/check checking 447e6a13cd68 on 565d0a848a85
2026-10-26 11:22:12 +01:00
mesh-tools c51a46b802 Merge pull request 'Build modules: run every package's tests of a level, with a fixture of two failing packages (issue 1921)' (#488) from fix/1921-every-failure-r into main 2026-10-26 10:18:08 +00:00
jschoubben 15a5fe09b6 mesh-build-go, mesh-build-node: run every package of a check's level after a failure (issue 1921)
mesh/delivery merged: source.main moved to novox/mesh-catalog@c51a46b802d3
mesh/delivery-group group fix/1921-every-failure-r merged: every member is merged
mesh/check checking 15a5fe09b62e on c51a46b802d3 | novox/mesh-catalog#488 in group fix/1921-every-failure-r
Pins mesh-sdk fix/1921-every-failure-r, whose check ask no longer stops a level at its first failing test step. A captured fixture with a failing test in each of two packages proves the build module's test step names both.
2026-10-26 10:58:07 +01:00
mesh-tools 3ded9e7800 Merge pull request 'Vault: name a hash record's secret in a field called of, so the runtime keeps it (issue 1945)' (#487) from fix/1945-vault-hash-field into main 2026-10-26 09:00:51 +00:00
jschoubben e7aaa8420e Vault: name a hash record's secret in a field called of, so the runtime accepts it
mesh/delivery merged: source.main moved to novox/mesh-catalog@3ded9e780043
mesh/check pass e7aaa8420eac on 3ded9e780043 | unit 2018 run, 2325 kept, 5 skipped | integration 2040 run, 2344 kept, 9 skipped | over bound: unit, in…
The runtime refuses a state field whose name ends in secret, so no bus-account hash could be written on novox. The test state now applies the same check.

Issue: 1945
2026-10-26 09:25:35 +01:00
mesh-tools 22381d7522 Merge pull request 'Vault: read own secrets' generations, and hash each bus account it is handed for the grants part (issue 1945, ADR 0547 clause 9)' (#486) from fix/1945-vault-own-read into main 2026-10-26 07:48:03 +00:00
145 changed files with 8256 additions and 248 deletions
+107 -9
View File
@@ -15,6 +15,13 @@ package main
// the desk says in its standing when that happened (`answers-lost-at`), and the router, which alone knows
// what is still open and unanswered, says that again under its handle — replacing the notification in place,
// so its buttons work again. Nothing resolved, silenced or cancelled comes back.
// - **A store that does not answer does not lose where a message is shown, while the desk runs.** A message
// is shown first and its notification ids kept after; when the state store does not answer then (the bus
// reloading its accounts, 2026-10-26 09:18: the very outage that raised the condition), the work is taken
// all the same and the ids are held in memory, recalled before the store, and kept at the start of the next
// work and at the minute's standing. Otherwise the clearing of that message finds no notification to
// replace and the urgent one stays on screen for good. A desk that restarts while it still holds some loses
// where those are shown; the log says so when they are held.
// - **Standing**: it can send when machines are named in its settings; which of them answered last is in its
// status. A message no machine took is said to the router as failed, and tried again by it.
@@ -93,6 +100,9 @@ type Desk struct {
pause map[string]pauseSeen
// lost is when each machine's notifier last started again: the answers its notifications waited for are lost.
lost map[string]time.Time
// unkept is what was shown and not yet kept in the store, because it did not answer: recalled from here
// before the store, and kept at the next piece of work.
unkept map[string]Shown
}
func (d *Desk) init() {
@@ -107,6 +117,9 @@ func (d *Desk) init() {
if d.lost == nil {
d.lost = map[string]time.Time{}
}
if d.unkept == nil {
d.unkept = map[string]Shown{}
}
if d.Now == nil {
d.Now = time.Now
}
@@ -138,6 +151,7 @@ func (d *Desk) Take(subject string, body json.RawMessage) error {
d.Logf("[desk-channel] work on %s could not be read; taken and dropped", subject)
return nil
}
d.KeepUnkept()
machines := d.Machines()
if m.Secret {
// Shown only where the notifier said lately it is not paused (the review of 2026-10-09): one held back
@@ -167,12 +181,16 @@ func (d *Desk) Take(subject string, body json.RawMessage) error {
switch {
case edit:
var err error
if before, err = d.Store.Get(m.Handle); err != nil {
if before, err = d.recall(m.Handle); err != nil {
return err
}
if before == nil {
d.Logf("[desk-channel] an edit of %s, which nothing here records as shown: shown as a new message; "+
"a notification it was to replace, if any, stays", m.Handle)
}
case !m.Secret:
var err error
if before, err = d.Store.Get(m.Handle); err != nil {
if before, err = d.recall(m.Handle); err != nil {
d.Logf("[desk-channel] what is shown under this handle could not be read (%v): shown as a new message", err)
before = nil
}
@@ -218,12 +236,91 @@ func (d *Desk) Take(subject string, body json.RawMessage) error {
// Shown once and kept nowhere of this module's (a link's code): no edit or re-showing reaches it.
return nil
}
if err := d.Store.Put(m.Handle, s); err != nil {
d.Logf("[desk-channel] shown, and what was shown could not be kept: an edit of it will not reach it: %v", err)
}
d.keep(m.Handle, s)
return nil
}
// keep keeps what a handle shows in the store. When the store does not answer it is held in memory, where an
// edit, a reminder and a click still find it, and kept later (KeepUnkept). The work is taken either way: offered
// again it would race the message's own later edits, and a redelivery is bounded where an outage is not.
func (d *Desk) keep(handle string, s Shown) {
err := d.Store.Put(handle, s)
d.mu.Lock()
defer d.mu.Unlock()
if err == nil {
delete(d.unkept, handle)
return
}
d.unkept[handle] = s
d.Logf("[desk-channel] shown, and where it is shown could not be kept yet (%v): held in memory, so its edit "+
"and clearing still reach it, and kept when the store answers; %d held, lost if the desk restarts first",
err, len(d.unkept))
}
// KeepUnkept keeps in the store what it did not answer for before: at the start of each piece of work and at
// the minute's standing. It stops at the first write the store does not answer, so one call waits on it once.
func (d *Desk) KeepUnkept() {
d.init()
d.mu.Lock()
held := make(map[string]Shown, len(d.unkept))
for h, s := range d.unkept {
held[h] = s
}
d.mu.Unlock()
for h, s := range held {
if err := d.Store.Put(h, s); err != nil {
return
}
d.mu.Lock()
if cur, ok := d.unkept[h]; ok && sameShown(cur, s) {
delete(d.unkept, h)
}
d.mu.Unlock()
d.Logf("[desk-channel] where %s is shown is kept now", h)
}
}
func (d *Desk) unkeptOf(handle string) (Shown, bool) {
d.mu.Lock()
defer d.mu.Unlock()
s, ok := d.unkept[handle]
return s, ok
}
// recall is what a handle shows: from memory when the store did not keep it yet, else from the store.
func (d *Desk) recall(handle string) (*Shown, error) {
if s, ok := d.unkeptOf(handle); ok {
return &s, nil
}
return d.Store.Get(handle)
}
// handles is every handle shown: the store's and those held in memory.
func (d *Desk) handles() ([]string, error) {
hs, err := d.Store.Handles()
d.mu.Lock()
defer d.mu.Unlock()
seen := map[string]bool{}
for _, h := range hs {
seen[h] = true
}
for h := range d.unkept {
if !seen[h] {
hs = append(hs, h)
}
}
if err != nil && len(d.unkept) == 0 {
return nil, err
}
return hs, nil
}
func sameShown(a, b Shown) bool {
x, err1 := json.Marshal(a)
y, err2 := json.Marshal(b)
return err1 == nil && err2 == nil && string(x) == string(y)
}
// show puts one message on one machine's notifier, in place of `replace` when it names one.
func (d *Desk) show(machine string, m asks.Message, replace int) (int, error) {
urgency := "normal"
@@ -308,12 +405,12 @@ func (d *Desk) Chosen(event, machine, ticket string) {
}
func (d *Desk) find(ticket string) (string, *Shown) {
handles, err := d.Store.Handles()
handles, err := d.handles()
if err != nil {
return "", nil
}
for _, h := range handles {
s, err := d.Store.Get(h)
s, err := d.recall(h)
if err != nil || s == nil {
continue
}
@@ -356,9 +453,9 @@ func (d *Desk) Started(event, machine string) int {
d.lost[machine] = d.Now()
d.mu.Unlock()
n := 0
if handles, err := d.Store.Handles(); err == nil {
if handles, err := d.handles(); err == nil {
for _, h := range handles {
if s, err := d.Store.Get(h); err == nil && s != nil && len(s.Message.Buttons) > 0 {
if s, err := d.recall(h); err == nil && s != nil && len(s.Message.Buttons) > 0 {
if _, here := s.On[machine]; here {
n++
}
@@ -472,6 +569,7 @@ func (d *Desk) Status() map[string]any {
"it proves nothing about who clicked: any process of the operator's account can choose an action, so it answers acknowledgements only",
"an answer is taken only on a machine its message was shown on; a message with buttons stays until dismissed",
"what it remembers is never shown again: when a notifier started again, the answers its notifications waited for are lost; the desk says when in its standing (answers-lost-at), and the router says again what is still open and unanswered, replacing those notifications in place — nothing resolved, silenced or cancelled comes back (issue 384)",
"a store that does not answer does not lose where a message is shown while the desk runs: it is held in memory, its edit and clearing still replace it in place, and it is kept when the store answers again; a desk restarted before then loses where those are shown, and its log said how many it held",
"it can send while a named machine's notifier said lately that it is not paused; a link's code is shown only on such a machine, kept nowhere, and sent secret so the notifier keeps it out of its history",
}}
if s.Why != "" {
@@ -3,6 +3,7 @@ package main
import (
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"strings"
@@ -363,9 +364,64 @@ func TestAMessageSaidAgainUnderItsHandleReplacesItsNotification(t *testing.T) {
}
}
// downStore is a store that does not answer while down: the bus reloading its accounts (2026-10-26 09:18).
type downStore struct {
memShown
down *bool
}
func (s downStore) Get(h string) (*Shown, error) {
if *s.down {
return nil, errors.New("nats: no responders available for request")
}
return s.memShown.Get(h)
}
func (s downStore) Put(h string, v Shown) error {
if *s.down {
return errors.New("nats: no responders available for request")
}
return s.memShown.Put(h, v)
}
// The review of PR 176: a replace that fails on a machine keeps that machine's notification on record, so a
// later edit or clearing still reaches it; failing everywhere, it is said to the router as failed.
// later edit or clearing still reaches it; failing everywhere, it is said to the router as failed. And
// (2026-10-26 09:18, an urgent "A new build of mesh-controller cannot be delivered" left on screen for hours
// after its clearing): a message shown while the store does not answer keeps where it is shown in memory, its
// work is taken, its clearing replaces it in place, and it is kept once the store answers.
func TestAFailedReplaceKeepsTheNotificationItWasToReplace(t *testing.T) {
// The raise, then its clearing, taken while the store does not answer; the store then stays down or comes
// back. Each is taken (never offered again, where a redelivered raise would follow its clearing), the
// clearing replaces the urgent notification in place, and the screen and the store end on the clearing.
for _, back := range []bool{false, true} {
t.Run(fmt.Sprintf("the store does not answer, back %v", back), func(t *testing.T) {
d, shown, _, store := newDesk(t, "laptop")
down := true
d.Store = downStore{store, &down}
raised, _ := json.Marshal(asks.Message{Handle: "m9", Title: "A new build cannot be delivered", Urgent: true})
if err := d.Take(asks.ChannelSubject(asks.Send, Kind), raised); err != nil {
t.Fatalf("the raise was not taken: %v", err)
}
cleared, _ := json.Marshal(asks.Message{Handle: "m9", Title: "Cleared: a new build can be delivered"})
if err := d.Take(asks.ChannelSubject(asks.Edit, Kind), cleared); err != nil {
t.Fatalf("the clearing was not taken: %v", err)
}
d.KeepUnkept() // the minute's standing, store still down
down = !back
d.KeepUnkept()
take(t, d, asks.Send, asks.Message{Handle: "m10", Title: "another"})
if len(*shown) != 3 || (*shown)[1].args["replace_id"] != 1 || (*shown)[1].args["summary"] != "Cleared: a new build can be delivered" {
t.Fatalf("the clearing did not replace the urgent notification: %+v", *shown)
}
s, kept := store["m9"]
if back != kept || (kept && s.Message.Title != "Cleared: a new build can be delivered") {
t.Fatalf("store back %v, kept %v: %+v", back, kept, store)
}
if got, _ := d.recall("m9"); got == nil || got.Message.Title != "Cleared: a new build can be delivered" {
t.Fatalf("recalled %+v", got)
}
})
}
d, shown, said, store := newDesk(t, "laptop", "desktop")
take(t, d, asks.Send, asks.Message{Handle: "m1", Title: "2 failed services on shanks"})
first := store["m1"].On
@@ -100,6 +100,7 @@ func run(d *Desk) {
}()
go func() {
for {
d.KeepUnkept()
d.SayStanding()
time.Sleep(time.Minute)
}
+47
View File
@@ -275,6 +275,53 @@ reported no key gets no entry, and its client says it waits. Keys titled `mesh:`
a key added by hand never is. Every consumer is made again every ten minutes and at once when a key changes.
`go test ./cmd/forge-account-provider ./internal/forgeaccount` runs it against a fake forge.
## The mesh's accounts read every repository (the operator, 2026-10-26)
The tools act as the plain account `mesh-tools` (ADR 0365 §2, issue 469), which writes to the organisation `novox`
through its team; the build modules fetch over ssh as `nox-builder` (ADR 0466 §3), in a team that reads the
organisation's code; the machines' agents use `nox-agent` (ADR 0410 §1). Any other repository each saw only when public,
so a private repository of a user answered every tool 404 and every build's fetch "not authorized to read". `readall.ts`
makes each of these accounts (one list, `meshAccounts` in `tools-account.ts`, whose two Go names
`internal/forgeaccount`'s test holds to its own constants) a collaborator that **reads only** on each repository it
cannot read: one repository at a time, no organisation membership, the least the forge offers. An account the forge
does not hold yet (the forge-account provider makes `nox-builder` and `nox-agent` when their first consumer asks) is
named and asked nothing; the start's pass is run again ten minutes apart until every account was there once.
- **When:** one pass over every repository of every user and organisation at the start (a forge not answering yet is
asked again a minute apart until one pass ran), and on the forge's system webhook: `forge.repository.created` and
`forge.fork` at once, and the first event of any repository this process has not seen readable (the forge may not
send a creation for every owner). Nothing polls for it.
- **Not caught:** a repository made private after it was seen readable. The forge sends no event for a change of
visibility, and the start's pass already saw it readable, so its events ask nothing. It is granted at the module's
next start, or now by `gitea_grant_read`; `gitea_unreadable_repos` shows it meanwhile. A repository whose permission
the forge will not answer is named as refused (or, in the listing, with permission `unknown`) and the pass goes on.
- **With what:** the site administrator's login (`MESH_GITEA_ADMIN_PASSWORD_FILE`), because adding a collaborator asks a
repository's administration, which on another user's repository only the site administrator holds. It is used for
five calls alone: list every repository, read one, whether an account exists, read an account's permission on one,
add an account as a read collaborator. The tools' own calls keep their token.
- **Tools:** `gitea_unreadable_repos` (per account, every repository it cannot read, the permission the forge gives
it there and why, a count per account and the accounts the forge does not hold) and `gitea_grant_read` (the same
grant by hand, now, on every repository or on those named, for every account or those named; each grant read back
from the forge, a refusal answered in the forge's own words).
### The merging account writes the repositories the mesh delivers
`mesh-tools` sets every commit status and merges; outside the organisation it only reads (above), so a pull request of
a user's repository could be checked and asked but never merged (issue 1945: car-hunter #2's status refused 403 after
the operator's Approve). `writes.ts` makes it a collaborator that **writes** on each repository that builds a registered
module, and on no other (the operator, 2026-10-26): the build modules' and the agents' accounts keep read.
- **Which:** the indexer's state `mesh-delivery-indexer.modules` (in this module's `reads`), one key per registered
module, each naming its repository. A repository of `novox` needs nothing: the team writes it. Every other repository
of a user stays read-only.
- **When:** the state's watch hands every module at the start and each one registered after; a repository not yet
written is granted then, read back from the forge, and said on the bus as `gitea.repo.writable` {owner, repo,
account}, on which mesh-delivery-gate reads that repository again. Nothing polls.
- **Tools:** `gitea_unwritable_repos` (every delivered repository the account cannot write, its permission and why)
and `gitea_grant_write` (the same grant by hand, on every delivered repository or those named; what it granted, what
was written already, what the forge refused in its own words, and the repositories left read-only and why).
`gitea_commit_statuses` answers `merge.writes`, whether the account may write the repository at all.
## Files it does not know
Each directory or archive this module declares says how it treats a file it did not write (ADR 0422, ADR 0416).
+155 -5
View File
@@ -50,6 +50,9 @@ export interface GiteaRepo {
default_branch?: string;
/** When anything last moved in it — a push, and so a merge. */
updated_at?: string;
/** Whether the account asking may write it (the forge's permissions.push or admin): a commit status and a merge need
* it. Absent when the forge did not say. */
writes?: boolean;
}
/** An issue, with its labels flattened to names. */
@@ -98,6 +101,10 @@ export interface GiteaPull {
html_url: string;
/** The description: where a pull request says it goes after another (`after: <repository>`, novox/hq ADR 0239). */
body?: string;
/** Whether the forge would merge it into its base now (gitea's `mergeable`): false while it is in conflict, while the
* forge is still working that out after its base moved, or while it is a work in progress; absent when the forge did not
* say. What a group's merge reads of every member before it merges any (novox/hq issue 1275). */
mergeable?: boolean;
}
/** An open pull request with what its base requires and what its head has: what gitea_open_pulls answers. */
@@ -789,14 +796,29 @@ export class GiteaClient {
/** Merge a pull request — never forced past its checks, and, given the head that was checked, only that head:
* the forge refuses the merge when the branch moved since (novox/hq ADR 0266). */
/** How long a merge waits before it asks again after the forge's 405 "Please try again later" (novox/hq issue 1275):
* the forge says so while it works out whether the pull request merges, for a few seconds after its base moved. */
static mergeRetryAfter: number[] = [1000, 2000, 4000];
async mergePullRequest(owner: string, repo: string, index: number, method = "merge", deleteBranch = false,
headCommit?: string): Promise<void> {
indexOf(index);
await this.request(`${repoPath(owner, repo)}/pulls/${index}/merge`, {
method: "POST",
body: JSON.stringify({ Do: method, delete_branch_after_merge: deleteBranch, force_merge: false,
...(headCommit ? { head_commit_id: headCommit } : {}) }),
});
for (let attempt = 0; ; attempt++) {
try {
await this.request(`${repoPath(owner, repo)}/pulls/${index}/merge`, {
method: "POST",
body: JSON.stringify({ Do: method, delete_branch_after_merge: deleteBranch, force_merge: false,
...(headCommit ? { head_commit_id: headCommit } : {}) }),
});
return;
} catch (err) {
// Only the forge's transient answer is asked again, briefly; every other refusal, and this one past the last
// wait, is the caller's to say. head_commit_id keeps a head that moved in between from landing.
const wait = GiteaClient.mergeRetryAfter[attempt];
if (wait === undefined || !tryAgainLater(err)) throw err;
await new Promise((r) => setTimeout(r, wait));
}
}
}
// ---- Mappers: the wire shape is broad and unstable; the mesh sees only these fields. ----
@@ -812,6 +834,8 @@ export class GiteaClient {
html_url: r.html_url,
default_branch: r.default_branch,
updated_at: r.updated_at ?? undefined,
...(r.permissions && typeof r.permissions === "object"
? { writes: r.permissions.push === true || r.permissions.admin === true } : {}),
};
}
@@ -842,6 +866,7 @@ export class GiteaClient {
updated_at: p.updated_at ?? undefined,
html_url: p.html_url,
body: typeof p.body === "string" ? p.body : undefined,
mergeable: typeof p.mergeable === "boolean" && !p.merged ? p.mergeable : undefined,
};
}
}
@@ -853,6 +878,25 @@ interface AdminResponse {
readonly body: any;
}
/** A repository as the site administrator sees it: what saying why an account cannot read it needs (readall.ts). */
export interface ForgeRepository {
full_name: string;
owner: string;
name: string;
private: boolean;
archived: boolean;
/** The owner's own visibility: public, limited or private. A public repository of an owner not public is hidden
* from whoever is no member. */
owner_visibility: string;
}
function forgeRepository(r: any): ForgeRepository {
const owner = String(r?.owner?.login ?? r?.owner?.username ?? "");
const name = String(r?.name ?? "");
return { full_name: String(r?.full_name ?? `${owner}/${name}`), owner, name, private: r?.private === true,
archived: r?.archived === true, owner_visibility: String(r?.owner?.visibility ?? "public") };
}
/**
* The forge's admin client, over **basic auth** — gitea's own code, living in the module, used only
* by the provisioner (novox/hq ADR 0048/0076).
@@ -970,6 +1014,105 @@ export class GiteaAdmin {
return repos;
}
// ---- The tools' account reads every repository (readall.ts) ----
/** Every repository on the forge, of every user and organisation: the search as the site administrator, whom the
* forge answers every repository, private ones and those of private owners included. Only a read. */
async everyRepository(): Promise<ForgeRepository[]> {
const all: ForgeRepository[] = [];
for (let page = 1; page <= 200; page++) {
const res = await this.request(`/repos/search?limit=50&page=${page}`);
if (res.status !== 200) GiteaAdmin.fail("/repos/search", res);
const list: any[] = Array.isArray(res.body?.data) ? res.body.data : [];
all.push(...list.map(forgeRepository));
if (list.length < 50) break;
}
return all;
}
/** One repository as the site administrator sees it, or null when the forge holds none by that name. */
async repository(owner: string, repo: string): Promise<ForgeRepository | null> {
const res = await this.request(repoPath(owner, repo));
if (res.status === 404) return null;
if (res.status !== 200) GiteaAdmin.fail(repoPath(owner, repo), res);
return forgeRepository(res.body);
}
/** Whether the forge holds an account of this name. */
async accountExists(user: string): Promise<boolean> {
const path = `/users/${encodeURIComponent(user)}`;
const res = await this.request(path);
if (res.status === 404) return false;
if (res.status !== 200) GiteaAdmin.fail(path, res);
return true;
}
/** What the forge gives an account on a repository, effective: none, read, write, admin or owner. It is the repository as
* that account sees it (the site administrator acts as the account with `sudo`), so access through an organisation's team
* counts. The collaborator endpoint is not used: it names only a collaborator's own permission, and says "read" of an
* account that writes through a team (novox/hq issue 1945). */
async repositoryPermission(owner: string, repo: string, user: string): Promise<string> {
const path = `${repoPath(owner, repo)}?sudo=${encodeURIComponent(user)}`;
const res = await this.request(path);
if (res.status === 404) return "none";
if (res.status !== 200) GiteaAdmin.fail(path, res);
const p = res.body?.permissions;
if (p?.admin === true) return res.body?.owner?.login === user ? "owner" : "admin";
if (p?.push === true) return "write";
return p?.pull === true ? "read" : "none";
}
/** Make an account a collaborator of a repository that reads only. The forge asks a repository's administrator for
* it, which on a repository of another user only the site administrator is. Idempotent: the forge answers 204 again. */
async addReadCollaborator(owner: string, repo: string, user: string): Promise<void> {
await this.addCollaborator(owner, repo, user, "read");
}
/** Make an account a collaborator of a repository that writes: pushes a branch, sets a commit status, merges (writes.ts:
* only on a repository that builds a registered module). An existing read collaborator is changed to write. */
async addWriteCollaborator(owner: string, repo: string, user: string): Promise<void> {
await this.addCollaborator(owner, repo, user, "write");
}
/** The rule that applies to main in a repository, as its administrator reads it: a rule named main, else the first glob
* rule covering main; null when none does. With the administrator's login. */
async mainRule(owner: string, repo: string): Promise<BranchProtection | null> {
const rules = await this.branchProtections(owner, repo);
return rules.find((r) => (r.rule_name ?? r.branch_name) === MAIN_BRANCH)
?? rules.find((r) => ruleCovers(String(r.rule_name ?? r.branch_name ?? ""), MAIN_BRANCH)) ?? null;
}
/** Put an account on main's merge whitelist in a repository of a user, and nothing else (the operator, 2026-10-26:
* write, the rule untouched). Only an existing rule whose merge whitelist is on is changed, and only its whitelist: the
* account is added to the logins already there. Push, status checks, approvals and the administrator override stay as
* the operator set them; with no rule, or the whitelist off, nothing is written (whoever writes may merge). Read back:
* the forge drops a whitelist login that cannot write the repository and still answers success, so a whitelist read back
* without the account fails loud. Answers the rule as the forge holds it after. With the administrator's login. */
async ensureMergerListed(owner: string, repo: string, account: string): Promise<{ changed: boolean; rule: BranchProtection | null }> {
const rule = await this.mainRule(owner, repo);
const listed = rule?.merge_whitelist_usernames ?? [];
if (!rule || rule.enable_merge_whitelist !== true || listed.includes(account)) return { changed: false, rule };
const name = String(rule.rule_name ?? rule.branch_name);
const path = `${repoPath(owner, repo)}/branch_protections/${encodeURIComponent(name)}`;
const res = await this.request(path, { method: "PATCH", body: JSON.stringify({ merge_whitelist_usernames: [...listed, account] }) });
if (res.status !== 200) GiteaAdmin.fail(path, res);
const after = (await this.branchProtections(owner, repo)).find((r) => (r.rule_name ?? r.branch_name) === name);
if (!after) throw new Error(`main's rule ${name} of ${owner}/${repo} was written, but the forge does not list it`);
const held = after.merge_whitelist_usernames ?? [];
if (after.enable_merge_whitelist === true && !held.includes(account)) {
throw new Error(`${account} was added to the merge whitelist of main's rule ${name} of ${owner}/${repo}, but the forge ` +
`holds [${held.join(", ")}]: it drops a login that cannot write the repository`);
}
return { changed: true, rule: after };
}
private async addCollaborator(owner: string, repo: string, user: string, permission: "read" | "write"): Promise<void> {
const path = `${repoPath(owner, repo)}/collaborators/${encodeURIComponent(user)}`;
const res = await this.request(path, { method: "PUT", body: JSON.stringify({ permission }) });
if (res.status === 204 || res.status === 200) return;
GiteaAdmin.fail(path, res);
}
/** Ensure the npm-owner org exists. 201 created, 2xx/404-then-created, and 422/409 (a concurrent
* create won the race) are all success. */
async ensureOrg(name: string): Promise<void> {
@@ -1160,3 +1303,10 @@ export function directoriesAbove(paths: string[]): string[] {
function encodePath(path: string): string {
return pathSegments("file path", path);
}
/** Whether an error is the forge's 405 "Please try again later" on a merge: its answer while the pull request's
* mergeability is being worked out (or it is in conflict, which a later ask says the same way). */
export function tryAgainLater(err: unknown): boolean {
const words = err instanceof Error ? err.message : String(err);
return / 405 /.test(words) && /try again later/i.test(words);
}
+50 -3
View File
@@ -58,8 +58,11 @@ import { CHECK_CONTEXT, CHECK_STATUS, REPO_CHECK_CONTEXT, commentFor, composedGa
import { GateWriter } from "./gate.js";
import { HOOK_PATH, HOOK_PORT, Pace, claimHead, ensureSystemHook, eventsTool, keptSecret, receiver, type ForgeEvent } from "./webhook.js";
import { keepSiblings, noteComposed, noteVerdict, readSiblings, siblingsFile, siblingsOf, weighSiblings } from "./siblings.js";
import { bindInBackground } from "./bind.js";
import { bindInBackground, bindUntilHeld } from "./bind.js";
import { announcementOf, keepTook, lateVerdictComment, outOfDateWords, sameCommit, tookFile, weighTrunks } from "./trunk.js";
import { ReadAll, readAllTools } from "./readall.js";
import { DeliveredWrites, INDEXER_MODULES, writeTools, type ModulesChange } from "./writes.js";
import { TOOLS_ACCOUNT, TOOLS_ORG, meshAccounts } from "./tools-account.js";
// Without a way to a token — configured, or mintable with the admin account (token.ts) — there is
// nothing to watch; log and stay quiet rather than crash the runtime. With one, the first poll mints
@@ -500,8 +503,21 @@ if (!gitea) pace.unheard("the forge is not watched: no way to a token", false);
// Said with `emit`, not the SDK's `keeps`, which the toolchain's SDK does not have yet: switch to `keeps` once that SDK is 0.1.15 or later (issue 1852).
await emit("states", statesAnnouncement(process.env.MESH_MODULE || "gitea", process.env.MESH_NODE));
// **The mesh's accounts read every repository** (readall.ts; the operator, 2026-10-26): the tools', the build modules'
// and the machines' agents' (meshAccounts, tools-account.ts), granted with the site
// administrator's login, which the module holds for setting the forge up and uses here for that grant alone. Made on
// first use, so a module whose admin login is missing still serves the rest and the tools say why.
let readAllOnce: ReadAll | null = null;
const readAll = (): ReadAll => (readAllOnce ??= new ReadAll(GiteaAdmin.fromEnv(), meshAccounts(process.env.MESH_GITEA_TOOLS_USER || TOOLS_ACCOUNT)));
// **The merging account writes every repository that builds a registered module** (writes.ts; the operator,
// 2026-10-26; issue 1945): which those are is the indexer's state, watched below. Made on first use, as readAll.
let writesOnce: DeliveredWrites | null = null;
const writes = (): DeliveredWrites => (writesOnce ??= new DeliveredWrites(GiteaAdmin.fromEnv(),
process.env.MESH_GITEA_TOOLS_USER || TOOLS_ACCOUNT, TOOLS_ORG, (l) => console.log(`[gitea] ${l}`), (type, body) => emit(type, body)));
// gitea_events is served whether or not the forge is watched: it says so (novox/hq issue 729).
registerModuleTools("gitea", () => [eventsTool(pace), ...(gitea ? [commitStatusTool({
registerModuleTools("gitea", () => [eventsTool(pace), ...readAllTools(readAll), ...writeTools(writes), ...(gitea ? [commitStatusTool({
setCommitStatus: (owner, repo, sha, status) => (gitea as GiteaClient).setCommitStatus(owner, repo, sha, status),
gates,
}), mergeTool(gitea, { gates })] : [])]);
@@ -543,7 +559,12 @@ async function listenForForge(client: GiteaClient): Promise<boolean> {
const server = receiver({ secret, port, heard: (kind, delivery) => pace.heard(kind, delivery),
refused: (reason, from) => pace.refused(reason, from),
publish: (event) => emit(event.name, event.body),
act: (event) => actOn(client, event) });
act: async (event) => {
// A repository made, or one not yet seen readable, is granted read before anything acts on it (readall.ts).
await Promise.resolve().then(() => readAll().onEvent(event)).catch((err) =>
console.error(`[gitea] whether the mesh's accounts read the repository of ${event.name} could not be settled — ${err instanceof Error ? err.message : err}`));
await actOn(client, event);
} });
await new Promise<void>((resolve, reject) => {
server.once("error", reject);
server.listen(port, host, () => resolve());
@@ -594,6 +615,32 @@ if (gitea) {
// of the node starts left gitea_commit_status, gitea_merge_pull_request and gitea_events served nowhere. Until it is
// listening, Pace keeps the polls at 30 s, so nothing is missed meanwhile; listenForForge says its own failure.
void listenForForge(client);
// The start's one pass over every repository (readall.ts), not awaited for the same reason. A forge not answering yet
// is asked again, a minute apart, until one pass has run; after that only the forge's events grant.
// An account of the list the forge does not hold yet (the forge-account provider makes nox-builder and nox-agent when
// their first consumer asks, and the forge sends no event for a user made) is looked for again ten minutes apart
// until every account was granted once; the line of each pass names it.
const readAllAtStart = (): void => void Promise.resolve().then(() => readAll().atStart()).then((r) => {
if (r.absent.length) setTimeout(readAllAtStart, 600_000);
}, (err) => {
console.error(`[gitea] whether the mesh's accounts read every repository could not be settled; asked again in a minute — ${err instanceof Error ? err.message : err}`);
setTimeout(readAllAtStart, 60_000);
});
try {
readAll();
readAllAtStart();
} catch (err) {
console.error(`[gitea] the mesh's accounts are not given read on every repository: no site administrator's login — ${err instanceof Error ? err.message : err}`);
}
// The indexer's modules, watched: the watch hands every key's latest value at its start and then each change, so a
// repository that builds a registered module is made writable at the start and when its first module is registered
// (writes.ts). A read not granted yet is said and asked again until it holds.
try {
const w = writes();
void bindUntilHeld(INDEXER_MODULES, () => state<unknown>(INDEXER_MODULES).watch((change) => w.take(change as ModulesChange)));
} catch (err) {
console.error(`[gitea] the merging account is not given write on the delivered repositories: no site administrator's login — ${err instanceof Error ? err.message : err}`);
}
// Main takes no push and only the tools' account merges, in every repository, kept and not only when a token is
// minted: at the start and every ten minutes after, so a repository made later gets it too (issue 948, ADR 0398 §2).
// Only while the setting keep-main-rules is true (mainrules.ts); false, the default, leaves every repository's rule as set.
@@ -1,6 +1,10 @@
package forgeaccount
import "testing"
import (
"os"
"regexp"
"testing"
)
func TestForgeUserOfAMachineAccountIsTheOneAgentUser(t *testing.T) {
if got := ForgeUser("g14", "machine"); got != "nox-agent" {
@@ -15,6 +19,18 @@ func TestForgeUserOfAMachineAccountIsTheOneAgentUser(t *testing.T) {
if got := NodeAccount("g14"); got != "agent-g14" {
t.Fatalf("the token tool's per-node account = %q", got)
}
// The accounts that read every repository (readall.ts, meshAccounts in tools-account.ts) name these two users: the
// list is held to the Go constants here, so the two cannot drift apart (the operator, 2026-10-26).
ts, err := os.ReadFile("../../tools-account.ts")
if err != nil {
t.Fatal(err)
}
for name, want := range map[string]string{"BUILDER_ACCOUNT": BuilderUser, "AGENT_ACCOUNT": AgentUser} {
m := regexp.MustCompile(`export const ` + name + ` = "([^"]+)";`).FindSubmatch(ts)
if m == nil || string(m[1]) != want {
t.Errorf("tools-account.ts %s = %q, want %q (internal/forgeaccount)", name, m, want)
}
}
}
func TestRemoveKeysTitledTakesOnlyTheMeshKeysOfThatTitle(t *testing.T) {
+3 -1
View File
@@ -40,6 +40,7 @@
],
"emits": [
"repo.created",
"repo.writable",
"issue.opened",
"pull.merged",
"pull.updated",
@@ -567,6 +568,7 @@
"mesh-build-go.keys",
"mesh-build-node.keys",
"mesh-build-image.keys",
"mesh-delivery-checker.owed"
"mesh-delivery-checker.owed",
"mesh-delivery-indexer.modules"
]
}
+263
View File
@@ -0,0 +1,263 @@
// The mesh's own forge accounts read every repository on the forge (the operator, 2026-10-26: "the mesh's forge account
// should get read access to all repositories"; and "why wouldn't the forge account get access to all repo's?").
//
// **Every account the mesh's parts use, from one list** (MESH_ACCOUNTS, tools-account.ts): the tools' `mesh-tools`, the
// build modules' `nox-builder` (their fetch over ssh, ADR 0466 §3) and the machines' `nox-agent` (ssh-client's key,
// ADR 0410 §1). The first pass granted the tools' account alone, so the build modules' fetch of jschoubben/car-hunter
// still failed on ace and g14 ("nox-builder ... is not authorized to read"). The two Go names are the ones of
// internal/forgeaccount, whose test holds the list to them. An account the forge does not hold yet (its consumer never
// asked) is said and left out, never a refusal per repository.
//
// Since issue 469 the tools act as the plain account `mesh-tools`, which writes to the organisation novox's
// repositories through its team (tools-account.ts). Every other repository it saw only when public: a private
// repository of a user (jschoubben/car-hunter, jschoubben/snake) answered its tools 404 and was missing from every
// list, while that user's public ones were there. So the module makes its account a collaborator that reads only on
// each repository it cannot read, the least the forge offers: one repository, read, no organisation membership.
//
// - **at the start**, one pass over every repository of every user and organisation;
// - **on the forge's own event** (the system webhook, webhook.ts): `forge.repository.created` and `forge.fork` at
// once, and the first event of any repository this process has not yet seen readable, since the forge may not
// send a repository's creation for every owner. Nothing polls for it.
//
// **Not caught: a repository made private later.** The forge sends no event for a change of visibility, and a public
// repository is already seen readable by the start's pass, so its events ask nothing. Such a repository is granted at
// the module's next start, or now by `gitea_grant_read`; `gitea_unreadable_repos` shows it meanwhile.
//
// **The grant needs the site administrator's login.** Adding a collaborator asks the repository's administration,
// which on another user's repository only the site administrator holds; the module holds that login for setting the
// forge up (MESH_GITEA_ADMIN_PASSWORD_FILE). Here it is used through `ReadAllForge` alone: list the repositories, read
// the account's permission on one, add the account as a collaborator that reads. Nothing else, and never for the tools'
// own calls, which keep their token.
//
// What the forge refuses (a repository that went meanwhile, an archived one if the forge will not take a collaborator
// on it) is said per repository and not retried; the tool `gitea_unreadable_repos` shows what is left, and
// `gitea_grant_read` grants again now.
import type { ForgeRepository } from "./client.js";
/** What the module asks of the forge for this, with the site administrator's login (GiteaAdmin, client.ts). */
export interface ReadAllForge {
everyRepository(): Promise<ForgeRepository[]>;
repository(owner: string, repo: string): Promise<ForgeRepository | null>;
/** Whether the forge holds the account at all. */
accountExists(user: string): Promise<boolean>;
repositoryPermission(owner: string, repo: string, user: string): Promise<string>;
addReadCollaborator(owner: string, repo: string, user: string): Promise<void>;
}
/** A repository an account cannot read, and why. */
export interface Unreadable {
account: string;
repository: string;
permission: string;
private: boolean;
archived: boolean;
why: string;
}
export interface Granted {
/** Repositories looked at (for each account). */
checked: number;
/** The accounts looked at: those the forge holds. */
accounts: string[];
/** Accounts of the list the forge does not hold yet: nothing is granted to them until it does. */
absent: string[];
/** Made readable now, by a read collaborator: each repository and the account. */
granted: { account: string; repository: string }[];
/** The forge refused the grant: each repository, the account and the forge's own words. */
refused: { account: string; repository: string; why: string }[];
}
const READS = new Set(["read", "write", "admin", "owner"]);
const said = (err: unknown): string => (err instanceof Error ? err.message : String(err));
/** Why the account cannot read a repository, from what the forge says of it. */
export function whyUnreadable(repo: ForgeRepository, account: string): string {
const parts: string[] = [];
if (repo.private) parts.push(`a private repository of ${repo.owner}, on which ${account} is no collaborator and in no team that covers it`);
else if (repo.owner_visibility !== "public") {
parts.push(`a public repository of ${repo.owner}, whose own visibility is ${repo.owner_visibility}: the forge shows it to members only`);
} else parts.push(`the forge answers ${account} no read on it`);
if (repo.archived) parts.push("archived");
return parts.join("; ");
}
export class ReadAll {
/** Per account, the repositories this process has seen it read, by full name: an event of a repository every account
* was seen reading asks nothing more. */
readonly readable = new Map<string, Set<string>>();
readonly accounts: readonly string[];
constructor(private readonly forge: ReadAllForge, accounts: readonly string[] | string,
private readonly log: (line: string) => void = (l) => console.log(`[gitea] ${l}`)) {
this.accounts = [...new Set(typeof accounts === "string" ? [accounts] : accounts)];
if (!this.accounts.length) throw new Error("no forge account to give read: the list is empty");
for (const a of this.accounts) this.readable.set(a, new Set());
}
/** The accounts of the list (or of those named) the forge holds, and those it does not hold yet. */
private async present(only?: string[]): Promise<{ accounts: string[]; absent: string[] }> {
const wanted = only && only.length ? only : [...this.accounts];
for (const a of wanted) if (!this.accounts.includes(a)) throw new Error(`"${a}" is not one of the mesh's accounts: ${this.accounts.join(", ")}`);
const accounts: string[] = [];
const absent: string[] = [];
for (const a of wanted) ((await this.forge.accountExists(a)) ? accounts : absent).push(a);
return { accounts, absent };
}
/** Every repository each account cannot read, and why, as the forge says it now. */
async unreadable(only?: string[]): Promise<{ checked: number; accounts: { account: string; unreadable: number }[]; absent: string[]; unreadable: Unreadable[] }> {
const { accounts, absent } = await this.present(only);
const repos = await this.forge.everyRepository();
const out: Unreadable[] = [];
for (const account of accounts) {
for (const repo of repos) {
try {
const u = await this.judge(repo, account);
if (u) out.push(u);
} catch (err) {
// A permission the forge would not answer is said for that repository; the rest are still looked at.
out.push({ account, repository: repo.full_name, permission: "unknown", private: repo.private, archived: repo.archived,
why: `the forge did not answer ${account}'s permission on it — ${said(err)}` });
}
}
}
return { checked: repos.length, accounts: accounts.map((account) => ({ account, unreadable: out.filter((u) => u.account === account).length })),
absent, unreadable: out };
}
/** Grant read on every repository an account cannot read (or on those named, `owner/name`), for every account of the
* list the forge holds (or for those named). */
async grant(only?: string[], accountsOnly?: string[]): Promise<Granted> {
const { accounts, absent } = await this.present(accountsOnly);
let repos: ForgeRepository[];
if (only && only.length) {
repos = [];
for (const full of only) {
const [owner, name, ...rest] = full.split("/");
if (!owner || !name || rest.length) throw new Error(`"${full}" is not a repository: say owner/name`);
const repo = await this.forge.repository(owner, name);
if (!repo) throw new Error(`the forge holds no repository ${full}`);
repos.push(repo);
}
} else repos = await this.forge.everyRepository();
const result: Granted = { checked: repos.length, accounts, absent, granted: [], refused: [] };
for (const account of accounts) {
for (const repo of repos) {
const repository = repo.full_name;
let u: Unreadable | null;
try {
u = await this.judge(repo, account);
} catch (err) {
// One repository whose permission cannot be read is a refusal of its own, never the end of the pass.
result.refused.push({ account, repository, why: `its permission could not be read — ${said(err)}` });
continue;
}
if (!u) continue;
try {
await this.forge.addReadCollaborator(repo.owner, repo.name, account);
} catch (err) {
result.refused.push({ account, repository, why: said(err) });
continue;
}
// Read back: the forge's own word that the account reads it now, never the grant's answer alone.
let after: string;
try {
after = await this.forge.repositoryPermission(repo.owner, repo.name, account);
} catch (err) {
result.refused.push({ account, repository, why: `granted, but its permission could not be read back — ${said(err)}` });
continue;
}
if (READS.has(after)) {
this.readable.get(account)!.add(repository);
result.granted.push({ account, repository });
} else {
result.refused.push({ account, repository, why: `the grant was answered, but the forge still gives ${account} "${after}"` });
}
}
}
return result;
}
/** The start's one pass: grant what is missing and say it. Answers the result; a failure is the caller's to say. */
async atStart(): Promise<Granted> {
const r = await this.grant();
const per = r.accounts.map((a) => {
const g = r.granted.filter((x) => x.account === a).map((x) => x.repository);
return `${a} ${g.length} granted read now${g.length ? ` (${g.join(", ")})` : ""}`;
});
this.log(`the mesh's accounts read every repository: ${r.checked} looked at; ${per.join("; ")}` +
`${r.absent.length ? `; not on the forge yet, so not granted: ${r.absent.join(", ")}` : ""}` +
`${r.refused.length ? `; ${r.refused.length} refused: ` + r.refused.map((x) => `${x.account} on ${x.repository} — ${x.why}`).join("; ") : ""}`);
return r;
}
/** A forge event: a repository made (or forked), or the first event of one some account was not seen reading, is
* granted read now to every account that cannot read it. */
async onEvent(event: { name: string; body: unknown }): Promise<void> {
const b = (event.body ?? {}) as { owner?: string; repo?: string };
if (!b.owner || !b.repo) return;
const full = `${b.owner}/${b.repo}`;
if (event.name === "forge.repository.deleted") {
for (const seen of this.readable.values()) seen.delete(full);
return;
}
const made = event.name === "forge.repository.created" || event.name === "forge.fork";
// Seen readable once by every account, its events ask nothing: so a repository made private after that is not
// caught here (the forge says no change of visibility), only at the next start or by gitea_grant_read.
if (!made && [...this.readable.values()].every((seen) => seen.has(full))) return;
const r = await this.grant([full]);
for (const x of r.granted) this.log(`${x.account} was given read on ${full}, on the forge's ${event.name}`);
for (const x of r.refused) this.log(`${x.account} could not be given read on ${x.repository} — ${x.why}`);
}
private async judge(repo: ForgeRepository, account: string): Promise<Unreadable | null> {
const permission = await this.forge.repositoryPermission(repo.owner, repo.name, account);
const seen = this.readable.get(account)!;
if (READS.has(permission)) {
seen.add(repo.full_name);
return null;
}
seen.delete(repo.full_name);
return { account, repository: repo.full_name, permission, private: repo.private, archived: repo.archived, why: whyUnreadable(repo, account) };
}
}
type Tool = { name: string; description: string; input: Record<string, unknown>; run: (args: Readonly<Record<string, unknown>>) => Promise<unknown> };
const strings = (v: unknown): string[] | undefined => (Array.isArray(v) ? v.map(String) : undefined);
/** The two tools: what each account cannot read and why, and the grant now. */
export function readAllTools(readAll: () => ReadAll): Tool[] {
return [
{
name: "gitea_unreadable_repos",
description: "Every repository on the forge one of the mesh's own forge accounts cannot read, and why, per account: " +
"the tools' (mesh-tools), the build modules' fetch over ssh (nox-builder) and the machines' agents' (nox-agent). " +
"For each, the permission the forge gives it there, whether the repository is private or archived, and in words " +
"the reason (a private repository of a user with the account no collaborator, an owner not public); a count per " +
"account, and the accounts the forge does not hold yet. Empty when every account reads every repository. Read " +
"with the site administrator's login, which alone sees every repository.",
input: {
accounts: { type: "array", items: { type: "string" }, description: "the accounts to look at; empty or absent, every one of the mesh's" },
},
run: async (args) => readAll().unreadable(strings(args.accounts)),
},
{
name: "gitea_grant_read",
description: "Give each of the mesh's own forge accounts (mesh-tools, nox-builder, nox-agent) read on every " +
"repository it cannot read now, or on the repositories named, or for the accounts named: each becomes a " +
"collaborator that reads only, the least the forge offers. The module does this itself at its start and on the " +
"forge's event of a repository made; this is the same act by hand, such as when a build's fetch says its account " +
"is not authorized to read a repository. Answers how many were looked at, which were granted to which account " +
"(each read back from the forge), which the forge refused, in its own words, and the accounts it does not hold " +
"yet. Needs the site administrator's login, used for this grant alone.",
input: {
repositories: { type: "array", items: { type: "string" }, description: "owner/name of each repository; empty or absent, every repository" },
accounts: { type: "array", items: { type: "string" }, description: "the accounts to grant; empty or absent, every one of the mesh's" },
},
run: async (args) => readAll().grant(strings(args.repositories), strings(args.accounts)),
},
];
}
+68
View File
@@ -0,0 +1,68 @@
import assert from "node:assert/strict";
import { test } from "node:test";
import { createServer } from "node:http";
// **The forge's 405 "Please try again later" on a merge is asked again briefly** (novox/hq issue 1275, 2026-10-26 13:40):
// mesh-delivery#73's merge was answered so a few seconds after its base moved, and counted as refused at once. The forge
// says it while it works out whether the pull request merges; the merge is asked again after a short wait, and every other
// refusal is answered at once. A pull request's own mergeable is carried, so a group can be read whole before it merges.
/** A forge whose merge of novox/hq#7 answers each of `answers` in turn (the last for ever), and whose pull request says
* `mergeable`. */
async function serve(answers: { status: number; body: string }[], mergeable?: boolean) {
const { GiteaClient } = await import("../client.ts");
let merges = 0;
const server = createServer((req, res) => {
const url = new URL(req.url ?? "", "http://x");
if (url.pathname === "/api/v1/repos/novox/hq/pulls/7/merge" && req.method === "POST") {
const a = answers[Math.min(merges++, answers.length - 1)];
res.statusCode = a.status;
return res.end(a.body);
}
if (url.pathname === "/api/v1/repos/novox/hq/pulls/7") {
return res.end(JSON.stringify({ number: 7, state: "open", merged: false, base: { ref: "main" },
head: { ref: "b", sha: "a".repeat(40) }, html_url: "h/7", ...(mergeable === undefined ? {} : { mergeable }) }));
}
res.statusCode = 404;
res.end("{}");
});
await new Promise<void>((r) => server.listen(0, r));
const client = new GiteaClient(`http://127.0.0.1:${(server.address() as any).port}`, "t");
GiteaClient.mergeRetryAfter = [1, 1, 1];
return { client, server, merges: () => merges };
}
const later = { status: 405, body: '{"message":"Please try again later"}' };
const ok = { status: 200, body: "" };
test("a merge answered 'try again later' while the forge works out its mergeability, then merged", async (t) => {
const cases: { name: string; answers: { status: number; body: string }[]; merged: boolean; asked: number }[] = [
{ name: "merged on the third ask", answers: [later, later, ok], merged: true, asked: 3 },
{ name: "still 'later' after the last wait: refused, with the forge's words", answers: [later], merged: false, asked: 4 },
{ name: "a refusal of another kind is answered at once", answers: [{ status: 409, body: '{"message":"head moved"}' }], merged: false, asked: 1 },
];
for (const c of cases) {
await t.test(c.name, async () => {
const { client, server, merges } = await serve(c.answers);
try {
const done = client.mergePullRequest("novox", "hq", 7, "merge", false, "a".repeat(40));
if (c.merged) await done;
else await assert.rejects(done, (err: Error) => /40[59]/.test(err.message));
assert.equal(merges(), c.asked);
} finally {
server.close();
}
});
}
});
test("a pull request carries the forge's mergeable, and none when the forge did not say", async () => {
for (const mergeable of [true, false, undefined]) {
const { client, server } = await serve([ok], mergeable);
try {
assert.equal((await client.getPullRequest("novox", "hq", 7)).mergeable, mergeable);
} finally {
server.close();
}
}
});
+293 -3
View File
@@ -16,7 +16,7 @@ import { join } from "node:path";
import { MintedToken } from "../dist/token.js";
import { GiteaClient } from "../dist/client.js";
import { TOOLS_ACCOUNT, TOOLS_TEAM_UNITS } from "../dist/tools-account.js";
import { TOOLS_ACCOUNT, TOOLS_TEAM_UNITS, BUILDER_ACCOUNT, AGENT_ACCOUNT, meshAccounts } from "../dist/tools-account.js";
import { repoCheckFacts } from "../dist/pulls.js";
const ADMIN = "mesh-admin";
@@ -26,6 +26,19 @@ interface Account { admin: boolean; password: string; active: boolean }
interface Token { id: number; name: string; sha1: string; user: string; scopes: string[] }
interface Team { id: number; name: string; units: any; members: Set<string> }
interface ForgeRepo { owner: string; name: string; full: string; private: boolean; archived?: boolean; ownerVisibility: string; collaborators: Map<string, string> }
const forgeRepos: ForgeRepo[] = [];
const userRules = new Map<string, any>();
const repoFaults = { refuse: new Set<string>(), permission: new Set<string>() };
function addRepo(full: string, isPrivate: boolean, ownerVisibility = "public"): ForgeRepo {
const [owner, name] = full.split("/");
const r = { owner, name, full, private: isPrivate, ownerVisibility, collaborators: new Map<string, string>() };
forgeRepos.push(r);
return r;
}
const repoJSON = (r: ForgeRepo) => ({ full_name: r.full, name: r.name, owner: { login: r.owner, visibility: r.ownerVisibility },
private: r.private, archived: r.archived === true });
function fakeForge() {
const users = new Map<string, Account>([[ADMIN, { admin: true, password: PASSWORD, active: true }]]);
const tokens: Token[] = [];
@@ -61,11 +74,18 @@ function fakeForge() {
return null;
};
const isAdmin = (u: string | null) => u !== null && users.get(u)?.admin === true;
const permissionOf = (u: string | null, r: ForgeRepo): string => {
if (isAdmin(u)) return "admin";
if (u === r.owner) return "owner";
if (r.owner === "novox" && teams.some((t) => t.members.has(u ?? "") && t.units.units_map?.["repo.code"] === "write")) return "write";
if (u && r.collaborators.has(u)) return r.collaborators.get(u)!;
return !r.private && r.ownerVisibility === "public" ? "read" : "none";
};
const server = createServer(async (req, res) => {
const url = new URL(req.url ?? "/", "http://fake");
const p = url.pathname.replace(/^\/api\/v1/, "");
const me = who(req);
const me = url.searchParams.has("sudo") && isAdmin(who(req)) ? url.searchParams.get("sudo") : who(req);
calls.push(`${req.method} ${p} as ${me}`);
if (me === null) return json(res, 401, { message: "unauthorized" });
@@ -149,6 +169,23 @@ function fakeForge() {
if (m[2] === "feat/unguarded") return json(res, 200, { name: m[2], protected: false });
return json(res, 404, { message: "branch does not exist" });
}
// A repository of a user: its rules are its administrator's (a site administrator, or a collaborator with admin).
m = p.match(/^\/repos\/(?!novox\/)([^/]+)\/([^/]+)\/branch_protections(?:\/(.*))?$/);
if (m) {
const key = `${m[1]}/${m[2]}`;
const repo = forgeRepos.find((r) => r.full === key);
if (!repo || !["admin", "owner"].includes(permissionOf(me, repo))) return json(res, 403, { message: "user should be an owner or a collaborator with admin write of a repository" });
if (req.method === "GET" && m[3] === undefined) return json(res, 200, userRules.has(key) ? [userRules.get(key)] : []);
// As gitea does: a whitelist login that cannot write the repository is dropped, and the write still answered.
const keep = (b: any) => (Array.isArray(b.merge_whitelist_usernames)
? { ...b, merge_whitelist_usernames: b.merge_whitelist_usernames.filter((u: string) => ["write", "admin", "owner"].includes(permissionOf(u, repo))) } : b);
if (req.method === "PATCH" && m[3] === "main") {
if (!userRules.has(key)) return json(res, 404, { message: "branch protection does not exist" });
Object.assign(userRules.get(key), keep(await read(req)));
return json(res, 200, userRules.get(key));
}
if (req.method === "POST") { const b = keep(await read(req)); userRules.set(key, b); return json(res, 201, b); }
}
// gitea's reqAdmin on the whole group: reading the rules is administration too.
m = p.match(/^\/repos\/novox\/([^/]+)\/branch_protections(?:\/(.*))?$/);
if (m) {
@@ -176,7 +213,37 @@ function fakeForge() {
if (!writes) return json(res, 403, { message: "write on code required" });
return json(res, 201, { context: (await read(req)).context, creator: { login: me } });
}
if (req.method === "GET" && p === "/repos/search") return json(res, 200, { data: [] });
// Every repository of every owner, and who reads it (readall.ts): the site administrator reads all; the tools' account
// a repository of novox through its team, a public one of a public owner, and one it is a collaborator of.
if (req.method === "GET" && p === "/repos/search") {
const page = Number(url.searchParams.get("page") ?? "1");
const seen = forgeRepos.filter((r) => permissionOf(me, r) !== "none");
return json(res, 200, { data: page === 1 ? seen.map(repoJSON) : [] });
}
m = p.match(/^\/repos\/([^/]+)\/([^/]+)(?:\/collaborators\/([^/]+)(\/permission)?)?$/);
if (m) {
const repo = forgeRepos.find((r) => r.owner === decodeURIComponent(m![1]) && r.name === decodeURIComponent(m![2]));
if (repo && req.method === "GET" && !m[3] && url.searchParams.has("sudo") && repoFaults.permission.has(repo.full)) return json(res, 500, { message: "the forge is not well" });
if (!repo || permissionOf(me, repo) === "none") return json(res, 404, { message: "repository does not exist" });
if (req.method === "GET" && !m[3]) {
const mine = permissionOf(me, repo);
return json(res, 200, { ...repoJSON(repo), permissions: { pull: true, push: ["write", "admin", "owner"].includes(mine), admin: ["admin", "owner"].includes(mine) } });
}
const user = decodeURIComponent(m[3] ?? "");
if (req.method === "GET" && m[4]) {
if (repoFaults.permission.has(repo.full)) return json(res, 500, { message: "the forge is not well" });
if (!isAdmin(me) && me !== user) return json(res, 403, { message: "only the user or an administrator" });
// As gitea answers: the collaborator's own permission, not what a team gives the account.
return json(res, 200, { permission: repo.collaborators.get(user) ?? "read" });
}
if (req.method === "PUT" && !m[4]) {
// gitea's reqAdmin: the repository's administration, which on a repository of another user the site administrator alone holds.
if (!isAdmin(me) && me !== repo.owner) return json(res, 403, { message: "user should be an owner or a collaborator with admin write of a repository" });
if (repoFaults.refuse.has(repo.full)) return json(res, 422, { message: "the forge would not have it" });
repo.collaborators.set(user, (await read(req))?.permission ?? "write");
return json(res, 204, null);
}
}
return json(res, 404, { message: "no such route in the fake" });
});
return new Promise<{ url: string; users: typeof users; tokens: typeof tokens; teams: typeof teams; calls: string[]; mainRules: typeof mainRules;
@@ -402,3 +469,226 @@ test("a mint with keep-main-rules false (the default) writes no branch protectio
const writes = forge.calls.slice(from).filter((c: string) => /branch_protections/.test(c) && !/^GET /.test(c) && !/^(GET|HEAD) /.test(c));
assert.deepEqual(writes, []);
});
// The mesh's accounts read every repository (readall.ts; the operator, 2026-10-26): jschoubben/car-hunter and
// jschoubben/snake, private repositories of a user, answered the tools 404 while that user's public ones were seen; and
// once the tools' account read them, the build modules' fetch as nox-builder still could not.
test("the start's pass gives each of the mesh's accounts read, as a collaborator that reads only, on each repository it could not read, and on no other; an account the forge does not hold is said, not refused", async () => {
const env = await environment();
await fresh(env).listRepos(); // the account and its team exist, as after any mint
forge.users.set(BUILDER_ACCOUNT, { admin: false, password: "unused", active: true }); // made by the forge-account provider
forge.users.delete(AGENT_ACCOUNT); // no machine's key asked for it yet
forgeRepos.splice(0);
addRepo("novox/hq", true);
addRepo("jschoubben/txt-game", false);
addRepo("jschoubben/car-hunter", true);
addRepo("jschoubben/snake", true);
addRepo("shy/open", false, "limited");
addRepo("elsewhere/secret", true);
const { GiteaAdmin } = await import("../dist/client.js");
const { ReadAll, readAllTools } = await import("../dist/readall.js");
const lines: string[] = [];
const readAll = new ReadAll(GiteaAdmin.fromEnv(env), meshAccounts(), (l: string) => lines.push(l));
assert.deepEqual(readAll.accounts, [TOOLS_ACCOUNT, BUILDER_ACCOUNT, AGENT_ACCOUNT]);
const [unreadableTool, grantTool] = readAllTools(() => readAll);
// What the tool says before, per account: the four the tools' account cannot read, each with why; the builders'
// account, in no write team here, also not novox/hq; nox-agent not on the forge.
const before = await unreadableTool.run({}) as any;
assert.equal(before.checked, 6);
const of = (account: string) => before.unreadable.filter((u: any) => u.account === account).map((u: any) => u.repository).sort();
assert.deepEqual(of(TOOLS_ACCOUNT), ["elsewhere/secret", "jschoubben/car-hunter", "jschoubben/snake", "shy/open"]);
assert.deepEqual(of(BUILDER_ACCOUNT), ["elsewhere/secret", "jschoubben/car-hunter", "jschoubben/snake", "novox/hq", "shy/open"]);
assert.deepEqual(before.accounts, [{ account: TOOLS_ACCOUNT, unreadable: 4 }, { account: BUILDER_ACCOUNT, unreadable: 5 }]);
assert.deepEqual(before.absent, [AGENT_ACCOUNT]);
assert.match(before.unreadable.find((u: any) => u.account === BUILDER_ACCOUNT && u.repository === "jschoubben/car-hunter").why,
/a private repository of jschoubben, on which nox-builder is no collaborator/);
const builderOnly = await unreadableTool.run({ accounts: [BUILDER_ACCOUNT] }) as any;
assert.deepEqual(builderOnly.accounts, [{ account: BUILDER_ACCOUNT, unreadable: 5 }]);
await assert.rejects(unreadableTool.run({ accounts: ["someone"] }), /not one of the mesh's accounts/);
assert.match(before.unreadable.find((u: any) => u.repository === "jschoubben/car-hunter").why,
/a private repository of jschoubben, on which mesh-tools is no collaborator/);
assert.match(before.unreadable.find((u: any) => u.repository === "shy/open").why, /visibility is limited/);
assert.deepEqual((await fresh(env).listAllRepos()).map((r) => r.full_name).sort(), ["jschoubben/txt-game", "novox/hq"]);
// The account's own token cannot grant itself: the forge asks the repository's administration (the reason for the admin login).
const own = (await readFile(join(env.MESH_GITEA_STATE_DIR, "token"), "utf8")).trim();
const self = await fetch(`${forge.url}/api/v1/repos/jschoubben/car-hunter/collaborators/${TOOLS_ACCOUNT}`,
{ method: "PUT", headers: { Authorization: `token ${own}`, "Content-Type": "application/json" }, body: JSON.stringify({ permission: "read" }) });
assert.equal(self.status, 404, "a repository the account cannot read is not even there for it");
const from = forge.calls.length;
const r = await readAll.atStart();
const grantedTo = (account: string) => r.granted.filter((x: any) => x.account === account).map((x: any) => x.repository).sort();
assert.deepEqual(grantedTo(TOOLS_ACCOUNT), ["elsewhere/secret", "jschoubben/car-hunter", "jschoubben/snake", "shy/open"]);
assert.deepEqual(grantedTo(BUILDER_ACCOUNT), ["elsewhere/secret", "jschoubben/car-hunter", "jschoubben/snake", "novox/hq", "shy/open"]);
assert.deepEqual(r.refused, []);
assert.deepEqual(r.absent, [AGENT_ACCOUNT]);
const puts = forge.calls.slice(from).filter((c) => c.startsWith("PUT "));
assert.equal(puts.length, 9, "the repositories each read already were left alone, and the absent account was asked nothing");
assert.ok(!puts.some((c) => c.includes(AGENT_ACCOUNT)));
assert.ok(puts.every((c) => c.endsWith(` as ${ADMIN}`)), "only the administrator's login grants");
for (const repo of forgeRepos.filter((x) => x.collaborators.size)) {
for (const [account, permission] of repo.collaborators) assert.equal(permission, "read", `${account} on ${repo.full} reads only`);
}
assert.equal(forgeRepos.find((x) => x.full === "jschoubben/car-hunter")!.collaborators.get(BUILDER_ACCOUNT), "read");
assert.match(lines.join("\n"), /6 looked at; mesh-tools 4 granted read now .*; nox-builder 5 granted read now .*; not on the forge yet, so not granted: nox-agent/);
// The account made later is granted by hand (or at the next pass), its rows like any other.
forge.users.set(AGENT_ACCOUNT, { admin: false, password: "unused", active: true });
const agent = await grantTool.run({ accounts: [AGENT_ACCOUNT], repositories: ["jschoubben/car-hunter"] }) as any;
assert.deepEqual(agent.granted, [{ account: AGENT_ACCOUNT, repository: "jschoubben/car-hunter" }]);
assert.deepEqual(agent.absent, []);
forge.users.delete(AGENT_ACCOUNT);
// After: the account's own token sees every repository, and the tool names none.
assert.equal((await fresh(env).listAllRepos()).length, 6);
assert.deepEqual(((await unreadableTool.run({})) as any).unreadable, []);
// Granting again asks nothing of the forge but reads.
const again = forge.calls.length;
assert.deepEqual(((await grantTool.run({})) as any).granted, []);
assert.ok(forge.calls.slice(again).every((c) => c.startsWith("GET ")));
});
test("on the forge's events: a repository made is granted read at once, a known readable one asks nothing, a refusal is said, not thrown", async () => {
const env = await environment();
await fresh(env).listRepos();
forgeRepos.splice(0);
addRepo("jschoubben/txt-game", false);
const { GiteaAdmin } = await import("../dist/client.js");
const { ReadAll, readAllTools } = await import("../dist/readall.js");
const lines: string[] = [];
forge.users.set(BUILDER_ACCOUNT, { admin: false, password: "unused", active: true });
const readAll = new ReadAll(GiteaAdmin.fromEnv(env), [TOOLS_ACCOUNT, BUILDER_ACCOUNT], (l: string) => lines.push(l));
await readAll.atStart();
const cases: { name: string; event: string; repo: string; make?: boolean; refuse?: boolean; unanswered?: boolean; asks: boolean; reads: string }[] = [
{ name: "a repository made", event: "forge.repository.created", repo: "jschoubben/new-private", make: true, asks: true, reads: "read" },
{ name: "a fork made", event: "forge.fork", repo: "someone/fork", make: true, asks: true, reads: "read" },
{ name: "a push to a repository seen readable", event: "forge.push", repo: "jschoubben/txt-game", asks: false, reads: "read" },
{ name: "a push to a private repository never seen", event: "forge.push", repo: "jschoubben/quiet", make: true, asks: true, reads: "read" },
{ name: "a repository the forge refuses the grant on", event: "forge.repository.created", repo: "jschoubben/stuck", make: true, refuse: true, asks: true, reads: "none" },
{ name: "a repository whose permission the forge will not answer", event: "forge.repository.created", repo: "jschoubben/mute", make: true, unanswered: true, asks: true, reads: "none" },
];
for (const c of cases) {
if (c.make) addRepo(c.repo, true);
if (c.refuse) repoFaults.refuse.add(c.repo);
if (c.unanswered) repoFaults.permission.add(c.repo);
const from = forge.calls.length;
const [owner, repo] = c.repo.split("/");
await readAll.onEvent({ name: c.event, body: { owner, repo } });
assert.equal(forge.calls.length > from, c.asks, `${c.name}: asks the forge`);
for (const account of [TOOLS_ACCOUNT, BUILDER_ACCOUNT]) {
assert.equal(forgeRepos.find((x) => x.full === c.repo)!.collaborators.get(account) ?? (c.make ? "none" : "read"), c.reads, `${c.name}: ${account}`);
}
}
assert.match(lines.join("\n"), /mesh-tools could not be given read on jschoubben\/stuck — .*422/);
assert.match(lines.join("\n"), /nox-builder was given read on someone\/fork, on the forge's forge.fork/);
const named = await (readAllTools(() => readAll)[1].run({ repositories: ["jschoubben/stuck"] })) as any;
assert.deepEqual(named.refused.map((x: any) => `${x.account} ${x.repository}`), [`${TOOLS_ACCOUNT} jschoubben/stuck`, `${BUILDER_ACCOUNT} jschoubben/stuck`]);
await assert.rejects(readAllTools(() => readAll)[1].run({ repositories: ["no-slash"] }), /not a repository/);
assert.match(lines.join("\n"), /could not be given read on jschoubben\/mute — its permission could not be read — .*500/);
// A whole pass goes past the repository it cannot judge: every other one is still granted, and the listing names it.
addRepo("jschoubben/after-mute", true);
const pass = await readAll.grant();
assert.ok(pass.granted.some((x: any) => x.repository === "jschoubben/after-mute" && x.account === BUILDER_ACCOUNT), "the pass did not stop at the repository it could not judge");
assert.ok(pass.refused.some((x: any) => x.repository === "jschoubben/mute" && /permission could not be read/.test(x.why)));
const listed = await readAllTools(() => readAll)[0].run({}) as any;
assert.equal(listed.unreadable.find((u: any) => u.repository === "jschoubben/mute").permission, "unknown");
repoFaults.refuse.clear();
repoFaults.permission.clear();
});
// The merging account writes every repository that builds a registered module, and no other (writes.ts; the operator,
// 2026-10-26; issue 1945): car-hunter #2 was approved and its commit status refused 403, mesh-tools reading only there.
test("the merging account is given write on each repository the indexer's modules name, read back and said; the operator's other repositories stay read-only; the tools say both", async () => {
const env = await environment();
const client = fresh(env);
await client.listRepos(); // the account and its team exist, as after any mint
forgeRepos.splice(0);
addRepo("novox/hq", true);
for (const full of ["jschoubben/car-hunter", "jschoubben/snake", "jschoubben/dotfiles"]) addRepo(full, true).collaborators.set(TOOLS_ACCOUNT, "read");
const { GiteaAdmin } = await import("../dist/client.js");
const { DeliveredWrites, writeTools, WRITABLE_EVENT } = await import("../dist/writes.js");
const lines: string[] = [];
const said: { type: string; body: any }[] = [];
const writes = new DeliveredWrites(GiteaAdmin.fromEnv(env), TOOLS_ACCOUNT, "novox", (l: string) => lines.push(l),
async (type: string, body: unknown) => { said.push({ type, body }); });
const [unwritableTool, grantTool] = writeTools(() => writes);
await assert.rejects(grantTool.run({}), /not known yet/, "nothing is granted before the indexer's modules were read");
assert.equal((await client.getRepo("jschoubben", "car-hunter")).writes, false, "the statuses' answer says the account cannot write");
// 1945/AC1: the watch hands the indexer's modules; a user's repository a module builds from is made writable, once.
await writes.take({ key: "car-hunter", op: "put", value: { v: 1, repository: "jschoubben/car-hunter", commit: "c" }, current: true });
await writes.take({ key: "gitea", op: "put", value: { v: 1, repository: "novox/hq", commit: "c" }, current: true });
await writes.take({ key: "broken", op: "put", value: { v: 1, repository: "no-slash" }, current: true });
const from = forge.calls.length;
await writes.take({ key: "car-hunter-2", op: "put", value: { v: 1, repository: "jschoubben/car-hunter" } });
assert.equal(forge.calls.length, from, "a repository already written asks nothing more");
const repo = (full: string) => forgeRepos.find((x) => x.full === full)!;
assert.equal(repo("jschoubben/car-hunter").collaborators.get(TOOLS_ACCOUNT), "write", "a user's repository is written, not administered");
assert.equal(userRules.has("jschoubben/car-hunter"), false, "a repository with no rule for main is given none");
assert.equal(repo("novox/hq").collaborators.size, 0, "the organisation's repository is written through its team, no collaborator");
assert.equal((await client.getRepo("jschoubben", "car-hunter")).writes, true);
assert.deepEqual(said, [{ type: WRITABLE_EVENT, body: { owner: "jschoubben", repo: "car-hunter", account: TOOLS_ACCOUNT } }]);
assert.match(lines.join("\n"), /mesh-tools was given write on jschoubben\/car-hunter, which builds the registered module car-hunter/);
assert.match(lines.join("\n"), /broken names no repository/);
// 1945/AC2: a delivered repository the forge refuses is said, in its words, by the listing and the grant; the operator's
// repository no module builds from is left read-only, named with why, and a grant naming it asks the forge nothing.
repoFaults.refuse.add("jschoubben/snake");
await writes.take({ key: "snake", op: "put", value: { v: 1, repository: "jschoubben/snake" } });
assert.match(lines.join("\n"), /could not be given write on jschoubben\/snake, which builds snake — .*422/);
const listed = await unwritableTool.run({}) as any;
assert.deepEqual(listed.unwritable.map((u: any) => [u.repository, u.permission]), [["jschoubben/snake", "read"]]);
assert.match(listed.unwritable[0].why, /builds snake.*gitea_grant_write/);
const all = await grantTool.run({}) as any;
assert.deepEqual(all.delivered, ["jschoubben/car-hunter", "jschoubben/snake", "novox/hq"]);
assert.deepEqual(all.already, ["jschoubben/car-hunter", "novox/hq"]);
assert.deepEqual(all.granted, []);
assert.match(all.refused.find((x: any) => x.repository === "jschoubben/snake").why, /422.*would not have it/);
assert.deepEqual(all.left_read_only.map((x: any) => x.repository), ["jschoubben/dotfiles"]);
assert.match(all.left_read_only[0].why, /builds no registered module/);
const puts = forge.calls.length;
const named = await grantTool.run({ repositories: ["jschoubben/dotfiles"] }) as any;
assert.deepEqual([named.granted, named.left_read_only.map((x: any) => x.repository)], [[], ["jschoubben/dotfiles"]]);
assert.ok(!forge.calls.slice(puts).some((c) => c.startsWith("PUT ")), "no write is asked on a repository no module builds from");
assert.equal(repo("jschoubben/dotfiles").collaborators.get(TOOLS_ACCOUNT), "read");
repoFaults.refuse.clear();
// The operator's rule is left as set (the operator, 2026-10-26: "Write, rule untouched"): only mesh-tools is added to its
// merge whitelist; a status the mesh does not set and required approvals are said as blocking the mesh's merge, unchanged.
const operators = { rule_name: "main", enable_push: true, enable_merge_whitelist: true, merge_whitelist_usernames: ["jschoubben"],
enable_status_check: true, status_check_contexts: ["mesh/check", "mesh/merge-gate", "ci/external"], required_approvals: 2,
block_admin_merge_override: false, require_signed_commits: true, block_on_outdated_branch: true, block_on_rejected_reviews: true };
userRules.set("jschoubben/snake", structuredClone(operators));
const after = await grantTool.run({ repositories: ["jschoubben/snake"] }) as any;
assert.deepEqual(after.granted, ["jschoubben/snake"]);
assert.equal(repo("jschoubben/snake").collaborators.get(TOOLS_ACCOUNT), "write");
assert.deepEqual(userRules.get("jschoubben/snake"), { ...operators, merge_whitelist_usernames: ["jschoubben", TOOLS_ACCOUNT] }, "only the whitelist changed");
assert.deepEqual(after.merge_whitelist, [{ repository: "jschoubben/snake", changed: true, listed: ["jschoubben", TOOLS_ACCOUNT] }]);
const blockedWhy = "writable, but main's rule blocks the mesh's merge: it requires the status ci/external, which the mesh does not set; " +
"it requires 2 approving reviews on the forge, which the gate does not give; it requires signed commits, which the mesh does not make; " +
"it refuses a branch behind main, which the mesh does not bring up to date; " +
"it refuses a merge after a review asked for changes, which the gate does not clear";
assert.deepEqual(after.merge_blocked, [{ repository: "jschoubben/snake", why: blockedWhy }]);
const again = await unwritableTool.run({}) as any;
assert.deepEqual([again.unwritable, again.merge_blocked], [[], [{ repository: "jschoubben/snake", why: blockedWhy }]]);
const writesBefore = forge.calls.filter((c) => /^PATCH .*branch_protections/.test(c)).length;
assert.deepEqual(((await grantTool.run({ repositories: ["jschoubben/snake"] })) as any).merge_whitelist[0].changed, false);
assert.equal(forge.calls.filter((c) => /^PATCH .*branch_protections/.test(c)).length, writesBefore, "a listed account is not written again");
// The mesh's own statuses never block: a rule requiring only them, with mesh-tools listed, is not reported.
userRules.set("jschoubben/snake", { rule_name: "main", enable_merge_whitelist: true, merge_whitelist_usernames: [TOOLS_ACCOUNT],
enable_status_check: true, status_check_contexts: ["mesh/check", "mesh/merge-gate", "mesh/repo-check"] });
assert.deepEqual(((await unwritableTool.run({})) as any).merge_blocked, []);
assert.deepEqual(((await grantTool.run({ repositories: ["jschoubben/snake"] })) as any).merge_blocked, []);
// The forge drops a whitelist login that cannot write and still answers 200: the whitelist is read back and that fails loud.
userRules.set("jschoubben/dotfiles", { rule_name: "main", enable_merge_whitelist: true, merge_whitelist_usernames: ["jschoubben"] });
await assert.rejects(GiteaAdmin.fromEnv(env).ensureMergerListed("jschoubben", "dotfiles", TOOLS_ACCOUNT),
/mesh-tools was added to the merge whitelist of main's rule main of jschoubben\/dotfiles, but the forge holds \[jschoubben\]: it drops a login that cannot write/);
userRules.delete("jschoubben/dotfiles");
// A module gone from the indexer leaves its repository out of the delivered list.
await writes.take({ key: "snake", op: "delete" });
assert.deepEqual(writes.delivered(), ["jschoubben/car-hunter", "novox/hq"]);
});
+3
View File
@@ -34,6 +34,9 @@ test("both statuses say the head and the trunk commit the verdict judged, first,
const [old] = statusesFor({ ...c, trunk: undefined, gate: { verdict: "pass", summary: "every machine composes" } });
assert.equal(old.description, "pass: every machine composes");
assert.equal(trunkNamed(old.description), undefined);
// ADR 0548 §8: a carried verdict's line names the head on the new trunk first, so the merge tool merges it on that trunk.
assert.equal(trunkNamed(`pass ${HEAD.slice(0, 12)} on ${MOVED.slice(0, 12)} | carried from ${TRUNK.slice(0, 12)}, mesh-catalog 0123456789ab | unit 0 run, 4 kept`),
MOVED.slice(0, 12));
});
test("a trunk's move puts an open head's merge check out of date, naming both commits, unless a check is on its way", () => {
+9
View File
@@ -22,6 +22,15 @@
import { keepMainRules } from "./mainrules.js";
export const TOOLS_ACCOUNT = "mesh-tools";
/** The build modules' account, whose keys fetch over ssh (internal/forgeaccount BuilderUser, ADR 0466 §3). */
export const BUILDER_ACCOUNT = "nox-builder";
/** The machines' agents' account, whose keys ssh-client reports (internal/forgeaccount AgentUser, ADR 0410 §1). */
export const AGENT_ACCOUNT = "nox-agent";
/** Every forge account the mesh's own parts use: each reads every repository on the forge (readall.ts; the operator,
* 2026-10-26). internal/forgeaccount's test holds the two Go names to its own constants. */
export function meshAccounts(tools: string = TOOLS_ACCOUNT): string[] {
return [tools, BUILDER_ACCOUNT, AGENT_ACCOUNT];
}
/** The organisation whose repositories are the mesh's, and the team through which the tools' account writes there. */
export const TOOLS_ORG = "novox";
export const TOOLS_TEAM = "mesh-tools";
+15 -4
View File
@@ -492,7 +492,7 @@ function allGiteaTools(gitea: GiteaClient, admin?: () => GiteaAdmin): ToolDefini
},
{
name: "gitea_commit_statuses",
description: "Read a commit's statuses — or a pull request's head's — as the forge keeps them: the newest of each context (mesh/check, mesh/delivery, …) with its state, description, link, when it was set and by whom; and whether the branch it merges into would let it merge, every status that branch's protection requires being a success. Reads only.",
description: "Read a commit's statuses — or a pull request's head's — as the forge keeps them: the newest of each context (mesh/check, mesh/delivery, …) with its state, description, link, when it was set and by whom; and whether the branch it merges into would let it merge, every status that branch's protection requires being a success; for a pull request, also the forge's own mergeable (false while it is in conflict with its base or the forge is still working that out); and merge.writes, whether the mesh's merging account may write the repository at all (false: its statuses and its merge are refused until gitea_grant_write). Reads only.",
input: {
owner: { type: "string", description: "the repository owner" },
repo: { type: "string", description: "the repository name" },
@@ -508,11 +508,12 @@ function allGiteaTools(gitea: GiteaClient, admin?: () => GiteaAdmin): ToolDefini
if (!owner || !repo) throw new Error("name the repository: owner and repo");
if (!Number.isInteger(number) || number < 0) throw new Error(`${args.number} is not a pull request's number`);
let pull: { number: number; title: string; state: string; merged: boolean; base?: string; head?: string;
head_sha?: string; merge_commit_sha?: string; merged_at?: string; html_url: string } | undefined;
head_sha?: string; merge_commit_sha?: string; merged_at?: string; html_url: string; mergeable?: boolean } | undefined;
if (number > 0) {
const p = await gitea.getPullRequest(owner, repo, number);
pull = { number: p.number, title: p.title, state: p.state, merged: p.merged, base: p.base, head: p.head,
head_sha: p.head_sha, merge_commit_sha: p.merge_commit_sha, merged_at: p.merged_at, html_url: p.html_url };
head_sha: p.head_sha, merge_commit_sha: p.merge_commit_sha, merged_at: p.merged_at, html_url: p.html_url,
...(p.mergeable === undefined ? {} : { mergeable: p.mergeable }) };
if (!ref) ref = p.head_sha ?? "";
if (!base) base = p.base ?? "";
if (!ref) throw new Error(`${owner}/${repo}#${number} names no head commit`);
@@ -527,8 +528,18 @@ function allGiteaTools(gitea: GiteaClient, admin?: () => GiteaAdmin): ToolDefini
} catch (err) {
unreadable = err instanceof Error ? err.message : String(err);
}
// Whether this account may write the repository at all (issue 1945): a commit status and a merge need it, and a
// repository of a user it only reads refuses both 403. Not said when the forge would not answer.
let writes: boolean | undefined;
try {
writes = (await gitea.getRepo(owner, repo)).writes;
} catch {
writes = undefined;
}
const merge = mergeableUnder(base, rule, statuses, unreadable);
return { commit: combined.sha || ref, pull, combined: combined.state, statuses,
merge: mergeableUnder(base, rule, statuses, unreadable) };
merge: writes === undefined ? merge : { ...merge, writes,
...(writes ? {} : { cannot_write: `the forge account cannot write ${owner}/${repo}: gitea_grant_write` }) } };
},
},
{
+321
View File
@@ -0,0 +1,321 @@
// The account that sets commit statuses and merges writes every repository that builds a registered module, and no other
// (the operator, 2026-10-26: "yes the mesh should get write access", on the repositories the mesh delivers; novox/hq issue
// 1945). car-hunter #2 (a repository of a user, not of the organisation novox) was approved, and mesh-delivery's first
// act, its commit status, was refused 403: the tools' account was only a collaborator that reads there (readall.ts).
//
// **Which account.** The tools' account, `mesh-tools`: the gitea module's own tools set every commit status and merge
// (gitea_commit_status, gitea_merge_pull_request) with its token. The build modules' `nox-builder` and the machines'
// `nox-agent` only fetch, and keep read (readall.ts).
//
// **Which repositories: the indexer's word, read over the bus.** mesh-delivery-indexer keeps every registered module in its
// state `modules` (mesh-sdk go/delivery ModulesBucket), each value naming the repository it builds from. This module reads
// that state (its manifest's `reads`) with a watch: the watch hands the latest value of every key when it starts, and then
// every change, so a repository is granted at the module's start and when a module of it is first registered. Nothing
// polls. A repository of the organisation needs nothing: the organisation's team writes it (tools-account.ts). Every other
// repository of a user (the operator's dotfiles, say) is left as it is: read only.
//
// **Every grant is read back** from the forge (its permission for the account), and said on the bus as
// `repo.writable` {owner, repo, account}: mesh-delivery-gate, which does not ask the operator's Approve of a pull request
// whose merge the forge would refuse, reads that repository again on it.
//
// **A repository of a user is written, and main's rule is left as the operator set it** (the operator, 2026-10-26, asked
// "Write, rule untouched"): the account is a write collaborator there, and only added to main's merge whitelist when that
// rule keeps one (client.ts ensureMergerListed). What else in main's rule stops the mesh's merge (a status the mesh does not
// set, approvals the gate does not give) is changed by nobody but said: both tools list such a repository as writable, but
// blocked, with why (merge_blocked).
//
// Not taken away: a repository whose last module goes from the indexer keeps its collaborator until a hand removes it;
// gitea_unwritable_repos and gitea_grant_write say the delivered list as it is now.
import type { ForgeRepository } from "./client.js";
import type { BranchProtection } from "./protection.js";
import { CHECK_CONTEXT, CHECK_STATUS, REPO_CHECK_CONTEXT } from "./pulls.js";
/** The indexer's state this module reads: one key per registered module, its value naming the repository. */
export const INDEXER_MODULES = "mesh-delivery-indexer.modules";
/** The event said once an account writes a repository it could not write before. */
export const WRITABLE_EVENT = "repo.writable";
/** What the module asks of the forge for this, with the site administrator's login (GiteaAdmin, client.ts). */
export interface WriteForge {
everyRepository(): Promise<ForgeRepository[]>;
repository(owner: string, repo: string): Promise<ForgeRepository | null>;
repositoryPermission(owner: string, repo: string, user: string): Promise<string>;
addWriteCollaborator(owner: string, repo: string, user: string): Promise<void>;
mainRule(owner: string, repo: string): Promise<BranchProtection | null>;
ensureMergerListed(owner: string, repo: string, account: string): Promise<{ changed: boolean; rule: BranchProtection | null }>;
}
/** The statuses every pull request the mesh delivers carries (mesh-delivery's checks); a rule requiring another blocks. */
const MESH_CONTEXTS = new Set([CHECK_STATUS, CHECK_CONTEXT, REPO_CHECK_CONTEXT]);
/** What in main's rule stops the mesh's merge, in words; empty when nothing does. Changed by nobody here: the operator's. */
export function mergeBlocks(rule: BranchProtection | null, account: string): string[] {
if (!rule) return [];
const blocks: string[] = [];
if (rule.enable_merge_whitelist === true && !(rule.merge_whitelist_usernames ?? []).includes(account)) {
blocks.push(`its merge whitelist [${(rule.merge_whitelist_usernames ?? []).join(", ")}] leaves out ${account} (gitea_grant_write adds it)`);
}
if (rule.enable_status_check === true) {
for (const c of rule.status_check_contexts ?? []) {
if (!MESH_CONTEXTS.has(c)) blocks.push(`it requires the status ${c}, which the mesh does not set`);
}
}
const approvals = Number(rule.required_approvals ?? 0);
if (approvals > 0) blocks.push(`it requires ${approvals} approving review${approvals === 1 ? "" : "s"} on the forge, which the gate does not give`);
if (rule.require_signed_commits === true) blocks.push("it requires signed commits, which the mesh does not make");
if (rule.block_on_outdated_branch === true) blocks.push("it refuses a branch behind main, which the mesh does not bring up to date");
if (rule.block_on_rejected_reviews === true) blocks.push("it refuses a merge after a review asked for changes, which the gate does not clear");
return blocks;
}
const blockedWhy = (blocks: string[]): string => `writable, but main's rule blocks the mesh's merge: ${blocks.join("; ")}`;
/** One change of the indexer's modules state, as the SDK's state watch hands it. */
export interface ModulesChange {
key: string;
op: "put" | "delete";
value?: unknown;
revision?: number;
current?: boolean;
}
export interface Written {
account: string;
/** The repositories that build a registered module, as the indexer says now. */
delivered: string[];
/** Made writable now, each read back from the forge. */
granted: string[];
/** Written already (a collaborator of a user's repository), or written through the organisation's team. */
already: string[];
/** Each repository of a user: whether the account was added to main's merge whitelist now, and the whitelist read back
* (null: main's rule keeps no whitelist, or there is no rule, and whoever writes may merge). */
merge_whitelist: { repository: string; changed: boolean; listed: string[] | null }[];
/** Writable, but main's rule (the operator's, left as set) stops the mesh's merge, and why. */
merge_blocked: { repository: string; why: string }[];
/** The forge refused, in its own words; or the repository is the organisation's, whose team should write it. */
refused: { repository: string; why: string }[];
/** Repositories the account cannot write that are left so, and why: they build no registered module. */
left_read_only: { repository: string; permission: string; why: string }[];
}
const WRITES = new Set(["write", "admin", "owner"]);
const said = (err: unknown): string => (err instanceof Error ? err.message : String(err));
function split(full: string): [string, string] {
const [owner, name, ...rest] = String(full ?? "").split("/");
if (!owner || !name || rest.length) throw new Error(`"${full}" is not a repository: say owner/name`);
return [owner, name];
}
export class DeliveredWrites {
/** Module to the repository it builds from, as the indexer's state says. */
readonly modules = new Map<string, string>();
/** Whether the watch has handed the indexer's state at all: until then, which repositories are delivered is not known. */
heard = false;
/** Repositories seen written by the account in this process: a module registered again asks nothing more. */
readonly writable = new Set<string>();
constructor(private readonly forge: WriteForge, readonly account: string, readonly org: string,
private readonly log: (line: string) => void = (l) => console.log(`[gitea] ${l}`),
private readonly emit: (type: string, body: unknown) => Promise<void> = async () => {}) {}
/** The repositories that build a registered module, sorted. */
delivered(): string[] {
return [...new Set(this.modules.values())].sort();
}
/** One change of the indexer's modules: a repository newly delivered is made writable now. Never throws. */
async take(change: ModulesChange): Promise<void> {
this.heard = true;
if (change.op === "delete") {
this.modules.delete(change.key);
return;
}
const repository = String((change.value as { repository?: unknown } | undefined)?.repository ?? "").trim();
try {
split(repository);
} catch {
this.log(`${INDEXER_MODULES} ${change.key} names no repository (${JSON.stringify(repository)}): no write is granted for it`);
return;
}
this.modules.set(change.key, repository);
if (this.writable.has(repository)) return;
try {
const r = await this.grant([repository]);
for (const g of r.granted) this.log(`${this.account} was given write on ${g}, which builds the registered module ${change.key}`);
for (const x of r.refused) this.log(`${this.account} could not be given write on ${x.repository}, which builds ${change.key} — ${x.why}`);
} catch (err) {
this.log(`whether ${this.account} writes ${repository} (module ${change.key}) could not be settled — ${said(err)}`);
}
}
/** Grant write on every delivered repository the account cannot write (or on those named, each of which must be
* delivered); with none named, also say which repositories it cannot write are left so, and why. */
async grant(only?: string[]): Promise<Written> {
if (!this.heard) {
throw new Error(`which repositories build a registered module is not known yet: ${INDEXER_MODULES} has not been read ` +
"(the watch is not granted or not answered); nothing is granted");
}
const delivered = this.delivered();
const result: Written = { account: this.account, delivered, granted: [], already: [], merge_whitelist: [], merge_blocked: [], refused: [], left_read_only: [] };
const named = only && only.length ? only.map((x) => String(x).trim()) : null;
const targets: string[] = [];
for (const full of named ?? delivered) {
split(full);
if (delivered.includes(full)) targets.push(full);
else result.left_read_only.push({ repository: full, permission: "unchanged",
why: "builds no registered module (the indexer names none from it), so the account is given no write there" });
}
for (const full of targets) {
const [owner, name] = split(full);
let permission: string;
try {
if (!(await this.forge.repository(owner, name))) {
result.refused.push({ repository: full, why: "the forge holds no repository by that name" });
continue;
}
permission = await this.forge.repositoryPermission(owner, name, this.account);
} catch (err) {
result.refused.push({ repository: full, why: `its permission could not be read — ${said(err)}` });
continue;
}
if (owner === this.org) {
if (WRITES.has(permission)) {
this.writable.add(full);
result.already.push(full);
continue;
}
// The organisation's repositories are written through its team (tools-account.ts), never a collaborator.
result.refused.push({ repository: full, why: `a repository of ${this.org}, which ${this.account} writes through the ` +
`team ${this.org}/${this.account}; the forge gives it "${permission}", so the team is not as it should be (it is ` +
"set again at the next mint of the tools' token)" });
continue;
}
// A repository of a user: the account writes it, and is put on main's merge whitelist when the rule keeps one.
let changed = false;
if (!WRITES.has(permission)) {
try {
await this.forge.addWriteCollaborator(owner, name, this.account);
} catch (err) {
result.refused.push({ repository: full, why: said(err) });
continue;
}
let after: string;
try {
after = await this.forge.repositoryPermission(owner, name, this.account);
} catch (err) {
result.refused.push({ repository: full, why: `granted, but its permission could not be read back — ${said(err)}` });
continue;
}
if (!WRITES.has(after)) {
result.refused.push({ repository: full, why: `the grant was answered, but the forge still gives ${this.account} "${after}"` });
continue;
}
changed = true;
}
try {
const { changed: listedNow, rule } = await this.forge.ensureMergerListed(owner, name, this.account);
result.merge_whitelist.push({ repository: full, changed: listedNow,
listed: rule?.enable_merge_whitelist === true ? (rule.merge_whitelist_usernames ?? []) : null });
const blocks = mergeBlocks(rule, this.account);
if (blocks.length > 0) result.merge_blocked.push({ repository: full, why: blockedWhy(blocks) });
} catch (err) {
result.refused.push({ repository: full, why: `${this.account} writes it, but could not be put on main's merge whitelist — ${said(err)}` });
continue;
}
this.writable.add(full);
if (!changed) {
result.already.push(full);
continue;
}
result.granted.push(full);
await this.emit(WRITABLE_EVENT, { owner, repo: name, account: this.account }).catch((err) =>
this.log(`${this.account} writes ${full} now, and that could not be said on the bus — ${said(err)}`));
}
if (!named) {
for (const repo of await this.forge.everyRepository()) {
if (repo.owner === this.org || delivered.includes(repo.full_name)) continue;
let permission: string;
try {
permission = await this.forge.repositoryPermission(repo.owner, repo.name, this.account);
} catch (err) {
permission = `unknown — ${said(err)}`;
}
if (WRITES.has(permission)) continue;
result.left_read_only.push({ repository: repo.full_name, permission,
why: "builds no registered module (the indexer names none from it), so the account is given no write there" });
}
}
return result;
}
/** Every delivered repository the account cannot write, as the forge says now, and why. */
async unwritable(): Promise<{ account: string; delivered: number; unwritable: { repository: string; permission: string; why: string }[];
merge_blocked: { repository: string; why: string }[] }> {
if (!this.heard) throw new Error(`which repositories build a registered module is not known yet: ${INDEXER_MODULES} has not been read`);
const delivered = this.delivered();
const out: { repository: string; permission: string; why: string }[] = [];
const blocked: { repository: string; why: string }[] = [];
for (const full of delivered) {
const [owner, name] = split(full);
let permission: string;
try {
permission = await this.forge.repositoryPermission(owner, name, this.account);
} catch (err) {
out.push({ repository: full, permission: "unknown", why: `the forge did not answer ${this.account}'s permission on it — ${said(err)}` });
continue;
}
if (WRITES.has(permission)) {
if (owner === this.org) continue; // the organisation's main rules are the mesh's own (mainrules.ts)
try {
const blocks = mergeBlocks(await this.forge.mainRule(owner, name), this.account);
if (blocks.length > 0) blocked.push({ repository: full, why: blockedWhy(blocks) });
} catch (err) {
blocked.push({ repository: full, why: `writable, but main's rule could not be read — ${said(err)}` });
}
continue;
}
const modules = [...this.modules].filter(([, r]) => r === full).map(([m]) => m).sort();
out.push({ repository: full, permission, why: `builds ${modules.join(", ")}, and the forge gives ${this.account} ` +
`"${permission}" on it: its commit status and its merge are refused until gitea_grant_write grants it` });
}
return { account: this.account, delivered: delivered.length, unwritable: out, merge_blocked: blocked };
}
}
type Tool = { name: string; description: string; input: Record<string, unknown>; run: (args: Readonly<Record<string, unknown>>) => Promise<unknown> };
/** The two tools: which delivered repositories the merging account cannot write, and the grant now. */
export function writeTools(writes: () => DeliveredWrites): Tool[] {
return [
{
name: "gitea_unwritable_repos",
description: "Every repository that builds a registered module (the indexer's modules, read over the bus) on which " +
"the mesh's merging forge account (mesh-tools, which sets commit statuses and merges) cannot write, with the " +
"permission the forge gives it there and why: such a pull request can be checked but never merged, and the " +
"delivery gate does not ask its Approve. Empty when it writes every one. Its merge_blocked lists each repository of " +
"a user it writes whose main's rule (the operator's, left as set) still stops the mesh's merge: a merge whitelist " +
"without it, a required status the mesh does not set, required approving reviews. Read with the site administrator's login.",
input: {},
run: async () => writes().unwritable(),
},
{
name: "gitea_grant_write",
description: "Give the mesh's merging forge account (mesh-tools) write, as a collaborator, on every repository of a " +
"user that builds a registered module, or on the repositories named (each must build one). Main's rule there is left " +
"as the operator set it; only when it keeps a merge whitelist is mesh-tools added to it, read back (merge_whitelist). " +
"What else in that rule stops the mesh's merge (a required status the mesh does not set, required approving reviews) " +
"is changed by nobody and said in merge_blocked. " +
"The module does this itself at its start and when a module of a new repository is registered; this is the same " +
"act by hand, such as when a merge or a commit status was refused 403. A repository of the organisation novox " +
"is written through its team and needs nothing. Answers what was granted (each read back from the forge), what " +
"was written already, what the forge refused in its own words, and which other repositories it left read-only " +
"and why (they build no registered module). Needs the site administrator's login, used for this grant alone.",
input: {
repositories: { type: "array", items: { type: "string" }, description: "owner/name of each repository; empty or absent, every delivered one" },
},
run: async (args) => writes().grant(Array.isArray(args.repositories) ? args.repositories.map(String) : undefined),
},
];
}
@@ -43,6 +43,7 @@ func (m *Module) askSteps(steps []StepTool) buildstep.AskSteps {
return delivery.ToolchainUsed{Version: installed, Recipe: m.Recipe.Version()}, nil
},
Kept: m.KeptLookup,
Has: m.Has,
Release: func(ws string) error { return m.spaces().Release(ws) },
Logf: m.Say,
}
@@ -96,7 +97,7 @@ func (m *Module) Puller(steps []StepTool, source buildstep.AskSource) *buildstep
}
}
return &buildstep.Puller{Source: source, Language: delivery.AskGo, Node: m.Node, Put: m.Steps, PausePut: m.Pauses, Load: load,
Capacity: capacity, WatchConfig: watch, Logf: m.Say,
Capacity: capacity, WatchConfig: watch, Logf: m.Say, Has: m.Has,
Run: func(ctx context.Context, a delivery.StepAsk) (any, error) {
if err := m.Check(ctx); err != nil {
return nil, err
@@ -115,3 +116,11 @@ func (m *Module) Puller(steps []StepTool, source buildstep.AskSource) *buildstep
func (m *Module) SeatTools(p *buildstep.Puller) []stdio.Tool {
return buildstep.SeatTools(Seat, "Go", p)
}
// StepEndTool ends one step left running (issue 1890); the module's steps state is the one it reads and writes.
const StepEndTool = "build_go_step_end"
// stepScan is the module's steps state as buildstep reads it.
func (m *Module) stepScan() buildstep.Scan {
return buildstep.ScanOf(stdio.State(delivery.StepsState))
}
@@ -183,12 +183,12 @@ func (m *Module) buildEnv(dir string) []string {
} else if modfile, ok := m.pinModfile(dir); ok {
mod += " -modfile=" + modfile // a pin on a pull request's head the fetch placed beside the workspace (pins.go)
}
return []string{
return append([]string{
"CGO_ENABLED=0",
"GOFLAGS=" + recipeFlags + " " + mod,
"GOTOOLCHAIN=local",
"GOMODCACHE=" + filepath.Join(m.Cache, "gomod"),
"GOCACHE=" + filepath.Join(m.Cache, "gobuild"),
"GOPROXY=off",
}
}, m.threadEnv()...)
}
@@ -13,6 +13,8 @@ import (
"path/filepath"
"strings"
"sync"
"git.novox.be/novox/mesh-sdk/go/forgekey"
)
// The runners of a digest or test step: this installed program, or the candidate's compiled from the workspace.
@@ -108,7 +110,7 @@ func (m *Module) mergeCommit(ctx context.Context, workspace string) (string, err
r := m.Run(ctx, nil, "git", "-C", filepath.Join(m.Work, workspace, "tree"), "rev-parse", "HEAD")
commit := strings.TrimSpace(r.Stdout)
if r.Status != 0 || r.Err != "" || commit == "" {
return "", fmt.Errorf("the workspace's commit: git rev-parse: %s", firstLine(r.Err+"\n"+r.Stderr))
return "", fmt.Errorf("the workspace's commit: git rev-parse: %s", forgekey.Said(r.Err+"\n"+r.Stderr))
}
return commit, nil
}
@@ -0,0 +1,36 @@
package main
import (
"git.novox.be/novox/mesh-sdk/go/buildcpu"
"git.novox.be/novox/mesh-sdk/go/buildstep"
)
// The processors a step may take (novox/hq ADR 0553, issue 1806). The node-engine holds this module's runtime, with
// every other build module's of the node, in one slice at idle priority under the node's one share of its processors:
// that quota bounds the total. Threads only size each step: the share the composer gives (buildstep.NodeLimits) over
// this module's own capacity, so a step does not wait in the kernel's throttle. Without the composer's share there is
// no slice to size a step to, and no thread limit is set.
// cpuLimits is what one step is held to now.
func (m *Module) cpuLimits() buildcpu.Limits {
slots := 1
switch {
case m.Capacity != nil:
slots = m.Capacity()
case m.Settings != "":
slots = buildstep.NodeCapacity(m.Settings, buildstep.DefaultCapacities, m.Node, nil)()
}
return buildstep.NodeLimits(slots)
}
// threadEnv is the environment (nothing without a share) every go command of a step is given for its threads: GOMAXPROCS sets its -p, the compiler's
// concurrency, a test binary's processors and -parallel.
func (m *Module) threadEnv() []string {
return buildcpu.Env(m.cpuLimits().Threads)
}
// cpuStatus is what the status tool says of it: the limits a step is given, and what the kernel holds this runtime
// to (its unit and its slice), read from its cgroup rather than assumed.
func (m *Module) cpuStatus() map[string]any {
return map[string]any{"limits": m.cpuLimits(), "in-force": buildcpu.ReadInForce()}
}
@@ -11,6 +11,7 @@ import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"io/fs"
@@ -19,6 +20,8 @@ import (
"sort"
"strconv"
"strings"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// DigestVersion is the form of the digest's text: changed whenever what the digest hashes changes in kind.
@@ -39,6 +42,33 @@ const besideVariable = "MESH_CHECK_BESIDE"
// test log) with each verdict.
const ReadsOutsideMarker = "reads-outside"
// NeedsMarker is the name of a file in a package's directory that names, one per line, what its integration tests need
// of the node they run on (issue 890; mesh-sdk delivery.AskNeeds): `node-build-node` for tests that read the Node build
// module's installed toolchain. The digest reports it (DigestResult.Needs); a check ask leaves such a package untested on
// a node that lacks it, and the checker asks it again of a node that has it. A marker beside the tests, as
// ReadsOutsideMarker is, and not a manifest field: it is a property of the package's tests, read only where they are
// run, and the manifest describes the module on the machines. The unit level is never given anything of the node (ADR
// 0533 §1), so a marker binds the integration level alone. Blank lines and lines starting with # are not read.
const NeedsMarker = "check-needs"
// readNeeds is what the marker in dir names; nil when there is none.
func readNeeds(dir string) ([]string, error) {
raw, err := os.ReadFile(filepath.Join(dir, NeedsMarker))
if errors.Is(err, fs.ErrNotExist) {
return nil, nil
}
if err != nil {
return nil, err
}
var needs []string
for _, l := range strings.Split(string(raw), "\n") {
if l = strings.TrimSpace(l); l != "" && !strings.HasPrefix(l, "#") {
needs = append(needs, l)
}
}
return needs, nil
}
// integrationTags are the build tags of the integration level, its run and its digest alike: integration, and isolation
// for the bubblewrap proofs (mesh-sdk's and the build modules'), which a build node can run.
var integrationTags = []string{"integration", "isolation"}
@@ -101,6 +131,8 @@ type DigestResult struct {
// ReadsOutside names the packages that carry the marker: each has a digest that no kept verdict can have (a fresh
// random one), so the check runs it every time.
ReadsOutside []string `json:"reads_outside,omitempty"`
// Needs are the packages whose tests at this level need something of the node, as their NeedsMarker names it.
Needs []delivery.PackageNeeds `json:"needs,omitempty"`
// JudgedBy is the program that ran the digest (ADR 0535 §5): "installed", or "candidate <merge commit>" under the
// runner workspace.
JudgedBy string `json:"judged_by,omitempty"`
@@ -206,12 +238,13 @@ func (m *Module) digestHere(ctx context.Context, workspace, level string) (Diges
if err != nil {
return DigestResult{}, err
}
if err := digestModule(&res, pkgs, tree, header, besideHeader); err != nil {
if err := digestModule(&res, pkgs, tree, header, besideHeader, level == LevelIntegration); err != nil {
return DigestResult{}, err
}
}
sort.Strings(res.WithoutTests)
sort.Strings(res.ReadsOutside)
sort.Slice(res.Needs, func(i, j int) bool { return res.Needs[i].Package < res.Needs[j].Package })
for _, p := range res.ReadsOutside {
if m.Say != nil {
m.Say("the %s level of %s always runs: its directory carries %s, so its tests read files outside it that no digest names", level, p, ReadsOutsideMarker)
@@ -256,7 +289,7 @@ func parsePackages(out string) (map[string]*goPackage, error) {
// digestModule adds to res the digest of each root package of one go.mod directory. A root is a package of the main
// module that is listed, not only a dependency, and not a variant of another.
func digestModule(res *DigestResult, pkgs map[string]*goPackage, tree string, header, beside []string) error {
func digestModule(res *DigestResult, pkgs map[string]*goPackage, tree string, header, beside []string, needs bool) error {
roots := []string{}
for path, p := range pkgs {
if p.ForTest == "" && !p.DepOnly && !p.Standard && !strings.HasSuffix(path, ".test") && p.Module != nil && p.Module.Main {
@@ -274,6 +307,15 @@ func digestModule(res *DigestResult, pkgs map[string]*goPackage, tree string, he
if testMain == nil {
return fmt.Errorf("go list gave no test binary for %s", path)
}
if needs {
n, err := readNeeds(root.Dir)
if err != nil {
return fmt.Errorf("%s: its %s: %w", path, NeedsMarker, err)
}
if len(n) > 0 {
res.Needs = append(res.Needs, delivery.PackageNeeds{Package: path, Needs: n})
}
}
if _, err := os.Stat(filepath.Join(root.Dir, ReadsOutsideMarker)); err == nil {
// The digest of a package that reads outside itself is one nothing can have kept: the key is a fresh random
// hex, so the ask finds no kept verdict and runs it, and what the checker writes under it is never read again.
@@ -213,6 +213,16 @@ func TestAPackageWithTheReadsOutsideMarkerIsNeverKept(t *testing.T) {
if first.Packages[a] != plain.Packages[a] || first.Packages[a] != second.Packages[a] {
t.Errorf("a has no marker and its digest moved: %s, %s, before %s", first.Packages[a], second.Packages[a], plain.Packages[a])
}
// Issue 890/AC3: a package whose directory carries the needs marker is named with its needs at the integration level,
// comments and blank lines not read, and at the unit level not at all.
edit(t, tree, "a/"+NeedsMarker, "# its integration tests copy the installed Node toolchain\n\nnode-build-node\n")
if n := digests(t, m, LevelIntegration).Needs; len(n) != 1 || n[0].Package != a || strings.Join(n[0].Needs, ",") != "node-build-node" {
t.Errorf("the integration level's needs = %+v, want %s needing node-build-node", n, a)
}
if n := digests(t, m, LevelUnit).Needs; len(n) != 0 {
t.Errorf("the unit level names needs: %+v", n)
}
}
// Issue 1935: a catalogue module's packages read ../../module.json and the module's files: a change to them changes the
@@ -18,6 +18,7 @@ import (
"time"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/forgekey"
)
var commitOK = regexp.MustCompile(`^[0-9a-f]{40}$`)
@@ -255,7 +256,7 @@ func (m *Module) downloadAndVerify(ctx context.Context, dir, d string) error {
func (m *Module) goModDirs(ctx context.Context, tree string) ([]string, error) {
r := m.Run(ctx, nil, "git", "-C", tree, "ls-files", "--", "go.mod", "*/go.mod")
if r.Status != 0 || r.Err != "" {
return nil, fmt.Errorf("git ls-files: %s", firstLine(r.Err+" "+r.Stderr))
return nil, fmt.Errorf("git ls-files: %s", forgekey.Said(r.Err+"\n"+r.Stderr))
}
dirs := []string{}
for _, l := range strings.Split(r.Stdout, "\n") {
@@ -292,7 +293,7 @@ func (m *Module) goEnv() []string {
if m.GoModCache != "" {
modcache = m.GoModCache
}
return []string{
return append([]string{
"GOTOOLCHAIN=local",
"GOFLAGS=-mod=mod",
"GOMODCACHE=" + modcache,
@@ -300,7 +301,7 @@ func (m *Module) goEnv() []string {
"GOPRIVATE=git.novox.be",
"GIT_TERMINAL_PROMPT=0",
"GIT_SSH_COMMAND=ssh -o BatchMode=yes",
}
}, m.threadEnv()...)
}
// goEnvFor is goEnv for a command run in dir: a directory that vendors (vendor/modules.txt) is read from its vendor
@@ -331,10 +332,11 @@ func (m *Module) clock() time.Time {
func (m *Module) now() time.Time { return m.clock() }
// causeLine is the first line of a go command's output that says something other than Go's own progress: `go:
// downloading ...` comes first when Go fetches a module, and the cause of a failure on a later line.
// downloading ...` comes first when Go fetches a module, and `# <package>` heads a compile error of go build or go vet
// (issue 1703: the reason said only the package), and the cause of a failure on a later line.
func causeLine(s string) string {
for _, l := range strings.Split(s, "\n") {
if l = strings.TrimSpace(l); l != "" && !strings.HasPrefix(l, "go: downloading ") {
if l = strings.TrimSpace(l); l != "" && !strings.HasPrefix(l, "go: downloading ") && !strings.HasPrefix(l, "# ") {
return l
}
}
@@ -302,3 +302,16 @@ func TestWorkAndCacheAreTheDirectoriesTheManifestPlacesOutsideTheHome(t *testing
}
}
}
// issue 1703: a go command's reason is its cause, not Go's progress nor go build's `# <package>` heading.
func TestCauseLineSkipsProgressAndThePackageHeading(t *testing.T) {
for in, want := range map[string]string{
"go: downloading x v1\n# git.example/m/cmd\n./main.go:3:2: undefined: y\n": "./main.go:3:2: undefined: y",
"\ngo: example.com/x@v1: reading: 404\n": "go: example.com/x@v1: reading: 404",
"go: downloading x v1\n": "go: downloading x v1",
} {
if got := causeLine(in); got != want {
t.Errorf("causeLine(%q) = %q, want %q", in, got, want)
}
}
}
@@ -5,10 +5,13 @@ import (
"os"
"path/filepath"
"reflect"
"runtime"
"strconv"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/buildcpu"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/forgekey"
"git.novox.be/novox/mesh-sdk/go/health"
@@ -28,6 +31,7 @@ func toolchainValues(env []string) []string {
// 1572/AC1 (ADR 0477 §1): every environment a step gives its go commands says GOTOOLCHAIN=local, once, and names no
// release.
func TestEveryStepEnvironmentSaysGotoolchainLocal(t *testing.T) {
t.Setenv("MESH_BUILD_CPU_SHARE", "50")
m := &Module{Cache: t.TempDir()}
vend := t.TempDir()
os.MkdirAll(filepath.Join(vend, "vendor"), 0o755)
@@ -41,6 +45,24 @@ func TestEveryStepEnvironmentSaysGotoolchainLocal(t *testing.T) {
if got := toolchainValues(env); len(got) != 1 || got[0] != "GOTOOLCHAIN=local" {
t.Errorf("%s: GOTOOLCHAIN settings %v, want exactly GOTOOLCHAIN=local", name, got)
}
// Issue 1806: and its threads, its part of the node's processor cap (probeEnv runs no build).
if procs := strings.Count(strings.Join(env, "\n"), "GOMAXPROCS="); name != "probeEnv" && procs != 1 {
t.Errorf("%s: GOMAXPROCS said %d times, want once: %v", name, procs, env)
}
}
// Without the composer's share no thread limit; with it, the share over its own three slots (ADR 0553 §4).
t.Setenv("MESH_BUILD_CPU_SHARE", "")
m = &Module{Cache: t.TempDir(), Node: "shanks", Capacity: func() int { return 3 }}
if l := m.cpuLimits(); l.Threads != 0 || strings.Contains(strings.Join(m.goEnv(), " "), "GOMAXPROCS=") {
t.Errorf("without the share: %+v, goEnv %v", l, m.goEnv())
}
t.Setenv("MESH_BUILD_CPU_SHARE", "50")
want := buildcpu.Threads(50, runtime.NumCPU(), 3)
if l := m.cpuLimits(); l.Slots != 3 || !strings.Contains(strings.Join(m.goEnv(), " "), "GOMAXPROCS="+strconv.Itoa(want)) {
t.Errorf("limits %+v, goEnv %v; want %d threads", l, m.goEnv(), want)
}
if st := m.cpuStatus(); st["limits"].(buildcpu.Limits) != m.cpuLimits() {
t.Errorf("the status says %v", st)
}
}
@@ -40,7 +40,7 @@ func (m *Module) isolation(workspace string, write bool) buildstep.Isolation {
Workspace: filepath.Join(m.Work, workspace),
Caches: m.caches(),
WriteCaches: write,
Env: []string{"PATH=" + path, "LC_ALL=C"},
Env: append([]string{"PATH=" + path, "LC_ALL=C"}, m.threadEnv()...),
}
}
@@ -11,11 +11,13 @@ import (
"encoding/json"
"fmt"
"os"
"os/exec"
"path/filepath"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// levelArg is the first argument that makes this binary a step's shut-off process: the integration level's, and, when
@@ -140,6 +142,49 @@ func (m *Module) nodeToolchains() []string {
return []string{dir}
}
// nodeToolchainMark is the file the Node build module writes in a toolchain directory once it is installed whole
// (mesh-build-node toolchain.go installedMark).
const nodeToolchainMark = ".installed"
// Has says whether this node has a need a check ask or a package's tests name (issue 890, ADR 0552; mesh-sdk
// delivery.AskNeeds), read from the node itself at every ask: node-build-node when the Node build module's toolchains
// directory, given to the integration level as nodeToolchains gives it, holds a toolchain installed whole (its mark and
// its package-lock.json readable by this account), and node, npm and bwrap are installed: what mesh-build-node's
// integration tests (realMachine) look for before they run. Any other need it lacks.
func (m *Module) Has(need string) bool {
if m.HasNeed != nil {
return m.HasNeed(need)
}
switch need {
case delivery.NeedNodeBuildNode:
dirs := m.nodeToolchains()
if len(dirs) == 0 {
return false
}
for _, p := range []string{"node", "npm", "bwrap"} {
if _, err := exec.LookPath(p); err != nil {
return false
}
}
entries, err := os.ReadDir(dirs[0])
if err != nil {
return false
}
for _, e := range entries {
dir := filepath.Join(dirs[0], e.Name())
if _, err := os.Stat(filepath.Join(dir, nodeToolchainMark)); err != nil {
continue
}
if f, err := os.Open(filepath.Join(dir, "package-lock.json")); err == nil {
f.Close()
return true
}
}
return false
}
return false
}
func lastLine(s string) string {
lines := strings.Split(strings.TrimSpace(s), "\n")
return strings.TrimSpace(lines[len(lines)-1])
@@ -41,6 +41,8 @@ type PackageResult struct {
Tests int `json:"tests"`
// NoTests is a package with no test files (at this level: a file with the other level's build tag is none).
NoTests bool `json:"no_tests,omitempty"`
// Skipped is how many of the package's tests skipped: a pass that skipped proved less than it says, so the checker keeps no verdict for it.
Skipped int `json:"skipped,omitempty"`
}
// TestTime is one test and the time it took.
@@ -227,6 +229,7 @@ func (r *reader) event(ev event) {
r.res.Tests++
if ev.Action == "skip" {
r.res.Skipped++
p.Skipped++
}
r.times = append(r.times, TestTime{Package: ev.Package, Test: ev.Test, Elapsed: ev.Elapsed})
if ev.Action == "fail" {
@@ -8,9 +8,11 @@ package main
import (
"context"
"fmt"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"os"
"os/user"
"strings"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/delivery"
@@ -70,7 +72,13 @@ func main() {
os.Exit(1)
}
steps := m.StepTools()
booted := time.Now()
puller := m.Puller(steps, nil)
m.Held = puller
if m.Node != "" {
// A step a previous process left running is ended (issue 1890); this process's own steps start after booted.
go buildstep.EndStaleAtStart(ctx, m.stepScan(), m.Node, booted, logf)
}
if m.Node != "" {
// The seat's asks, pulled once Serve has the runtime's channel; a pull before it is said and tried again.
go func() { _ = puller.Pull(ctx) }()
@@ -9,6 +9,7 @@ import (
"syscall"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/forgekey"
)
// mirrorLocks hold one mutex per mirror of this process; the lock file beside the mirror holds off another process.
@@ -67,7 +68,7 @@ func (m *Module) dissociate(ctx context.Context, dir string) error {
return nil // no mirror was used
}
if r := m.Run(ctx, []string{"GIT_TERMINAL_PROMPT=0"}, "git", "-C", dir, "repack", "-a", "-d", "-q"); r.Status != 0 || r.Err != "" {
return fmt.Errorf("git repack in %s: %s", dir, firstLine(r.Err+" "+r.Stderr))
return fmt.Errorf("git repack in %s: %s", dir, forgekey.Said(r.Err+"\n"+r.Stderr))
}
return os.Remove(alternates)
}
@@ -2,6 +2,7 @@ package main
import (
"encoding/json"
"fmt"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"os"
"os/exec"
@@ -121,20 +122,35 @@ func TestTwoUpdatesOfOneMirrorNeverRunAtOnce(t *testing.T) {
}
}
// A mirror that cannot be updated does not fail the fetch: the fetch goes on without it.
// A mirror that cannot be updated does not fail the fetch: the fetch goes on without it, and the line saying so carries
// git's own words. The forge's refusal of a repository its account cannot read came out as "git -C: error:" (ace,
// 2026-10-26: git writes a bare "error:" line before the forge's message).
func TestAMirrorThatCannotBeUpdatedLetsTheFetchGoOnWithoutIt(t *testing.T) {
f := &fakeStep{gomods: "go.mod\n"}
f.rule = func(line string) (r runAnswer, ok bool) {
if strings.Contains(line, "mirror") && strings.Contains(line, " fetch ") {
return runAnswer{Status: 1, Stderr: "fatal: no space left\n"}, true
return runAnswer{Status: 128, Stderr: "error: \nerror: User: 22:nox-builder with Key: 30:mesh:ace.mesh-build-go is not authorized to read jschoubben/car-hunter.\nerror: \nfatal: Could not read from remote repository.\n"}, true
}
return r, false
}
m, states := fetchModule(t, f)
var mu sync.Mutex
var said []string
m.Say = func(format string, a ...any) {
mu.Lock()
defer mu.Unlock()
said = append(said, fmt.Sprintf(format, a...))
}
ws := fetchOnce(t, m, states, fetchArgs).Workspace
if _, err := os.Stat(filepath.Join(m.Work, ws, "tree", ".git", "objects", "info", "alternates")); err == nil {
t.Errorf("the workspace borrows from a mirror that is not there")
}
mu.Lock()
defer mu.Unlock()
all := strings.Join(said, "\n")
if !strings.Contains(all, "is not used: git fetch: error: User: 22:nox-builder with Key: 30:mesh:ace.mesh-build-go is not authorized to read jschoubben/car-hunter.; fatal: Could not read from remote repository.") {
t.Errorf("the mirror's line lost git's words:\n%s", all)
}
}
func sleepABit() { time.Sleep(30 * time.Millisecond) }
@@ -28,6 +28,8 @@ type Module struct {
// Toolchains and Steps are the Put of the states of those names.
Toolchains, Steps buildstep.Put
// ReadSteps and ReadToolchainsState read those two states back for the read tools (issue 1562).
// Held is the puller whose running asks a step-end tool will not end (issue 1890); nil in a test.
Held *buildstep.Puller
ReadSteps, ReadToolchainsState StateRead
// Pauses is the Put of the pauses state, which keeps the node's pause key without an age; ReadPauses reads it back.
Pauses buildstep.Put
@@ -73,6 +75,8 @@ type Module struct {
// NodeToolchains is where the Node build module keeps its installed toolchains, bound read-only into the
// integration level (ADR 0533 §1); NodeToolchains when empty. A test points it elsewhere.
NodeToolchains string
// HasNeed answers Has in place of the node's own reading; a test's.
HasNeed func(need string) bool
// StartBus starts the throwaway bus of a run in dir and answers its address and what stops it (the module's own
// program, StartBus in bus.go, when nil).
StartBus func(ctx context.Context, dir string) (url string, stop func(), err error)
@@ -121,6 +125,7 @@ func (m *Module) Start(ctx context.Context) {
m.Say("the toolchain: %v", err)
}
m.CleanWorkspaces(time.Now())
buildstep.EndStaleRound(m.stepScan(), m.Node, m.Say) // asks past the step bound, on the module's own round (issue 1890)
select {
case <-ctx.Done():
return
@@ -184,13 +189,14 @@ func (m *Module) Short(run func(ctx context.Context, args map[string]any) (any,
}
// Status is what the module reports of itself: the node, the recipe, the toolchain found and why it refuses, if it
// does.
// does, and the processors its steps may take (issue 1806).
func (m *Module) Status(ctx context.Context) map[string]any {
err := m.Check(ctx)
installed, _ := m.Probe(ctx)
out := map[string]any{
"node": m.Node, "recipe": m.Recipe.Version(), "installed": installed,
"postgres-major": m.Recipe.PostgresMajor, "nats-server": m.Recipe.NatsServer, "refusing": err != nil,
"cpu": m.cpuStatus(),
}
if err != nil {
out["why"] = err.Error()
@@ -115,7 +115,7 @@ func TestAGoThatCannotBeRunRefusesEveryStepAndHealthIsUnhealthy(t *testing.T) {
{Name: "short_probe", Run: run},
}, m.StepTools()...)
for _, tool := range m.Tools(steps) {
if tool.Name == StatusTool || tool.Name == HealthTool || tool.Name == StepReadTool || tool.Name == StepsReadTool || tool.Name == ToolchainsReadTool || tool.Name == ReleaseTool {
if tool.Name == StatusTool || tool.Name == HealthTool || tool.Name == StepReadTool || tool.Name == StepsReadTool || tool.Name == ToolchainsReadTool || tool.Name == ReleaseTool || tool.Name == StepEndTool {
continue
}
var reason string
@@ -71,6 +71,47 @@ func TestANodeWithoutNodeToolchainsRunsTheLevelWithNothingMoreBound(t *testing.T
if slices.Contains(s.roBinds, missing) {
t.Errorf("a missing %s is bound: %v", missing, s.roBinds)
}
// Issue 890/AC4: such a node lacks node-build-node; one with an installed toolchain and node and npm has it; nothing
// else is had.
bin := t.TempDir()
for _, p := range []string{"node", "npm", "bwrap"} {
if err := os.WriteFile(filepath.Join(bin, p), []byte("#!/bin/sh\n"), 0o755); err != nil {
t.Fatal(err)
}
}
t.Setenv("PATH", bin)
m := &Module{NodeToolchains: missing}
if m.Has(delivery.NeedNodeBuildNode) {
t.Error("a node with no Node toolchain has node-build-node")
}
installed := t.TempDir()
m.NodeToolchains = installed
if m.Has(delivery.NeedNodeBuildNode) {
t.Error("a node with an empty toolchains directory has node-build-node")
}
half := filepath.Join(installed, "0.1.0")
if err := os.Mkdir(half, 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(half, "package-lock.json"), []byte("{}"), 0o644); err != nil {
t.Fatal(err)
}
if m.Has(delivery.NeedNodeBuildNode) {
t.Error("a toolchain without its installed mark counts")
}
if err := os.WriteFile(filepath.Join(half, nodeToolchainMark), nil, 0o644); err != nil {
t.Fatal(err)
}
if !m.Has(delivery.NeedNodeBuildNode) || m.Has("gpu") {
t.Errorf("with a toolchain installed: node-build-node %v, gpu %v; want true, false", m.Has(delivery.NeedNodeBuildNode), m.Has("gpu"))
}
if err := os.Remove(filepath.Join(bin, "bwrap")); err != nil {
t.Fatal(err)
}
if m.Has(delivery.NeedNodeBuildNode) {
t.Error("a node with no bubblewrap installed has node-build-node")
}
}
// 1891 (ADR 0533 §1): the unit level is not given them.
@@ -101,3 +101,25 @@ func TestALoggedPanicIsNotTheCause(t *testing.T) { // 1901/AC4
t.Errorf("1901/AC4: failures = %+v, want the timeout's panic as the cause", res.Failed)
}
}
// A package counts its own skipped tests, so the checker withholds a kept verdict from that package alone (issue 890).
func TestEachPackageCountsItsOwnSkippedTests(t *testing.T) {
log := `{"Action":"run","Package":"m/a","Test":"TestA"}
{"Action":"skip","Package":"m/a","Test":"TestA","Elapsed":0}
{"Action":"run","Package":"m/b","Test":"TestB"}
{"Action":"pass","Package":"m/b","Test":"TestB","Elapsed":0}
{"Action":"pass","Package":"m/a","Elapsed":0}
{"Action":"pass","Package":"m/b","Elapsed":0}
`
res := &TestResult{}
rd := newReader(res, ".")
rd.read(log)
rd.finish()
got := map[string]int{}
for _, p := range res.Packages {
got[p.Package] = p.Skipped
}
if res.Skipped != 1 || got["m/a"] != 1 || got["m/b"] != 0 {
t.Errorf("skipped = %d, per package %v, want 1 with m/a 1 and m/b 0", res.Skipped, got)
}
}
@@ -15,6 +15,8 @@ import (
"path/filepath"
"regexp"
"strings"
"git.novox.be/novox/mesh-sdk/go/forgekey"
)
const forgePrefix = "git.novox.be/novox/"
@@ -125,7 +127,7 @@ func (m *Module) placePin(ctx context.Context, root string, p pin, failure strin
repository := "novox/" + p.repo
r := m.Run(ctx, []string{"GIT_TERMINAL_PROMPT=0"}, "git", "-c", "credential.helper=", "ls-remote", m.remote(repository), "refs/pull/*/head")
if r.Status != 0 || r.Err != "" {
return "", fmt.Errorf("the pin of %s at commit %s: listing the pull requests of %s: %s", p.module, p.short, repository, firstLine(r.Err+" "+r.Stderr))
return "", fmt.Errorf("the pin of %s at commit %s: listing the pull requests of %s: %s", p.module, p.short, repository, forgekey.Said(r.Err+"\n"+r.Stderr))
}
commit, ref := "", ""
for _, l := range strings.Split(r.Stdout, "\n") {
@@ -128,6 +128,7 @@ type StepLine struct {
Started string `json:"started"`
Ended string `json:"ended,omitempty"`
Reason string `json:"reason,omitempty"`
What string `json:"what,omitempty"`
}
// RecentSteps are this node's steps, the latest started first, at most limit (ten when limit is not above zero). Only
@@ -180,7 +181,7 @@ func (m *Module) RecentSteps(limit int) (lines []StepLine, cut bool, err error)
}
out := make([]StepLine, 0, len(steps))
for _, s := range steps {
l := StepLine{ID: s.ID, Tool: s.Tool, State: s.State, Started: s.Started.UTC().Format("2006-01-02T15:04:05Z"), Reason: s.Reason}
l := StepLine{ID: s.ID, Tool: s.Tool, State: s.State, Started: s.Started.UTC().Format("2006-01-02T15:04:05Z"), Reason: s.Reason, What: s.What}
if !s.Ended.IsZero() {
l.Ended = s.Ended.UTC().Format("2006-01-02T15:04:05Z")
}
@@ -96,7 +96,7 @@ func TestStepsToolListsRecentStepsOfThisNode(t *testing.T) {
t0 := time.Date(2026, 10, 21, 9, 0, 0, 0, time.UTC)
putStep(t, steps, delivery.Step{ID: "a", Tool: "build_go_fetch", Node: "ace", State: delivery.StepPassed, Started: t0, Ended: t0, Result: json.RawMessage(`1`)})
putStep(t, steps, delivery.Step{ID: "b", Tool: "build_go_test", Node: "ace", State: delivery.StepRunning, Started: t0.Add(time.Minute)})
putStep(t, steps, delivery.Step{ID: "c", Tool: "build_go_build", Node: "ace", State: delivery.StepFailed, Started: t0.Add(2 * time.Minute), Reason: "boom"})
putStep(t, steps, delivery.Step{ID: "c", Tool: "build_go_build", Node: "ace", State: delivery.StepFailed, Started: t0.Add(2 * time.Minute), Reason: "boom", What: "check ask, novox/x@abc"})
putStep(t, steps, delivery.Step{ID: "d", Tool: "build_go_build", Node: "g14", State: delivery.StepPassed, Started: t0.Add(time.Hour)})
m := readModule(steps, fakeState{})
out, err := tool(t, m, StepsReadTool)(map[string]any{})
@@ -111,7 +111,7 @@ func TestStepsToolListsRecentStepsOfThisNode(t *testing.T) {
if !slices.Equal(ids, []string{"c", "b", "a"}) {
t.Errorf("ids = %v, want c b a", ids)
}
if lines[0].Reason != "boom" || lines[0].State != delivery.StepFailed {
if lines[0].Reason != "boom" || lines[0].What != "check ask, novox/x@abc" || lines[0].State != delivery.StepFailed {
t.Errorf("first = %+v", lines[0])
}
out, _ = tool(t, m, StepsReadTool)(map[string]any{"limit": float64(1)})
@@ -103,6 +103,33 @@ func TestUnitLevelNamesTheFailingTestAndFailsTheLevel(t *testing.T) {
}
}
// Issue 1921: a fixture with a failing test in each of two packages: one level's run names both, each with its own
// cause, and counts every test, the passing one beside a failure too. go test runs every package to its end; nothing
// stops at the first failure.
func TestUnitLevelNamesEveryFailingTestOfEveryPackage(t *testing.T) {
m, states, ws, calls := workspaceFor(t, "twofailing")
res, st := runUnit(t, m, states, ws)
if st.State != delivery.StepPassed || res.Passed {
t.Fatalf("step %s (%s), result %+v: want the step done and the level failed", st.State, st.Reason, res)
}
var got []string
for _, f := range res.Failed {
got = append(got, f.Package+"."+f.Test+": "+f.Cause)
}
want := []string{"fixture/twofailing/one.TestOneIsTwo: One() = 1, want 2", "fixture/twofailing/two.TestTwoIsThree: Two() = 2, want 3"}
if strings.Join(got, "\n") != strings.Join(want, "\n") {
t.Errorf("failed\n%s\nwant\n%s", strings.Join(got, "\n"), strings.Join(want, "\n"))
}
if res.Tests != 3 {
t.Errorf("%d tests counted, want 3", res.Tests)
}
for _, c := range *calls {
if strings.Contains(c.line, " test ") && strings.Contains(c.line, "-failfast") {
t.Errorf("go test ran with -failfast: %s", c.line)
}
}
}
// 1290/AC2: a fixture whose tests take more than the bound (lowered by the module's option, not by waiting five
// seconds): the level fails naming the bound and the slowest tests; under the real bound the same run passes.
func TestUnitLevelOverItsBoundFailsNamingTheBoundAndTheSlowestTests(t *testing.T) {
@@ -1,6 +1,7 @@
Each file is the standard output of `GOTOOLCHAIN=go1.27.1 GOFLAGS=-mod=mod go test -count=1 -json ./...` (go1.27.1)
run in the fixture module of the same name under ../repos, captured on 2026-10-20 for issue 1290. The unit
tests feed them to the parser through a fake runner, so no test here runs `go test`.
tests feed them to the parser through a fake runner, so no test here runs `go test`. twofailing.json was captured the
same way on 2026-10-24 for issue 1921: two packages with a failing test each, and a passing test beside one of them.
golist-external.json is `GOTOOLCHAIN=go1.27.1 GOFLAGS=-mod=mod go list -json=ImportPath,Standard,DepOnly,Module
git.novox.be/novox/mesh-sdk/go/buildstep` run in this module (go1.27.1, 2026-10-20 for issue 1291): a package of a module
@@ -0,0 +1,23 @@
{"Time":"2026-10-24T23:45:37.721718705+02:00","Action":"start","Package":"fixture/twofailing/one"}
{"Time":"2026-10-24T23:45:37.723685152+02:00","Action":"run","Package":"fixture/twofailing/one","Test":"TestOneIsTwo"}
{"Time":"2026-10-24T23:45:37.723726472+02:00","Action":"output","Package":"fixture/twofailing/one","Test":"TestOneIsTwo","Output":"=== RUN TestOneIsTwo\n","OutputType":"frame"}
{"Time":"2026-10-24T23:45:37.723743682+02:00","Action":"output","Package":"fixture/twofailing/one","Test":"TestOneIsTwo","Output":" one_test.go:6: checking One\n"}
{"Time":"2026-10-24T23:45:37.723750622+02:00","Action":"output","Package":"fixture/twofailing/one","Test":"TestOneIsTwo","Output":" one_test.go:8: One() = 1, want 2\n","OutputType":"error"}
{"Time":"2026-10-24T23:45:37.723759302+02:00","Action":"output","Package":"fixture/twofailing/one","Test":"TestOneIsTwo","Output":"--- FAIL: TestOneIsTwo (0.00s)\n","OutputType":"frame"}
{"Time":"2026-10-24T23:45:37.723889303+02:00","Action":"start","Package":"fixture/twofailing/two"}
{"Time":"2026-10-24T23:45:37.723762972+02:00","Action":"fail","Package":"fixture/twofailing/one","Test":"TestOneIsTwo","Elapsed":0}
{"Time":"2026-10-24T23:45:37.723952274+02:00","Action":"run","Package":"fixture/twofailing/one","Test":"TestOneIsOne"}
{"Time":"2026-10-24T23:45:37.723955484+02:00","Action":"output","Package":"fixture/twofailing/one","Test":"TestOneIsOne","Output":"=== RUN TestOneIsOne\n","OutputType":"frame"}
{"Time":"2026-10-24T23:45:37.723959304+02:00","Action":"output","Package":"fixture/twofailing/one","Test":"TestOneIsOne","Output":"--- PASS: TestOneIsOne (0.00s)\n","OutputType":"frame"}
{"Time":"2026-10-24T23:45:37.723964914+02:00","Action":"pass","Package":"fixture/twofailing/one","Test":"TestOneIsOne","Elapsed":0}
{"Time":"2026-10-24T23:45:37.723967704+02:00","Action":"output","Package":"fixture/twofailing/one","Output":"FAIL\n","OutputType":"frame"}
{"Time":"2026-10-24T23:45:37.724225126+02:00","Action":"output","Package":"fixture/twofailing/one","Output":"FAIL\tfixture/twofailing/one\t0.002s\n","OutputType":"frame"}
{"Time":"2026-10-24T23:45:37.724231936+02:00","Action":"fail","Package":"fixture/twofailing/one","Elapsed":0.003}
{"Time":"2026-10-24T23:45:37.725752949+02:00","Action":"run","Package":"fixture/twofailing/two","Test":"TestTwoIsThree"}
{"Time":"2026-10-24T23:45:37.725796309+02:00","Action":"output","Package":"fixture/twofailing/two","Test":"TestTwoIsThree","Output":"=== RUN TestTwoIsThree\n","OutputType":"frame"}
{"Time":"2026-10-24T23:45:37.725808239+02:00","Action":"output","Package":"fixture/twofailing/two","Test":"TestTwoIsThree","Output":" two_test.go:7: Two() = 2, want 3\n","OutputType":"error"}
{"Time":"2026-10-24T23:45:37.72581671+02:00","Action":"output","Package":"fixture/twofailing/two","Test":"TestTwoIsThree","Output":"--- FAIL: TestTwoIsThree (0.00s)\n","OutputType":"frame"}
{"Time":"2026-10-24T23:45:37.72582026+02:00","Action":"fail","Package":"fixture/twofailing/two","Test":"TestTwoIsThree","Elapsed":0}
{"Time":"2026-10-24T23:45:37.72582549+02:00","Action":"output","Package":"fixture/twofailing/two","Output":"FAIL\n","OutputType":"frame"}
{"Time":"2026-10-24T23:45:37.726319344+02:00","Action":"output","Package":"fixture/twofailing/two","Output":"FAIL\tfixture/twofailing/two\t0.002s\n","OutputType":"frame"}
{"Time":"2026-10-24T23:45:37.726362544+02:00","Action":"fail","Package":"fixture/twofailing/two","Elapsed":0.002}
@@ -0,0 +1,3 @@
module fixture/twofailing
go 1.25.0
@@ -0,0 +1,3 @@
package one
func One() int { return 1 }
@@ -0,0 +1,16 @@
package one
import "testing"
func TestOneIsTwo(t *testing.T) {
t.Log("checking One")
if One() != 2 {
t.Fatalf("One() = %d, want 2", One())
}
}
func TestOneIsOne(t *testing.T) {
if One() != 1 {
t.Fatal("one")
}
}
@@ -0,0 +1,3 @@
package two
func Two() int { return 2 }
@@ -0,0 +1,9 @@
package two
import "testing"
func TestTwoIsThree(t *testing.T) {
if Two() != 3 {
t.Errorf("Two() = %d, want 3", Two())
}
}
@@ -2,6 +2,7 @@ package main
import (
"context"
"git.novox.be/novox/mesh-sdk/go/buildstep"
stdio "git.novox.be/novox/mesh-sdk/go"
)
@@ -40,6 +41,7 @@ func (m *Module) Tools(steps []StepTool) []stdio.Tool {
tools = append(tools, t)
}
tools = append(tools, m.ReadTools()...)
tools = append(tools, buildstep.StepEndTool(StepEndTool, m.Node, m.stepScan(), m.Held))
tools = append(tools, m.releaseTool())
return append(tools,
stdio.Tool{Name: StatusTool,
+1 -1
View File
@@ -3,7 +3,7 @@ module meshbuildgo
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025175617-518dae5ec9c8
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026142629-ff81bdf8cc8f
github.com/distribution/distribution/v3 v3.1.2
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025175617-518dae5ec9c8 h1:x4mvY7yiGTP5lnzxbiwyj1o24p05hAmzZP7xHg3XX2M=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025175617-518dae5ec9c8/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026142629-ff81bdf8cc8f h1:GYLCvz2x+JNUOTbXzyh2QwzX0sTzvRlXR7lGocWjk3A=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026142629-ff81bdf8cc8f/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/alicebob/miniredis/v2 v2.35.0 h1:QwLphYqCEAo1eu1TqPRN2jgVMPBweeQcR21jeqDCONI=
github.com/alicebob/miniredis/v2 v2.35.0/go.mod h1:TcL7YfarKPGDAthEtl5NBeHZfeUQj6OXMm/+iu5cLMM=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
+1
View File
@@ -39,6 +39,7 @@
"build_go_release",
"build_go_step",
"build_go_steps",
"build_go_step_end",
"build_go_toolchains",
"build_go_status",
"build_go_health"
@@ -66,3 +66,11 @@ func (m *Module) Puller(steps []StepTool, source buildstep.AskSource) *buildstep
func (m *Module) SeatTools(p *buildstep.Puller) []stdio.Tool {
return buildstep.SeatTools(Seat, "image", p)
}
// StepEndTool ends one step left running (issue 1890); the module's steps state is the one it reads and writes.
const StepEndTool = "build_image_step_end"
// stepScan is the module's steps state as buildstep reads it.
func (m *Module) stepScan() buildstep.Scan {
return buildstep.ScanOf(stdio.State(delivery.StepsState))
}
@@ -155,14 +155,18 @@ func (m *Module) build(ctx context.Context, args map[string]any) (any, error) {
}
local := localName(workspace, manifest.Module, art.Name)
argv := append(m.storeFlags(), "bud", "--isolation", "oci", "--tag", local, "--file", dockerfile)
// --network pasta: a RUN step's network is pasta's, in user space. buildah's default for a rootless RUN is a
// netavark bridge whose nftables rules cannot be applied for an account without a user session, so every RUN
// failed before it started ("setup network: netavark: nftables error"; issue 1703, reproduced with buildah 1.45.2).
argv := append(m.storeFlags(), "bud", "--isolation", "oci", "--network", "pasta", "--tag", local, "--file", dockerfile)
if art.Target != "" {
argv = append(argv, "--target", art.Target)
}
argv = append(argv, imageArgs(manifest)...)
argv = append(argv, contextDir)
if r := m.Run(ctx, m.env(), "buildah", argv...); r.Status != 0 || r.Err != "" {
return nil, fmt.Errorf("buildah bud of %s: %s", art.Name, lastLine(r.Err+"\n"+r.Stderr+"\n"+r.Stdout))
m.sayFailure("buildah bud of "+art.Name+" of "+manifest.Module, r)
return nil, fmt.Errorf("buildah bud of %s: %s", art.Name, cause(r))
}
defer m.remove(local)
digest, err := m.localDigest(ctx, local)
@@ -204,8 +208,11 @@ func localName(workspace, module, artifact string) string {
func (m *Module) localDigest(ctx context.Context, name string) (string, error) {
r := m.Run(ctx, m.env(), "buildah", append(m.storeFlags(), "images", "--no-trunc", "--format", "{{.Digest}}", name)...)
digest := strings.TrimSpace(r.Stdout)
if r.Status != 0 || r.Err != "" || !digestOK.MatchString(digest) {
return "", fmt.Errorf("the digest of the image %s could not be read: %s", name, lastLine(r.Err+"\n"+r.Stderr+"\n"+r.Stdout))
if r.Status != 0 || r.Err != "" {
return "", fmt.Errorf("the digest of the image %s could not be read: %s", name, cause(r))
}
if !digestOK.MatchString(digest) {
return "", fmt.Errorf("the digest of the image %s could not be read: buildah images answered %q, not a digest", name, cutFront(digest, CauseBytes))
}
return digest, nil
}
@@ -216,17 +223,6 @@ var digestOK = regexp.MustCompile(`^sha256:[0-9a-f]{64}$`)
func (m *Module) remove(name string) {
r := m.Run(m.Ctx, m.env(), "buildah", append(m.storeFlags(), "rmi", "--force", name)...)
if (r.Status != 0 || r.Err != "") && m.Say != nil {
m.Say("the image %s could not be removed from the store: %s", name, lastLine(r.Err+"\n"+r.Stderr))
m.Say("the image %s could not be removed from the store: %s", name, cause(r))
}
}
// lastLine is the last line of a command's output that says something: buildah's error is its last line.
func lastLine(s string) string {
lines := strings.Split(s, "\n")
for i := len(lines) - 1; i >= 0; i-- {
if l := strings.TrimSpace(lines[i]); l != "" {
return l
}
}
return ""
}
@@ -2,6 +2,7 @@ package main
import (
"encoding/json"
"git.novox.be/novox/mesh-sdk/go/forgekey"
"os"
"path/filepath"
"strings"
@@ -69,6 +70,7 @@ func setup(t *testing.T, edit func(string) string) (*Module, *fake, *keptStates)
// Dockerfile of the module's own directory and that directory as the context, the target, the pinned base as a build
// argument; no mount, no credential file, a runtime directory of its own, and it answers the image's digest.
func TestBuildRunsBuildahRootlessAndAnswersTheDigest(t *testing.T) {
t.Setenv("MESH_BUILD_CPU_SHARE", "50") // the composer's share (ADR 0553): buildah is given its threads
m, f, states := setup(t, same)
res, st := runBuild(t, m, states, nil)
if st.State != delivery.StepPassed {
@@ -83,7 +85,7 @@ func TestBuildRunsBuildahRootlessAndAnswersTheDigest(t *testing.T) {
}
dir := filepath.Join(m.Work, "ws1", "tree", "modules", "demo")
for _, want := range []string{
"--root " + m.Store + "/root", "--runroot " + m.Store + "/run", "--isolation oci",
"--root " + m.Store + "/root", "--runroot " + m.Store + "/run", "--isolation oci", "--network pasta",
"--file " + dir + "/Dockerfile", "--target final",
"--build-arg DEMO_BASE=docker.io/library/busybox@sha256:0000000000000000000000000000000000000000000000000000000000000001",
} {
@@ -110,6 +112,9 @@ func TestBuildRunsBuildahRootlessAndAnswersTheDigest(t *testing.T) {
if !strings.Contains(env, "XDG_RUNTIME_DIR="+m.Store+"/xdg") {
t.Errorf("the environment of %q lacks the module's own runtime directory: %s", c.line, env)
}
if !strings.Contains(env, "GOMAXPROCS=") {
t.Errorf("the environment of %q gives buildah no thread limit under a share (ADR 0553): %s", c.line, env)
}
}
// The image is removed from the store once the digest is read, so the store does not grow with every build.
if len(f.called("rmi")) != 1 {
@@ -117,16 +122,50 @@ func TestBuildRunsBuildahRootlessAndAnswersTheDigest(t *testing.T) {
}
}
// 1703c: a failing Dockerfile step fails the step, saying buildah's last line.
func TestBuildFailsWithBuildahsLastLine(t *testing.T) {
m, f, states := setup(t, same)
f.failBud = "Error: building at STEP \"RUN false\": exit status 1"
_, st := runBuild(t, m, states, nil)
if st.State != delivery.StepFailed || !strings.Contains(st.Reason, `STEP "RUN false"`) {
t.Errorf("step = %+v", st)
// 1703c, issue 1703: a failing Dockerfile step fails the step, its reason saying buildah's error from stderr, never the
// last STEP line buildah writes on stdout; nothing is pushed.
func TestBuildFailsWithBuildahsError(t *testing.T) {
steps := "STEP 1/3: FROM docker.io/library/alpine\nSTEP 2/3: RUN apk add --no-cache ca-certificates\nfetch https://dl-cdn.alpinelinux.org/x\n"
long := strings.Repeat("x", 2*CauseBytes)
cases := map[string]struct {
ran forgekey.Ran
want []string
wantNot []string
}{
"buildah's RUN failure": {forgekey.Ran{Status: 1, Stdout: steps + "STEP 3/3: COPY . /app\n",
Stderr: "error running container: from /usr/bin/crun creating container for [/bin/sh -c apk add]: sd-bus call: Interactive authentication required.\nError: building at STEP \"RUN apk add --no-cache ca-certificates\": while running runtime: exit status 1\n"},
[]string{"exit status 1", "Interactive authentication required", `building at STEP "RUN apk add`}, []string{"STEP 3/3", "COPY"}},
"a failed RUN": {forgekey.Ran{Status: 1, Stdout: steps, Stderr: "Error: building at STEP \"RUN false\": exit status 1\n"},
[]string{`STEP "RUN false"`}, []string{"dl-cdn"}},
"a long stderr is bounded, its end kept": {forgekey.Ran{Status: 125, Stdout: steps, Stderr: "early noise\n" + long + "\nError: the end\n"},
[]string{"Error: the end"}, []string{"early noise", long}},
"nothing on stderr": {forgekey.Ran{Status: 2, Stdout: steps}, []string{"exit status 2", "nothing on stderr", "dl-cdn"}, nil},
"never ran": {forgekey.Ran{Err: "ENOENT"}, []string{"ENOENT", "wrote nothing"}, nil},
}
if len(f.called("push")) != 0 {
t.Error("a failed build pushed")
for name, c := range cases {
m, f, states := setup(t, same)
f.failBud = c.ran
_, st := runBuild(t, m, states, nil)
if st.State != delivery.StepFailed {
t.Errorf("%s: step = %+v, want failed", name, st)
continue
}
for _, w := range c.want {
if !strings.Contains(st.Reason, w) {
t.Errorf("%s: the reason %q lacks %q", name, st.Reason, w)
}
}
for _, w := range c.wantNot {
if strings.Contains(st.Reason, w) {
t.Errorf("%s: the reason %q carries %q", name, st.Reason, w)
}
}
if len(st.Reason) > CauseBytes+200 {
t.Errorf("%s: the reason is %d bytes, more than its bound", name, len(st.Reason))
}
if len(f.called("push")) != 0 {
t.Errorf("%s: a failed build pushed", name)
}
}
}
@@ -0,0 +1,90 @@
package main
// Why a command failed, as the command said it (issue 1703). buildah writes its progress (STEP n/m, the RUN's own
// output) on stdout and its error on stderr, so the last line of both together is a STEP line and hides the error:
// a failure's reason is stderr's last lines, never stdout's, and the whole tail of stderr is said in the journal.
import (
"fmt"
"strings"
"unicode/utf8"
"git.novox.be/novox/mesh-sdk/go/forgekey"
)
// CauseBytes bounds what a failure's reason carries of the command's output; TailBytes bounds what the journal gets.
const (
CauseBytes = 600
TailBytes = 4096
)
// cause says why a command failed: its exit status (or why it never ran to one) and the last lines it wrote on stderr,
// at most CauseBytes of them, joined with " | ". Only when stderr is empty does stdout's tail stand in, said as such.
func cause(r forgekey.Ran) string {
head := exitOf(r)
if said := tailLines(r.Stderr, CauseBytes, " | "); said != "" {
return head + ": " + said
}
if said := tailLines(r.Stdout, CauseBytes, " | "); said != "" {
return head + ", nothing on stderr; stdout ends: " + said
}
return head + ", and it wrote nothing"
}
// exitOf is how a command ended: why it never ran to an exit status, or the status.
func exitOf(r forgekey.Ran) string {
if r.Err != "" {
return r.Err
}
return fmt.Sprintf("exit status %d", r.Status)
}
// tailLines is the last non-empty lines of s, trimmed, joined with sep, at most max bytes: the lines are taken whole
// from the end while they fit, and when even the last one does not, its end is kept behind "…".
func tailLines(s string, max int, sep string) string {
var lines []string
for _, l := range strings.Split(s, "\n") {
if l = strings.TrimSpace(l); l != "" {
lines = append(lines, l)
}
}
var keep []string
n := 0
for i := len(lines) - 1; i >= 0; i-- {
add := len(lines[i])
if len(keep) > 0 {
add += len(sep)
}
if n+add > max {
if len(keep) == 0 {
keep = append(keep, "…"+cutFront(lines[i], max-len("…")))
}
break
}
keep = append([]string{lines[i]}, keep...)
n += add
}
return strings.Join(keep, sep)
}
// cutFront is the last at most max bytes of s, starting on a whole character.
func cutFront(s string, max int) string {
if max <= 0 {
return ""
}
if len(s) <= max {
return s
}
s = s[len(s)-max:]
for len(s) > 0 && !utf8.RuneStart(s[0]) {
s = s[1:]
}
return s
}
// sayFailure writes the tail of a failed command's stderr (and of its stdout, where buildah's RUN output goes) to the
// journal, so what the reason had to leave out can still be read there.
func (m *Module) sayFailure(what string, r forgekey.Ran) {
m.say("%s failed (%s); stderr ends:\n%s\nstdout ends:\n%s", what, exitOf(r),
tailLines(r.Stderr, TailBytes, "\n"), tailLines(r.Stdout, TailBytes, "\n"))
}
@@ -0,0 +1,36 @@
package main
import (
"git.novox.be/novox/mesh-sdk/go/buildcpu"
"git.novox.be/novox/mesh-sdk/go/buildstep"
)
// The processors a step may take (novox/hq ADR 0553, issue 1806). The node-engine holds this module's runtime, with
// every other build module's of the node, in one slice at idle priority under the node's one share of its processors:
// that quota bounds the total. Threads only size each step: the share the composer gives (buildstep.NodeLimits) over
// this module's own capacity, so a step does not wait in the kernel's throttle. Without the composer's share there is
// no slice to size a step to, and no thread limit is set.
// cpuLimits is what one step is held to now.
func (m *Module) cpuLimits() buildcpu.Limits {
slots := 1
switch {
case m.Capacity != nil:
slots = m.Capacity()
case m.Settings != "":
slots = buildstep.NodeCapacity(m.Settings, buildstep.DefaultCapacities, m.Node, nil)()
}
return buildstep.NodeLimits(slots)
}
// threadEnv is the environment (nothing without a share) buildah is given for its threads (it is a Go program). What a build's RUN lines start
// runs in the container's own environment, inside the runtime's cgroup: the slice's quota and the nice level bind it.
func (m *Module) threadEnv() []string {
return buildcpu.Env(m.cpuLimits().Threads)
}
// cpuStatus is what the status tool says of it: the limits a step is given, and what the kernel holds this runtime
// to (its unit and its slice), read from its cgroup rather than assumed.
func (m *Module) cpuStatus() map[string]any {
return map[string]any{"limits": m.cpuLimits(), "in-force": buildcpu.ReadInForce()}
}
@@ -70,8 +70,8 @@ type fake struct {
calls []cmdRecord
digest string // what `buildah images` prints
pushes int
refuse int // the first n pushes answer "connection refused"
failBud string
refuse int // the first n pushes answer "connection refused"
failBud forgekey.Ran // what a failing buildah bud answers; a zero Status is a passing bud
// digestFile: "" writes the digest, "none" writes no file, "empty" writes an empty one.
digestFile string
// registry: repository:tag -> digest
@@ -116,8 +116,8 @@ func (f *fake) run(ctx context.Context, env []string, name string, args ...strin
case name == "buildah" && strings.Contains(line, "--version"):
return forgekey.Ran{Stdout: "buildah version 1.45.1 (image-spec 1.1.0, runtime-spec 1.1.0)\n"}
case name == "buildah" && strings.Contains(line, " bud "):
if f.failBud != "" {
return forgekey.Ran{Status: 1, Stderr: "STEP 1/2: FROM x\n" + f.failBud + "\n"}
if f.failBud.Status != 0 || f.failBud.Err != "" {
return f.failBud
}
case name == "buildah" && strings.Contains(line, " images "):
return forgekey.Ran{Stdout: f.digest + "\n"}
@@ -203,6 +203,8 @@ func TestARealBuildIsRootlessWithNoSocketAndNoWayOut(t *testing.T) {
imageModule(t, m, "FROM "+base+"\nRUN echo b > /proc/sysrq-trigger\n", nil)
if _, st := buildReal(t, m, states); st.State != delivery.StepFailed {
t.Errorf("a write to the host's /proc/sysrq-trigger passed: %+v", st)
} else if strings.Contains(st.Reason, "setup network") {
t.Errorf("the RUN never started, so the write was not what failed (issue 1703): %s", st.Reason)
}
}
@@ -9,9 +9,11 @@ package main
import (
"context"
"fmt"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"os"
"os/user"
"strings"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/delivery"
@@ -57,7 +59,13 @@ func main() {
os.Exit(1)
}
steps := m.StepTools()
booted := time.Now()
puller := m.Puller(steps, nil)
m.Held = puller
if m.Node != "" {
// A step a previous process left running is ended (issue 1890); this process's own steps start after booted.
go buildstep.EndStaleAtStart(ctx, m.stepScan(), m.Node, booted, logf)
}
if m.Node != "" {
// The seat's asks, pulled once Serve has the runtime's channel; a pull before it is said and tried again.
go func() { _ = puller.Pull(ctx) }()
@@ -22,7 +22,9 @@ type Module struct {
// Account and UID are the account the module runs as; Home is its home.
Account, UID, Home string
// Steps is the Put of the steps state; ReadSteps reads it back for the read tools.
Steps buildstep.Put
Steps buildstep.Put
// Held is the puller whose running asks a step-end tool will not end (issue 1890); nil in a test.
Held *buildstep.Puller
ReadSteps StateRead
// Pauses is the Put of the pauses state, which keeps the node's pause key without an age; ReadPauses reads it back.
Pauses buildstep.Put
@@ -104,6 +106,7 @@ func (m *Module) Start(ctx context.Context) {
m.Say("this machine cannot build images: %v", err)
}
m.CleanWorkspaces(m.clock())
buildstep.EndStaleRound(m.stepScan(), m.Node, m.Say) // asks past the step bound, on the module's own round (issue 1890)
select {
case <-ctx.Done():
return
@@ -128,7 +131,8 @@ func (m *Module) Health(ctx context.Context) health.Answer {
// Status is what the module reports of itself.
func (m *Module) Status(ctx context.Context) map[string]any {
l, err := m.look(ctx)
out := map[string]any{"node": m.Node, "account": m.Account, "buildah": l.Buildah, "subuids": l.UIDs, "subgids": l.GIDs, "refusing": err != nil}
out := map[string]any{"node": m.Node, "account": m.Account, "buildah": l.Buildah, "subuids": l.UIDs, "subgids": l.GIDs, "refusing": err != nil,
"cpu": m.cpuStatus()}
if err != nil {
out["why"] = err.Error()
}
@@ -117,7 +117,7 @@ func (m *Module) storeFlags() []string {
// error, and the next look refuses with it.
func (m *Module) env() []string {
_ = m.ensureRuntimeDir()
return []string{"XDG_RUNTIME_DIR=" + m.runtimeDir()}
return append([]string{"XDG_RUNTIME_DIR=" + m.runtimeDir()}, m.threadEnv()...)
}
// runtimeDir is buildah's runtime directory, inside the store.
@@ -15,6 +15,8 @@ import (
"strings"
"syscall"
"time"
"git.novox.be/novox/mesh-sdk/go/forgekey"
)
// RegistryWait is how long a call waits for a registry that refuses connections, longer than the nightly collection
@@ -47,13 +49,15 @@ func (m *Module) push(ctx context.Context, workspace, module, local string, res
dest := addr + "/" + repository + ":" + tag
digestFile := filepath.Join(m.Work, workspace, "pushed."+tag)
argv := append(m.storeFlags(), "push", "--tls-verify=false", "--digestfile", digestFile, local, "docker://"+dest)
var last forgekey.Ran
if err := m.waitForRegistry(ctx, addr, "buildah push "+dest, func() error {
r := m.Run(ctx, m.env(), "buildah", argv...)
if r.Status != 0 || r.Err != "" {
return errors.New(lastLine(r.Err + "\n" + r.Stderr + "\n" + r.Stdout))
last = m.Run(ctx, m.env(), "buildah", argv...)
if last.Status != 0 || last.Err != "" {
return errors.New(cause(last))
}
return nil
}); err != nil {
m.sayFailure("buildah push "+dest, last)
return nil, fmt.Errorf("pushing %s: %w", dest, err)
}
var served string
@@ -127,6 +127,7 @@ type StepLine struct {
Started string `json:"started"`
Ended string `json:"ended,omitempty"`
Reason string `json:"reason,omitempty"`
What string `json:"what,omitempty"`
}
// RecentSteps are this node's steps, the latest started first, at most limit (ten when limit is not above zero). Only
@@ -179,7 +180,7 @@ func (m *Module) RecentSteps(limit int) (lines []StepLine, cut bool, err error)
}
out := make([]StepLine, 0, len(steps))
for _, s := range steps {
l := StepLine{ID: s.ID, Tool: s.Tool, State: s.State, Started: s.Started.UTC().Format("2006-01-02T15:04:05Z"), Reason: s.Reason}
l := StepLine{ID: s.ID, Tool: s.Tool, State: s.State, Started: s.Started.UTC().Format("2006-01-02T15:04:05Z"), Reason: s.Reason, What: s.What}
if !s.Ended.IsZero() {
l.Ended = s.Ended.UTC().Format("2006-01-02T15:04:05Z")
}
@@ -2,6 +2,7 @@ package main
import (
"context"
"git.novox.be/novox/mesh-sdk/go/buildstep"
stdio "git.novox.be/novox/mesh-sdk/go"
)
@@ -44,6 +45,7 @@ func (m *Module) Tools(steps []StepTool) []stdio.Tool {
tools = append(tools, t)
}
tools = append(tools, m.ReadTools()...)
tools = append(tools, buildstep.StepEndTool(StepEndTool, m.Node, m.stepScan(), m.Held))
tools = append(tools, m.releaseTool())
return append(tools,
stdio.Tool{Name: StatusTool,
+1 -1
View File
@@ -3,7 +3,7 @@ module meshbuildimage
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025175617-518dae5ec9c8
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026142629-ff81bdf8cc8f
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
)
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025175617-518dae5ec9c8 h1:x4mvY7yiGTP5lnzxbiwyj1o24p05hAmzZP7xHg3XX2M=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025175617-518dae5ec9c8/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026142629-ff81bdf8cc8f h1:GYLCvz2x+JNUOTbXzyh2QwzX0sTzvRlXR7lGocWjk3A=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026142629-ff81bdf8cc8f/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
+1
View File
@@ -33,6 +33,7 @@
"build_image_release",
"build_image_step",
"build_image_steps",
"build_image_step_end",
"build_image_status",
"build_image_health"
],
@@ -74,3 +74,11 @@ func (m *Module) Puller(steps []StepTool, source buildstep.AskSource) *buildstep
func (m *Module) SeatTools(p *buildstep.Puller) []stdio.Tool {
return buildstep.SeatTools(Seat, "Node", p)
}
// StepEndTool ends one step left running (issue 1890); the module's steps state is the one it reads and writes.
const StepEndTool = "build_node_step_end"
// stepScan is the module's steps state as buildstep reads it.
func (m *Module) stepScan() buildstep.Scan {
return buildstep.ScanOf(stdio.State(delivery.StepsState))
}
@@ -17,11 +17,11 @@ import (
"git.novox.be/novox/mesh-sdk/go/forgekey"
)
// realMachine gives the tests the real toolchain of the module's own lock and makes two workspaces holding one fixture
// bundle. The toolchain is the one this node's installed Node build module keeps (ADR 0533 §1): the integration level
// is given its toolchains read-only and no network, so it is copied into the test's own cache from there. Only where
// no toolchain is installed (a desk) is it installed with `npm ci --ignore-scripts` from the mesh's registry, as the
// module does, which needs the network.
// realMachine gives the tests the real toolchain of the module's own lock and makes three workspaces holding one fixture
// bundle. The toolchain is the one this node's installed Node build module keeps (ADR 0533 §1), copied into the test's
// own cache: the integration level is given its toolchains read-only and no network, so it is never installed from the
// registry here. Where no Node build module has installed one (a node without it, a desk), the test is skipped by
// name: an install from the public registry inside a shut-off level only fails, after 70 s of npm's retries.
func realMachine(t *testing.T) (*Module, string) {
t.Helper()
for _, p := range []string{"node", "npm", "bwrap"} { // bwrap: tsc, esbuild and the scripts run shut off (1309)
@@ -44,16 +44,19 @@ func realMachine(t *testing.T) (*Module, string) {
run := forgekey.ExecRunnerFor(0)
m := &Module{Node: "ace", Recipe: lock, Work: work, Run: run, Ctx: context.Background(),
Toolchain: &Toolchain{Paths: Paths{Home: home, Cache: cache}, Run: run, Registry: func() (Registry, error) {
return Registry{URL: "https://git.novox.be/api/packages/novox/npm/"}, nil
return Registry{}, fmt.Errorf("the integration test installs no toolchain from a registry: it has no network")
}}}
withStore(t, m)
if from := installedToolchain(t, filepath.Join(CacheDir, "toolchain"), filepath.Join(home, "toolchain", "package-lock.json")); from != "" {
copyToolchain(t, from, m.Toolchain.Paths.Dir(m.Recipe.Version()))
from := installedToolchain(t, filepath.Join(CacheDir, "toolchain"), filepath.Join(home, "toolchain", "package-lock.json"))
if from == "" {
t.Skipf("NOT RUN: no Node toolchain is installed under %s for this account to read (no Node build module on this node); "+
"the integration level has no network to install one, so this runs on a node that holds the Node build module", filepath.Join(CacheDir, "toolchain"))
}
if err := m.Toolchain.Ensure(context.Background(), m.Recipe.Version()); err != nil {
t.Fatalf("no toolchain of this lock is installed under %s, and installing it from the registry failed: %v", filepath.Join(CacheDir, "toolchain"), err)
copyToolchain(t, from, m.Toolchain.Paths.Dir(m.Recipe.Version()))
if err := m.Toolchain.Ensure(context.Background(), m.Recipe.Version()); err != nil { // present: opens it, installs nothing
t.Fatalf("the toolchain copied from %s cannot be used: %v", from, err)
}
for _, id := range []string{"w1", "w2"} {
for _, id := range []string{"w1", "w2", "w3"} {
for p, body := range map[string]string{
"modules/m/module.json": `{"module":"m","build":{"artifacts":[{"name":"tools","kind":"bundle","language":"typescript","entrypoints":["tools/index.js"]}]}}`,
"modules/m/package.json": `{"name":"m","type":"module","scripts":{"test":"node -e \"process.exit(0)\""}}`,
@@ -65,10 +68,36 @@ func realMachine(t *testing.T) (*Module, string) {
return m, work
}
// 1490/AC2: a fixture TypeScript bundle importing @novox/mesh-sdk, built twice by the real tsc and esbuild, has equal
// digests, and its archive holds the bundled .js, the .serve.mjs launcher and package.json.
func TestAFixtureBundleBuiltTwiceHasOneDigest(t *testing.T) {
// 1490/AC2, 1491 and issue 1869, in one test so the toolchain is copied once: the real type-check and test script of the
// fixture pass the unit level; a package whose test imports its own build output (as mesh-sdk's do) passes the unit level,
// the test step emitting it with the real tsc, shut off, before the test script runs; and the fixture TypeScript bundle
// importing @novox/mesh-sdk, built twice by the real tsc and esbuild, has equal digests, its archive holding the bundled
// .js, the .serve.mjs launcher and package.json. The unit levels run in w3, so the two builds' workspaces stay alike.
func TestTheFixturePassesTheUnitLevelAndBuildsTwiceToOneDigest(t *testing.T) {
m, work := realMachine(t)
out, err := m.test(context.Background(), map[string]any{"workspace": "w3", "module": "modules/m", "level": "unit"})
if err != nil {
t.Fatal(err)
}
if r := out.(*TestResult); !r.Passed || r.NoTests || len(r.Ran) != 2 {
t.Errorf("the fixture's unit level: result = %+v", r)
}
for p, body := range map[string]string{
"modules/p/package.json": `{"name":"p","type":"module","scripts":{"build":"tsc","test":"node --test test/a.test.mjs","prepare":"tsc"}}`,
"modules/p/tsconfig.json": `{"compilerOptions":{"module":"NodeNext","moduleResolution":"NodeNext","target":"ES2022","outDir":"dist","rootDir":"src"},"include":["src"]}`,
"modules/p/src/index.ts": "export const answer: number = 42;\n",
"modules/p/test/a.test.mjs": "import { test } from 'node:test';\nimport assert from 'node:assert';\nimport { answer } from '../dist/index.js';\n" +
"test('the build is there', () => assert.equal(answer, 42));\n",
} {
write(t, filepath.Join(work, "w3", "tree", filepath.FromSlash(p)), body, 0o644)
}
out, err = m.test(context.Background(), map[string]any{"workspace": "w3", "module": "modules/p", "level": "unit"})
if err != nil {
t.Fatal(err)
}
if r := out.(*TestResult); !r.Passed || !slices.Equal(r.Ran, []string{"build", "test"}) {
t.Errorf("the package importing its build: result = %+v", r)
}
var digests []string
for _, id := range []string{"w1", "w2"} {
out, err := m.build(context.Background(), map[string]any{"workspace": id, "module": "modules/m", "artifact": "tools"})
@@ -89,43 +118,6 @@ func TestAFixtureBundleBuiltTwiceHasOneDigest(t *testing.T) {
}
}
// 1491: the real type-check and test script of the same fixture pass the unit level.
func TestTheFixturePassesTheUnitLevel(t *testing.T) {
m, _ := realMachine(t)
out, err := m.test(context.Background(), map[string]any{"workspace": "w1", "module": "modules/m", "level": "unit"})
if err != nil {
t.Fatal(err)
}
if r := out.(*TestResult); !r.Passed || r.NoTests || len(r.Ran) != 2 {
t.Errorf("result = %+v", r)
}
}
// Issue 1869: a package whose test imports its own build output (as mesh-sdk's do) passes the unit level: the test step
// emits it with the real tsc, shut off, before the test script runs.
func TestAPackageWhoseTestsImportItsBuildPassesTheUnitLevel(t *testing.T) {
if _, err := exec.LookPath("bwrap"); err != nil {
t.Skip("NOT RUN: no bubblewrap installed where this check runs; every build node has it (the module declares it)")
}
m, work := realMachine(t)
for p, body := range map[string]string{
"modules/p/package.json": `{"name":"p","type":"module","scripts":{"build":"tsc","test":"node --test test/a.test.mjs","prepare":"tsc"}}`,
"modules/p/tsconfig.json": `{"compilerOptions":{"module":"NodeNext","moduleResolution":"NodeNext","target":"ES2022","outDir":"dist","rootDir":"src"},"include":["src"]}`,
"modules/p/src/index.ts": "export const answer: number = 42;\n",
"modules/p/test/a.test.mjs": "import { test } from 'node:test';\nimport assert from 'node:assert';\nimport { answer } from '../dist/index.js';\n" +
"test('the build is there', () => assert.equal(answer, 42));\n",
} {
write(t, filepath.Join(work, "w1", "tree", filepath.FromSlash(p)), body, 0o644)
}
out, err := m.test(context.Background(), map[string]any{"workspace": "w1", "module": "modules/p", "level": "unit"})
if err != nil {
t.Fatal(err)
}
if r := out.(*TestResult); !r.Passed || !slices.Equal(r.Ran, []string{"build", "test"}) {
t.Errorf("result = %+v", r)
}
}
// installedToolchain is the toolchain directory under toolchains that this node's Node build module installed: the
// one installed from the lock at lockPath, byte for byte, or, when the change carries a lock no delivered release has
// installed yet, the one installed last, said in the test's log (ADR 0533 §3: a new lock is proven by the module that
@@ -267,3 +267,38 @@ func TestTheLaunchersAreTheSDKsRule(t *testing.T) {
t.Errorf("launchers %v, mesh-sdk's rule %v", got, want)
}
}
// issue 1703: a failed command's reason keeps stderr's end and stdout's start, each bounded, so a long stdout (npm's
// progress) never pushes npm's error out, and tsc's first error on stdout stays in.
func TestRanFailedKeepsTheError(t *testing.T) {
noise := strings.Repeat("npm progress line\n", 200)
cases := map[string]struct {
ran forgekey.Ran
want []string
wantNot []string
}{
"passed": {forgekey.Ran{Stdout: "ok"}, nil, nil},
"npm, long stdout": {forgekey.Ran{Status: 1, Stdout: noise, Stderr: "npm ERR! early\n" + noise + "npm ERR! code E404\n"}, []string{"status 1", "npm ERR! code E404"}, []string{"npm ERR! early"}},
"tsc on stdout": {forgekey.Ran{Status: 2, Stdout: "src/a.ts(1,7): error TS2322: wrong\n" + noise}, []string{"error TS2322"}, nil},
"never ran": {forgekey.Ran{Err: "ENOENT"}, []string{"ENOENT"}, nil},
}
for name, c := range cases {
got := ranFailed(c.ran)
if c.want == nil && got != "" {
t.Errorf("%s: %q, want none", name, got)
}
for _, w := range c.want {
if !strings.Contains(got, w) {
t.Errorf("%s: %q lacks %q", name, got, w)
}
}
for _, w := range c.wantNot {
if strings.Contains(got, w) {
t.Errorf("%s: %q carries %q", name, got, w)
}
}
if len(got) > failedBytes+100 {
t.Errorf("%s: %d bytes, past its bound", name, len(got))
}
}
}
@@ -0,0 +1,3 @@
# Its integration tests copy the Node toolchain the Node build module installed on this node and run node and npm
# (realMachine, build_integration_test.go): they run only where the node holds node-build-node (issue 890).
node-build-node
@@ -0,0 +1,36 @@
package main
import (
"git.novox.be/novox/mesh-sdk/go/buildcpu"
"git.novox.be/novox/mesh-sdk/go/buildstep"
)
// The processors a step may take (novox/hq ADR 0553, issue 1806). The node-engine holds this module's runtime, with
// every other build module's of the node, in one slice at idle priority under the node's one share of its processors:
// that quota bounds the total. Threads only size each step: the share the composer gives (buildstep.NodeLimits) over
// this module's own capacity, so a step does not wait in the kernel's throttle. Without the composer's share there is
// no slice to size a step to, and no thread limit is set.
// cpuLimits is what one step is held to now.
func (m *Module) cpuLimits() buildcpu.Limits {
slots := 1
switch {
case m.Capacity != nil:
slots = m.Capacity()
case m.Settings != "":
slots = buildstep.NodeCapacity(m.Settings, buildstep.DefaultCapacities, m.Node, nil)()
}
return buildstep.NodeLimits(slots)
}
// threadEnv is the environment (nothing without a share) every command of a step is given for its threads: GOMAXPROCS for esbuild and every Go
// program, UV_THREADPOOL_SIZE for Node's worker pool.
func (m *Module) threadEnv() []string {
return buildcpu.Env(m.cpuLimits().Threads)
}
// cpuStatus is what the status tool says of it: the limits a step is given, and what the kernel holds this runtime
// to (its unit and its slice), read from its cgroup rather than assumed.
func (m *Module) cpuStatus() map[string]any {
return map[string]any{"limits": m.cpuLimits(), "in-force": buildcpu.ReadInForce()}
}
@@ -212,7 +212,7 @@ func (m *Module) fetchInto(ctx context.Context, req buildstep.FetchRequest, grou
func (m *Module) typeScriptModules(ctx context.Context, tree string) (dirs []string, packages map[string]bool, err error) {
r := m.Run(ctx, m.baseEnv(), "git", "-C", tree, "ls-files", "--", "module.json", "*/module.json")
if r.Status != 0 || r.Err != "" {
return nil, nil, fmt.Errorf("git ls-files: %s", firstLine(r.Err+" "+r.Stderr))
return nil, nil, fmt.Errorf("git ls-files: %s", forgekey.Said(r.Err+"\n"+r.Stderr))
}
packages = map[string]bool{}
for _, l := range strings.Split(r.Stdout, "\n") {
@@ -4,6 +4,7 @@ import (
"context"
"os"
"path/filepath"
"slices"
"strings"
"testing"
"time"
@@ -225,6 +226,7 @@ func mustReadDir(t *testing.T, dir string) []string {
// are given PATH, HOME and the locale and their own words; and the real clean runner, run on `env` here, prints only
// those.
func TestNoCommandOfTheStepSeesATokenOfTheProcess(t *testing.T) {
t.Setenv("MESH_BUILD_CPU_SHARE", "50")
t.Setenv("MESH_BUS_CREDENTIAL", "hunter2")
t.Setenv("NPM_TOKEN", "hunter3")
files := typeScriptModule(`{"dependencies":{"lodash":"^4"}}`)
@@ -245,6 +247,11 @@ func TestNoCommandOfTheStepSeesATokenOfTheProcess(t *testing.T) {
if len(c.env) < 3 || !strings.HasPrefix(c.env[0], "PATH=") || !strings.HasPrefix(c.env[1], "HOME=") || c.env[2] != "LC_ALL=C" {
t.Errorf("%s was given %v, want PATH, HOME and the locale first", c.name, c.env)
}
// And its threads, its part of the node's share the composer gave (ADR 0553).
if !slices.ContainsFunc(c.env, func(e string) bool { return strings.HasPrefix(e, "GOMAXPROCS=") }) ||
!slices.ContainsFunc(c.env, func(e string) bool { return strings.HasPrefix(e, "UV_THREADPOOL_SIZE=") }) {
t.Errorf("%s was given %v, without its threads", c.name, c.env)
}
}
// The runner main wires: what it hands a command is exactly what the step gave.
ran := forgekey.ExecRunnerClean(10*time.Second)(context.Background(), append(m.baseEnv(), "ONLY=this"), "env")
@@ -35,7 +35,7 @@ func (m *Module) isolation(root string, write bool) buildstep.Isolation {
ReadOnly: []string{m.toolchainDir()},
Caches: []string{m.npmCache()},
WriteCaches: write,
Env: []string{"PATH=" + pathOrDefault(), "LC_ALL=C"},
Env: append([]string{"PATH=" + pathOrDefault(), "LC_ALL=C"}, m.threadEnv()...),
}
}
@@ -9,10 +9,12 @@ package main
import (
"context"
"fmt"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"os"
"os/user"
"path/filepath"
"strings"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/delivery"
@@ -69,7 +71,13 @@ func main() {
os.Exit(1)
}
steps := m.StepTools()
booted := time.Now()
puller := m.Puller(steps, nil)
m.Held = puller
if m.Node != "" {
// A step a previous process left running is ended (issue 1890); this process's own steps start after booted.
go buildstep.EndStaleAtStart(ctx, m.stepScan(), m.Node, booted, logf)
}
if m.Node != "" {
// The seat's asks, pulled once Serve has the runtime's channel; a pull before it is said and tried again.
go func() { _ = puller.Pull(ctx) }()
@@ -42,6 +42,8 @@ type Module struct {
// Toolchains and Steps are the Put of the states of those names.
Toolchains, Steps buildstep.Put
// ReadSteps and ReadToolchainsState read those two states back for the read tools (issue 1562).
// Held is the puller whose running asks a step-end tool will not end (issue 1890); nil in a test.
Held *buildstep.Puller
ReadSteps, ReadToolchainsState StateRead
// Pauses is the Put of the pauses state, which keeps the node's pause key without an age; ReadPauses reads it back.
Pauses buildstep.Put
@@ -137,6 +139,7 @@ func (m *Module) Start(ctx context.Context) {
m.Say("the toolchain: %v", err)
}
m.CleanWorkspaces(m.now())
buildstep.EndStaleRound(m.stepScan(), m.Node, m.Say) // asks past the step bound, on the module's own round (issue 1890)
select {
case <-ctx.Done():
return
@@ -202,7 +205,8 @@ func (m *Module) Status(ctx context.Context) map[string]any {
err := m.Check(ctx)
installed, _ := m.Probe(ctx)
r := m.recipe()
out := map[string]any{"node": m.Node, "recipe": r.Version(), "installed": installed, "refusing": err != nil}
out := map[string]any{"node": m.Node, "recipe": r.Version(), "installed": installed, "refusing": err != nil,
"cpu": m.cpuStatus()}
if m.Toolchain != nil {
out["toolchain-dir"] = m.Toolchain.Paths.Dir(r.Version())
out["toolchain-installed"] = m.Toolchain.Present(r.Version())
@@ -227,7 +231,7 @@ const defaultPath = "/usr/local/bin:/usr/bin:/bin"
// files and git configuration) and a fixed locale. A command is given these and its own words, never this process's
// environment, which holds the module's bus credentials (ADR 0466 §4).
func (m *Module) baseEnv() []string {
return []string{"PATH=" + pathOrDefault(), "HOME=" + m.Toolchain.homeOrEmpty(), "LC_ALL=C"}
return append([]string{"PATH=" + pathOrDefault(), "HOME=" + m.Toolchain.homeOrEmpty(), "LC_ALL=C"}, m.threadEnv()...)
}
func pathOrDefault() string {
@@ -117,7 +117,7 @@ func TestANodeThatCannotBeRunRefusesEveryStepAndHealthIsUnhealthy(t *testing.T)
{Name: "short_probe", Run: run},
}, (&Module{}).StepTools()...)
for _, tool := range m.Tools(steps) {
if tool.Name == StatusTool || tool.Name == HealthTool || tool.Name == StepReadTool || tool.Name == StepsReadTool || tool.Name == ToolchainsReadTool || tool.Name == ReleaseTool {
if tool.Name == StatusTool || tool.Name == HealthTool || tool.Name == StepReadTool || tool.Name == StepsReadTool || tool.Name == ToolchainsReadTool || tool.Name == ReleaseTool || tool.Name == StepEndTool {
continue
}
var reason string
@@ -128,6 +128,7 @@ type StepLine struct {
Started string `json:"started"`
Ended string `json:"ended,omitempty"`
Reason string `json:"reason,omitempty"`
What string `json:"what,omitempty"`
}
// RecentSteps are this node's steps, the latest started first, at most limit (ten when limit is not above zero). Only
@@ -180,7 +181,7 @@ func (m *Module) RecentSteps(limit int) (lines []StepLine, cut bool, err error)
}
out := make([]StepLine, 0, len(steps))
for _, s := range steps {
l := StepLine{ID: s.ID, Tool: s.Tool, State: s.State, Started: s.Started.UTC().Format("2006-01-02T15:04:05Z"), Reason: s.Reason}
l := StepLine{ID: s.ID, Tool: s.Tool, State: s.State, Started: s.Started.UTC().Format("2006-01-02T15:04:05Z"), Reason: s.Reason, What: s.What}
if !s.Ended.IsZero() {
l.Ended = s.Ended.UTC().Format("2006-01-02T15:04:05Z")
}
@@ -96,7 +96,7 @@ func TestStepsToolListsRecentStepsOfThisNode(t *testing.T) {
t0 := time.Date(2026, 10, 21, 9, 0, 0, 0, time.UTC)
putStep(t, steps, delivery.Step{ID: "a", Tool: "build_node_fetch", Node: "ace", State: delivery.StepPassed, Started: t0, Ended: t0, Result: json.RawMessage(`1`)})
putStep(t, steps, delivery.Step{ID: "b", Tool: "build_node_test", Node: "ace", State: delivery.StepRunning, Started: t0.Add(time.Minute)})
putStep(t, steps, delivery.Step{ID: "c", Tool: "build_node_build", Node: "ace", State: delivery.StepFailed, Started: t0.Add(2 * time.Minute), Reason: "boom"})
putStep(t, steps, delivery.Step{ID: "c", Tool: "build_node_build", Node: "ace", State: delivery.StepFailed, Started: t0.Add(2 * time.Minute), Reason: "boom", What: "check ask, novox/x@abc"})
putStep(t, steps, delivery.Step{ID: "d", Tool: "build_node_build", Node: "g14", State: delivery.StepPassed, Started: t0.Add(time.Hour)})
m := readModule(steps, fakeState{})
out, err := tool(t, m, StepsReadTool)(map[string]any{})
@@ -111,7 +111,7 @@ func TestStepsToolListsRecentStepsOfThisNode(t *testing.T) {
if !slices.Equal(ids, []string{"c", "b", "a"}) {
t.Errorf("ids = %v, want c b a", ids)
}
if lines[0].Reason != "boom" || lines[0].State != delivery.StepFailed {
if lines[0].Reason != "boom" || lines[0].What != "check ask, novox/x@abc" || lines[0].State != delivery.StepFailed {
t.Errorf("first = %+v", lines[0])
}
out, _ = tool(t, m, StepsReadTool)(map[string]any{"limit": float64(1)})
@@ -2,6 +2,7 @@ package main
import (
"context"
"git.novox.be/novox/mesh-sdk/go/buildstep"
stdio "git.novox.be/novox/mesh-sdk/go"
)
@@ -40,6 +41,7 @@ func (m *Module) Tools(steps []StepTool) []stdio.Tool {
tools = append(tools, t)
}
tools = append(tools, m.ReadTools()...)
tools = append(tools, buildstep.StepEndTool(StepEndTool, m.Node, m.stepScan(), m.Held))
tools = append(tools, m.releaseTool())
return append(tools,
stdio.Tool{Name: StatusTool,
@@ -180,14 +180,28 @@ func (m *Module) runIn(ctx context.Context, root string, write bool, dir string,
return run(ctx, extra, "env", append([]string{"-C", dir, name}, args...)...)
}
// ranFailed says why a command did not pass, or "" when it did.
// ranFailed says why a command did not pass, or "" when it did (issue 1703): its exit status, the end of what it wrote
// on stderr (npm's error is its last lines) and the start of what it wrote on stdout (tsc writes its errors there,
// first first), each bounded, so a long stdout never pushes stderr's error out of the reason.
func ranFailed(r forgekey.Ran) string {
if r.Status == 0 && r.Err == "" {
return ""
}
out := strings.TrimSpace(r.Stdout + "\n" + r.Stderr)
if len(out) > 1500 {
out = out[:1500] + " ..."
why := fmt.Sprintf("status %d %s", r.Status, r.Err)
if e := strings.TrimSpace(r.Stderr); e != "" {
if len(e) > failedBytes/2 {
e = "... " + e[len(e)-failedBytes/2:]
}
why += ": stderr: " + e
}
return fmt.Sprintf("status %d %s: %s", r.Status, r.Err, out)
if o := strings.TrimSpace(r.Stdout); o != "" {
if len(o) > failedBytes/2 {
o = o[:failedBytes/2] + " ..."
}
why += ": stdout: " + o
}
return why
}
// failedBytes bounds what ranFailed carries of a command's output, stderr and stdout half each.
const failedBytes = 1500
+1 -1
View File
@@ -3,7 +3,7 @@ module meshbuildnode
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025175617-518dae5ec9c8
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026142629-ff81bdf8cc8f
github.com/distribution/distribution/v3 v3.1.2
github.com/nats-io/nats.go v1.54.0
)
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025175617-518dae5ec9c8 h1:x4mvY7yiGTP5lnzxbiwyj1o24p05hAmzZP7xHg3XX2M=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025175617-518dae5ec9c8/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026142629-ff81bdf8cc8f h1:GYLCvz2x+JNUOTbXzyh2QwzX0sTzvRlXR7lGocWjk3A=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026142629-ff81bdf8cc8f/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/alicebob/miniredis/v2 v2.35.0 h1:QwLphYqCEAo1eu1TqPRN2jgVMPBweeQcR21jeqDCONI=
github.com/alicebob/miniredis/v2 v2.35.0/go.mod h1:TcL7YfarKPGDAthEtl5NBeHZfeUQj6OXMm/+iu5cLMM=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
+1
View File
@@ -34,6 +34,7 @@
"build_node_release",
"build_node_step",
"build_node_steps",
"build_node_step_end",
"build_node_toolchains",
"build_node_status",
"build_node_health"
+8
View File
@@ -36,6 +36,14 @@ names every refusal and rejection at once, and the pull request is told at once;
`needs: new-head` and names its author: the branch, and the session a `Pushed-by: <who@node>` line in the pull
request's description names.
## A merge the forge account cannot carry out is not asked (issue 1945)
Before the operator's Approve is asked, the gate reads through `gitea_commit_statuses` whether the forge's merging
account writes each repository the ask covers (`merge.writes`). Where it does not, nothing is asked: the delivery waits
on "the forge account cannot write <repo>: gitea_grant_write" (`show`), raised at once as a condition. The read holds
until the gitea module's `gitea.repo.writable` (its grant, read back) or ten minutes, then the ask is made again; an
approved merge of that repository the forge refused is asked again on the same event (`writes.go`).
## The seat's hand-over
The gate took the seat `mesh-delivery` over from the module `mesh-delivery` (ADR 0449 §4, ADR 0450 §6, ADR 0454) and carried its
@@ -0,0 +1,246 @@
package main
import (
"fmt"
"strings"
"sync"
"testing"
"time"
sdk "git.novox.be/novox/mesh-sdk/go/delivery"
)
// One check per delivery (novox/hq ADR 0548), the gate's part: nothing merges into a repository while a check of other heads
// judges it (§2); an approved merge holds a turn on its repositories (§3); a held merge past its bound is a stalled line, read
// and handed by two tools (§4); an approved delivery whose trunk moved merges on its carried verdict with no second ask (§8).
const otherHead = "c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3"
// approvedLone is a lone delivery into oneRepo whose check passed, approved by the operator (the world approves at once).
func approvedLone(w *world) string {
w.t.Helper()
p := pr(oneRepo, 1, oneHead, "feat/one")
requiresCheck(w, p)
w.h.PullUpdated(p)
putCheckerVerdict(w, "pass", headOf(oneRepo, 1, oneHead))
w.h.CommitStatus(checkStatus(oneRepo, oneHead, "success", oneTrunk))
return IDOf(oneRepo, oneHead)
}
// memTurns is the bucket turns in memory; it logs each write with the merges the fake bridge had made by then.
type memTurns struct {
mu sync.Mutex
m map[string]sdk.Turn
log []string
merges func() int
}
func newMemTurns(w *world) *memTurns {
return &memTurns{m: map[string]sdk.Turn{}, merges: func() int { return len(w.forge.merges) }}
}
func (f *memTurns) All() (map[string]sdk.Turn, error) {
f.mu.Lock()
defer f.mu.Unlock()
out := map[string]sdk.Turn{}
for k, v := range f.m {
out[k] = v
}
return out, nil
}
func (f *memTurns) Put(t sdk.Turn) error {
if _, err := sdk.EncodeTurn(t); err != nil {
return err
}
f.mu.Lock()
defer f.mu.Unlock()
f.m[t.Repository] = t
f.log = append(f.log, fmt.Sprintf("put %s after %d merges", t.Repository, f.merges()))
return nil
}
func (f *memTurns) Delete(repository string) error {
f.mu.Lock()
defer f.mu.Unlock()
delete(f.m, repository)
f.log = append(f.log, fmt.Sprintf("delete %s after %d merges", repository, f.merges()))
return nil
}
func (f *memTurns) said() string { f.mu.Lock(); defer f.mu.Unlock(); return strings.Join(f.log, "; ") }
func TestOneCheckPerDeliveryAtTheGate(t *testing.T) {
for name, run := range map[string]func(t *testing.T){
// §2: a merge into a repository a running verdict names is not written to merges, and is written on that verdict's pass.
"an approved merge waits for a running check on its repository and merges on its end": func(t *testing.T) {
w := newWorld(t)
putCheckerVerdict(w, "running", headOf(oneRepo, 9, otherHead))
id := approvedLone(w)
w.settleAll()
if w.op.count() != 1 || len(w.forge.merges) != 0 {
t.Fatalf("asked %d times, merges %v: want the Approve asked and the merge held", w.op.count(), w.forge.merges)
}
d := w.delivery(id)
if d.Ready == nil || !strings.Contains(d.Ready.Waits, "waits for the check of g") || !strings.Contains(d.Ready.Waits, oneRepo) ||
d.Ready.HeldSince == nil {
t.Fatalf("the merge does not say what it waits for: %+v", d.Ready)
}
// §9: deliveries says what it waits for.
for _, l := range w.h.Deliveries("", "", "", false) {
if l.ID == id && !strings.Contains(l.Waits, "approved; waits for the check of g") {
t.Errorf("deliveries says it waits for %q", l.Waits)
}
}
// §4: past 30 minutes the held merge is a stalled line naming the check it waits for and since when.
for _, l := range w.h.Stalled() {
if l.ID == id {
t.Fatalf("a stalled line within the bound: %+v", l)
}
}
w.later(turnBound + time.Minute)
w.settleAll()
var line *StalledLine
for _, l := range w.h.Stalled() {
if l.ID == id {
line = &l
}
}
if line == nil || line.Bound != "30m0s" || !strings.Contains(line.Says, "waits for the check of g") || !strings.Contains(line.Says, "held since") ||
!strings.Contains(line.Says, "checker_status") || !strings.Contains(line.H2, "delivery_hand_now") {
t.Fatalf("the stalled line of the held merge = %+v", line)
}
putCheckerVerdict(w, "pass", headOf(oneRepo, 9, otherHead))
w.settleAll()
if len(w.forge.merges) != 1 || w.op.count() != 1 {
t.Fatalf("after the check's end: merges %v, asked %d times; want merged with no second ask", w.forge.merges, w.op.count())
}
},
"a running check on another repository holds nothing": func(t *testing.T) {
w := newWorld(t)
putCheckerVerdict(w, "running", headOf(oneRepo2, 9, otherHead))
approvedLone(w)
w.settleAll()
if len(w.forge.merges) != 1 {
t.Fatalf("merges %v: a check of another repository held the merge (%s)", w.forge.merges, w.logs())
}
},
"the delivery's own running verdict does not hold it": func(t *testing.T) {
w := newWorld(t)
w.op.auto = false
id := approvedLone(w)
w.settleAll()
putCheckerVerdict(w, "running", headOf(oneRepo, 1, oneHead)) // its own heads judged again (a carry)
for _, k := range w.op.open() {
w.op.answer(k, OptionApprove)
}
w.settleAll()
if d := w.delivery(id); len(w.forge.merges) != 1 || d.Ready == nil || d.Ready.HeldSince != nil {
t.Fatalf("merges %v, run %+v: its own check held it", w.forge.merges, d.Ready)
}
},
// §3: the turn is written before the verdicts are read for the merge, and deleted after the last merge.
"a lone merge's turn is written before it is handed and deleted once it merged": func(t *testing.T) {
w := newWorld(t)
turns := newMemTurns(w)
w.h.Turns = turns
approvedLone(w)
w.settleAll()
if len(w.forge.merges) != 1 || turns.said() != "put "+oneRepo+" after 0 merges; delete "+oneRepo+" after 1 merges" {
t.Fatalf("merges %v; the turns were %q", w.forge.merges, turns.said())
}
},
"a group's turns stand on every member's repository until its last member merged": func(t *testing.T) {
w := newWorld(t)
turns := newMemTurns(w)
w.h.Turns = turns
a, b := pr(oneRepo, 1, oneHead, "feat/two"), pr(oneRepo2, 2, oneHead2, "feat/two")
requiresCheck(w, a, b)
w.h.PullUpdated(a)
w.h.PullUpdated(b)
w.h.wantForge()
putCheckerVerdict(w, "pass", headOf(oneRepo, 1, oneHead), headOf(oneRepo2, 2, oneHead2))
w.h.CommitStatus(checkStatus(oneRepo, oneHead, "success", oneTrunk))
w.h.CommitStatus(checkStatus(oneRepo2, oneHead2, "success", oneTrunk))
for range 3 {
w.settleAll()
}
want := "put " + oneRepo2 + " after 0 merges; put " + oneRepo + " after 0 merges; delete " + oneRepo2 + " after 2 merges; delete " + oneRepo + " after 2 merges"
if len(w.forge.merges) != 2 || turns.said() != want {
t.Fatalf("merges %v; the turns were %q, want %q", w.forge.merges, turns.said(), want)
}
},
// §8: an approved delivery whose trunk moved goes back to proposed, waits for its carried verdict rather than being
// handed on the old one, and merges on it with no second ask.
"an approved merge whose trunk moved merges on its carried verdict with no second ask": func(t *testing.T) {
w := newWorld(t)
putCheckerVerdict(w, "running", headOf(oneRepo, 9, otherHead)) // holds the approved merge (§2)
id := approvedLone(w)
w.settleAll()
if w.op.count() != 1 || len(w.forge.merges) != 0 {
t.Fatalf("asked %d times, merges %v; want approved and held", w.op.count(), w.forge.merges)
}
w.h.PullMerged(merged(oneRepo, 5, otherHead, oneTrunk2)) // a merge at the forge moves the trunk
putCheckerVerdict(w, "pass", headOf(oneRepo, 9, otherHead))
w.settleAll()
if len(w.forge.merges) != 0 {
t.Fatalf("merges %v: handed on a verdict judged on the old trunk", w.forge.merges)
}
d := w.delivery(id)
if d.Ready == nil || !strings.Contains(d.Ready.Waits, "being carried") {
t.Fatalf("after the trunk moved it waits for %q, want its verdict carried", d.Ready.Waits)
}
// The checker's carried line: the head on the new trunk, carried from the old one.
carried := checkStatus(oneRepo, oneHead, "success", oneTrunk2)
carried.Description = "pass " + oneHead[:12] + " on " + oneTrunk2[:12] + " | carried from " + oneTrunk[:12] + " | unit 0 run, 4 kept"
if head, trunk := checkLine(carried.Description); head != oneHead[:12] || trunk != oneTrunk2[:12] {
t.Fatalf("the carried line reads as %s on %s", head, trunk)
}
if head, trunk := checkLine("failed: a test"); head != "" || trunk != "" {
t.Fatalf("a line that names no commits gave %q %q", head, trunk)
}
putCheckerVerdict(w, "pass", headOf(oneRepo, 1, oneHead))
w.h.CommitStatus(carried)
w.settleAll()
if w.op.count() != 1 {
t.Errorf("the Approve was asked %d times, want once: it stands through the carry", w.op.count())
}
if d := w.delivery(id); d.Check == nil || d.Check.Trunk != oneTrunk2[:12] || len(w.forge.merges) != 1 {
t.Errorf("after the carry: verdict %+v, merges %v; want merged on the carried verdict", d.Check, w.forge.merges)
}
},
// §4: delivery_turns reads every turn with the check it waits for; delivery_hand_now hands a held merge at once on the
// operator's word, and refuses one no check holds.
"the turns are read, and a held merge is handed now on the operator's word": func(t *testing.T) {
w := newWorld(t)
w.h.Turns = newMemTurns(w)
putCheckerVerdict(w, "running", headOf(oneRepo, 9, otherHead))
id := approvedLone(w)
w.settleAll()
read := w.h.TurnsRead()
lines := read["turns"].([]TurnLine)
if len(lines) != 1 || lines[0].Repository != oneRepo || len(lines[0].Turn.Waiting) != 1 || lines[0].Turn.Waiting[0].WaitsFor == "" ||
!strings.Contains(lines[0].Waits["feat/one"].(string), "the check g.") || read["written"] != true {
t.Fatalf("delivery_turns = %+v", read)
}
if _, err := w.h.HandNow(id, ""); err == nil || !strings.Contains(err.Error(), "why") {
t.Errorf("a hand without the operator's why: %v", err)
}
if _, err := w.h.HandNow(IDOf(oneRepo2, oneHead2), "x"); err == nil {
t.Error("an unknown delivery was handed")
}
if _, err := w.h.HandNow(id, "the operator, on telegram: merge it now"); err != nil {
t.Fatal(err)
}
w.settleAll()
if len(w.forge.merges) != 1 || w.delivery(id).Ready.HandNow == "" {
t.Fatalf("merges %v after the hand", w.forge.merges)
}
if _, err := w.h.HandNow(id, "again"); err == nil {
t.Error("a merged delivery was handed again")
}
},
} {
t.Run(name, run)
}
}
@@ -339,6 +339,9 @@ func (h *Holder) approval(d *Delivery, now time.Time) (string, func()) {
return "the operator cannot be asked from here", nil
}
members := h.askMembers(d)
if waits := h.writeWaits(d, members, now); waits != "" {
return waits, nil // asked only once the forge account can carry the merge out (writes.go, issue 1945)
}
group := ""
if len(members) > 1 {
group = d.Group
@@ -390,12 +393,12 @@ func (h *Holder) approval(d *Delivery, now time.Time) (string, func()) {
if gid == "" {
break
}
says, err := h.notASuccess(m)
says, forge, err := h.notASuccess(m)
if err != nil {
readErr = err
break
}
if says != "" {
if says != "" && !forge { // the forge's merge answer is weighed at the merge, where it is asked again (group_merge.go)
holdMember, holdSays = m.id, says
break
}
@@ -423,6 +426,16 @@ func (h *Holder) approval(d *Delivery, now time.Time) (string, func()) {
h.mu.Unlock()
}
}
// Nothing is asked that the forge account could not merge (writes.go, issue 1945): said at the next look.
if says := h.cannotWrite(reads); says != "" {
stale()
h.mu.Lock()
defer h.mu.Unlock()
undo()
h.Logf("[mesh-delivery] %s passed its checks and the operator is not asked: %s", d.ID, says)
h.lookAt(h.Now(), "an ask not made: the forge account cannot write a repository it covers")
return
}
if readErr != nil || holdSays != "" {
stale()
h.mu.Lock()
@@ -224,6 +224,9 @@ func (h *Holder) nextStall(now time.Time) time.Time {
consider(r.RefusedSince.Add(h.answeredRefusalBound(d.Repository)))
consider(r.RefusedSince.Add(outOfDateFor))
}
if r.HeldSince != nil {
consider(r.HeldSince.Add(turnBound))
}
}
return next
}
@@ -101,7 +101,9 @@ type fakeForge struct {
viewCalls int
// What Statuses reads: the pull requests by owner/repo#number, the base's required statuses, whether the
// protection cannot be read, and what it was asked.
pulls map[string]ForgePull
pulls map[string]ForgePull
// unwritable are the repositories (owner/repo) the forge's merging account cannot write (issue 1945).
unwritable map[string]bool
required []string
protectionDown bool
asked []string
@@ -220,6 +222,10 @@ func (f *fakeForge) Statuses(owner, repo string, number int, commit, base string
base = "main"
}
out.Merge = ForgeMerge{Branch: base, Protected: f.required != nil, Required: f.required}
if f.unwritable[owner+"/"+repo] {
no := false
out.Merge.Writes, out.Merge.CannotWrite = &no, "the forge account cannot write "+owner+"/"+repo+": gitea_grant_write"
}
if f.protectionDown {
out.Merge.Says = "the protection of " + base + " could not be read"
return out, nil
@@ -2,6 +2,7 @@ package main
import (
"fmt"
"sort"
"strings"
"time"
)
@@ -28,26 +29,46 @@ func readOf(d *Delivery, head string) memberRead {
return memberRead{id: d.ID, owner: d.Owner(), repo: d.Repo(), base: d.Base, head: head, number: d.Number}
}
// notAsSuccess reads a member's required statuses at its head and says, in the forge's words, which are not a success;
// empty when all are. An error is a
// forgeMergeGrace is how long the forge may say a member does not merge now before the group is held for it (issue 1275):
// the forge says so for a few seconds after a base moved, while it works out whether the pull request still merges; within
// the grace the merge is asked again every forgeMergeAgain. Past it, the member is in conflict with its trunk: held, said
// urgent, and asked again on the next event (a new head, a status).
const (
forgeMergeGrace = 2 * time.Minute
forgeMergeAgain = 5 * time.Second
)
// notASuccess reads a member's required statuses at its head and says, in the forge's words, which are not a success,
// and whether the forge says the pull request merges into its base now (its head still the one approved, and not in
// conflict); empty when all hold. forge says the reason is the forge's merge answer rather than a status. An error is a
// read that could not be made: the member then counts as not mergeable.
func (h *Holder) notASuccess(m memberRead) (says string, err error) {
func (h *Holder) notASuccess(m memberRead) (says string, forge bool, err error) {
fc, err := h.Forge.Statuses(m.owner, m.repo, m.number, m.head, m.base)
if err != nil {
return "", err
return "", false, err
}
if fc.Merge.Mergeable == nil {
return "", fmt.Errorf("the forge could not say whether %s may merge: %s", m.id, fc.Merge.Says)
return "", false, fmt.Errorf("the forge could not say whether %s may merge: %s", m.id, fc.Merge.Says)
}
if *fc.Merge.Mergeable {
return "", nil
if !*fc.Merge.Mergeable {
var parts []string
for _, b := range fc.Merge.Blocking {
parts = append(parts, fmt.Sprintf("%s (%s)", b.Context, b.State))
}
return fmt.Sprintf("%s requires %s; not a success: %s", fc.Merge.Branch, strings.Join(fc.Merge.Required, ", "),
strings.Join(parts, ", ")), false, nil
}
var parts []string
for _, b := range fc.Merge.Blocking {
parts = append(parts, fmt.Sprintf("%s (%s)", b.Context, b.State))
if p := fc.Pull; p != nil && m.number > 0 {
if p.HeadSHA != "" && !sameCommit(p.HeadSHA, m.head) {
return fmt.Sprintf("its head is now %s, not the %s approved", shortOf(p.HeadSHA, 12), shortOf(m.head, 12)), false, nil
}
if p.Mergeable != nil && !*p.Mergeable {
base := firstOf(p.Base, m.base)
return fmt.Sprintf("the forge says it does not merge into %s now (in conflict with it, a work in progress, or still "+
"being worked out after %s moved)", base, base), true, nil
}
}
return fmt.Sprintf("%s requires %s; not a success: %s", fc.Merge.Branch, strings.Join(fc.Merge.Required, ", "),
strings.Join(parts, ", ")), nil
return "", false, nil
}
// unmergedLive are a group's members not yet merged and not over, in the group's order. Under the lock.
@@ -61,10 +82,13 @@ func (h *Holder) unmergedLive(g *Group) []*Delivery {
return out
}
// groupMergeable reads every member's required statuses at the head the Approve covers, and says whether all are a success.
// When one is not, or cannot be read, nothing is handed to the bridge: the group is held, said urgent (ADR 0441 §2, §4), and
// the target's turn is given back so the next event that decides whether it may merge asks it again. A read that could not
// be made is a failed ask, tried again after askAgainAfter. Outside the lock.
// groupMergeable reads every member's required statuses, its head and the forge's merge answer at the head the Approve
// covers, and says whether all hold: no member is merged unless every member merges now (issue 1275, 2026-10-26: two of
// three merged and the third refused). When one does not, or cannot be read, nothing is handed to the bridge: the group is
// held, said urgent (ADR 0441 §2, §4), and the target's turn is given back so the next event that decides whether it may
// merge asks it again. A read that could not be made is a failed ask, tried again after askAgainAfter; the forge saying a
// member does not merge now is asked again every forgeMergeAgain for forgeMergeGrace before the group is held. Outside
// the lock.
func (h *Holder) groupMergeable(gid, target string, reads []memberRead) bool {
release := func() {
if d := h.deliveries[target]; d != nil && d.Ready != nil {
@@ -73,7 +97,7 @@ func (h *Holder) groupMergeable(gid, target string, reads []memberRead) bool {
}
}
for _, m := range reads {
says, err := h.notASuccess(m)
says, forge, err := h.notASuccess(m)
if err == nil && says == "" {
continue
}
@@ -85,6 +109,25 @@ func (h *Holder) groupMergeable(gid, target string, reads []memberRead) bool {
h.mu.Unlock()
return false
}
if forge {
if h.unmergeableSince == nil {
h.unmergeableSince = map[string]time.Time{}
}
key, now := gid+" "+m.id, h.Now()
first, seen := h.unmergeableSince[key]
if !seen {
first = now
h.unmergeableSince[key] = now
h.Logf("[mesh-delivery] group %s: nothing is merged yet: %s at %s: %s; asked again every %s for %s", gid, m.id,
shortOf(m.head, 12), says, forgeMergeAgain, forgeMergeGrace)
}
if now.Sub(first) < forgeMergeGrace {
release()
h.lookAt(now.Add(forgeMergeAgain), "a group's member the forge was still weighing")
h.mu.Unlock()
return false
}
}
if g := h.groups[gid]; g != nil {
if g.Hold == nil || g.Hold.Kind != "blocked" || g.Hold.Says != says || g.Hold.Member != m.id {
h.Logf("[mesh-delivery] group %s is approved and NOT merged: %s at %s: %s", gid, m.id, shortOf(m.head, 12), says)
@@ -99,13 +142,18 @@ func (h *Holder) groupMergeable(gid, target string, reads []memberRead) bool {
g.Hold = &GroupHold{Kind: "blocked", Member: m.id, Says: says, Since: since}
h.keepGroup(g)
}
// No timer: the status turning success is an event on the forge (CommitStatus, mergeAgainNow), and it asks the
// merge again at once (issue 1275: a pass that stops part way resumes on an event).
// No timer: the status turning success, or a new head, is an event on the forge (CommitStatus, mergeAgainNow), and
// it asks the merge again at once (issue 1275: a pass that stops part way resumes on an event).
release()
h.mu.Unlock()
return false
}
h.mu.Lock()
for key := range h.unmergeableSince {
if strings.HasPrefix(key, gid+" ") {
delete(h.unmergeableSince, key)
}
}
if g := h.groups[gid]; g != nil && g.Hold != nil {
g.Hold = nil
h.keepGroup(g)
@@ -114,6 +162,98 @@ func (h *Holder) groupMergeable(gid, target string, reads []memberRead) bool {
return true
}
// passWaits is why an approved merge is not handed yet because another approved merge's pass is under way into a
// repository it goes into; empty when none is (issue 1275, 2026-10-26 13:40: group feat/890-placement merged
// mesh-delivery#72 between the members of fix/734-lost-ask, whose mesh-delivery#73 then conflicted). A pass is under way
// from its first member handed until every member is merged; one half merged stays under way while it is refused or held,
// until it is whole, declined, or its remaining members are gone or asked anew (a new head is a new Approve). The first
// merge handed has the repositories; the other is handed once that pass ends, on the event that ends it. This holds
// whether or not mesh/check gates the merge: it waits for a merge, never for a check (ADR 0551). Under the lock.
func (h *Holder) passWaits(d *Delivery) string {
c := h.covering(d)
if c == nil || c.Approved == nil || c.AskID == "" {
return ""
}
mine := h.unmergedRepos(c)
for _, p := range h.passes() {
if p.run.AskID == c.AskID || !h.passUnderWay(p.run) {
continue
}
for repo := range h.unmergedRepos(p.run) {
if mine[repo] {
return fmt.Sprintf("approved; waits for the merge of %s into %s to end, so no other merge lands there between "+
"its members (issue 1275)", p.name, repo)
}
}
}
return ""
}
// pass is an approved merge not merged whole: the run holding its Approve and the name of its group (or pull request).
type pass struct {
run *ReadyRun
name string
}
// passes are the approved merges not merged whole and not declined, each ask once. Unlike approvedMerges it keeps one whose
// members merged or were refused: a half-merged pass still holds its repositories (issue 1275). Under the lock.
func (h *Holder) passes() []pass {
ids := make([]string, 0, len(h.deliveries))
for id := range h.deliveries {
ids = append(ids, id)
}
sort.Strings(ids)
seen := map[string]bool{}
var out []pass
for _, id := range ids {
d := h.deliveries[id]
r := d.Ready
if r == nil || r.Approved == nil || r.Declined != nil || r.AskID == "" || len(r.Heads) == 0 || seen[r.AskID] ||
d.Base != MergeBranch || h.allMerged(r.Heads) {
continue
}
seen[r.AskID] = true
out = append(out, pass{run: r, name: askGroup(d)})
}
return out
}
// unmergedRepos are the repositories (lowercase) of the deliveries an Approve covers that are not merged and may still be
// merged on it: not over (closed, held), at the head approved, and covered by this Approve, not a newer one. Under the lock.
func (h *Holder) unmergedRepos(r *ReadyRun) map[string]bool {
out := map[string]bool{}
for id, sha := range r.Heads {
m := h.deliveries[id]
if m == nil || m.MergedAs != "" || (m.Ready != nil && m.Ready.Merged != nil) || m.State.Final() || !sameCommit(m.Commit, sha) {
continue
}
if c := h.covering(m); c == nil || c.AskID != r.AskID {
continue // asked anew (its new head, or a new Approve): that ask is the pass now
}
out[strings.ToLower(m.Repository)] = true
}
return out
}
// passUnderWay is whether an approved merge's pass has started and not ended: a member is handed to the bridge, or some
// are merged and others may still merge on this Approve, refused or held or not (issue 1275, review: a held half-merged
// group that let another group start, then asked again, waited on that group while it waited on it). Under the lock.
func (h *Holder) passUnderWay(r *ReadyRun) bool {
if len(h.unmergedRepos(r)) == 0 {
return false
}
for id := range r.Heads {
m := h.deliveries[id]
if m == nil {
continue
}
if m.MergedAs != "" || (m.Ready != nil && m.Ready.Merged != nil) || (m.Ready != nil && m.Ready.merging) {
return true
}
}
return false
}
// groupLines are the groups that are held or half merged, as urgent stalled lines the gate says as conditions (conditions.go):
// one for each, on its first member not merged, until every member is merged or a person decides (issue 1275). A member
// the merge tool refused is said by readyLines, with its group; this adds a group held for a required status, and a
@@ -149,9 +289,10 @@ func (h *Holder) groupLines(now time.Time) []StalledLine {
}
out = append(out, StalledLine{ID: g.Hold.Member, Number: numberOf(h.deliveries[g.Hold.Member]), State: first.State,
For: now.Sub(g.Hold.Since).Round(time.Second).String(), Bound: "0s", Severity: "urgent",
H2: "none: the required status turns success, which asks the merge again — or the operator's decision",
Says: fmt.Sprintf("group %s %s (merged %d of %d): %s waits on a required status "+
"at its head: %s. Needed: that status turns success (its check asked again), or the operator decides.",
H2: "none: the required status turns success or a new head merges, which asks the merge again — or the operator's decision",
Says: fmt.Sprintf("group %s %s (merged %d of %d): %s does not merge now "+
"at its head: %s. Needed: that status turns success (its check asked again), a new head that merges "+
"(rebased on its trunk), or the operator decides.",
g.ID, stands, merged, len(h.membersOf(g)), g.Hold.Member, g.Hold.Says)})
case merged > 0 && !refused && !running:
out = append(out, StalledLine{ID: first.ID, Number: first.Number, State: first.State,
@@ -177,3 +318,69 @@ func holdMember(g *Group) string {
}
return g.Hold.Member
}
// MergeAgain is the tool delivery_merge_again: on the operator's word, the merge of an approved delivery the forge refused,
// or of a group held part way, is asked again at once at the heads the Approve named (issue 1275): a refusal the forge
// meant as "later", a conflict cleared on the trunk, or a refusal repeated until the gate stopped asking. A delivery whose
// head moved is asked again by its new head, never by this. Refused, naming why, for one that is not an approved merge
// waiting after a refusal or a hold.
func (h *Holder) MergeAgain(id, why string) (string, error) {
h.mu.Lock()
defer h.mu.Unlock()
d := h.deliveries[id]
if d == nil {
return "", fmt.Errorf("no delivery %q", id)
}
if strings.TrimSpace(why) == "" {
return "", fmt.Errorf("say why: the merge asked again is the operator's word, kept in the log")
}
c := h.covering(d)
if c == nil || c.Approved == nil || c.Declined != nil || c.Heads[d.ID] != d.Commit {
return "", fmt.Errorf("%s is not approved at its head %s: nothing to ask again (a new head is asked on its own)", id, shortOf(d.Commit, 12))
}
if d.MergedAs != "" || (d.Ready != nil && d.Ready.Merged != nil) {
return "", fmt.Errorf("%s is merged already", id)
}
g := h.groups[d.Group]
if d.Group == "" {
g = nil
}
held := g != nil && g.Hold != nil && g.Hold.Kind == "blocked"
refused := false
for hid := range c.Heads {
if m := h.deliveries[hid]; m != nil && m.Ready != nil && m.Ready.Merged == nil && m.Ready.RefusedSince != nil {
refused = true
}
}
if !held && !refused {
return "", fmt.Errorf("%s was neither refused nor held: %s", id, firstOf(d.readyWords(), "it is merged at its turn"))
}
if waits := h.passWaits(d); waits != "" {
return "", fmt.Errorf("%s is not asked again yet: it %s; it is asked at that pass's end", id, strings.TrimPrefix(waits, "approved; "))
}
for hid := range c.Heads {
if m := h.deliveries[hid]; m != nil && m.Ready != nil && m.Ready.Merged == nil {
m.Ready.Repeats, m.Ready.LastTry, m.Ready.merging = 0, time.Time{}, false
h.keep(m)
}
}
if held {
g.Hold = nil
h.keepGroup(g)
}
for key := range h.unmergeableSince {
if g != nil && strings.HasPrefix(key, g.ID+" ") {
delete(h.unmergeableSince, key)
}
}
h.Logf("[mesh-delivery] %s: its merge is asked again on the operator's word, at the heads approved: %s", id, why)
return fmt.Sprintf("%s's merge is asked again at the next look, every member read first; nothing merges unless every member merges now", id), nil
}
// readyWords is what an approved delivery's run says it waits for, if anything.
func (d *Delivery) readyWords() string {
if d.Ready == nil {
return ""
}
return d.Ready.Waits
}
@@ -12,6 +12,7 @@ import (
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
sdk "git.novox.be/novox/mesh-sdk/go/delivery"
)
// The holder of the mesh-delivery seat (novox/hq ADR 0239, to-be 47): the one writer of deliveries and
@@ -75,8 +76,19 @@ type Holder struct {
// unchecked are the open pull requests' heads seen without a merge check (unchecked.go).
unchecked map[string]unchecked
// checker are the checker's verdicts by the hash of the heads they judged, as its verdicts bucket holds them
// (onestatus.go, issue 1302).
checker map[string]checkerVerdict
// (onestatus.go, issue 1302); runningSince when this owner first saw each running (ADR 0548 §2, turns.go).
checker map[string]checkerVerdict
runningSince map[string]time.Time
// unmergeableSince is when the forge first said a group's member does not merge now, by group and member (issue 1275,
// group_merge.go): within forgeMergeGrace that is the forge still working out mergeability, asked again shortly.
unmergeableSince map[string]time.Time
// writeReads is whether the forge's merging account writes each repository, as last read (writes.go, issue 1945).
writeReads map[string]*writeRead
// Turns is the gate's bucket turns (turns.go, ADR 0548 §3); nil keeps none. turnsHeld are the turns as written there,
// by repository, read once at the first look (turnsLoaded).
Turns TurnStore
turnsHeld map[string]sdk.Turn
turnsLoaded bool
// tried counts the owed effects Flush tried since this owner started, by how each ended (novox/hq issue 407).
tried owedTries
// forgeWanted is whether the forge's open pull requests are to be read at the next React: a check may have
@@ -126,6 +138,7 @@ func (h *Holder) init() {
h.lostRead = map[string]time.Time{}
h.unchecked = map[string]unchecked{}
h.checker = map[string]checkerVerdict{}
h.runningSince, h.turnsHeld = map[string]time.Time{}, map[string]sdk.Turn{}
h.started = h.Now()
}
@@ -0,0 +1,347 @@
package main
import (
"errors"
"fmt"
"sort"
"strings"
"testing"
sdk "git.novox.be/novox/mesh-sdk/go/delivery"
)
// Issue 1275, met a third time (2026-10-26 13:40, group fix/734-lost-ask): the operator approved mesh-sdk#92,
// mesh-controller#472 and mesh-delivery#73; the first two merged, the forge answered #73's merge 405 "Please try again
// later", and in the seconds between, group feat/890-placement merged mesh-delivery#72, so #73 then conflicted. A group is
// merged only when every member merges now, and no other approved merge lands in a repository between its members.
const (
wSDK, wCtl, wDel = "novox/mesh-sdk", "novox/mesh-controller", "novox/mesh-delivery"
wH92, wH472, wH73 = "3203358c4cb5fa5bed223f7ce85577bbb70a346e", "ccc7ba334c82554d809b5b38d8eb0eb5f36be7be", "bf574172154a9f6ed7d4e790643e0dcab2a10a91"
wH72, wH73b = "72727272727272727272727272727272727272aa", "b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0"
forgeLater = "Gitea API /repos/novox/mesh-delivery/pulls/73/merge: 405 {\"message\":\"Please try again later\"}"
conflictWords = "does not merge into main now"
headMovedWords = "its head is now"
)
var lostAsk = []member{
{wSDK, 92, wH92, "go/buildstep/ask.go"},
{wCtl, 472, wH472, "internal/grants.go"},
{wDel, 73, wH73, "modules/mesh-delivery-checker/internal/check/lostask.go,modules/mesh-delivery-checker/go.mod"},
}
// forgePull says what the forge answers of a pull request: its head and whether it merges now.
func forgePull(w *world, repository string, number int, head string, mergeable bool) {
w.forge.mu.Lock()
defer w.forge.mu.Unlock()
m := mergeable
w.forge.pulls[fmt.Sprintf("%s#%d", repository, number)] = ForgePull{Number: number, State: "open", Base: "main", HeadSHA: head, Mergeable: &m}
}
func TestAnApprovedGroupMergesNoMemberUnlessEveryMemberMergesNow(t *testing.T) {
approve := func(t *testing.T) *world {
w := approvedGroupOf(t, lostAsk...)
allGreenFor(w, lostAsk...)
for _, m := range lostAsk {
forgePull(w, m.repo, m.number, m.head, true)
}
return w
}
cases := []struct {
name string
// before is the forge as the Approve lands; after, once the grace has passed (nil: as before).
before, after func(w *world)
// merged is how many members merge in the end; line what the urgent line says, when one stands.
merged int
line string
}{
{name: "a member in conflict with its trunk: none merged, the group held and said urgent", merged: 0, line: conflictWords,
before: func(w *world) {
forgePull(w, wDel, 73, wH73, false)
w.forge.refuseRepo = map[string]error{wDel: errors.New(forgeLater)} // what the merge tool would answer
}},
{name: "the forge still weighing a member: none merged, then every member once it says yes", merged: 3,
before: func(w *world) { forgePull(w, wDel, 73, wH73, false) },
after: func(w *world) { forgePull(w, wDel, 73, wH73, true) }},
{name: "a member's head moved after the Approve: none merged, said urgent", merged: 0, line: headMovedWords,
before: func(w *world) { forgePull(w, wDel, 73, wH73b, true) }},
{name: "every member merges now: all merged in one pass", merged: 3, before: func(*world) {}},
}
// The recovery of a group that ended half merged: its remaining member's new head, or the operator's word.
t.Run("a half-merged group is made whole by its remaining member's new head", halfMergedMadeWholeByANewHead)
t.Run("a member refused part way is asked again on the operator's word", refusedPartWayAskedAgainOnTheOperatorsWord)
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
w := approve(t)
c.before(w)
w.op.answer(w.op.open()[0], OptionApprove)
w.settleAll()
if c.after == nil && c.merged == 3 {
if len(w.forge.merges) != 3 {
t.Fatalf("merged %v, wanted all three", w.forge.merges)
}
return
}
if len(w.forge.merges) != 0 {
t.Fatalf("a member was merged while another does not merge now: %v", w.forge.merges)
}
if c.after != nil {
c.after(w)
}
w.later(forgeMergeGrace + time30s)
w.settleAll()
if len(w.forge.merges) != c.merged {
t.Fatalf("merged %v, wanted %d", w.forge.merges, c.merged)
}
lines := urgentLines(w)
if c.line == "" {
if len(lines) != 0 {
t.Fatalf("a group merged whole still says %+v", lines)
}
return
}
if len(lines) != 1 || !strings.Contains(lines[0].Says, "feat/g") ||
!strings.Contains(lines[0].Says, IDOf(wDel, wH73)) || !strings.Contains(lines[0].Says, c.line) ||
!strings.Contains(lines[0].Says, "nothing is merged") {
t.Fatalf("the held group is not one urgent line naming the member and why: %+v", lines)
}
})
}
}
// handing are the pull requests the gate has handed the forge bridge and the bridge has not answered, in every merges key.
func handing(w *world) []sdk.MergePull {
w.merges.mu.Lock()
defer w.merges.mu.Unlock()
var out []sdk.MergePull
for _, v := range w.merges.values {
for _, p := range v.Pulls {
if p.State == sdk.MergeMerging {
out = append(out, p)
}
}
}
sort.Slice(out, func(i, j int) bool { return out[i].Repository < out[j].Repository })
return out
}
func handedWords(ps []sdk.MergePull) string {
var out []string
for _, p := range ps {
out = append(out, fmt.Sprintf("%s#%d", p.Repository, p.Number))
}
return strings.Join(out, " ")
}
// Two approved merges into one repository: the one whose pass started has the repository until every member merged, and
// the other is handed only then (2026-10-26: mesh-delivery#72 of feat/890-placement merged between fix/734-lost-ask's
// members). This holds while mesh/check gates no merge too (ADR 0551): it waits for a merge, never for a check.
func TestTwoApprovedMergesIntoOneRepositoryDoNotInterleave(t *testing.T) {
for _, gates := range []bool{true, false} {
t.Run(fmt.Sprintf("check-gates-merge %v", gates), func(t *testing.T) {
w := approvedGroupOf(t, lostAsk...)
if !gates {
ungated(w)
}
w.merges.manual = true
allGreenFor(w, lostAsk...)
w.op.answer(w.op.open()[0], OptionApprove)
w.settleAll()
if got := handedWords(handing(w)); got != "novox/mesh-sdk#92" {
t.Fatalf("handed %q, wanted the group's first member", got)
}
// feat/890-placement: a lone pull request into mesh-delivery, approved while the group's pass is under way.
w.h.PullUpdated(pr(wDel, 72, wH72, "feat/890-placement", "modules/mesh-delivery-checker/go.mod"))
w.h.Checked(judged(wDel, 72, wH72, "pass", gTrunk0))
setChecks(w, wDel, wH72, "success", "success")
w.settleAll()
if open := w.op.open(); len(open) != 1 {
t.Fatalf("the lone pull request is asked once: %v", open)
} else {
w.op.answer(open[0], OptionApprove)
}
w.settleAll()
var order []string
steps := []struct {
repo string
number int
head string
}{{wSDK, 92, wH92}, {wCtl, 472, wH472}, {wDel, 73, wH73}}
if !gates {
// While the check gates the merge, #72's verdict went out of date with #73's merge and it waits for its
// check against the new trunk; while it gates none, #72 is handed once the group's pass ended.
steps = append(steps, struct {
repo string
number int
head string
}{wDel, 72, wH72})
}
for _, next := range steps {
handed := handing(w)
if len(handed) != 1 || handed[0].Repository != next.repo || handed[0].Number != next.number {
t.Fatalf("after %v: handed %q, wanted only %s#%d", order, handedWords(handed), next.repo, next.number)
}
merge := strings.Repeat(string(rune('a'+next.number%20)), 40)
bridgeMerged(w, next.repo, next.number, next.head, merge)
w.h.PullMerged(merged(next.repo, next.number, next.head, merge))
w.settleAll()
order = append(order, fmt.Sprintf("%s#%d", next.repo, next.number))
}
if left := handing(w); len(left) != 0 {
t.Fatalf("still handed: %q", handedWords(left))
}
})
}
}
// A group that ended half merged (its last member refused after the others merged) is made whole by its remaining member's
// new head: rebased on its trunk, checked, composed with the merged members, asked once and merged, and the urgent line
// is cleared (issue 1275, the recovery of 2026-10-26 13:40).
func halfMergedMadeWholeByANewHead(t *testing.T) {
w := approvedGroupOf(t, lostAsk...)
allGreenFor(w, lostAsk...)
w.forge.refuseRepo = map[string]error{wDel: errors.New(forgeLater)}
w.op.answer(w.op.open()[0], OptionApprove)
w.settleAll()
if len(w.forge.merges) != 3 || len(urgentLines(w)) != 1 {
t.Fatalf("the case: two merged and the third refused, said urgent: %v %+v", w.forge.merges, urgentLines(w))
}
w.forge.refuseRepo = nil
// #73 rebased on its trunk: a new head, checked, composed with the members merged.
w.h.PullUpdated(pr(wDel, 73, wH73b, "feat/g", "modules/mesh-delivery-checker/go.mod"))
w.h.Checked(judged(wDel, 73, wH73b, "pass", gT1))
setChecks(w, wDel, wH73b, "success", "success")
w.settleAll()
open := w.op.open()
if len(open) != 1 {
t.Fatalf("the remaining member's new head is asked once: %v", open)
}
w.op.answer(open[0], OptionApprove)
w.settleAll()
if last := w.forge.merges[len(w.forge.merges)-1]; len(w.forge.merges) != 4 || !strings.Contains(last, "mesh-delivery#73@"+wH73b) {
t.Fatalf("the new head is not merged alone: %v", w.forge.merges)
}
if lines := urgentLines(w); len(lines) != 0 {
t.Fatalf("the group is whole and still said: %+v", lines)
}
}
// delivery_merge_again (issue 1275, the tools rule): a member refused after its siblings merged, with no new head and no
// trunk move to ask it again, is asked again on the operator's word, at the head approved; a delivery neither refused nor
// held is refused, naming why.
func refusedPartWayAskedAgainOnTheOperatorsWord(t *testing.T) {
w := approvedGroupOf(t, lostAsk...)
allGreenFor(w, lostAsk...)
if _, err := w.h.MergeAgain(IDOf(wDel, wH73), "too early"); err == nil {
t.Fatal("a delivery not approved is asked again")
}
w.forge.refuseRepo = map[string]error{wDel: errors.New(forgeLater)}
w.op.answer(w.op.open()[0], OptionApprove)
w.settleAll()
w.forge.refuseRepo = nil
w.settleAll()
if len(w.forge.merges) != 3 {
t.Fatalf("the case: the third member refused and not asked again by itself: %v", w.forge.merges)
}
if _, err := w.h.MergeAgain(IDOf(wDel, wH73), ""); err == nil {
t.Fatal("asked again with no why")
}
if _, err := w.h.MergeAgain(IDOf(wDel, wH73), "the forge said try again later"); err != nil {
t.Fatal(err)
}
w.settleAll()
if len(w.forge.merges) != 4 || len(urgentLines(w)) != 0 {
t.Fatalf("not merged on the operator's word: %v %+v", w.forge.merges, urgentLines(w))
}
if _, err := w.h.MergeAgain(IDOf(wDel, wH73), "again"); err == nil || !strings.Contains(err.Error(), "merged already") {
t.Fatalf("a merged delivery is asked again: %v", err)
}
}
// approveSecondGroup makes a second group of these members on branch, checked, composed together and approved; the world
// already holds group feat/g.
func approveSecondGroup(t *testing.T, w *world, branch string, ms ...member) {
t.Helper()
for _, m := range ms {
w.h.PullUpdated(pr(m.repo, m.number, m.head, branch, strings.Split(m.paths, ",")...))
w.h.Checked(judged(m.repo, m.number, m.head, "pass", gTrunk0))
}
allGreenFor(w, ms...)
w.settleAll()
g := w.h.groups[branch]
if g == nil || g.Check == nil {
t.Fatalf("no composed check was asked for %s: %+v", branch, g)
}
c := CheckedEvent{Group: branch, ID: g.Check.Asked, Verdict: "pass", Summary: "composed"}
for _, m := range ms {
owner, repo, _ := strings.Cut(m.repo, "/")
c.Members = append(c.Members, struct {
Owner string `json:"owner"`
Repo string `json:"repo"`
Commit string `json:"commit"`
Trunk string `json:"trunk,omitempty"`
}{Owner: owner, Repo: repo, Commit: m.head, Trunk: gTrunk0[:12]})
}
w.h.Checked(c)
w.settleAll()
for _, key := range w.op.open() {
if strings.Contains(key, strings.ReplaceAll(branch, "/", "_")) || strings.Contains(key, branch) {
w.op.answer(key, OptionApprove)
}
}
}
// A group held half merged keeps its repositories until it is whole or declined: another group sharing one waits, and the
// operator's word that asks the held group again never leaves the two waiting on each other (issue 1275, review round 1:
// a2 waited for group B, b2 waited for group A, for ever). And the operator's word is refused while another pass it would
// wait for is under way, naming it.
func TestAHeldHalfMergedGroupKeepsItsRepositoriesAndNoTwoPassesWaitOnEachOther(t *testing.T) {
groupA := []member{lostAsk[0], lostAsk[2]} // mesh-sdk#92, then mesh-delivery#73
groupB := []member{lostAsk[1], {wDel, 72, wH72, "modules/mesh-delivery-checker/go.mod"}}
t.Run("half merged and held: the other group waits, and both merge once the held one is asked again", func(t *testing.T) {
w := approvedGroupOf(t, groupA...)
allGreenFor(w, groupA...)
w.forge.refuseRepo = map[string]error{wDel: errors.New(forgeLater)}
w.op.answer(w.op.open()[0], OptionApprove)
w.settleAll()
if len(w.forge.merges) != 2 {
t.Fatalf("the case: mesh-sdk#92 merged, mesh-delivery#73 refused: %v", w.forge.merges)
}
approveSecondGroup(t, w, "feat/890-placement", groupB...)
w.settleAll()
if len(w.forge.merges) != 2 {
t.Fatalf("group feat/890-placement merged while feat/g is half merged into a repository both go into: %v", w.forge.merges)
}
w.forge.refuseRepo = nil
if _, err := w.h.MergeAgain(IDOf(wDel, wH73), "the forge said try again later"); err != nil {
t.Fatal(err)
}
w.settleAll()
want := []string{"mesh-sdk#92", "mesh-delivery#73", "mesh-delivery#73", "mesh-controller#472", "mesh-delivery#72"}
if len(w.forge.merges) != len(want) {
t.Fatalf("merged %v, wanted %v: the held group first, then the other", w.forge.merges, want)
}
for i, m := range want {
if !strings.Contains(w.forge.merges[i], m+"@") {
t.Fatalf("merged %v, wanted %v in that order", w.forge.merges, want)
}
}
})
t.Run("the operator's word is refused while another pass it would wait for is under way", func(t *testing.T) {
w := approvedGroupOf(t, groupA...)
allGreenFor(w, groupA...)
w.forge.refuseRepo = map[string]error{wSDK: errors.New(forgeLater)} // refused first: nothing of feat/g merged
w.op.answer(w.op.open()[0], OptionApprove)
w.settleAll()
w.merges.manual = true
approveSecondGroup(t, w, "feat/890-placement", groupB...)
w.settleAll()
if got := handedWords(handing(w)); got != "novox/mesh-controller#472" {
t.Fatalf("the other group's pass did not start: handed %q", got)
}
_, err := w.h.MergeAgain(IDOf(wSDK, wH92), "asked again")
if err == nil || !strings.Contains(err.Error(), "feat/890-placement") {
t.Fatalf("asked again while a pass it waits for is under way: %v", err)
}
})
}
@@ -6,7 +6,7 @@ import "git.novox.be/novox/mesh-sdk/go/announce"
// issue 1790): the module says it itself, from its code, before it serves, and a reader is granted what it said
// (clause 3). A state is readable when another module's manifest reads it (keeps_test.go names the readers). keeps_test.go holds the list to the
// manifest's `state`, and keeps_integration_test.go that the module announces it on a bus.
var Kept = []announce.State{{Name: "asks", Readable: true}, {Name: "conditions"}, {Name: "deliveries"}, {Name: "groups"}, {Name: "merges", Readable: true}, {Name: "owed", Readable: true}, {Name: "pulls", Readable: true}, {Name: "source", Readable: true}}
var Kept = []announce.State{{Name: "asks", Readable: true}, {Name: "conditions"}, {Name: "deliveries"}, {Name: "groups"}, {Name: "merges", Readable: true}, {Name: "owed", Readable: true}, {Name: "pulls", Readable: true}, {Name: "source", Readable: true}, {Name: "turns", Readable: true}}
// announceKept says Kept through keeps, which is stdio.Keeps.
func announceKept(keeps func(...announce.State) error) error { return keeps(Kept...) }
@@ -17,6 +17,9 @@ var readers = map[string][]string{
"pulls": {"build-agent"}, // in mesh-controller
"owed": {"mesh-delivery-forge-bridge"},
"source": {"mesh-delivery-indexer", "mesh-delivery-checker"},
// ADR 0548 §3, §4: the checker reads the merge turns and claims no check of a repository one names; the gate alone writes
// them (a state is written by its owner's account alone, ADR 0495 §3: announcing it readable grants a read, never a write).
"turns": {"mesh-delivery-checker"},
}
// 1790/S: the states the module announces are the states its manifest declares, no more, no fewer, and the readable
@@ -34,7 +34,7 @@ func (l *listening) get() string { l.mu.Lock(); defer l.mu.Unlock(); return l.n
// The events this owner follows: the forge's pull requests. The forge's status events say that a check answered (events.go).
var followed = []string{"gitea.pull.updated", "gitea.pull.merged", "gitea.pull.closed",
"gitea.forge.status", "gitea.forge.pull-request.edited", "gitea.forge.pull-request.closed",
forgePattern, ConfigChangedEvent}
forgePattern, ConfigChangedEvent, writableEvent}
// forgePattern is the forge bridge's messages on FORGE (novox/hq ADR 0437 §8): the gate follows a merge and the refusal of one.
const forgePattern = "mesh-delivery-forge-bridge.forge.**"
@@ -65,6 +65,7 @@ func main() {
Operator: kvOperator{},
Merges: kvMerges{},
Source: kvSource{},
Turns: kvTurns{},
Owed: kvOwed{},
Pulls: kvPulls{},
// The gate says what stalled to the operator itself (issue 1275, ADR 0437 §9).
@@ -218,6 +219,10 @@ func take(h *Holder, e stdio.Envelope) error {
}
h.CommitStatus(s)
h.KeepCheckVerdict(s)
case writableEvent:
if err := h.WriteGranted(e.Body); err != nil {
h.Refuse(err)
}
case ConfigChangedEvent:
var c ConfigChange
if err := json.Unmarshal(e.Body, &c); err != nil {
@@ -357,6 +362,38 @@ func tools(h *Holder, l *listening) []stdio.Tool {
}
return h.Close(strArg(a, "id"), strArg(a, "why"))
}},
{Name: "delivery_turns",
Description: "Every merge turn (novox/hq ADR 0548 §3): per repository, the approved merges that wait to merge into it in the order of " +
"their Approve, the check of another delivery each waits for and since when, whether the gate wrote the turn to its bucket " +
"turns yet (the checker claims no check of a repository a turn names), and the bound past which a held merge is a stalled " +
"line. Read-only.",
Run: func(map[string]any) (any, error) { return h.TurnsRead(), nil }},
{Name: "delivery_hand_now",
Description: "On the operator's word, hand an approved merge held for another delivery's running check to the merge at once " +
"(novox/hq ADR 0548 §4): the check it waited for runs on and is carried to the new trunk after the merge. Refused for a " +
"delivery that is not an approved merge held by a check, naming why. The why is kept on the delivery.",
Input: map[string]any{"type": "object", "properties": map[string]any{"id": str("the delivery's id (owner/repository@commit)"),
"why": str("the operator's word, kept on the delivery")}, "required": []string{"id", "why"}},
Run: func(a map[string]any) (any, error) {
if err := need(a, "id", "why"); err != nil {
return nil, err
}
return h.HandNow(strArg(a, "id"), strArg(a, "why"))
}},
{Name: "delivery_merge_again",
Description: "On the operator's word, ask again at once the merge of an approved delivery the forge refused, or of an approved " +
"group held or half merged, at the heads the Approve named (novox/hq issue 1275): every member is read first and " +
"nothing merges unless every member merges now. For a refusal that was the forge's \"try again later\", a conflict " +
"cleared on the trunk, or a refusal the gate stopped asking after it repeated. A member whose head moved is asked by its " +
"new head, never by this. Refused, naming why, for a delivery that is not an approved merge waiting after a refusal or a hold.",
Input: map[string]any{"type": "object", "properties": map[string]any{"id": str("the delivery's id (owner/repository@commit), any member of the group"),
"why": str("the operator's word, kept in the log")}, "required": []string{"id", "why"}},
Run: func(a map[string]any) (any, error) {
if err := need(a, "id", "why"); err != nil {
return nil, err
}
return h.MergeAgain(strArg(a, "id"), strArg(a, "why"))
}},
{Name: "delivery_status",
Description: "Whether this owner works: how many deliveries in each state, the groups, what is owed to " +
"the forge and the bus and not yet done, how many owed effects were tried since it started (done, failed " +
@@ -91,9 +91,10 @@ func TestItClaimsTheDeliverySeatAndFollowsWhatItHandles(t *testing.T) {
if !reflect.DeepEqual(m.Emits, []string{"transition", "group", "merged"}) {
t.Fatalf("emits %v", m.Emits)
}
// Eight buckets, and no `asked` (the teller asks, ADR 0437 §6): ADR 0450 §3, and the conditions it raised (issue 1275).
// Nine buckets, and no `asked` (the teller asks, ADR 0437 §6): ADR 0450 §3, the conditions it raised (issue 1275) and the
// merge turns (ADR 0548 §3).
buckets := m.buckets()
if !reflect.DeepEqual(buckets, []string{"deliveries", "groups", "pulls", "asks", "merges", "owed", "source", "conditions"}) {
if !reflect.DeepEqual(buckets, []string{"deliveries", "groups", "pulls", "asks", "merges", "owed", "source", "conditions", "turns"}) {
t.Fatalf("state %v", buckets)
}
// What it reads: the teller's answers and the checker's verdicts (issue 1302). The carry from today's module's buckets, the
@@ -72,8 +72,12 @@ func (h *Holder) CheckerVerdictChanged(key string, value []byte) {
if i := strings.LastIndex(key, "."); i >= 0 {
hash = key[i+1:]
}
if h.runningSince == nil {
h.runningSince = map[string]time.Time{}
}
if len(value) == 0 || string(value) == "null" {
delete(h.checker, hash)
delete(h.runningSince, hash)
h.Poke()
return
}
@@ -85,6 +89,11 @@ func (h *Holder) CheckerVerdictChanged(key string, value []byte) {
if v.Heads == "" {
v.Heads = hash
}
if prev, ok := h.checker[v.Heads]; v.State == checkerRunning && (!ok || prev.State != checkerRunning) {
h.runningSince[v.Heads] = h.Now() // a check starts: a merge it holds says since when (ADR 0548 §2)
} else if v.State != checkerRunning {
delete(h.runningSince, v.Heads)
}
h.checker[v.Heads] = v
h.Poke()
}
@@ -199,14 +199,3 @@ func TestAClosedPullRequestAsksNoForgetChecks(t *testing.T) {
w.h.PullUpdated(p)
w.h.PullClosed(p)
}
// 1302: the mesh/check line gives the head and the trunk it judged.
func TestTheCheckLineGivesHeadAndTrunk(t *testing.T) {
head, trunk := checkLine("pass " + oneHead[:12] + " on " + oneTrunk[:12] + " | unit 4 run, 0 kept | group g of 2 pull requests")
if head != oneHead[:12] || trunk != oneTrunk[:12] {
t.Fatalf("head %q trunk %q", head, trunk)
}
if head, trunk = checkLine("failed: a test"); head != "" || trunk != "" {
t.Fatalf("a line that names no commits gave %q %q", head, trunk)
}
}
@@ -48,6 +48,9 @@ type ForgePull struct {
// MergedAt is when it merged, as the forge says it (RFC 3339); empty when not merged or not said.
MergedAt string `json:"merged_at,omitempty"`
HTMLURL string `json:"html_url,omitempty"`
// Mergeable is the forge's own word on whether it merges into its base now (gitea's mergeable): false while it is in
// conflict with its base, or while the forge works that out after the base moved; nil when the forge did not say.
Mergeable *bool `json:"mergeable,omitempty"`
}
// ForgeStatus is one status of a commit, the newest of its context.
@@ -73,6 +76,10 @@ type ForgeMerge struct {
} `json:"blocking"`
Mergeable *bool `json:"mergeable"`
Says string `json:"says"`
// Writes is whether the forge's merging account may write the repository at all (issue 1945): false, its statuses and
// its merge are refused, and CannotWrite says so with the tool that grants it. Nil when the forge holder did not say.
Writes *bool `json:"writes,omitempty"`
CannotWrite string `json:"cannot_write,omitempty"`
}
// Store keeps deliveries and groups: one key each, in the module's own state (ADR 0201).
@@ -121,6 +121,18 @@ type ReadyRun struct {
Declined *time.Time `json:"declined,omitempty"`
Warrant string `json:"warrant,omitempty"`
Refused string `json:"refused,omitempty"`
// One check per delivery (novox/hq ADR 0548 §2, §4, turns.go): HeldSince is when the approved merge began to wait for the
// check of other heads HeldFor names (its verdict key), running on a repository it goes into; HandNow the operator's word
// that handed it at once past that check.
HeldSince *time.Time `json:"held_since,omitempty"`
HeldFor string `json:"held_for,omitempty"`
HandNow string `json:"hand_now,omitempty"`
// CannotWrite is why the operator is not asked: the forge's merging account cannot write a repository the ask covers
// (writes.go, issue 1945); CannotWriteSince since when. Said at once as a condition.
CannotWrite string `json:"cannot_write,omitempty"`
CannotWriteSince *time.Time `json:"cannot_write_since,omitempty"`
}
// MergeBranch is the one branch this owner merges into (novox/hq ADR 0398 §4): a merge into any other is its author's.
@@ -244,6 +256,12 @@ func (h *Holder) readyAsks(now time.Time) []func() {
if waits == "" {
waits, approvalAsk = h.approval(d, now)
}
if waits == "" && gates {
waits = h.turnWaits(d, now) // nothing merges into a repository while a check of other heads judges it (ADR 0548 §2)
}
if waits == "" {
waits = h.passWaits(d) // no merge lands in a repository between another approved group's members (issue 1275)
}
if waits != r.Waits {
r.Waits = waits
h.keep(d)
@@ -267,6 +285,7 @@ func (h *Holder) readyAsks(now time.Time) []func() {
asks = append(asks, a)
}
}
asks = append(asks, h.turnWrites()...) // the merge turns, after the holds of this look (ADR 0548 §3)
asks = append(asks, h.settleMerges()...)
asks = append(asks, h.settleForgeMerges()...)
asks = append(asks, h.withdrawnKeys()...)
@@ -376,6 +395,9 @@ func (h *Holder) readyWaits(d *Delivery) string {
case d.State == Proposed && d.outOfDate() != "" && d.AskedAgain != nil:
return "its check against the trunk's head is asked and has not answered"
case d.State == Proposed && d.outOfDate() != "" && d.Check.Passes() && h.covering(d) != nil:
// Approved, and its trunk moved: the checker carries its verdict to the trunk's head by its package digests (ADR 0548
// §5, §8). The Approve stands and is not asked again; the merge waits for the carried verdict, not a fresh check.
return "approved; its verdict is being carried to the trunk's head " + shortOf(d.TrunkNow, 12) + " (" + d.outOfDate() + ")"
case d.State == Proposed && d.outOfDate() != "":
return "its verdict is out of date: its check against the trunk's head is asked before the operator is"
case d.State == Proposed && d.AskedAgain != nil:
@@ -503,6 +525,12 @@ func (h *Holder) readyLines(now time.Time) []StalledLine {
out = append(out, StalledLine{ID: d.ID, Number: d.Number, State: d.State, For: now.Sub(r.Seen).Round(time.Second).String(),
Bound: headHeardBound.String(), H2: "none: a later announcement of the pull request's head (the merge tool announces it again, the forge's announcer restarts, or a new push) or an edit of its description; a recheck does not fill it — the operator's",
Says: fmt.Sprintf("%s#%d passed its checks at %s and waits for %s", d.Repository, d.Number, shortOf(r.Head, 12), waitsForHead)})
case r.CannotWrite != "" && r.Approved == nil && r.CannotWriteSince != nil:
out = append(out, StalledLine{ID: d.ID, Number: d.Number, State: d.State,
For: now.Sub(*r.CannotWriteSince).Round(time.Second).String(), Bound: "0s",
H2: "none: the forge account given write on the repository (gitea_grant_write; the gitea module grants it itself once a module of it is registered), which asks the operator then",
Says: fmt.Sprintf("%s#%d passed its checks at %s and the operator is not asked for its Approve: %s",
d.Repository, d.Number, shortOf(r.Head, 12), r.CannotWrite)})
case r.Failed != "" && d.State == Rejected:
// Its cure is its author's new head: listed here at once (`show`), told on the pull request at once, and the
// operator's only once no new head came within the author's bound (novox/hq ADR 0505 §5).
@@ -0,0 +1,454 @@
package main
import (
"encoding/json"
"fmt"
"sort"
"strings"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
sdk "git.novox.be/novox/mesh-sdk/go/delivery"
)
// **One check per delivery** (novox/hq ADR 0548 §2 to §4, issue 568). While a check runs, nothing merges into a repository it
// judges, and an approved merge holds a turn on its repositories so no new check starts there:
//
// - the hold (§2): an approved merge is handed to the forge bridge only when no verdict of the checker's verdicts bucket in
// the state running names, among its trunk heads, a repository the merge goes into; the verdict of the merge's own heads
// excepted. The gate watches that bucket already (onestatus.go), so the verdict leaving running gives the waiting merge its
// turn at once, with no timer. Siblings a check reads beside are not held (§2). Only while mesh/check gates the merge;
// - the turn (§3): before it reads the verdicts for a merge, the gate writes one key per repository the merge goes into in its
// bucket turns (`<owner>.<name>`, sdk.TurnsKey): the deliveries that wait there in the order of their Approve, the check
// each waits for and since when; for a group on every member's repository until its last member merged. The checker reads
// the bucket and claims no check of a repository a turn names. The key goes when the last delivery in it merged or left;
// - the bound and the tools (§4): a merge held past turnBound is a stalled line; delivery_turns reads every turn and
// delivery_hand_now hands a waiting merge at once on the operator's word, the check it waited for then carried.
// turnBound is how long an approved merge may wait for another delivery's check before it is a stalled line (ADR 0548 §4).
const turnBound = 30 * time.Minute
// TurnStore is the gate's bucket turns: one key per repository, written by the gate alone.
type TurnStore interface {
// All reads every turn, by repository (lowercase).
All() (map[string]sdk.Turn, error)
Put(t sdk.Turn) error
Delete(repository string) error
}
// kvTurns is the bucket turns over the runtime.
type kvTurns struct{}
func (kvTurns) All() (map[string]sdk.Turn, error) {
keys, err := stdio.State("turns").Keys()
if err != nil {
return nil, err
}
out := map[string]sdk.Turn{}
for _, k := range keys {
e, err := stdio.State("turns").Get(k)
if err != nil {
return nil, err
}
if e == nil {
continue
}
repository, err := sdk.ParseTurnsKey(k)
if err != nil {
continue // not a turn's key: never the gate's, left
}
t, err := sdk.DecodeTurn(e.Value)
if err != nil {
t = sdk.Turn{Repository: repository} // unreadable: rewritten or deleted at the next look
}
out[repository] = t
}
return out, nil
}
func (kvTurns) Put(t sdk.Turn) error {
key, err := sdk.TurnsKey(t.Repository)
if err != nil {
return err
}
raw, err := sdk.EncodeTurn(t)
if err != nil {
return err
}
_, err = stdio.State("turns").Put(key, json.RawMessage(raw))
return err
}
func (kvTurns) Delete(repository string) error {
key, err := sdk.TurnsKey(repository)
if err != nil {
return err
}
return stdio.State("turns").Delete(key)
}
// approvedMerge is an approved merge not yet done: the run holding its Approve, the deliveries it covers, the hash of their
// heads as the checker keys their verdict (lowercase, onestatus.go headsOf), and the repositories it goes into (lowercase).
type approvedMerge struct {
owner *Delivery
run *ReadyRun
group string
heads string
repos []string
}
// approvedMerges are the merges the operator approved that are not done, in the order of their Approve: each ask once, its
// run on the delivery that asked it. A declined, refused, merged, rejected or final one has left. Under the lock.
func (h *Holder) approvedMerges() []approvedMerge {
var out []approvedMerge
seen := map[string]bool{}
ids := make([]string, 0, len(h.deliveries))
for id := range h.deliveries {
ids = append(ids, id)
}
sort.Strings(ids) // the same owner of an ask at every look, whichever member's run holds it
for _, id := range ids {
d := h.deliveries[id]
r := d.Ready
if r == nil || r.Approved == nil || r.Declined != nil || r.AskID == "" || len(r.Heads) == 0 || seen[r.AskID] ||
d.State.Final() || d.State == Rejected || d.Base != MergeBranch || r.RefusedSince != nil || h.allMerged(r.Heads) {
continue
}
seen[r.AskID] = true
var heads []sdk.Head
repos := map[string]bool{}
for id, sha := range r.Heads {
m := h.deliveries[id]
if m == nil {
continue
}
heads = append(heads, sdk.Head{Repository: strings.ToLower(m.Repository), Number: m.Number, Head: strings.ToLower(sha)})
repos[strings.ToLower(m.Repository)] = true
}
hash, err := sdk.HeadsHash(heads)
if err != nil {
continue
}
am := approvedMerge{owner: d, run: r, group: askGroup(d), heads: hash}
for repo := range repos {
am.repos = append(am.repos, repo)
}
sort.Strings(am.repos)
out = append(out, am)
}
sort.SliceStable(out, func(i, j int) bool {
if !out[i].run.Approved.Equal(*out[j].run.Approved) {
return out[i].run.Approved.Before(*out[j].run.Approved)
}
return out[i].heads < out[j].heads
})
return out
}
// mergeOfDelivery is the approved merge covering d, if one is approved.
func (h *Holder) mergeOfDelivery(d *Delivery) (approvedMerge, bool) {
c := h.covering(d)
if c == nil || c.Approved == nil {
return approvedMerge{}, false
}
for _, am := range h.approvedMerges() {
// By the ask, not the run: a group's members may each hold a run of the same ask, and the merge is the ask's.
if am.run.AskID == c.AskID {
return am, true
}
}
return approvedMerge{}, false
}
// runningCheck is a check of the checker in the state running that names repo among its trunk heads, its heads not own: its
// verdict key and since when the gate has seen it run. False when none runs there. Under the lock.
func (h *Holder) runningCheck(repo, own string) (key string, v checkerVerdict, since time.Time, ok bool) {
hashes := make([]string, 0, len(h.checker))
for hash := range h.checker {
hashes = append(hashes, hash)
}
sort.Strings(hashes)
for _, hash := range hashes {
v := h.checker[hash]
if v.State != checkerRunning || v.Heads == own {
continue
}
for r := range v.Trunk {
if strings.EqualFold(r, repo) {
key, err := sdk.CheckerVerdictKey(v.Group, v.Heads)
if err != nil {
key = v.Group + "." + v.Heads
}
return key, v, h.runningSince[v.Heads], true
}
}
}
return "", checkerVerdict{}, time.Time{}, false
}
// turnWaits is what an approved merge waits for before it is handed (ADR 0548 §2, §3): its turns written, then the end of
// every check of other heads running on a repository it goes into. Empty when it may be handed. It keeps on the run since
// when it is held and for which check, for the stalled line and the tools. Under the lock.
func (h *Holder) turnWaits(d *Delivery, now time.Time) string {
am, ok := h.mergeOfDelivery(d)
if !ok {
return ""
}
r := am.run
if h.Turns != nil {
if !h.turnsLoaded {
return "approved; its merge turn waits for the turns the gate wrote before to be read"
}
for _, repo := range am.repos {
if !turnNames(h.turnsHeld[repo], am.heads) {
return "approved; its merge turn on " + repo + " is being written, so no check starts there before it merges"
}
}
}
if r.HandNow != "" {
if r.HeldSince != nil {
r.HeldSince, r.HeldFor = nil, ""
h.keep(am.owner)
}
return "" // handed now on the operator's word: the check it waited for is carried after the merge
}
for _, repo := range am.repos {
key, v, since, running := h.runningCheck(repo, am.heads)
if !running {
continue
}
if r.HeldSince == nil || r.HeldFor != key {
at := now
r.HeldSince, r.HeldFor = &at, key
h.keep(am.owner)
h.Logf("[mesh-delivery] %s is approved and held: the check %s runs on %s, and nothing merges there while it does (ADR 0548 §2)", am.owner.ID, key, repo)
}
words := fmt.Sprintf("approved; waits for the check of %s (heads %s) on %s", v.Group, v.Heads, repo)
if !since.IsZero() {
words += ", running since " + since.UTC().Format(time.RFC3339)
}
return words + " (novox/hq ADR 0548 §2)"
}
if r.HeldSince != nil {
r.HeldSince, r.HeldFor = nil, ""
h.keep(am.owner)
}
return ""
}
// turnNames says whether a turn names the heads.
func turnNames(t sdk.Turn, heads string) bool {
for _, w := range t.Waiting {
if w.Heads == heads {
return true
}
}
return false
}
// wantedTurns are the turns the approved merges hold now, by repository: each merge on every repository it goes into, in
// the order of their Approve, with the check of other heads it waits for there. Under the lock.
func (h *Holder) wantedTurns() map[string]sdk.Turn {
out := map[string]sdk.Turn{}
for _, am := range h.approvedMerges() {
for _, repo := range am.repos {
t := out[repo]
t.Repository = repo
w := sdk.TurnWait{Group: am.group, Heads: am.heads, Ask: am.run.AskID, Approved: am.run.Approved.UTC(), Since: am.run.Approved.UTC()}
if key, _, since, ok := h.runningCheck(repo, am.heads); ok {
w.WaitsFor = key
if !since.IsZero() {
w.WaitsSince = since.UTC()
}
}
t.Waiting = append(t.Waiting, w)
out[repo] = t
}
}
return out
}
// turnWrites are the writes that bring the bucket turns to what the approved merges hold: a turn written where it changed,
// a key deleted where its last delivery merged or left. The first look reads what a gate before this one wrote. Under the
// lock; the writes run outside it, and each that lands is kept and asks for a look.
func (h *Holder) turnWrites() []func() {
if h.Turns == nil {
return nil
}
if !h.turnsLoaded {
return []func(){func() {
held, err := h.Turns.All()
h.mu.Lock()
defer h.mu.Unlock()
if err != nil {
h.Logf("[mesh-delivery] the merge turns written before cannot be read, so no merge is handed yet: %v; asked again in %s", err, askAgainAfter)
h.again("the merge turns written before, read")
return
}
h.turnsHeld, h.turnsLoaded = held, true
h.Poke()
}}
}
want := map[string]sdk.Turn{}
if h.checkGates() {
want = h.wantedTurns() // while the check gates no merge, no merge waits for one, and no turn holds a check
}
var out []func()
repos := make([]string, 0, len(want)+len(h.turnsHeld))
for repo := range want {
repos = append(repos, repo)
}
for repo := range h.turnsHeld {
if _, ok := want[repo]; !ok {
repos = append(repos, repo)
}
}
sort.Strings(repos)
for _, repo := range repos {
t, wanted := want[repo]
held, had := h.turnsHeld[repo]
switch {
case wanted && had && sameTurnValue(held, t):
case wanted:
out = append(out, func() {
err := h.Turns.Put(t)
h.mu.Lock()
defer h.mu.Unlock()
if err != nil {
h.Logf("[mesh-delivery] the merge turn on %s cannot be written: %v; asked again in %s", repo, err, askAgainAfter)
h.again("a merge turn, written")
return
}
h.turnsHeld[repo] = t
h.Poke()
})
default:
out = append(out, func() {
err := h.Turns.Delete(repo)
h.mu.Lock()
defer h.mu.Unlock()
if err != nil {
h.Logf("[mesh-delivery] the merge turn on %s cannot be deleted: %v; asked again in %s", repo, err, askAgainAfter)
h.again("a merge turn, deleted")
return
}
delete(h.turnsHeld, repo)
h.Logf("[mesh-delivery] the merge turn on %s is deleted: no approved merge waits on it, so the checker may start a check there", repo)
h.Poke()
})
}
}
return out
}
// sameTurnValue says whether two turns hold the same value, the version aside.
func sameTurnValue(a, b sdk.Turn) bool {
a.V, b.V = 0, 0
return sdk.SameTurn(a, b)
}
// turnLines are the stalled lines of approved merges held past turnBound by another delivery's check (ADR 0548 §4). Under the
// lock.
func (h *Holder) turnLines(now time.Time) []StalledLine {
var out []StalledLine
for _, d := range h.deliveries {
r := d.Ready
if r == nil || r.HeldSince == nil || r.Merged != nil || d.MergedAs != "" || d.State.Final() || now.Sub(*r.HeldSince) < turnBound {
continue
}
words := r.Waits
if words == "" {
words = "approved; waits for the check " + r.HeldFor
}
out = append(out, StalledLine{ID: d.ID, Number: d.Number, State: d.State, For: now.Sub(*r.HeldSince).Round(time.Second).String(),
Bound: turnBound.String(),
H2: "none: the check it waits for ends, or the operator hands the merge now with delivery_hand_now — the operator's",
Says: fmt.Sprintf("%s#%d at %s is %s, held since %s; the step that check is at is read with the checker's checker_status",
d.Repository, d.Number, shortOf(r.Head, 12), words, r.HeldSince.UTC().Format(time.RFC3339))})
}
return out
}
// TurnLine is one repository's turn as delivery_turns reads it: the turn as the gate holds it and, for each delivery waiting,
// the check it waits for in words.
type TurnLine struct {
Repository string `json:"repository"`
Turn sdk.Turn `json:"turn"`
Waits map[string]any `json:"waits,omitempty"`
}
// TurnsRead is the tool delivery_turns: every turn the gate holds, with the check each waiting delivery waits for, since when,
// and whether the merge was handed now on the operator's word.
func (h *Holder) TurnsRead() map[string]any {
h.mu.Lock()
defer h.mu.Unlock()
want := h.wantedTurns()
repos := make([]string, 0, len(want))
for repo := range want {
repos = append(repos, repo)
}
sort.Strings(repos)
lines := []TurnLine{}
for _, repo := range repos {
l := TurnLine{Repository: repo, Turn: want[repo], Waits: map[string]any{}}
for _, w := range want[repo].Waiting {
says := "nothing: no check of other heads runs on " + repo
if w.WaitsFor != "" {
says = "the check " + w.WaitsFor
if !w.WaitsSince.IsZero() {
says += ", running since " + w.WaitsSince.Format(time.RFC3339)
}
}
l.Waits[w.Group] = says
}
lines = append(lines, l)
}
out := map[string]any{"turns": lines, "check-gates-merge": h.checkGates(), "bound": turnBound.String()}
if h.Turns != nil {
out["written"] = h.turnsLoaded
var unwritten []string
for _, repo := range repos {
if held, ok := h.turnsHeld[repo]; !ok || !sameTurnValue(held, want[repo]) {
unwritten = append(unwritten, repo)
}
}
for repo := range h.turnsHeld {
if _, ok := want[repo]; !ok {
unwritten = append(unwritten, repo+" (to delete)")
}
}
sort.Strings(unwritten)
if len(unwritten) > 0 {
out["not-yet-written"] = unwritten
}
}
return out
}
// HandNow is the tool delivery_hand_now (ADR 0548 §4): on the operator's word an approved merge held for another delivery's
// check is handed at once; that check runs on and is carried to the new trunk after the merge. Refused for a delivery that is
// not an approved merge waiting, naming why.
func (h *Holder) HandNow(id, why string) (string, error) {
h.mu.Lock()
defer h.mu.Unlock()
d := h.deliveries[id]
if d == nil {
return "", fmt.Errorf("no delivery %q", id)
}
if strings.TrimSpace(why) == "" {
return "", fmt.Errorf("say why: the hand is the operator's word, kept on the delivery")
}
am, ok := h.mergeOfDelivery(d)
if !ok {
return "", fmt.Errorf("%s is not an approved merge waiting: nothing to hand (its state is %s)", id, d.State)
}
if am.run.HeldSince == nil {
return "", fmt.Errorf("%s waits for no check of another delivery: %s", id, firstOf(am.run.Waits, "it is handed at its turn"))
}
am.run.HandNow = why
h.keep(am.owner)
h.mergeAgainNow(d)
h.Logf("[mesh-delivery] %s is handed now on the operator's word, past the check %s it waited for, which is carried after the merge: %s",
id, am.run.HeldFor, why)
return fmt.Sprintf("%s is handed to the merge at the next look; the check %s it waited for runs on and is carried to the new trunk", id, am.run.HeldFor), nil
}

Some files were not shown because too many files have changed in this diff Show More