From 513de0e3a8c19935bbd4ca87ab9eaec2c461b7ca Mon Sep 17 00:00:00 2001 From: Marcos Date: Tue, 18 Aug 2026 04:17:32 +0000 Subject: [PATCH] docs(pilot): document AI-USAGE label, token attribution, webhook denylist MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - new "AI-USAGE label" section: what gets reported, the attributed-vs-measured distinction, PRAGENT_USAGE_ALWAYS, regression-clean without the label. - new "Webhook fires on any PR update (except closed)" section: denylist semantics + why the AI-REVIEW gate + sha dedupe keep it safe. - arch diagram gate updated (allowlist → denylist; AI-USAGE side-gate). - opencode engine step 4: --format json + parse_opencode_events → (text, usage). Co-Authored-By: Claude --- pilot/README-webhook.md | 62 ++++++++++++++++++++++++++++++++++++----- 1 file changed, 55 insertions(+), 7 deletions(-) diff --git a/pilot/README-webhook.md b/pilot/README-webhook.md index b900a83..cfebbd1 100644 --- a/pilot/README-webhook.md +++ b/pilot/README-webhook.md @@ -8,13 +8,13 @@ service, which gates on the `AI-REVIEW` label and runs the same review core. ## Architecture ``` -PR opened/pushed/labeled "AI-REVIEW" (any repo under a covered owner) +PR opened/pushed/labeled/edited/… (any repo under a covered owner) │ Gitea user-level webhook (events: pull_request) ▼ Service pragent-webhook.pragent.svc.cluster.local (ClusterIP, ns pragent) - │ HMAC-verify (X-Gitea-Signature) → gate: action in {opened, - │ reopened, synchronize/synchronized, labeled/label_updated} + │ HMAC-verify (X-Gitea-Signature) → gate: action ≠ closed │ AND pull_request.labels ∋ AI-REVIEW + │ (report_usage ← pull_request.labels ∋ AI-USAGE, optional) ▼ ai_review.review_pr() (same core the CI-step uses) 1. fetch existing reviews → dedupe: skip if a review already carries @@ -60,6 +60,50 @@ No workflow file, no repo secret, no act-runner needed. (The owner must already be covered by a user-level webhook — see below. If not, do the one-time per-owner setup first.) +## AI-USAGE label — token-usage reporting (optional, opt-in) + +A review always fires on `AI-REVIEW`. Adding a second label **`AI-USAGE`** on +the same PR opts the review into appending a token-usage report: + +- a `## 🔋 AI usage` section on the review summary body with the **measured** + review total — input / output / reasoning / cache read+write / total tokens, + agent step count, wall-clock duration, estimated cost, the model, and a scope + note (the agent reviews a whole-repo checkout at the head sha, so input + tokens include files read beyond the diff); +- a per-finding attribution table (severity · location · ≈out tok · %); +- a `🪙 ~N tok (X% · attributed output)` line at the foot of each inline + comment. + +**Attributed, not measured.** One opencode agent pass produces *all* findings, +so there is no native per-finding token metering. The per-comment / per-row +counts are the review's measured **output** tokens split by each finding's +rendered-body weight (`len(problem)+len(fix)+len(suggestion)`) — an honest +attribution, labelled as such. The totals are real measurements summed from +opencode's `step_finish` events. + +`PRAGENT_USAGE_ALWAYS=1` on the Deployment forces usage reporting on for every +review (testing / a future default-on) regardless of the label. + +Without `AI-USAGE` (regression): no usage section, no 🪙 lines — behaviour +identical to before the feature. The usage section is part of the review body, +so it's covered by the existing sha-marker dedupe. + +## Webhook fires on any PR update (except `closed`) + +The receiver uses a **denylist**, not an allowlist: it reviews on every +`pull_request` action **except `closed`** — `opened`, `reopened`, +`synchronize`/`synchronized`, `labeled`/`label_updated`, `edited` (title/body), +`ready_for_review` (draft→ready), `assigned`, `review_requested`, `milestone`, +… . This is safe because of two downstream gates: + +- the **AI-REVIEW label gate** — payload `labels` reflect current state, so an + `unlabeled` that *removed* AI-REVIEW fails the gate (no review); an + `unlabeled` of another label still passes; +- the **sha dedupe** — any same-sha re-fire (title edit, assignee, milestone, + a label toggle of another label…) is skipped, so the only newly-effective + actions are ones that change the head sha (`synchronize`, already covered) or + move a draft to ready (`ready_for_review`) on an un-reviewed sha. + ## Repo-local focus: `.pr-review.json` (optional) Drop a `.pr-review.json` at the repo root (committed on the PR's branch, or on @@ -160,13 +204,17 @@ cramped model call. `pilot/opencode_review.py` is the glue: `.pr-review.json`, prior reviews, sha, anchor hint). 3. `drop_factory` — copies `opencode.json` + `.opencode/` (agents/skills/commands) into the workdir as the project config. -4. `run_opencode` — `opencode run --pure --agent pragent --dir - --model headroom/glm-5.2:cloud` headlessly; returns the agent's stdout. +4. `run_opencode` — `opencode run --pure --format json --agent pragent + --dir --model headroom/glm-5.2:cloud` headlessly. `--format json` + emits NDJSON events: `parse_opencode_events` reconstructs the assistant text + from `text` events and sums tokens/cost/steps from every `step_finish` event. + Returns `(text, usage)`. It does **no Gitea I/O and no parsing** — `review_pr` parses the stdout into `(summary, findings)`, validates findings against diff anchors, and posts. So -all v2 logic (dedupe marker, anchor validation, ```suggestion fencing, posting) -is reused and never depends on the model remembering it. +all v2 logic (dedupe marker, anchor validation, language-tagged suggestion +fencing, posting, optional AI-USAGE attribution) is reused and never depends on +the model remembering it. The factory lives in the pragent repo root: `opencode.json` (provider/model/ permission) + `.opencode/` (agents, skills, commands). It is **both** the