feat: token-usage reporting (AI-USAGE label) #6

Merged
gitea_admin merged 3 commits from feat/token-usage into main 2026-08-18 04:32:19 +00:00

E2E + deliverable PR. Adds per-review + per-comment token accounting, gated by the new AI-USAGE label. See commit. Label both AI-REVIEW + AI-USAGE to see the usage section + per-comment 🪙 lines.

E2E + deliverable PR. Adds per-review + per-comment token accounting, gated by the new AI-USAGE label. See commit. Label both AI-REVIEW + AI-USAGE to see the usage section + per-comment 🪙 lines.
pragent-bot bot added 1 commit 2026-08-18 04:15:59 +00:00
Add per-review + per-comment token accounting, surfaced only when a PR carries
the new AI-USAGE label (on top of the existing AI-REVIEW trigger).

opencode_review:
- run_opencode now uses `--format json`; parse_opencode_events reconstructs the
  assistant text from `text` events and sums tokens/cost/steps from every
  `step_finish` event (tolerant of noise / missing fields).
- run() measures duration_s around the opencode call and returns (text, usage).
- changed_files(diff) extracts the `+++ b/` paths; the brief now lists them
  under a "Changed files" focus block so the agent grounds findings in the
  diff's neighbourhood instead of unbounded whole-repo walks.

ai_review:
- format_usage_section renders a `## AI usage` block: measured totals
  (in/out/reasoning/cache/cost/steps/duration), the whole-repo scope note, and
  an attributed per-finding table. Per-comment counts are output tokens split by
  each finding's body weight — labelled "attributed" since one model pass
  produces all findings.
- inline_comment_body appends `🪙 ~N tok (X% · attributed output)` when
  attribution is present.
- review_pr gains report_usage; compute_attribution stashes _tok_attrib/_tok_pct.
- format_review_body inserts the usage section between summary and findings.

webhook_server:
- Fire on every pull_request action except `closed` (denylist, was an allowlist)
  — the AI-REVIEW gate + sha dedupe keep this safe.
- AI-USAGE label detection + PRAGENT_USAGE_ALWAYS env drive report_usage.

.opencode factory + review-methodology skill: new "Ground findings in context"
step — read callers/imports/sibling functions per changed file (1-3 files per
finding), no unbounded walks.

Tests: parse_opencode_events (text+usage sum, malformed tolerance, none-usage),
changed_files, compute_attribution math, inline 🪙 line, format_usage_section
totals/table/cost, format_review_body ordering. 68 passing.

Co-Authored-By: Claude <noreply@anthropic.com>
pragent-bot bot added the AI-REVIEWAI-USAGE labels 2026-08-18 04:16:09 +00:00
gitea_admin added 1 commit 2026-08-18 04:17:35 +00:00
- new "AI-USAGE label" section: what gets reported, the attributed-vs-measured
  distinction, PRAGENT_USAGE_ALWAYS, regression-clean without the label.
- new "Webhook fires on any PR update (except closed)" section: denylist
  semantics + why the AI-REVIEW gate + sha dedupe keep it safe.
- arch diagram gate updated (allowlist → denylist; AI-USAGE side-gate).
- opencode engine step 4: --format json + parse_opencode_events → (text, usage).

Co-Authored-By: Claude <noreply@anthropic.com>
pragent-bot bot reviewed 2026-08-18 04:20:41 +00:00
pragent-bot bot left a comment

🤖 AI Review · pragent pilot · glm-5.2:cloud · 08783456

Adds token-usage reporting (per-review totals + per-comment attributed 🪙 lines) gated by a new AI-USAGE label, plus a changed-files focus list in the brief and a broadened webhook action gate (allowlist→denylist). Diff is ~340 lines of Python across the review core, opencode engine, webhook server, and tests; not security-sensitive. All 68 tests pass and files parse cleanly. Three issues: a parser contract violation that can abort a review, a hardcoded provider cost note, and missing tests for the changed webhook gate logic.

🔋 AI usage

  • model: glm-5.2:cloud · engine: opencode · agent steps: 23 · duration: 268.4s
  • tokens: 1007651 in · 15099 out · 0 reasoning · cache 0 read / 0 write → 1022750 total
  • est. cost: $0.00 (on-network glm-5.2:cloud via headroom — no per-token charge)
  • scope: whole-repo checkout at head sha (agent can read any file + run linters, not just the diff) — input tokens include files read beyond the diff
  • per-comment tokens are attributed (one model pass produces all findings; output split by each finding's body weight)
severity location ≈out tok %
HIGH pilot/webhook_server.py:48 4727 31%
MEDIUM pilot/opencode_review.py:324 5043 33%
LOW pilot/ai_review.py:213 5330 35%

3 inline comment(s) posted below.

🤖 **AI Review** · pragent pilot · glm-5.2:cloud · `08783456` Adds token-usage reporting (per-review totals + per-comment attributed 🪙 lines) gated by a new AI-USAGE label, plus a changed-files focus list in the brief and a broadened webhook action gate (allowlist→denylist). Diff is ~340 lines of Python across the review core, opencode engine, webhook server, and tests; not security-sensitive. All 68 tests pass and files parse cleanly. Three issues: a parser contract violation that can abort a review, a hardcoded provider cost note, and missing tests for the changed webhook gate logic. ## 🔋 AI usage - model: `glm-5.2:cloud` · engine: opencode · agent steps: 23 · duration: 268.4s - tokens: 1007651 in · 15099 out · 0 reasoning · cache 0 read / 0 write → 1022750 total - est. cost: $0.00 (on-network glm-5.2:cloud via headroom — no per-token charge) - scope: whole-repo checkout at head sha (agent can read any file + run linters, not just the diff) — input tokens include files read beyond the diff - per-comment tokens are *attributed* (one model pass produces all findings; output split by each finding's body weight) | severity | location | ≈out tok | % | |---|---|---:|---:| | HIGH | `pilot/webhook_server.py:48` | 4727 | 31% | | MEDIUM | `pilot/opencode_review.py:324` | 5043 | 33% | | LOW | `pilot/ai_review.py:213` | 5330 | 35% | _3 inline comment(s) posted below._ <!-- pragent:sha=087834565d9cd026cd2c05acd17339e169d60894 -->
@@ -161,0 +210,4 @@
dur_s = f"{dur}s" if dur is not None else "?"
cost = usage.get("cost") or 0.0
cost_s = f"${cost:.4f}" if cost else "$0.00"
cost_note = (

[LOW] cost_note hardcodes "(on-network glm-5.2:cloud via headroom — no per-token charge)" keyed off not cost rather than the actual provider/model, so it mislabels any non-headroom model, and also mislabels a billed provider whose run cost $0.0000 (e.g. cached tokens) as "no per-token charge".

Fix: Drop the provider-specific note, or derive it from the model/provider config instead of the cost value; at minimum key it on the model name passed in, not on not cost.

cost_note = (
    f"(billed by provider: {model})" if cost else "(no per-token charge reported)"
)

🪙 ~5330 tok (35% · attributed output)

**[LOW]** cost_note hardcodes "(on-network glm-5.2:cloud via headroom — no per-token charge)" keyed off `not cost` rather than the actual provider/model, so it mislabels any non-headroom model, and also mislabels a billed provider whose run cost $0.0000 (e.g. cached tokens) as "no per-token charge". Fix: Drop the provider-specific note, or derive it from the model/provider config instead of the cost value; at minimum key it on the model name passed in, not on `not cost`. ```python cost_note = ( f"(billed by provider: {model})" if cost else "(no per-token charge reported)" ) ``` 🪙 ~5330 tok (35% · attributed output)
@@ -236,0 +321,4 @@
if isinstance(tok, dict):
saw_step = True
usage["steps"] += 1
usage["input"] += int(tok.get("input") or 0)

[MEDIUM] int(tok.get("input") or 0) raises ValueError when a token field is a non-int string (e.g. "90.5"), violating the docstring's "Never raises" promise; run_opencode trusts that contract and a single bad event line aborts the whole parse, losing the assistant text.

Fix: Wrap each token coercion in a try/except (or use a safe _to_int helper that falls back to 0 on TypeError/ValueError) so malformed values are skipped, not fatal.

def _to_int(v) -> int:
    try:
        return int(v)
    except (TypeError, ValueError):
        return 0

🪙 ~5043 tok (33% · attributed output)

**[MEDIUM]** int(tok.get("input") or 0) raises ValueError when a token field is a non-int string (e.g. "90.5"), violating the docstring's "Never raises" promise; run_opencode trusts that contract and a single bad event line aborts the whole parse, losing the assistant text. Fix: Wrap each token coercion in a try/except (or use a safe _to_int helper that falls back to 0 on TypeError/ValueError) so malformed values are skipped, not fatal. ```python def _to_int(v) -> int: try: return int(v) except (TypeError, ValueError): return 0 ``` 🪙 ~5043 tok (33% · attributed output)
@@ -46,0 +45,4 @@
# the label gate (payload `labels` reflect current state). Gitea emits
# GitHub-style `action` names (`labeled`, `synchronize`) even though the
# `X-Gitea-Event-Type` header uses `label_updated` / `synchronized`.
SKIP_ACTIONS = {"closed"}

[HIGH] No tests cover the changed webhook gate logic: the _labels_have refactor, SKIP_ACTIONS denylist, AI_USAGE_LABEL detection, and report_usage plumbing are all new/changed with zero test coverage (no webhook tests exist in the repo). Gate logic is exactly what needs tests.

Fix: Add tests/pilot/test_webhook_server.py covering _labels_have (dict + bare-string labels), the SKIP_ACTIONS gate (closed skipped, other actions pass), AI-USAGE label detection, and report_usage env-override + label detection.

🪙 ~4727 tok (31% · attributed output)

**[HIGH]** No tests cover the changed webhook gate logic: the _labels_have refactor, SKIP_ACTIONS denylist, AI_USAGE_LABEL detection, and report_usage plumbing are all new/changed with zero test coverage (no webhook tests exist in the repo). Gate logic is exactly what needs tests. Fix: Add tests/pilot/test_webhook_server.py covering _labels_have (dict + bare-string labels), the SKIP_ACTIONS gate (closed skipped, other actions pass), AI-USAGE label detection, and report_usage env-override + label detection. 🪙 ~4727 tok (31% · attributed output)
pragent-bot bot reviewed 2026-08-18 04:21:08 +00:00
pragent-bot bot left a comment

🤖 AI Review · pragent pilot · glm-5.2:cloud · 513de0e3

Adds opt-in token-usage reporting (AI-USAGE label): measured review totals + attributed per-comment 🪙 lines, via opencode --format json NDJSON parsing. Also broadens the webhook from an action allowlist to a closed-only denylist. Tests pass (68) and ruff is clean of new diagnostics. Risk is low-medium: the denylist broadening is the riskiest behavioral change and has no test coverage; attribution rounding can make the per-finding table not sum to the stated output total.

🔋 AI usage

  • model: glm-5.2:cloud · engine: opencode · agent steps: 15 · duration: 210.8s
  • tokens: 577960 in · 5764 out · 0 reasoning · cache 0 read / 0 write → 583724 total
  • est. cost: $0.00 (on-network glm-5.2:cloud via headroom — no per-token charge)
  • scope: whole-repo checkout at head sha (agent can read any file + run linters, not just the diff) — input tokens include files read beyond the diff
  • per-comment tokens are attributed (one model pass produces all findings; output split by each finding's body weight)
severity location ≈out tok %
MEDIUM pilot/webhook_server.py:100 1733 30%
LOW pilot/ai_review.py:193 2603 45%
LOW pilot/ai_review.py:912 1428 25%

2 inline comment(s) posted below.

  • [LOW] pilot/ai_review.py:912 — The CI run() entry point calls review_pr without report_usage, so AI-USAGE reporting only works via the webhook path — but the README implies the feature is generally available and the CI path is described as reusing the same review core. — fix: Either pass report_usage=bool(os.environ.get("PRAGENT_USAGE_ALWAYS")) in run() so CI can opt in, or note in the README that AI-USAGE is webhook-only.
🤖 **AI Review** · pragent pilot · glm-5.2:cloud · `513de0e3` Adds opt-in token-usage reporting (AI-USAGE label): measured review totals + attributed per-comment 🪙 lines, via opencode --format json NDJSON parsing. Also broadens the webhook from an action allowlist to a `closed`-only denylist. Tests pass (68) and ruff is clean of new diagnostics. Risk is low-medium: the denylist broadening is the riskiest behavioral change and has no test coverage; attribution rounding can make the per-finding table not sum to the stated output total. ## 🔋 AI usage - model: `glm-5.2:cloud` · engine: opencode · agent steps: 15 · duration: 210.8s - tokens: 577960 in · 5764 out · 0 reasoning · cache 0 read / 0 write → 583724 total - est. cost: $0.00 (on-network glm-5.2:cloud via headroom — no per-token charge) - scope: whole-repo checkout at head sha (agent can read any file + run linters, not just the diff) — input tokens include files read beyond the diff - per-comment tokens are *attributed* (one model pass produces all findings; output split by each finding's body weight) | severity | location | ≈out tok | % | |---|---|---:|---:| | MEDIUM | `pilot/webhook_server.py:100` | 1733 | 30% | | LOW | `pilot/ai_review.py:193` | 2603 | 45% | | LOW | `pilot/ai_review.py:912` | 1428 | 25% | _2 inline comment(s) posted below._ - **[LOW]** `pilot/ai_review.py:912` — The CI `run()` entry point calls `review_pr` without `report_usage`, so AI-USAGE reporting only works via the webhook path — but the README implies the feature is generally available and the CI path is described as reusing the same review core. — fix: Either pass `report_usage=bool(os.environ.get("PRAGENT_USAGE_ALWAYS"))` in `run()` so CI can opt in, or note in the README that AI-USAGE is webhook-only. <!-- pragent:sha=513de0e3a8c19935bbd4ca87ab9eaec2c461b7ca -->
@@ -161,0 +190,4 @@
f["_tok_pct"] = 1.0 / len(findings)
return
for f, w in zip(findings, weights):
f["_tok_attrib"] = int(round(output_tokens * w / total_w))

[LOW] int(round(output_tokens * w / total_w)) per finding can make the per-finding ≈out tok table column sum to a value ≠ the stated measured output total (off by 1–N), so the table is internally inconsistent with the headline output figure.

Fix: Compute all but the last finding's attribution by rounding, then set the last finding's to output_tokens - sum(others) so the column always sums to the measured total.

for f, w in zip(findings[:-1], weights[:-1]):
        f["_tok_attrib"] = int(round(output_tokens * w / total_w))
        f["_tok_pct"] = w / total_w
    last_w = weights[-1]
    findings[-1]["_tok_attrib"] = output_tokens - sum(f["_tok_attrib"] for f in findings[:-1])
    findings[-1]["_tok_pct"] = last_w / total_w

🪙 ~2603 tok (45% · attributed output)

**[LOW]** `int(round(output_tokens * w / total_w))` per finding can make the per-finding `≈out tok` table column sum to a value ≠ the stated measured output total (off by 1–N), so the table is internally inconsistent with the headline output figure. Fix: Compute all but the last finding's attribution by rounding, then set the last finding's to `output_tokens - sum(others)` so the column always sums to the measured total. ```python for f, w in zip(findings[:-1], weights[:-1]): f["_tok_attrib"] = int(round(output_tokens * w / total_w)) f["_tok_pct"] = w / total_w last_w = weights[-1] findings[-1]["_tok_attrib"] = output_tokens - sum(f["_tok_attrib"] for f in findings[:-1]) findings[-1]["_tok_pct"] = last_w / total_w ``` 🪙 ~2603 tok (45% · attributed output)
@@ -88,3 +98,3 @@
repo = repo_obj.get("full_name") or ""
if action not in REVIEW_ACTIONS:
if action in SKIP_ACTIONS:

[MEDIUM] The allowlist→denylist broadening (fire on every action except closed) is the riskiest behavioral change in the PR but has zero test coverage — a regression here could fire on merged PRs or cause review storms, and the safety argument (sha-dedupe + label gate) is never verified by a test.

Fix: Add tests for _handle_pull_request: closed is skipped, a non-closed action with AI-REVIEW label proceeds, and an unlabeled payload whose labels no longer contain AI-REVIEW is skipped.

🪙 ~1733 tok (30% · attributed output)

**[MEDIUM]** The allowlist→denylist broadening (fire on every action except `closed`) is the riskiest behavioral change in the PR but has zero test coverage — a regression here could fire on merged PRs or cause review storms, and the safety argument (sha-dedupe + label gate) is never verified by a test. Fix: Add tests for `_handle_pull_request`: `closed` is skipped, a non-`closed` action with AI-REVIEW label proceeds, and an `unlabeled` payload whose labels no longer contain AI-REVIEW is skipped. 🪙 ~1733 tok (30% · attributed output)
pragent-bot bot removed the AI-USAGE label 2026-08-18 04:23:01 +00:00
gitea_admin added 1 commit 2026-08-18 04:23:06 +00:00
pragent-bot bot reviewed 2026-08-18 04:25:53 +00:00
pragent-bot bot left a comment

🤖 AI Review · pragent pilot · glm-5.2:cloud · d3dcd9c8

Adds opt-in token-usage reporting (AI-USAGE label): measured review totals + attributed per-comment 🪙 lines, via opencode --format json NDJSON parsing, plus a changed-files focus list in the brief and a webhook gate broadened from an action allowlist to a closed-only denylist. ~340 lines of Python; 68 tests pass and files compile cleanly. Risk is low — the two prior reviews already raised the main issues (untested webhook denylist, hardcoded cost-note model name, CI run() not passing report_usage, attribution rounding). Two new minor findings: a regex false-positive on diff body lines starting with +++ b/, and a misleading 0 total when the opencode event omits the total field.

2 inline comment(s) posted below.

🤖 **AI Review** · pragent pilot · glm-5.2:cloud · `d3dcd9c8` Adds opt-in token-usage reporting (AI-USAGE label): measured review totals + attributed per-comment 🪙 lines, via opencode --format json NDJSON parsing, plus a changed-files focus list in the brief and a webhook gate broadened from an action allowlist to a closed-only denylist. ~340 lines of Python; 68 tests pass and files compile cleanly. Risk is low — the two prior reviews already raised the main issues (untested webhook denylist, hardcoded cost-note model name, CI run() not passing report_usage, attribution rounding). Two new minor findings: a regex false-positive on diff body lines starting with `+++ b/`, and a misleading `0 total` when the opencode event omits the total field. _2 inline comment(s) posted below._ <!-- pragent:sha=d3dcd9c8e57baf5239bc6319ad99245cd6d4f1b9 -->
@@ -145,1 +147,4 @@
# Matches unified-diff new-file path headers: `+++ b/path` (and `+++ /dev/null`
# for deletions, which we skip). Captures the path after the `b/` prefix.
_NEW_FILE_HEADER_RE = re.compile(r"^\+\+\+ b/(.+?)\s*$")

[LOW] changed_files() scans every line of the raw diff for +++ b/ without distinguishing file headers from hunk body lines, so an added diff line whose content begins with +++ b/ (e.g. a patch that itself contains diff text) is falsely extracted as a changed file and injected into the brief's focus list.

Fix: Only treat a +++ b/ line as a header when it is a real file header — i.e. not preceded by a hunk + content marker. Track whether a hunk is open and skip matches inside hunk bodies, or require the line to follow a diff --git header.

out = []
    seen = set()
    in_hunk = False
    for line in (diff or "").splitlines():
        if line.startswith("@@"):
            in_hunk = True
            continue
        if line.startswith("diff --git"):
            in_hunk = False
        if in_hunk:
            continue
        if not line.startswith("+++ b/"):
            continue
        m = _NEW_FILE_HEADER_RE.match(line)
        if not m:
            continue
        path = m.group(1).strip()
        if path and path not in seen:
            seen.add(path)
            out.append(path)
    return sorted(out)
**[LOW]** changed_files() scans every line of the raw diff for `+++ b/` without distinguishing file headers from hunk body lines, so an added diff line whose content begins with `+++ b/` (e.g. a patch that itself contains diff text) is falsely extracted as a changed file and injected into the brief's focus list. Fix: Only treat a `+++ b/` line as a header when it is a real file header — i.e. not preceded by a hunk `+` content marker. Track whether a hunk is open and skip matches inside hunk bodies, or require the line to follow a `diff --git` header. ```python out = [] seen = set() in_hunk = False for line in (diff or "").splitlines(): if line.startswith("@@"): in_hunk = True continue if line.startswith("diff --git"): in_hunk = False if in_hunk: continue if not line.startswith("+++ b/"): continue m = _NEW_FILE_HEADER_RE.match(line) if not m: continue path = m.group(1).strip() if path and path not in seen: seen.add(path) out.append(path) return sorted(out) ```
@@ -236,0 +328,4 @@
if isinstance(cache, dict):
usage["cache_read"] += int(cache.get("read") or 0)
usage["cache_write"] += int(cache.get("write") or 0)
usage["total"] += int(tok.get("total") or 0)

[LOW] usage["total"] is only summed from an explicit total field in the event; if opencode ever emits a step_finish with input/output but no total, the usage section reports 0 total alongside non-zero in/out figures, which is misleading.

Fix: Fall back to input+output+reasoning when total is absent, or derive total as the sum of the components when the field is missing.

tot = tok.get("total")
                if tok.get("total") is None:
                    tok_total = usage["input"] + usage["output"] + usage["reasoning"]
                else:
                    tok_total = int(tok_total or 0)
                usage["total"] += tok_total
**[LOW]** usage["total"] is only summed from an explicit `total` field in the event; if opencode ever emits a step_finish with input/output but no `total`, the usage section reports `0 total` alongside non-zero in/out figures, which is misleading. Fix: Fall back to input+output+reasoning when `total` is absent, or derive total as the sum of the components when the field is missing. ```python tot = tok.get("total") if tok.get("total") is None: tok_total = usage["input"] + usage["output"] + usage["reasoning"] else: tok_total = int(tok_total or 0) usage["total"] += tok_total ```
gitea_admin merged commit ace47d3899 into main 2026-08-18 04:32:19 +00:00
Sign in to join this conversation.
No Reviewers
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: gitea_admin/pragent#6