Marcos
2613b3e3af
PR #7 ran under the AI-USAGE label and reported real numbers: 28 agent steps, 348s, 2,071,025 input / 17,303 output tokens, and zero cache reads or writes. The model predicted ~$0.73 on Opus 5 for that tier. The measurement prices it at $10.79 — the model was ~15x low. Two wrong assumptions: - Step count and per-step growth. `full` assumed 12 steps and 1,200 tokens per tool result; the run did 28 steps averaging ~3,300. Cost is roughly quadratic in steps, so this compounds. Tier defaults are re-derived from the measured per-step growth rather than from guesses. - Caching. The model defaulted to prompt caching on. The headroom/glm-5.2 path reports 0 read / 0 write, so the stable prefix is paid at full input price on every step. Budget with caching off until that column is nonzero. Adds OBSERVED_RUNS as an append-only calibration anchor, an observed-runs section in the report, and a regression test asserting the model stays within 2.5x of the measurement — so the next drift is caught by the suite rather than by a surprising invoice. Corrected blended figures at 350 PRs/month: ~$1,740 Opus 5, ~$1,755 GPT-5.6 Sol, ~$696 Sonnet 5, ~$348 Haiku 4.5, ~$70 GPT-5.6 Luna. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B11e8TZZxJyzHW7jj7KWUN
pragent
An extensible, forge-agnostic PR review framework. Not a product — a toolkit that teams extend with their own review dimensions.
Status: design approved; framework build deferred. A pilot is live on
glm-5.2:cloud with two delivery paths:
- Central webhook service (preferred, least per-repo setup): a Gitea
user-level webhook posts PR events to an always-on in-cluster service that
gates on the
AI-REVIEWlabel. Onboarding a repo = addpragent-botcollaborator + create the label + label a PR. Seepilot/README-webhook.md. - CI-step (legacy): a per-repo Gitea Action fetches the reviewer script at
runtime. See
pilot/README.md.
The framework design remains at
docs/plans/2026-08-04-pragent-design.md;
the pilot is its bootstrap and will be superseded by pragent review when the
framework build resumes.
What it is
pragent runs as a CI step. It reads a pull request, decides how much attention the
change deserves, runs the analyzers that apply, and posts ranked findings back to the
forge.
pragent init # one-time repo scan → .pragent/profile.yml (committed, reviewable)
pragent review # the CI step: tier → analyze → aggregate → publish
pragent explain # why did this PR get this tier / these findings?
pragent replay # re-run a past PR against a new prompt or model (the eval loop)
pragent doctor # config, credentials, and adapter health
Why not CodeRabbit / Greptile / Qodo
Those are good products with fixed review dimensions and per-seat pricing. pragent
targets the case where a platform team needs to add its own dimensions — an internal
compliance rule, a service-catalog ownership check, a house performance idiom — without
forking a vendor's reviewer. Cost lands in the same range (~$25/dev/month at 350 PRs/mo
for 20 devs), but the analyzers, the data, and the analytics are yours.
Attention tiers
Every PR is classified before any expensive work happens. Deterministic rules decide first; an ambiguous case gets one cheap model call as tie-breaker.
| Tier | What it means | Cost/PR |
|---|---|---|
trivial |
lockfile bumps, generated code, docs typos | ~$0.005 |
lite |
small change, no risk paths | ~$0.08 |
full |
the default for real changes | ~$0.80–2.00 |
oversized |
too big to review whole; structural summary + deep pass on the hot subset | ~$5 ceiling |
Every tier decision records why, so a surprising outcome is explainable rather than mysterious.
What it costs
The pilot runs on glm-5.2:cloud through the on-network headroom proxy, so today it
bills nothing per token — but the token work is real, and pilot/cost_model.py
prices it against published API rates. Factory prompt sizes are measured from the
files in this repo; the per-tier workloads are calibrated against runs actually
measured through the AI-USAGE label (OBSERVED_RUNS in that file).
The measured anchor. The hardening PR (#7, 16 files / ~1100 changed lines,
tier full) took 28 agent steps and 348s, and consumed 2,071,025 input and
17,303 output tokens — with zero cache reads or writes, because the current
headroom/glm path does no prompt caching. Priced elsewhere, that single review is:
| Model | that review | blended per PR | 350 PRs/month |
|---|---|---|---|
| Claude Opus 5 | $10.79 | ~$4.97 | ~$1,740 |
| GPT-5.6 Sol | $10.87 | ~$5.02 | ~$1,755 |
| Claude Sonnet 5 | $4.32 | ~$1.99 | ~$696 |
| GPT-5.6 Terra | $4.35 | ~$2.01 | ~$702 |
| Claude Haiku 4.5 | $2.16 | ~$0.99 | ~$348 |
| GPT-5.6 Luna | $0.43 | ~$0.20 | ~$70 |
Blended figures use a 5/35/55/5 tier mix with caching off, matching what is
actually observed. Run python3 pilot/cost_model.py --help for other mixes and
volumes.
Two things dominate, and neither is the diff:
- The loop resends its context every step. 28 steps over a ~17k-token diff
produced 2M input tokens. Cost is roughly quadratic in step count, which is why
attention-tieringcaps steps, file reads and subagent fan-out per tier. - Prompt caching is worth about a third of the bill and is currently not
happening. Any move to a paid provider should confirm the
cache_readcolumn goes nonzero before budgeting.
An earlier version of this model assumed 12 steps and caching on, and was ~15x
low. The lesson is in the file: budget from OBSERVED_RUNS, not from the tier
table, and append a row every time a real review reports usage.
Extension points
Five, all documented in the design doc. Teams override or add; nobody forks.
- Analyzers — drop a YAML + prompt in
.pragent/analyzers/, or install from npm - Forge adapters — Gitea, GitLab, GitHub, local diff
- Tier policy — thresholds and the path risk map, per repo or per org
- Profile enrichers — extend what
pragent initlearns about a repo - Emitter sinks — JSONL by default, OpenTelemetry, or your own
Org config can lock keys, so a repo cannot quietly disable the security analyzer.
Design principles
- Polyglot by construction. Language knowledge lives in the repo profile, not in the reviewer. A new language is a profile change, not a core change.
- One shared prompt prefix. All analyzers for a PR share a byte-identical cached prefix. This is what makes fan-out affordable; it is enforced, not hoped for.
- Everything is traceable. Tier reasons, token counts, cost, latency, and finding outcomes are recorded per run. False-positive rate is measurable per analyzer.
- Fail open. A budget ceiling or an analyzer crash yields a partial review with a clear note, never a blocked pipeline with no explanation.
- The reviewed code is untrusted input. The reviewer runs an agent over a
branch anyone with PR access can write. So it holds no credentials in its
environment, the checkout is stripped of files an agent runtime would load as
instructions, PR-authored text is fenced as data, and reviewer config is read
from the base branch. See "Threat model" in
pilot/README-webhook.md.
Stack
TypeScript + Node, built on the pi agent SDK.
Shipped as an npm package and an OCI image, so CI runners need no local Node install.
Roadmap
- Walking skeleton — local diff, one analyzer, rules-only tiering
- Gitea end to end — adapter, Woodpecker step, PR comments, status checks
- Profile + full tier —
pragent init, shared-prefix caching, analyzer fan-out - Extensibility hardening — plugin loading, config layering,
explain/replay - Second forge — GitLab adapter, Jenkins recipe
- Analytics maturity — OTel export, feedback loop, eval harness
License
TBD.