Marcos
5d28f97080
Found by running the reviewer against this branch. The second review of PR #7 ran a full agent loop — 330s of a 540s budget, no timeout — and ended without a parseable ```json block. The code discarded the entire run and posted "AI review produced no parseable output.", losing minutes of work and millions of tokens for a message that tells the maintainer nothing and gives me nothing to debug. Three changes on that path: - salvage_summary() keeps the agent's prose (fenced blocks stripped, tail kept because the conclusion is written last) and posts it under an explicit banner saying it is unstructured and its line numbers were never validated against the diff. A partial review honestly labelled beats no review. - The raw output's length and last 600 chars go to stderr, so the next occurrence is diagnosable from pod logs instead of invisible. - The AI-USAGE section is still rendered. The label asked for it and the tokens were spent either way; dropping the measurement on the failure path is how the cost model stops getting calibration data exactly when it is most interesting. Not fixed here: why the agent went off-format. The likely cause is the 40-step cap in the agent definition being reached on a larger diff (the successful run used 28), which wants either a higher cap or a step-budget warning in the prompt. Needs the next occurrence's stderr to confirm rather than guess. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B11e8TZZxJyzHW7jj7KWUN
pragent
An extensible, forge-agnostic PR review framework. Not a product — a toolkit that teams extend with their own review dimensions.
Status: design approved; framework build deferred. A pilot is live on
glm-5.2:cloud with two delivery paths:
- Central webhook service (preferred, least per-repo setup): a Gitea
user-level webhook posts PR events to an always-on in-cluster service that
gates on the
AI-REVIEWlabel. Onboarding a repo = addpragent-botcollaborator + create the label + label a PR. Seepilot/README-webhook.md. - CI-step (legacy): a per-repo Gitea Action fetches the reviewer script at
runtime. See
pilot/README.md.
The framework design remains at
docs/plans/2026-08-04-pragent-design.md;
the pilot is its bootstrap and will be superseded by pragent review when the
framework build resumes.
What it is
pragent runs as a CI step. It reads a pull request, decides how much attention the
change deserves, runs the analyzers that apply, and posts ranked findings back to the
forge.
pragent init # one-time repo scan → .pragent/profile.yml (committed, reviewable)
pragent review # the CI step: tier → analyze → aggregate → publish
pragent explain # why did this PR get this tier / these findings?
pragent replay # re-run a past PR against a new prompt or model (the eval loop)
pragent doctor # config, credentials, and adapter health
Why not CodeRabbit / Greptile / Qodo
Those are good products with fixed review dimensions and per-seat pricing. pragent
targets the case where a platform team needs to add its own dimensions — an internal
compliance rule, a service-catalog ownership check, a house performance idiom — without
forking a vendor's reviewer. Cost lands in the same range (~$25/dev/month at 350 PRs/mo
for 20 devs), but the analyzers, the data, and the analytics are yours.
Attention tiers
Every PR is classified before any expensive work happens. Deterministic rules decide first; an ambiguous case gets one cheap model call as tie-breaker.
| Tier | What it means | Cost/PR |
|---|---|---|
trivial |
lockfile bumps, generated code, docs typos | ~$0.005 |
lite |
small change, no risk paths | ~$0.08 |
full |
the default for real changes | ~$0.80–2.00 |
oversized |
too big to review whole; structural summary + deep pass on the hot subset | ~$5 ceiling |
Every tier decision records why, so a surprising outcome is explainable rather than mysterious.
What it costs
The pilot runs on glm-5.2:cloud through the on-network headroom proxy, so today it
bills nothing per token — but the token work is real, and pilot/cost_model.py
prices it against published API rates. Factory prompt sizes are measured from the
files in this repo; the per-tier workloads are calibrated against runs actually
measured through the AI-USAGE label (OBSERVED_RUNS in that file).
The measured anchor. The hardening PR (#7, 16 files / ~1100 changed lines,
tier full) took 28 agent steps and 348s, and consumed 2,071,025 input and
17,303 output tokens — with zero cache reads or writes, because the current
headroom/glm path does no prompt caching. Priced elsewhere, that single review is:
| Model | that review | blended per PR | 350 PRs/month |
|---|---|---|---|
| Claude Opus 5 | $10.79 | ~$4.97 | ~$1,740 |
| GPT-5.6 Sol | $10.87 | ~$5.02 | ~$1,755 |
| Claude Sonnet 5 | $4.32 | ~$1.99 | ~$696 |
| GPT-5.6 Terra | $4.35 | ~$2.01 | ~$702 |
| Claude Haiku 4.5 | $2.16 | ~$0.99 | ~$348 |
| GPT-5.6 Luna | $0.43 | ~$0.20 | ~$70 |
Blended figures use a 5/35/55/5 tier mix with caching off, matching what is
actually observed. Run python3 pilot/cost_model.py --help for other mixes and
volumes.
Two things dominate, and neither is the diff:
- The loop resends its context every step. 28 steps over a ~17k-token diff
produced 2M input tokens. Cost is roughly quadratic in step count, which is why
attention-tieringcaps steps, file reads and subagent fan-out per tier. - Prompt caching is worth about a third of the bill and is currently not
happening. Any move to a paid provider should confirm the
cache_readcolumn goes nonzero before budgeting.
An earlier version of this model assumed 12 steps and caching on, and was ~15x
low. The lesson is in the file: budget from OBSERVED_RUNS, not from the tier
table, and append a row every time a real review reports usage.
Extension points
Five, all documented in the design doc. Teams override or add; nobody forks.
- Analyzers — drop a YAML + prompt in
.pragent/analyzers/, or install from npm - Forge adapters — Gitea, GitLab, GitHub, local diff
- Tier policy — thresholds and the path risk map, per repo or per org
- Profile enrichers — extend what
pragent initlearns about a repo - Emitter sinks — JSONL by default, OpenTelemetry, or your own
Org config can lock keys, so a repo cannot quietly disable the security analyzer.
Design principles
- Polyglot by construction. Language knowledge lives in the repo profile, not in the reviewer. A new language is a profile change, not a core change.
- One shared prompt prefix. All analyzers for a PR share a byte-identical cached prefix. This is what makes fan-out affordable; it is enforced, not hoped for.
- Everything is traceable. Tier reasons, token counts, cost, latency, and finding outcomes are recorded per run. False-positive rate is measurable per analyzer.
- Fail open. A budget ceiling or an analyzer crash yields a partial review with a clear note, never a blocked pipeline with no explanation.
- The reviewed code is untrusted input. The reviewer runs an agent over a
branch anyone with PR access can write. So it holds no credentials in its
environment, the checkout is stripped of files an agent runtime would load as
instructions, PR-authored text is fenced as data, and reviewer config is read
from the base branch. See "Threat model" in
pilot/README-webhook.md.
Stack
TypeScript + Node, built on the pi agent SDK.
Shipped as an npm package and an OCI image, so CI runners need no local Node install.
Roadmap
- Walking skeleton — local diff, one analyzer, rules-only tiering
- Gitea end to end — adapter, Woodpecker step, PR comments, status checks
- Profile + full tier —
pragent init, shared-prefix caching, analyzer fan-out - Extensibility hardening — plugin loading, config layering,
explain/replay - Second forge — GitLab adapter, Jenkins recipe
- Analytics maturity — OTel export, feedback loop, eval harness
License
TBD.