pragent/docs/plans/2026-08-04-pragent-design.md

pragent — Design

Date: 2026-08-04 Status: Approved, then revised the same day after a prior-art review — see docs/research/2026-08-04-prior-art-ai-code-review.md. Phase 0 (evaluate the closest existing tool) now precedes implementation. Sections 6 and 7 below are the revisions.

Problem

Existing AI PR reviewers (CodeRabbit, Greptile, Qodo) are closed products with fixed review dimensions and per-seat pricing. We need a framework, not a product: enterprise-grade, polyglot, forge-agnostic, and extensible by product teams who want to add their own review dimensions without forking the tool.

Decisions

Question	Decision	Why
Forge	Gitea first, GitLab (GitLab CI or Jenkins) next	Matches current infra; forge adapter interface keeps the second one cheap
Delivery	CLI binary run as a CI step	Stateless, no HA service to run, secrets stay in the runner, portable across forges
Agent harness	pi SDK (earendil-works/pi)	Agent loop as a library; we own orchestration, tiering, analytics
Runtime	TypeScript + Node; ship as npm package and OCI image	pi is TS; the image means project runners need no Node
Tiering	Deterministic rules first, cheap LLM triage as tie-breaker	Repeatable and auditable; adapts on semantic risk
Onboarding	pragent init writes a committed .pragent/profile.yml	Reviewable in a PR, no external index to run or keep fresh
Analytics	Structured JSONL per run + optional OpenTelemetry export	Vendor-neutral; no service required to start

1. Architecture

┌─ pragent CLI (npm pkg + OCI image) ─────────────────────┐
│  commands: init | review | explain | replay | doctor    │
└──────────────────────┬──────────────────────────────────┘
                       │
   ┌───────────────────┼────────────────────┬─────────────┐
   │                   │                    │             │
Forge Adapter     Triage Engine        Analyzer Bus   Emitter
(gitea|gitlab|    (rules → cheap        (plugins)     (JSONL+OTel)
 github|jenkins|   LLM tiebreak)             │
 local-diff)            │                    │
   │                    ▼                    ▼
   │              tier: trivial       security / quality /
   │                    lite          docs-consistency /
   │                    full          performance / custom
   │                    oversized            │
   │                                         ▼
   │                              pi SDK AgentSession per analyzer
   │                              (model pinned per analyzer + tier)
   ▼                                         │
Findings ◄───── Aggregator (dedupe, rank, confidence gate) ◄┘
   │
   ▼
Publisher → inline comments / summary / exit code

Analyzers only ever see profile + diff + tier budget. A new language means no core change — language support lives in the profile scan and per-language tool detection, not in the reviewer.

Five extension points

1. Analyzer plugin — declares id, applicable tiers, languages/globs, prompt, tools, output schema. Dropped in .pragent/analyzers/ or installed from npm.
2. Forge adapter — getDiff / getContext / publish / setStatus. Gitea day 1; GitLab + Jenkins next; local adapter for the dev loop.
3. Tier policy — rule file per repo/org; overrides thresholds and the path risk map.
4. Profile enrichers — extend pragent init (e.g. service-catalog ownership lookup).
5. Emitter sinks — JSONL default; add OTel/Datadog/whatever.

Config layering

org defaults → repo .pragent/config.yml → PR labels/commands → CLI flags

Org policy can mark keys locked, so a repo cannot downgrade e.g. the security analyzer.

2. Tiering and cost model

Priced at Anthropic rates: Opus 5 $5/$25 per MTok, Sonnet 5 $3/$15, Haiku 4.5 $1/$5; cache read 0.1×, cache write 1.25×, Batch API −50%.

Key lever: every analyzer in a PR shares one prompt prefix (profile + diff + pulled context). The first analyzer pays the cache write; the rest pay 0.1×. This is what makes fan-out affordable, and it is an architectural constraint, not an optimization — the prefix must be byte-identical across analyzer calls, verified in CI.

Tier	Trigger	Engine	Cost/PR
trivial	lockfile-only, generated, docs typo, <20 LOC off risk paths	rules only (or 1 Haiku triage)	~$0.005
lite	<150 LOC, no risk paths	1 Sonnet pass	~$0.08
full	default; risk paths or >150 LOC	4–5 Opus analyzers, shared cached prefix	~$0.80 cold, ~$2.00 with agentic file reads
oversized	>2k LOC or >50 files	structural summary + deep review on hot subset, hard task_budget	~$5 ceiling

Full-tier breakdown: 53k prefix × 1.25 write = $0.33 → 3 further analyzers × 53k × 0.1 = $0.08 → unique prompts $0.04 → 12k output @ $25 = $0.30 → aggregator $0.07 ≈ $0.82.

Team of 20 devs, ~350 PRs/month, 1.7 review runs per PR: ~$510/month ≈ $25/dev, plus a one-off $5–20 per repo for pragent init. Comparable to CodeRabbit ($30/dev) and Qodo ($19/seat), with ownership of the analyzers, the data, and the analytics.

Cost knobs in config: tier→model map; effort per analyzer; 1h cache TTL on busy repos; Batch API (−50%) for nightly re-scans, analytics backfills, and eval runs (never for the blocking review path); hard per-PR spend ceiling plus task_budget, failing open with a "budget exceeded, partial review" comment.

Main variance driver: agentic file reading ($0.80 → $2.00). Tool-call budget per analyzer is a first-class config key.

3. Repo profile and analyzers

.pragent/profile.yml (written by pragent init, committed)

version: 1
generated_at: 2026-08-04T00:00:00Z
generator: pragent 0.1.0
languages:
  - { name: typescript, share: 0.72, roots: [src/, packages/] }
  - { name: go,         share: 0.21, roots: [services/ingest/] }
build:
  install: pnpm install --frozen-lockfile
  test:    pnpm test
  lint:    pnpm lint
  typecheck: pnpm tsc --noEmit
modules:
  - { path: packages/auth,    role: security-critical, owners: ["@platform"] }
  - { path: packages/billing, role: money-path,        owners: ["@payments"] }
conventions:
  error_handling: "Result<T,E> in TS; wrapped errors in Go"
  test_layout: "co-located *.test.ts; Go table tests"
  docs: { adr: docs/adr/, api: docs/api/, runbooks: docs/runbooks/ }
risk_paths: ["**/auth/**", "**/migrations/**", "infra/**", "**/*.tf"]
existing_tooling: { linters: [eslint, golangci-lint], sast: [semgrep], ci: woodpecker }

Refresh is a PR (pragent init --refresh), so profile drift is reviewable. A staleness warning fires when the profile predates N commits or a new language appears.

Analyzer contract

# .pragent/analyzers/security.yml
id: security
tiers: [lite, full, oversized]
languages: ["*"]
paths: ["**"]
model: { tier_full: claude-opus-5, tier_lite: claude-sonnet-5, effort: high }
tools: [read_file, grep, list_deps]
tool_budget: 12
prompt: ./prompts/security.md
output_schema: finding[]   # {file, line, severity, confidence, category, claim, fix}

Ships with: security, code-quality, docs-consistency (does the change contradict the docs it touches, and do public API changes update docs?), performance, tests (coverage of the changed behaviour, not line coverage). Teams add their own the same way — there is no privileged built-in path.

Aggregator: dedupes across analyzers by (file, line-window, category), drops findings below the repo's confidence gate, ranks by severity × confidence, caps comment count, and routes the remainder into the run record so nothing is silently lost.

4. Analytics and traceability

Every run emits one JSONL record, plus one per finding:

{"run_id":"...","repo":"...","pr":412,"commit":"...","tier":"full",
 "tier_reason":"rule:risk_path(**/auth/**)","analyzers":["security","tests"],
 "tokens":{"in":54200,"cached":41000,"out":11800},"cost_usd":0.83,
 "latency_ms":48200,"findings":{"total":7,"posted":4,"suppressed":3},
 "verdict":"changes_requested","models":{"security":"claude-opus-5"},
 "pragent_version":"0.1.0","profile_hash":"sha256:..."}

Traceability: tier_reason names the exact rule or triage call that chose the tier; pragent explain <run_id> reconstructs the decision path; pragent replay <run_id> re-runs the same inputs against a new prompt or model — the eval loop.

Same event schema feeds OTel spans (one per analyzer) for teams with a collector. A dashboard is a later, optional consumer of the same schema — it is not in scope now.

Feedback signal: a resolved/👎 reaction on a posted comment writes back a finding_outcome record. That is what makes false-positive rate measurable per analyzer, which is what makes the confidence gates tunable rather than guessed.

5. Rollout phases

Phase 0 — Evaluate ai-code-review (do this first). Red Hat's MIT-licensed ai-code-review already implements Phases 1–2 of this design: four forge clients (GitLab, GitHub, Forgejo, local git), six AI providers, CI integration, a committed repo-context file. Run it on real repos in the Gitea setup for a week before writing pragent code. Done when: we can name, from use rather than speculation, which of our differentiators (tiering, analyzer bus, analytics) are worth building, and whether to build them standalone, on top of that CLI, or as upstream contributions. See docs/research/2026-08-04-prior-art-ai-code-review.md for the full assessment.

Phases 1–6 below stand as written if Phase 0 concludes we build standalone. If it concludes we extend an existing base, Phases 1–2 mostly disappear and Phases 3–6 become the whole project.

1. Walking skeleton — local forge adapter, one analyzer, rules-only tiering, JSONL emitter. Runs on a local diff, prints findings. No network beyond the model API.
2. Gitea end-to-end — Gitea adapter, Woodpecker step, PR comments, status checks.
3. Profile + full tier — pragent init, shared-prefix caching, analyzer fan-out, aggregator.
4. Extensibility hardening — plugin loading from .pragent/analyzers/ and npm, config layering with locked org keys, explain / replay.
5. Second forge — GitLab adapter, Jenkins runner recipe. Proves the abstraction.
6. Analytics maturity — OTel export, feedback loop, per-analyzer eval harness.

6. Revisions from the prior-art review

Seven requirements the original design missed. They apply on every path — standalone, fork, or contribution — so they are part of the design now rather than a backlog.

6.1 Prior-comment context and synthesis (highest priority)

Before the main review, fetch all existing comments and reviews on the PR, including resolved ones, and compress them with a cheap model (Haiku-class) into a short summary of what has already been said, fixed, or explicitly rejected. Feed that summary to the analyzers.

This closes a hole the original design created. The cost model assumes ~1.7 review runs per PR — every push re-reviews. Without prior-comment context, the second run repeats the first run's findings and re-argues points a human already dismissed. That is the fastest way for an AI reviewer to get muted, and it was designed in by accident.

Cost impact is favourable: one Haiku call (~$0.01) to compress a thread, against re-posting findings that get ignored. Config: reviewContext.enabled, reviewContext.synthesisModel, reviewContext.maxTokens.

6.2 Team context document

teamContextFile — a local path or a URL — carrying organization-wide review guidance: security requirements, house conventions, compliance language. It outranks the repo profile, so one document steers every repo without copying. This is the missing half of "roll it out across many projects"; org config layering handles thresholds, this handles judgment.

6.3 Deterministic skip conditions

The tier engine gains free, pre-model skips beyond paths and sizes: draft/WIP MRs, WIP in the commit message, wip/ branch prefixes, bot authors, tagged MRs. Each records its tier_reason like any other rule. Reviewing a draft PR at full tier is a pure waste the original rules would not have caught.

6.4 Self-hosted forge configuration

forgeUrl per adapter, sslVerify, sslCertPath. A self-hosted GitLab behind a corporate CA is the normal case for the company deployment this targets, and the original design had no way to express it.

6.5 Provider matrix as a requirement

The ModelClient port must be exercised by more than one provider before we claim it is a port. Required: Anthropic, plus Ollama or another local runtime — "the diff never leaves our network" is a procurement requirement for regulated repos, not a preference. Vertex/Bedrock variants follow the same shape.

6.6 Per-provider input clamp

Alongside the file and line caps, clamp total diff characters with a provider-aware default (roughly: 150k Anthropic, 200k Gemini, 100k OpenAI, 50k Ollama). The oversized tier decides strategy; the clamp is the backstop that keeps a pathological diff from blowing the context window regardless of tier.

6.7 Smaller additions

• PR summary alongside findings — a short description of what the change does
• --dry-run with mock responses, so a team can wire the pipeline before buying keys
• Gitea adapter targets Forgejo too — Forgejo is a Gitea fork with a compatible API; this is close to free and doubles the addressable forges
• Library-docs enrichment (à la Context7) as a profile-enricher plugin, not core

7. What remains genuinely ours

After the revisions above, the differentiation is narrower and clearer than the original design implied. Existing tools — Red Hat's CLI, CodeRabbit, Greptile, Qodo — do adapters, providers, prompts, and skip logic. None of them do:

1. Graded attention with recorded provenance. Four tiers, each decision naming the rule that fired. Everyone else has binary skip-or-review.
2. An analyzer plugin bus. Per-analyzer model, effort, and tool budget, fanned out over a shared cached prefix — so a team adds a review dimension without forking anything.
3. Measurement. Run records, explain, replay, and finding-outcome feedback, which together make false-positive rate per analyzer a number you can query and a prompt change something you can A/B. No tool in this category can answer "did last week's prompt change help?"
4. Org-locked policy. Config a repo cannot downgrade.

If Phase 0 shows an existing base covers everything else well, these four are the project — and they may be worth contributing upstream rather than shipping standalone.

Non-goals (for now)

• Central webhook service (CLI-only until a team actually needs zero-setup onboarding)
• Vector/graph codebase index (profile first; index only as an opt-in plugin)
• Auto-fix commits (findings and suggestions only — writing to branches is a later, separately-gated decision)
• Web dashboard (event schema first; dashboard is a downstream consumer)

Bootstrap: the pilot (2026-08-17)

Before the framework build, a minimal pilot was shipped to get a working AI review bot live immediately (user decision 2026-08-17). It is deliberately not this design:

• Single Python script (pilot/ai_review.py, stdlib only) fetched at runtime by a per-repo Gitea Action (pilot/workflow-template.yml).
• One model, glm-5.2:cloud via the on-network headroom proxy — no tiering, no analyzer fan-out, no shared-prefix caching, no provider matrix.
• Trigger = AI-REVIEW label on a PR (deterministic skip, the cheapest tier rule, but expressed as a Gitea Actions if gate rather than the tier engine).
• Fail-open, comment-only review; no inline line comments, no status checks, no analytics JSONL, no explain/replay.
• Re-posts on every push (no prior-comment synthesis — §6.1 — yet).

The pilot validates the delivery model (CI-step + bot user, not a central webhook) and the on-network provider path. When the framework build resumes, pilot/ai_review.py is replaced by pragent review; the per-repo workflow stays, just calling the CLI instead of curling the script. See pilot/README.md.

Central webhook service (2026-08-17) — partially reverses the non-goal

User decision 2026-08-17: ship a central webhook service so "add the bot + label a PR" is the only per-repo action (no workflow file, no secret, no runner). This walks back the "central webhook service = non-goal" line above, for the pilot only — the framework's CLI-step delivery model is unchanged.

Implementation notes (see pilot/README-webhook.md):

• Gitea 1.26.1 system webhooks are broken: POST /admin/hooks returns 201 but the hook never persists (GET /admin/hooks lists 0, no delivery). The one-webhook-per-instance ideal is not achievable on this version. Fallback: user-level webhooks — one webhook per repo-owner, fires for every repo that user owns. Few owners on this instance, so near-equivalent.
• Gitea SSRF allow-list blocks webhook delivery to in-cluster hosts by default; required a scoped [webhook] ALLOWED_HOST_LIST addition to app.ini (via the helm inline-config secret) + a gitea pod restart.
• The webhook receiver (pilot/webhook_server.py, stdlib only) HMAC-verifies the delivery, gates on AI-REVIEW label + PR action, and calls the same ai_review.review_pr() core the CI-step uses — one review path, two triggers.
• The bot stays a normal user (not site admin); it must be a Write collaborator on each reviewed repo.