pragent/docs/plans/2026-08-04-pragent-design.md

# pragent — Design

**Date:** 2026-08-04
**Status:** Approved, then revised the same day after a prior-art review — see
`docs/research/2026-08-04-prior-art-ai-code-review.md`. **Phase 0 (evaluate the closest
existing tool) now precedes implementation.** Sections 6 and 7 below are the revisions.

## Problem

Existing AI PR reviewers (CodeRabbit, Greptile, Qodo) are closed products with fixed
review dimensions and per-seat pricing. We need a **framework**, not a product:
enterprise-grade, polyglot, forge-agnostic, and extensible by product teams who want
to add their own review dimensions without forking the tool.

## Decisions

| Question | Decision | Why |
|---|---|---|
| Forge | Gitea first, GitLab (GitLab CI or Jenkins) next | Matches current infra; forge adapter interface keeps the second one cheap |
| Delivery | CLI binary run as a CI step | Stateless, no HA service to run, secrets stay in the runner, portable across forges |
| Agent harness | `pi` SDK (`earendil-works/pi`) | Agent loop as a library; we own orchestration, tiering, analytics |
| Runtime | TypeScript + Node; ship as npm package **and** OCI image | `pi` is TS; the image means project runners need no Node |
| Tiering | Deterministic rules first, cheap LLM triage as tie-breaker | Repeatable and auditable; adapts on semantic risk |
| Onboarding | `pragent init` writes a committed `.pragent/profile.yml` | Reviewable in a PR, no external index to run or keep fresh |
| Analytics | Structured JSONL per run + optional OpenTelemetry export | Vendor-neutral; no service required to start |

## 1. Architecture

```
┌─ pragent CLI (npm pkg + OCI image) ─────────────────────┐
│  commands: init | review | explain | replay | doctor    │
└──────────────────────┬──────────────────────────────────┘
                       │
   ┌───────────────────┼────────────────────┬─────────────┐
   │                   │                    │             │
Forge Adapter     Triage Engine        Analyzer Bus   Emitter
(gitea|gitlab|    (rules → cheap        (plugins)     (JSONL+OTel)
 github|jenkins|   LLM tiebreak)             │
 local-diff)            │                    │
   │                    ▼                    ▼
   │              tier: trivial       security / quality /
   │                    lite          docs-consistency /
   │                    full          performance / custom
   │                    oversized            │
   │                                         ▼
   │                              pi SDK AgentSession per analyzer
   │                              (model pinned per analyzer + tier)
   ▼                                         │
Findings ◄───── Aggregator (dedupe, rank, confidence gate) ◄┘
   │
   ▼
Publisher → inline comments / summary / exit code
```

Analyzers only ever see **profile + diff + tier budget**. A new language means no core
change — language support lives in the profile scan and per-language tool detection,
not in the reviewer.

### Five extension points

1. **Analyzer plugin** — declares `id`, applicable `tiers`, `languages`/globs, prompt,
   tools, output schema. Dropped in `.pragent/analyzers/` or installed from npm.
2. **Forge adapter** — `getDiff / getContext / publish / setStatus`. Gitea day 1;
   GitLab + Jenkins next; `local` adapter for the dev loop.
3. **Tier policy** — rule file per repo/org; overrides thresholds and the path risk map.
4. **Profile enrichers** — extend `pragent init` (e.g. service-catalog ownership lookup).
5. **Emitter sinks** — JSONL default; add OTel/Datadog/whatever.

### Config layering

`org defaults → repo .pragent/config.yml → PR labels/commands → CLI flags`

Org policy can mark keys **locked**, so a repo cannot downgrade e.g. the security analyzer.

## 2. Tiering and cost model

Priced at Anthropic rates: Opus 5 $5/$25 per MTok, Sonnet 5 $3/$15, Haiku 4.5 $1/$5;
cache read 0.1×, cache write 1.25×, Batch API −50%.

**Key lever:** every analyzer in a PR shares one prompt prefix (profile + diff + pulled
context). The first analyzer pays the cache write; the rest pay 0.1×. This is what makes
fan-out affordable, and it is an architectural constraint, not an optimization — the
prefix must be byte-identical across analyzer calls, verified in CI.

| Tier | Trigger | Engine | Cost/PR |
|---|---|---|---|
| trivial | lockfile-only, generated, docs typo, <20 LOC off risk paths | rules only (or 1 Haiku triage) | ~$0.005 |
| lite | <150 LOC, no risk paths | 1 Sonnet pass | ~$0.08 |
| full | default; risk paths or >150 LOC | 4–5 Opus analyzers, shared cached prefix | ~$0.80 cold, ~$2.00 with agentic file reads |
| oversized | >2k LOC or >50 files | structural summary + deep review on hot subset, hard `task_budget` | ~$5 ceiling |

Full-tier breakdown: 53k prefix × 1.25 write = $0.33 → 3 further analyzers × 53k × 0.1 =
$0.08 → unique prompts $0.04 → 12k output @ $25 = $0.30 → aggregator $0.07 ≈ **$0.82**.

**Team of 20 devs, ~350 PRs/month, 1.7 review runs per PR:** ~$510/month ≈ **$25/dev**,
plus a one-off $5–20 per repo for `pragent init`. Comparable to CodeRabbit ($30/dev) and
Qodo ($19/seat), with ownership of the analyzers, the data, and the analytics.

**Cost knobs in config:** tier→model map; `effort` per analyzer; 1h cache TTL on busy
repos; Batch API (−50%) for nightly re-scans, analytics backfills, and eval runs (never
for the blocking review path); hard per-PR spend ceiling plus `task_budget`, failing open
with a "budget exceeded, partial review" comment.

**Main variance driver:** agentic file reading ($0.80 → $2.00). Tool-call budget per
analyzer is a first-class config key.

## 3. Repo profile and analyzers

### `.pragent/profile.yml` (written by `pragent init`, committed)

```yaml
version: 1
generated_at: 2026-08-04T00:00:00Z
generator: pragent 0.1.0
languages:
  - { name: typescript, share: 0.72, roots: [src/, packages/] }
  - { name: go,         share: 0.21, roots: [services/ingest/] }
build:
  install: pnpm install --frozen-lockfile
  test:    pnpm test
  lint:    pnpm lint
  typecheck: pnpm tsc --noEmit
modules:
  - { path: packages/auth,    role: security-critical, owners: ["@platform"] }
  - { path: packages/billing, role: money-path,        owners: ["@payments"] }
conventions:
  error_handling: "Result<T,E> in TS; wrapped errors in Go"
  test_layout: "co-located *.test.ts; Go table tests"
  docs: { adr: docs/adr/, api: docs/api/, runbooks: docs/runbooks/ }
risk_paths: ["**/auth/**", "**/migrations/**", "infra/**", "**/*.tf"]
existing_tooling: { linters: [eslint, golangci-lint], sast: [semgrep], ci: woodpecker }
```

Refresh is a PR (`pragent init --refresh`), so profile drift is reviewable. A staleness
warning fires when the profile predates N commits or a new language appears.

### Analyzer contract

```yaml
# .pragent/analyzers/security.yml
id: security
tiers: [lite, full, oversized]
languages: ["*"]
paths: ["**"]
model: { tier_full: claude-opus-5, tier_lite: claude-sonnet-5, effort: high }
tools: [read_file, grep, list_deps]
tool_budget: 12
prompt: ./prompts/security.md
output_schema: finding[]   # {file, line, severity, confidence, category, claim, fix}
```

Ships with: `security`, `code-quality`, `docs-consistency` (does the change contradict
the docs it touches, and do public API changes update docs?), `performance`, `tests`
(coverage of the changed behaviour, not line coverage). Teams add their own the same way
— there is no privileged built-in path.

**Aggregator:** dedupes across analyzers by `(file, line-window, category)`, drops
findings below the repo's confidence gate, ranks by severity × confidence, caps comment
count, and routes the remainder into the run record so nothing is silently lost.

## 4. Analytics and traceability

Every run emits one JSONL record, plus one per finding:

```json
{"run_id":"...","repo":"...","pr":412,"commit":"...","tier":"full",
 "tier_reason":"rule:risk_path(**/auth/**)","analyzers":["security","tests"],
 "tokens":{"in":54200,"cached":41000,"out":11800},"cost_usd":0.83,
 "latency_ms":48200,"findings":{"total":7,"posted":4,"suppressed":3},
 "verdict":"changes_requested","models":{"security":"claude-opus-5"},
 "pragent_version":"0.1.0","profile_hash":"sha256:..."}
```

Traceability: `tier_reason` names the exact rule or triage call that chose the tier;
`pragent explain <run_id>` reconstructs the decision path; `pragent replay <run_id>`
re-runs the same inputs against a new prompt or model — the eval loop.

Same event schema feeds OTel spans (one per analyzer) for teams with a collector. A
dashboard is a later, optional consumer of the same schema — it is not in scope now.

**Feedback signal:** a resolved/👎 reaction on a posted comment writes back a
`finding_outcome` record. That is what makes false-positive rate measurable per analyzer,
which is what makes the confidence gates tunable rather than guessed.

## 5. Rollout phases

**Phase 0 — Evaluate `ai-code-review` (do this first).** Red Hat's MIT-licensed
`ai-code-review` already implements Phases 1–2 of this design: four forge clients
(GitLab, GitHub, Forgejo, local git), six AI providers, CI integration, a committed
repo-context file. Run it on real repos in the Gitea setup for a week before writing
pragent code. *Done when:* we can name, from use rather than speculation, which of our
differentiators (tiering, analyzer bus, analytics) are worth building, and whether to
build them standalone, on top of that CLI, or as upstream contributions. See
`docs/research/2026-08-04-prior-art-ai-code-review.md` for the full assessment.

Phases 1–6 below stand as written **if** Phase 0 concludes we build standalone. If it
concludes we extend an existing base, Phases 1–2 mostly disappear and Phases 3–6 become
the whole project.

1. **Walking skeleton** — `local` forge adapter, one analyzer, rules-only tiering, JSONL
   emitter. Runs on a local diff, prints findings. No network beyond the model API.
2. **Gitea end-to-end** — Gitea adapter, Woodpecker step, PR comments, status checks.
3. **Profile + full tier** — `pragent init`, shared-prefix caching, analyzer fan-out,
   aggregator.
4. **Extensibility hardening** — plugin loading from `.pragent/analyzers/` and npm,
   config layering with locked org keys, `explain` / `replay`.
5. **Second forge** — GitLab adapter, Jenkins runner recipe. Proves the abstraction.
6. **Analytics maturity** — OTel export, feedback loop, per-analyzer eval harness.

## 6. Revisions from the prior-art review

Seven requirements the original design missed. They apply on every path — standalone,
fork, or contribution — so they are part of the design now rather than a backlog.

### 6.1 Prior-comment context and synthesis (highest priority)

Before the main review, fetch **all** existing comments and reviews on the PR, including
resolved ones, and compress them with a **cheap model** (Haiku-class) into a short summary
of what has already been said, fixed, or explicitly rejected. Feed that summary to the
analyzers.

This closes a hole the original design created. The cost model assumes ~1.7 review runs
per PR — every push re-reviews. Without prior-comment context, the second run repeats the
first run's findings and re-argues points a human already dismissed. That is the fastest
way for an AI reviewer to get muted, and it was designed in by accident.

Cost impact is favourable: one Haiku call (~$0.01) to compress a thread, against re-posting
findings that get ignored. Config: `reviewContext.enabled`, `reviewContext.synthesisModel`,
`reviewContext.maxTokens`.

### 6.2 Team context document

`teamContextFile` — a local path **or a URL** — carrying organization-wide review guidance:
security requirements, house conventions, compliance language. It outranks the repo
profile, so one document steers every repo without copying. This is the missing half of
"roll it out across many projects"; org config layering handles thresholds, this handles
judgment.

### 6.3 Deterministic skip conditions

The tier engine gains free, pre-model skips beyond paths and sizes: **draft/WIP MRs**,
`WIP` in the commit message, `wip/` branch prefixes, bot authors, tagged MRs. Each records
its `tier_reason` like any other rule. Reviewing a draft PR at full tier is a pure waste
the original rules would not have caught.

### 6.4 Self-hosted forge configuration

`forgeUrl` per adapter, `sslVerify`, `sslCertPath`. A self-hosted GitLab behind a corporate
CA is the normal case for the company deployment this targets, and the original design had
no way to express it.

### 6.5 Provider matrix as a requirement

The `ModelClient` port must be exercised by more than one provider before we claim it is a
port. Required: Anthropic, plus **Ollama or another local runtime** — "the diff never
leaves our network" is a procurement requirement for regulated repos, not a preference.
Vertex/Bedrock variants follow the same shape.

### 6.6 Per-provider input clamp

Alongside the file and line caps, clamp total diff characters with a provider-aware
default (roughly: 150k Anthropic, 200k Gemini, 100k OpenAI, 50k Ollama). The `oversized`
tier decides *strategy*; the clamp is the backstop that keeps a pathological diff from
blowing the context window regardless of tier.

### 6.7 Smaller additions

- **PR summary** alongside findings — a short description of what the change does
- **`--dry-run`** with mock responses, so a team can wire the pipeline before buying keys
- **Gitea adapter targets Forgejo too** — Forgejo is a Gitea fork with a compatible API;
  this is close to free and doubles the addressable forges
- **Library-docs enrichment** (à la Context7) as a profile-enricher plugin, not core

## 7. What remains genuinely ours

After the revisions above, the differentiation is narrower and clearer than the original
design implied. Existing tools — Red Hat's CLI, CodeRabbit, Greptile, Qodo — do adapters,
providers, prompts, and skip logic. None of them do:

1. **Graded attention with recorded provenance.** Four tiers, each decision naming the rule
   that fired. Everyone else has binary skip-or-review.
2. **An analyzer plugin bus.** Per-analyzer model, effort, and tool budget, fanned out over
   a shared cached prefix — so a team adds a review dimension without forking anything.
3. **Measurement.** Run records, `explain`, `replay`, and finding-outcome feedback, which
   together make false-positive rate per analyzer a number you can query and a prompt change
   something you can A/B. No tool in this category can answer "did last week's prompt change
   help?"
4. **Org-locked policy.** Config a repo cannot downgrade.

If Phase 0 shows an existing base covers everything else well, these four are the project —
and they may be worth contributing upstream rather than shipping standalone.

## Non-goals (for now)

- Central webhook service (CLI-only until a team actually needs zero-setup onboarding)
- Vector/graph codebase index (profile first; index only as an opt-in plugin)
- Auto-fix commits (findings and suggestions only — writing to branches is a later,
  separately-gated decision)
- Web dashboard (event schema first; dashboard is a downstream consumer)

## Bootstrap: the pilot (2026-08-17)

Before the framework build, a minimal **pilot** was shipped to get a working AI
review bot live immediately (user decision 2026-08-17). It is deliberately not
this design:

- Single Python script (`pilot/ai_review.py`, stdlib only) fetched at runtime by
  a per-repo Gitea Action (`pilot/workflow-template.yml`).
- One model, `glm-5.2:cloud` via the on-network headroom proxy — no tiering, no
  analyzer fan-out, no shared-prefix caching, no provider matrix.
- Trigger = `AI-REVIEW` label on a PR (deterministic skip, the cheapest tier
  rule, but expressed as a Gitea Actions `if` gate rather than the tier engine).
- Fail-open, comment-only review; no inline line comments, no status checks,
  no analytics JSONL, no `explain`/`replay`.
- Re-posts on every push (no prior-comment synthesis — §6.1 — yet).

The pilot validates the delivery model (CI-step + bot user, not a central
webhook) and the on-network provider path. When the framework build resumes,
`pilot/ai_review.py` is replaced by `pragent review`; the per-repo workflow
stays, just calling the CLI instead of curling the script. See `pilot/README.md`.

### Central webhook service (2026-08-17) — partially reverses the non-goal

User decision 2026-08-17: ship a central webhook service so "add the bot +
label a PR" is the only per-repo action (no workflow file, no secret, no runner).
This walks back the "central webhook service = non-goal" line above, for the
pilot only — the framework's CLI-step delivery model is unchanged.

Implementation notes (see `pilot/README-webhook.md`):

- **Gitea 1.26.1 system webhooks are broken**: `POST /admin/hooks` returns `201`
  but the hook never persists (`GET /admin/hooks` lists 0, no delivery). The
  one-webhook-per-instance ideal is not achievable on this version. Fallback:
  **user-level webhooks** — one webhook per repo-owner, fires for every repo
  that user owns. Few owners on this instance, so near-equivalent.
- **Gitea SSRF allow-list** blocks webhook delivery to in-cluster hosts by
  default; required a scoped `[webhook] ALLOWED_HOST_LIST` addition to
  `app.ini` (via the helm inline-config secret) + a gitea pod restart.
- The webhook receiver (`pilot/webhook_server.py`, stdlib only) HMAC-verifies
  the delivery, gates on `AI-REVIEW` label + PR action, and calls the same
  `ai_review.review_pr()` core the CI-step uses — one review path, two triggers.
- The bot stays a **normal user** (not site admin); it must be a Write
  collaborator on each reviewed repo.