pragent/docs/plans/2026-08-04-pragent-design.md

# pragent — Design

**Date:** 2026-08-04
**Status:** Approved (brainstorming complete, ready for implementation planning)

## Problem

Existing AI PR reviewers (CodeRabbit, Greptile, Qodo) are closed products with fixed
review dimensions and per-seat pricing. We need a **framework**, not a product:
enterprise-grade, polyglot, forge-agnostic, and extensible by product teams who want
to add their own review dimensions without forking the tool.

## Decisions

| Question | Decision | Why |
|---|---|---|
| Forge | Gitea first, GitLab (GitLab CI or Jenkins) next | Matches current infra; forge adapter interface keeps the second one cheap |
| Delivery | CLI binary run as a CI step | Stateless, no HA service to run, secrets stay in the runner, portable across forges |
| Agent harness | `pi` SDK (`earendil-works/pi`) | Agent loop as a library; we own orchestration, tiering, analytics |
| Runtime | TypeScript + Node; ship as npm package **and** OCI image | `pi` is TS; the image means project runners need no Node |
| Tiering | Deterministic rules first, cheap LLM triage as tie-breaker | Repeatable and auditable; adapts on semantic risk |
| Onboarding | `pragent init` writes a committed `.pragent/profile.yml` | Reviewable in a PR, no external index to run or keep fresh |
| Analytics | Structured JSONL per run + optional OpenTelemetry export | Vendor-neutral; no service required to start |

## 1. Architecture

```
┌─ pragent CLI (npm pkg + OCI image) ─────────────────────┐
│  commands: init | review | explain | replay | doctor    │
└──────────────────────┬──────────────────────────────────┘
                       │
   ┌───────────────────┼────────────────────┬─────────────┐
   │                   │                    │             │
Forge Adapter     Triage Engine        Analyzer Bus   Emitter
(gitea|gitlab|    (rules → cheap        (plugins)     (JSONL+OTel)
 github|jenkins|   LLM tiebreak)             │
 local-diff)            │                    │
   │                    ▼                    ▼
   │              tier: trivial       security / quality /
   │                    lite          docs-consistency /
   │                    full          performance / custom
   │                    oversized            │
   │                                         ▼
   │                              pi SDK AgentSession per analyzer
   │                              (model pinned per analyzer + tier)
   ▼                                         │
Findings ◄───── Aggregator (dedupe, rank, confidence gate) ◄┘
   │
   ▼
Publisher → inline comments / summary / exit code
```

Analyzers only ever see **profile + diff + tier budget**. A new language means no core
change — language support lives in the profile scan and per-language tool detection,
not in the reviewer.

### Five extension points

1. **Analyzer plugin** — declares `id`, applicable `tiers`, `languages`/globs, prompt,
   tools, output schema. Dropped in `.pragent/analyzers/` or installed from npm.
2. **Forge adapter** — `getDiff / getContext / publish / setStatus`. Gitea day 1;
   GitLab + Jenkins next; `local` adapter for the dev loop.
3. **Tier policy** — rule file per repo/org; overrides thresholds and the path risk map.
4. **Profile enrichers** — extend `pragent init` (e.g. service-catalog ownership lookup).
5. **Emitter sinks** — JSONL default; add OTel/Datadog/whatever.

### Config layering

`org defaults → repo .pragent/config.yml → PR labels/commands → CLI flags`

Org policy can mark keys **locked**, so a repo cannot downgrade e.g. the security analyzer.

## 2. Tiering and cost model

Priced at Anthropic rates: Opus 5 $5/$25 per MTok, Sonnet 5 $3/$15, Haiku 4.5 $1/$5;
cache read 0.1×, cache write 1.25×, Batch API −50%.

**Key lever:** every analyzer in a PR shares one prompt prefix (profile + diff + pulled
context). The first analyzer pays the cache write; the rest pay 0.1×. This is what makes
fan-out affordable, and it is an architectural constraint, not an optimization — the
prefix must be byte-identical across analyzer calls, verified in CI.

| Tier | Trigger | Engine | Cost/PR |
|---|---|---|---|
| trivial | lockfile-only, generated, docs typo, <20 LOC off risk paths | rules only (or 1 Haiku triage) | ~$0.005 |
| lite | <150 LOC, no risk paths | 1 Sonnet pass | ~$0.08 |
| full | default; risk paths or >150 LOC | 4–5 Opus analyzers, shared cached prefix | ~$0.80 cold, ~$2.00 with agentic file reads |
| oversized | >2k LOC or >50 files | structural summary + deep review on hot subset, hard `task_budget` | ~$5 ceiling |

Full-tier breakdown: 53k prefix × 1.25 write = $0.33 → 3 further analyzers × 53k × 0.1 =
$0.08 → unique prompts $0.04 → 12k output @ $25 = $0.30 → aggregator $0.07 ≈ **$0.82**.

**Team of 20 devs, ~350 PRs/month, 1.7 review runs per PR:** ~$510/month ≈ **$25/dev**,
plus a one-off $5–20 per repo for `pragent init`. Comparable to CodeRabbit ($30/dev) and
Qodo ($19/seat), with ownership of the analyzers, the data, and the analytics.

**Cost knobs in config:** tier→model map; `effort` per analyzer; 1h cache TTL on busy
repos; Batch API (−50%) for nightly re-scans, analytics backfills, and eval runs (never
for the blocking review path); hard per-PR spend ceiling plus `task_budget`, failing open
with a "budget exceeded, partial review" comment.

**Main variance driver:** agentic file reading ($0.80 → $2.00). Tool-call budget per
analyzer is a first-class config key.

## 3. Repo profile and analyzers

### `.pragent/profile.yml` (written by `pragent init`, committed)

```yaml
version: 1
generated_at: 2026-08-04T00:00:00Z
generator: pragent 0.1.0
languages:
  - { name: typescript, share: 0.72, roots: [src/, packages/] }
  - { name: go,         share: 0.21, roots: [services/ingest/] }
build:
  install: pnpm install --frozen-lockfile
  test:    pnpm test
  lint:    pnpm lint
  typecheck: pnpm tsc --noEmit
modules:
  - { path: packages/auth,    role: security-critical, owners: ["@platform"] }
  - { path: packages/billing, role: money-path,        owners: ["@payments"] }
conventions:
  error_handling: "Result<T,E> in TS; wrapped errors in Go"
  test_layout: "co-located *.test.ts; Go table tests"
  docs: { adr: docs/adr/, api: docs/api/, runbooks: docs/runbooks/ }
risk_paths: ["**/auth/**", "**/migrations/**", "infra/**", "**/*.tf"]
existing_tooling: { linters: [eslint, golangci-lint], sast: [semgrep], ci: woodpecker }
```

Refresh is a PR (`pragent init --refresh`), so profile drift is reviewable. A staleness
warning fires when the profile predates N commits or a new language appears.

### Analyzer contract

```yaml
# .pragent/analyzers/security.yml
id: security
tiers: [lite, full, oversized]
languages: ["*"]
paths: ["**"]
model: { tier_full: claude-opus-5, tier_lite: claude-sonnet-5, effort: high }
tools: [read_file, grep, list_deps]
tool_budget: 12
prompt: ./prompts/security.md
output_schema: finding[]   # {file, line, severity, confidence, category, claim, fix}
```

Ships with: `security`, `code-quality`, `docs-consistency` (does the change contradict
the docs it touches, and do public API changes update docs?), `performance`, `tests`
(coverage of the changed behaviour, not line coverage). Teams add their own the same way
— there is no privileged built-in path.

**Aggregator:** dedupes across analyzers by `(file, line-window, category)`, drops
findings below the repo's confidence gate, ranks by severity × confidence, caps comment
count, and routes the remainder into the run record so nothing is silently lost.

## 4. Analytics and traceability

Every run emits one JSONL record, plus one per finding:

```json
{"run_id":"...","repo":"...","pr":412,"commit":"...","tier":"full",
 "tier_reason":"rule:risk_path(**/auth/**)","analyzers":["security","tests"],
 "tokens":{"in":54200,"cached":41000,"out":11800},"cost_usd":0.83,
 "latency_ms":48200,"findings":{"total":7,"posted":4,"suppressed":3},
 "verdict":"changes_requested","models":{"security":"claude-opus-5"},
 "pragent_version":"0.1.0","profile_hash":"sha256:..."}
```

Traceability: `tier_reason` names the exact rule or triage call that chose the tier;
`pragent explain <run_id>` reconstructs the decision path; `pragent replay <run_id>`
re-runs the same inputs against a new prompt or model — the eval loop.

Same event schema feeds OTel spans (one per analyzer) for teams with a collector. A
dashboard is a later, optional consumer of the same schema — it is not in scope now.

**Feedback signal:** a resolved/👎 reaction on a posted comment writes back a
`finding_outcome` record. That is what makes false-positive rate measurable per analyzer,
which is what makes the confidence gates tunable rather than guessed.

## 5. Rollout phases

1. **Walking skeleton** — `local` forge adapter, one analyzer, rules-only tiering, JSONL
   emitter. Runs on a local diff, prints findings. No network beyond the model API.
2. **Gitea end-to-end** — Gitea adapter, Woodpecker step, PR comments, status checks.
3. **Profile + full tier** — `pragent init`, shared-prefix caching, analyzer fan-out,
   aggregator.
4. **Extensibility hardening** — plugin loading from `.pragent/analyzers/` and npm,
   config layering with locked org keys, `explain` / `replay`.
5. **Second forge** — GitLab adapter, Jenkins runner recipe. Proves the abstraction.
6. **Analytics maturity** — OTel export, feedback loop, per-analyzer eval harness.

## Non-goals (for now)

- Central webhook service (CLI-only until a team actually needs zero-setup onboarding)
- Vector/graph codebase index (profile first; index only as an opt-in plugin)
- Auto-fix commits (findings and suggestions only — writing to branches is a later,
  separately-gated decision)
- Web dashboard (event schema first; dashboard is a downstream consumer)