pragent/docs/plans/2026-08-04-pragent-design.md

pragent — Design

Date: 2026-08-04 Status: Approved (brainstorming complete, ready for implementation planning)

Problem

Existing AI PR reviewers (CodeRabbit, Greptile, Qodo) are closed products with fixed review dimensions and per-seat pricing. We need a framework, not a product: enterprise-grade, polyglot, forge-agnostic, and extensible by product teams who want to add their own review dimensions without forking the tool.

Decisions

Question	Decision	Why
Forge	Gitea first, GitLab (GitLab CI or Jenkins) next	Matches current infra; forge adapter interface keeps the second one cheap
Delivery	CLI binary run as a CI step	Stateless, no HA service to run, secrets stay in the runner, portable across forges
Agent harness	pi SDK (earendil-works/pi)	Agent loop as a library; we own orchestration, tiering, analytics
Runtime	TypeScript + Node; ship as npm package and OCI image	pi is TS; the image means project runners need no Node
Tiering	Deterministic rules first, cheap LLM triage as tie-breaker	Repeatable and auditable; adapts on semantic risk
Onboarding	pragent init writes a committed .pragent/profile.yml	Reviewable in a PR, no external index to run or keep fresh
Analytics	Structured JSONL per run + optional OpenTelemetry export	Vendor-neutral; no service required to start

1. Architecture

┌─ pragent CLI (npm pkg + OCI image) ─────────────────────┐
│  commands: init | review | explain | replay | doctor    │
└──────────────────────┬──────────────────────────────────┘
                       │
   ┌───────────────────┼────────────────────┬─────────────┐
   │                   │                    │             │
Forge Adapter     Triage Engine        Analyzer Bus   Emitter
(gitea|gitlab|    (rules → cheap        (plugins)     (JSONL+OTel)
 github|jenkins|   LLM tiebreak)             │
 local-diff)            │                    │
   │                    ▼                    ▼
   │              tier: trivial       security / quality /
   │                    lite          docs-consistency /
   │                    full          performance / custom
   │                    oversized            │
   │                                         ▼
   │                              pi SDK AgentSession per analyzer
   │                              (model pinned per analyzer + tier)
   ▼                                         │
Findings ◄───── Aggregator (dedupe, rank, confidence gate) ◄┘
   │
   ▼
Publisher → inline comments / summary / exit code

Analyzers only ever see profile + diff + tier budget. A new language means no core change — language support lives in the profile scan and per-language tool detection, not in the reviewer.

Five extension points

1. Analyzer plugin — declares id, applicable tiers, languages/globs, prompt, tools, output schema. Dropped in .pragent/analyzers/ or installed from npm.
2. Forge adapter — getDiff / getContext / publish / setStatus. Gitea day 1; GitLab + Jenkins next; local adapter for the dev loop.
3. Tier policy — rule file per repo/org; overrides thresholds and the path risk map.
4. Profile enrichers — extend pragent init (e.g. service-catalog ownership lookup).
5. Emitter sinks — JSONL default; add OTel/Datadog/whatever.

Config layering

org defaults → repo .pragent/config.yml → PR labels/commands → CLI flags

Org policy can mark keys locked, so a repo cannot downgrade e.g. the security analyzer.

2. Tiering and cost model

Priced at Anthropic rates: Opus 5 $5/$25 per MTok, Sonnet 5 $3/$15, Haiku 4.5 $1/$5; cache read 0.1×, cache write 1.25×, Batch API −50%.

Key lever: every analyzer in a PR shares one prompt prefix (profile + diff + pulled context). The first analyzer pays the cache write; the rest pay 0.1×. This is what makes fan-out affordable, and it is an architectural constraint, not an optimization — the prefix must be byte-identical across analyzer calls, verified in CI.

Tier	Trigger	Engine	Cost/PR
trivial	lockfile-only, generated, docs typo, <20 LOC off risk paths	rules only (or 1 Haiku triage)	~$0.005
lite	<150 LOC, no risk paths	1 Sonnet pass	~$0.08
full	default; risk paths or >150 LOC	4–5 Opus analyzers, shared cached prefix	~$0.80 cold, ~$2.00 with agentic file reads
oversized	>2k LOC or >50 files	structural summary + deep review on hot subset, hard task_budget	~$5 ceiling

Full-tier breakdown: 53k prefix × 1.25 write = $0.33 → 3 further analyzers × 53k × 0.1 = $0.08 → unique prompts $0.04 → 12k output @ $25 = $0.30 → aggregator $0.07 ≈ $0.82.

Team of 20 devs, ~350 PRs/month, 1.7 review runs per PR: ~$510/month ≈ $25/dev, plus a one-off $5–20 per repo for pragent init. Comparable to CodeRabbit ($30/dev) and Qodo ($19/seat), with ownership of the analyzers, the data, and the analytics.

Cost knobs in config: tier→model map; effort per analyzer; 1h cache TTL on busy repos; Batch API (−50%) for nightly re-scans, analytics backfills, and eval runs (never for the blocking review path); hard per-PR spend ceiling plus task_budget, failing open with a "budget exceeded, partial review" comment.

Main variance driver: agentic file reading ($0.80 → $2.00). Tool-call budget per analyzer is a first-class config key.

3. Repo profile and analyzers

.pragent/profile.yml (written by pragent init, committed)

version: 1
generated_at: 2026-08-04T00:00:00Z
generator: pragent 0.1.0
languages:
  - { name: typescript, share: 0.72, roots: [src/, packages/] }
  - { name: go,         share: 0.21, roots: [services/ingest/] }
build:
  install: pnpm install --frozen-lockfile
  test:    pnpm test
  lint:    pnpm lint
  typecheck: pnpm tsc --noEmit
modules:
  - { path: packages/auth,    role: security-critical, owners: ["@platform"] }
  - { path: packages/billing, role: money-path,        owners: ["@payments"] }
conventions:
  error_handling: "Result<T,E> in TS; wrapped errors in Go"
  test_layout: "co-located *.test.ts; Go table tests"
  docs: { adr: docs/adr/, api: docs/api/, runbooks: docs/runbooks/ }
risk_paths: ["**/auth/**", "**/migrations/**", "infra/**", "**/*.tf"]
existing_tooling: { linters: [eslint, golangci-lint], sast: [semgrep], ci: woodpecker }

Refresh is a PR (pragent init --refresh), so profile drift is reviewable. A staleness warning fires when the profile predates N commits or a new language appears.

Analyzer contract

# .pragent/analyzers/security.yml
id: security
tiers: [lite, full, oversized]
languages: ["*"]
paths: ["**"]
model: { tier_full: claude-opus-5, tier_lite: claude-sonnet-5, effort: high }
tools: [read_file, grep, list_deps]
tool_budget: 12
prompt: ./prompts/security.md
output_schema: finding[]   # {file, line, severity, confidence, category, claim, fix}

Ships with: security, code-quality, docs-consistency (does the change contradict the docs it touches, and do public API changes update docs?), performance, tests (coverage of the changed behaviour, not line coverage). Teams add their own the same way — there is no privileged built-in path.

Aggregator: dedupes across analyzers by (file, line-window, category), drops findings below the repo's confidence gate, ranks by severity × confidence, caps comment count, and routes the remainder into the run record so nothing is silently lost.

4. Analytics and traceability

Every run emits one JSONL record, plus one per finding:

{"run_id":"...","repo":"...","pr":412,"commit":"...","tier":"full",
 "tier_reason":"rule:risk_path(**/auth/**)","analyzers":["security","tests"],
 "tokens":{"in":54200,"cached":41000,"out":11800},"cost_usd":0.83,
 "latency_ms":48200,"findings":{"total":7,"posted":4,"suppressed":3},
 "verdict":"changes_requested","models":{"security":"claude-opus-5"},
 "pragent_version":"0.1.0","profile_hash":"sha256:..."}

Traceability: tier_reason names the exact rule or triage call that chose the tier; pragent explain <run_id> reconstructs the decision path; pragent replay <run_id> re-runs the same inputs against a new prompt or model — the eval loop.

Same event schema feeds OTel spans (one per analyzer) for teams with a collector. A dashboard is a later, optional consumer of the same schema — it is not in scope now.

Feedback signal: a resolved/👎 reaction on a posted comment writes back a finding_outcome record. That is what makes false-positive rate measurable per analyzer, which is what makes the confidence gates tunable rather than guessed.

5. Rollout phases

1. Walking skeleton — local forge adapter, one analyzer, rules-only tiering, JSONL emitter. Runs on a local diff, prints findings. No network beyond the model API.
2. Gitea end-to-end — Gitea adapter, Woodpecker step, PR comments, status checks.
3. Profile + full tier — pragent init, shared-prefix caching, analyzer fan-out, aggregator.
4. Extensibility hardening — plugin loading from .pragent/analyzers/ and npm, config layering with locked org keys, explain / replay.
5. Second forge — GitLab adapter, Jenkins runner recipe. Proves the abstraction.
6. Analytics maturity — OTel export, feedback loop, per-analyzer eval harness.

Non-goals (for now)

• Central webhook service (CLI-only until a team actually needs zero-setup onboarding)
• Vector/graph codebase index (profile first; index only as an opt-in plugin)
• Auto-fix commits (findings and suggestions only — writing to branches is a later, separately-gated decision)
• Web dashboard (event schema first; dashboard is a downstream consumer)