docs: initial pragent design and README
Extensible PR review framework: CLI-in-CI delivery, pi SDK agent loop, deterministic-plus-LLM tiering, committed repo profile, JSONL/OTel analytics. Design doc records the decisions, cost model, and rollout phases. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011Ye1KNFMkkUtmzTypHXkoK
This commit is contained in:
Marcos
@@ -0,0 +1,201 @@
|
||||
# pragent — Design
|
||||
|
||||
**Date:** 2026-08-04
|
||||
**Status:** Approved (brainstorming complete, ready for implementation planning)
|
||||
|
||||
## Problem
|
||||
|
||||
Existing AI PR reviewers (CodeRabbit, Greptile, Qodo) are closed products with fixed
|
||||
review dimensions and per-seat pricing. We need a **framework**, not a product:
|
||||
enterprise-grade, polyglot, forge-agnostic, and extensible by product teams who want
|
||||
to add their own review dimensions without forking the tool.
|
||||
|
||||
## Decisions
|
||||
|
||||
| Question | Decision | Why |
|
||||
|---|---|---|
|
||||
| Forge | Gitea first, GitLab (GitLab CI or Jenkins) next | Matches current infra; forge adapter interface keeps the second one cheap |
|
||||
| Delivery | CLI binary run as a CI step | Stateless, no HA service to run, secrets stay in the runner, portable across forges |
|
||||
| Agent harness | `pi` SDK (`earendil-works/pi`) | Agent loop as a library; we own orchestration, tiering, analytics |
|
||||
| Runtime | TypeScript + Node; ship as npm package **and** OCI image | `pi` is TS; the image means project runners need no Node |
|
||||
| Tiering | Deterministic rules first, cheap LLM triage as tie-breaker | Repeatable and auditable; adapts on semantic risk |
|
||||
| Onboarding | `pragent init` writes a committed `.pragent/profile.yml` | Reviewable in a PR, no external index to run or keep fresh |
|
||||
| Analytics | Structured JSONL per run + optional OpenTelemetry export | Vendor-neutral; no service required to start |
|
||||
|
||||
## 1. Architecture
|
||||
|
||||
```
|
||||
┌─ pragent CLI (npm pkg + OCI image) ─────────────────────┐
|
||||
│ commands: init | review | explain | replay | doctor │
|
||||
└──────────────────────┬──────────────────────────────────┘
|
||||
│
|
||||
┌───────────────────┼────────────────────┬─────────────┐
|
||||
│ │ │ │
|
||||
Forge Adapter Triage Engine Analyzer Bus Emitter
|
||||
(gitea|gitlab| (rules → cheap (plugins) (JSONL+OTel)
|
||||
github|jenkins| LLM tiebreak) │
|
||||
local-diff) │ │
|
||||
│ ▼ ▼
|
||||
│ tier: trivial security / quality /
|
||||
│ lite docs-consistency /
|
||||
│ full performance / custom
|
||||
│ oversized │
|
||||
│ ▼
|
||||
│ pi SDK AgentSession per analyzer
|
||||
│ (model pinned per analyzer + tier)
|
||||
▼ │
|
||||
Findings ◄───── Aggregator (dedupe, rank, confidence gate) ◄┘
|
||||
│
|
||||
▼
|
||||
Publisher → inline comments / summary / exit code
|
||||
```
|
||||
|
||||
Analyzers only ever see **profile + diff + tier budget**. A new language means no core
|
||||
change — language support lives in the profile scan and per-language tool detection,
|
||||
not in the reviewer.
|
||||
|
||||
### Five extension points
|
||||
|
||||
1. **Analyzer plugin** — declares `id`, applicable `tiers`, `languages`/globs, prompt,
|
||||
tools, output schema. Dropped in `.pragent/analyzers/` or installed from npm.
|
||||
2. **Forge adapter** — `getDiff / getContext / publish / setStatus`. Gitea day 1;
|
||||
GitLab + Jenkins next; `local` adapter for the dev loop.
|
||||
3. **Tier policy** — rule file per repo/org; overrides thresholds and the path risk map.
|
||||
4. **Profile enrichers** — extend `pragent init` (e.g. service-catalog ownership lookup).
|
||||
5. **Emitter sinks** — JSONL default; add OTel/Datadog/whatever.
|
||||
|
||||
### Config layering
|
||||
|
||||
`org defaults → repo .pragent/config.yml → PR labels/commands → CLI flags`
|
||||
|
||||
Org policy can mark keys **locked**, so a repo cannot downgrade e.g. the security analyzer.
|
||||
|
||||
## 2. Tiering and cost model
|
||||
|
||||
Priced at Anthropic rates: Opus 5 $5/$25 per MTok, Sonnet 5 $3/$15, Haiku 4.5 $1/$5;
|
||||
cache read 0.1×, cache write 1.25×, Batch API −50%.
|
||||
|
||||
**Key lever:** every analyzer in a PR shares one prompt prefix (profile + diff + pulled
|
||||
context). The first analyzer pays the cache write; the rest pay 0.1×. This is what makes
|
||||
fan-out affordable, and it is an architectural constraint, not an optimization — the
|
||||
prefix must be byte-identical across analyzer calls, verified in CI.
|
||||
|
||||
| Tier | Trigger | Engine | Cost/PR |
|
||||
|---|---|---|---|
|
||||
| trivial | lockfile-only, generated, docs typo, <20 LOC off risk paths | rules only (or 1 Haiku triage) | ~$0.005 |
|
||||
| lite | <150 LOC, no risk paths | 1 Sonnet pass | ~$0.08 |
|
||||
| full | default; risk paths or >150 LOC | 4–5 Opus analyzers, shared cached prefix | ~$0.80 cold, ~$2.00 with agentic file reads |
|
||||
| oversized | >2k LOC or >50 files | structural summary + deep review on hot subset, hard `task_budget` | ~$5 ceiling |
|
||||
|
||||
Full-tier breakdown: 53k prefix × 1.25 write = $0.33 → 3 further analyzers × 53k × 0.1 =
|
||||
$0.08 → unique prompts $0.04 → 12k output @ $25 = $0.30 → aggregator $0.07 ≈ **$0.82**.
|
||||
|
||||
**Team of 20 devs, ~350 PRs/month, 1.7 review runs per PR:** ~$510/month ≈ **$25/dev**,
|
||||
plus a one-off $5–20 per repo for `pragent init`. Comparable to CodeRabbit ($30/dev) and
|
||||
Qodo ($19/seat), with ownership of the analyzers, the data, and the analytics.
|
||||
|
||||
**Cost knobs in config:** tier→model map; `effort` per analyzer; 1h cache TTL on busy
|
||||
repos; Batch API (−50%) for nightly re-scans, analytics backfills, and eval runs (never
|
||||
for the blocking review path); hard per-PR spend ceiling plus `task_budget`, failing open
|
||||
with a "budget exceeded, partial review" comment.
|
||||
|
||||
**Main variance driver:** agentic file reading ($0.80 → $2.00). Tool-call budget per
|
||||
analyzer is a first-class config key.
|
||||
|
||||
## 3. Repo profile and analyzers
|
||||
|
||||
### `.pragent/profile.yml` (written by `pragent init`, committed)
|
||||
|
||||
```yaml
|
||||
version: 1
|
||||
generated_at: 2026-08-04T00:00:00Z
|
||||
generator: pragent 0.1.0
|
||||
languages:
|
||||
- { name: typescript, share: 0.72, roots: [src/, packages/] }
|
||||
- { name: go, share: 0.21, roots: [services/ingest/] }
|
||||
build:
|
||||
install: pnpm install --frozen-lockfile
|
||||
test: pnpm test
|
||||
lint: pnpm lint
|
||||
typecheck: pnpm tsc --noEmit
|
||||
modules:
|
||||
- { path: packages/auth, role: security-critical, owners: ["@platform"] }
|
||||
- { path: packages/billing, role: money-path, owners: ["@payments"] }
|
||||
conventions:
|
||||
error_handling: "Result<T,E> in TS; wrapped errors in Go"
|
||||
test_layout: "co-located *.test.ts; Go table tests"
|
||||
docs: { adr: docs/adr/, api: docs/api/, runbooks: docs/runbooks/ }
|
||||
risk_paths: ["**/auth/**", "**/migrations/**", "infra/**", "**/*.tf"]
|
||||
existing_tooling: { linters: [eslint, golangci-lint], sast: [semgrep], ci: woodpecker }
|
||||
```
|
||||
|
||||
Refresh is a PR (`pragent init --refresh`), so profile drift is reviewable. A staleness
|
||||
warning fires when the profile predates N commits or a new language appears.
|
||||
|
||||
### Analyzer contract
|
||||
|
||||
```yaml
|
||||
# .pragent/analyzers/security.yml
|
||||
id: security
|
||||
tiers: [lite, full, oversized]
|
||||
languages: ["*"]
|
||||
paths: ["**"]
|
||||
model: { tier_full: claude-opus-5, tier_lite: claude-sonnet-5, effort: high }
|
||||
tools: [read_file, grep, list_deps]
|
||||
tool_budget: 12
|
||||
prompt: ./prompts/security.md
|
||||
output_schema: finding[] # {file, line, severity, confidence, category, claim, fix}
|
||||
```
|
||||
|
||||
Ships with: `security`, `code-quality`, `docs-consistency` (does the change contradict
|
||||
the docs it touches, and do public API changes update docs?), `performance`, `tests`
|
||||
(coverage of the changed behaviour, not line coverage). Teams add their own the same way
|
||||
— there is no privileged built-in path.
|
||||
|
||||
**Aggregator:** dedupes across analyzers by `(file, line-window, category)`, drops
|
||||
findings below the repo's confidence gate, ranks by severity × confidence, caps comment
|
||||
count, and routes the remainder into the run record so nothing is silently lost.
|
||||
|
||||
## 4. Analytics and traceability
|
||||
|
||||
Every run emits one JSONL record, plus one per finding:
|
||||
|
||||
```json
|
||||
{"run_id":"...","repo":"...","pr":412,"commit":"...","tier":"full",
|
||||
"tier_reason":"rule:risk_path(**/auth/**)","analyzers":["security","tests"],
|
||||
"tokens":{"in":54200,"cached":41000,"out":11800},"cost_usd":0.83,
|
||||
"latency_ms":48200,"findings":{"total":7,"posted":4,"suppressed":3},
|
||||
"verdict":"changes_requested","models":{"security":"claude-opus-5"},
|
||||
"pragent_version":"0.1.0","profile_hash":"sha256:..."}
|
||||
```
|
||||
|
||||
Traceability: `tier_reason` names the exact rule or triage call that chose the tier;
|
||||
`pragent explain <run_id>` reconstructs the decision path; `pragent replay <run_id>`
|
||||
re-runs the same inputs against a new prompt or model — the eval loop.
|
||||
|
||||
Same event schema feeds OTel spans (one per analyzer) for teams with a collector. A
|
||||
dashboard is a later, optional consumer of the same schema — it is not in scope now.
|
||||
|
||||
**Feedback signal:** a resolved/👎 reaction on a posted comment writes back a
|
||||
`finding_outcome` record. That is what makes false-positive rate measurable per analyzer,
|
||||
which is what makes the confidence gates tunable rather than guessed.
|
||||
|
||||
## 5. Rollout phases
|
||||
|
||||
1. **Walking skeleton** — `local` forge adapter, one analyzer, rules-only tiering, JSONL
|
||||
emitter. Runs on a local diff, prints findings. No network beyond the model API.
|
||||
2. **Gitea end-to-end** — Gitea adapter, Woodpecker step, PR comments, status checks.
|
||||
3. **Profile + full tier** — `pragent init`, shared-prefix caching, analyzer fan-out,
|
||||
aggregator.
|
||||
4. **Extensibility hardening** — plugin loading from `.pragent/analyzers/` and npm,
|
||||
config layering with locked org keys, `explain` / `replay`.
|
||||
5. **Second forge** — GitLab adapter, Jenkins runner recipe. Proves the abstraction.
|
||||
6. **Analytics maturity** — OTel export, feedback loop, per-analyzer eval harness.
|
||||
|
||||
## Non-goals (for now)
|
||||
|
||||
- Central webhook service (CLI-only until a team actually needs zero-setup onboarding)
|
||||
- Vector/graph codebase index (profile first; index only as an opt-in plugin)
|
||||
- Auto-fix commits (findings and suggestions only — writing to branches is a later,
|
||||
separately-gated decision)
|
||||
- Web dashboard (event schema first; dashboard is a downstream consumer)
|
||||
Reference in New Issue
Block a user