docs: initial pragent design and README

Extensible PR review framework: CLI-in-CI delivery, pi SDK agent loop, deterministic-plus-LLM tiering, committed repo profile, JSONL/OTel analytics. Design doc records the decisions, cost model, and rollout phases. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011Ye1KNFMkkUtmzTypHXkoK
2026-08-04 16:35:35 +00:00
commit 1c0c576e40
3 changed files with 292 additions and 0 deletions
@@ -0,0 +1,6 @@
+node_modules/
+dist/
+*.token
+.pragent/cache/
+*.jsonl
+.env
@@ -0,0 +1,85 @@
+# pragent
+
+An extensible, forge-agnostic PR review framework. Not a product — a toolkit that teams
+extend with their own review dimensions.
+
+**Status:** design approved, implementation not started. See
+[`docs/plans/2026-08-04-pragent-design.md`](docs/plans/2026-08-04-pragent-design.md).
+
+## What it is
+
+`pragent` runs as a CI step. It reads a pull request, decides how much attention the
+change deserves, runs the analyzers that apply, and posts ranked findings back to the
+forge.
+
+```
+pragent init      # one-time repo scan → .pragent/profile.yml (committed, reviewable)
+pragent review    # the CI step: tier → analyze → aggregate → publish
+pragent explain   # why did this PR get this tier / these findings?
+pragent replay    # re-run a past PR against a new prompt or model (the eval loop)
+pragent doctor    # config, credentials, and adapter health
+```
+
+## Why not CodeRabbit / Greptile / Qodo
+
+Those are good products with fixed review dimensions and per-seat pricing. `pragent`
+targets the case where a platform team needs to **add its own dimensions** — an internal
+compliance rule, a service-catalog ownership check, a house performance idiom — without
+forking a vendor's reviewer. Cost lands in the same range (~$25/dev/month at 350 PRs/mo
+for 20 devs), but the analyzers, the data, and the analytics are yours.
+
+## Attention tiers
+
+Every PR is classified before any expensive work happens. Deterministic rules decide
+first; an ambiguous case gets one cheap model call as tie-breaker.
+
+| Tier | What it means | Cost/PR |
+|---|---|---|
+| `trivial` | lockfile bumps, generated code, docs typos | ~$0.005 |
+| `lite` | small change, no risk paths | ~$0.08 |
+| `full` | the default for real changes | ~$0.80–2.00 |
+| `oversized` | too big to review whole; structural summary + deep pass on the hot subset | ~$5 ceiling |
+
+Every tier decision records *why*, so a surprising outcome is explainable rather than
+mysterious.
+
+## Extension points
+
+Five, all documented in the design doc. Teams override or add; nobody forks.
+
+1. **Analyzers** — drop a YAML + prompt in `.pragent/analyzers/`, or install from npm
+2. **Forge adapters** — Gitea, GitLab, GitHub, local diff
+3. **Tier policy** — thresholds and the path risk map, per repo or per org
+4. **Profile enrichers** — extend what `pragent init` learns about a repo
+5. **Emitter sinks** — JSONL by default, OpenTelemetry, or your own
+
+Org config can lock keys, so a repo cannot quietly disable the security analyzer.
+
+## Design principles
+
+- **Polyglot by construction.** Language knowledge lives in the repo profile, not in the
+  reviewer. A new language is a profile change, not a core change.
+- **One shared prompt prefix.** All analyzers for a PR share a byte-identical cached
+  prefix. This is what makes fan-out affordable; it is enforced, not hoped for.
+- **Everything is traceable.** Tier reasons, token counts, cost, latency, and finding
+  outcomes are recorded per run. False-positive rate is measurable per analyzer.
+- **Fail open.** A budget ceiling or an analyzer crash yields a partial review with a
+  clear note, never a blocked pipeline with no explanation.
+
+## Stack
+
+TypeScript + Node, built on the [`pi`](https://github.com/badlogic/pi-mono) agent SDK.
+Shipped as an npm package and an OCI image, so CI runners need no local Node install.
+
+## Roadmap
+
+1. Walking skeleton — local diff, one analyzer, rules-only tiering
+2. Gitea end to end — adapter, Woodpecker step, PR comments, status checks
+3. Profile + full tier — `pragent init`, shared-prefix caching, analyzer fan-out
+4. Extensibility hardening — plugin loading, config layering, `explain` / `replay`
+5. Second forge — GitLab adapter, Jenkins recipe
+6. Analytics maturity — OTel export, feedback loop, eval harness
+
+## License
+
+TBD.
@@ -0,0 +1,201 @@
+# pragent — Design
+
+**Date:** 2026-08-04
+**Status:** Approved (brainstorming complete, ready for implementation planning)
+
+## Problem
+
+Existing AI PR reviewers (CodeRabbit, Greptile, Qodo) are closed products with fixed
+review dimensions and per-seat pricing. We need a **framework**, not a product:
+enterprise-grade, polyglot, forge-agnostic, and extensible by product teams who want
+to add their own review dimensions without forking the tool.
+
+## Decisions
+
+| Question | Decision | Why |
+|---|---|---|
+| Forge | Gitea first, GitLab (GitLab CI or Jenkins) next | Matches current infra; forge adapter interface keeps the second one cheap |
+| Delivery | CLI binary run as a CI step | Stateless, no HA service to run, secrets stay in the runner, portable across forges |
+| Agent harness | `pi` SDK (`earendil-works/pi`) | Agent loop as a library; we own orchestration, tiering, analytics |
+| Runtime | TypeScript + Node; ship as npm package **and** OCI image | `pi` is TS; the image means project runners need no Node |
+| Tiering | Deterministic rules first, cheap LLM triage as tie-breaker | Repeatable and auditable; adapts on semantic risk |
+| Onboarding | `pragent init` writes a committed `.pragent/profile.yml` | Reviewable in a PR, no external index to run or keep fresh |
+| Analytics | Structured JSONL per run + optional OpenTelemetry export | Vendor-neutral; no service required to start |
+
+## 1. Architecture
+
+```
+┌─ pragent CLI (npm pkg + OCI image) ─────────────────────┐
+│  commands: init | review | explain | replay | doctor    │
+└──────────────────────┬──────────────────────────────────┘
+                       │
+   ┌───────────────────┼────────────────────┬─────────────┐
+   │                   │                    │             │
+Forge Adapter     Triage Engine        Analyzer Bus   Emitter
+(gitea|gitlab|    (rules → cheap        (plugins)     (JSONL+OTel)
+ github|jenkins|   LLM tiebreak)             │
+ local-diff)            │                    │
+   │                    ▼                    ▼
+   │              tier: trivial       security / quality /
+   │                    lite          docs-consistency /
+   │                    full          performance / custom
+   │                    oversized            │
+   │                                         ▼
+   │                              pi SDK AgentSession per analyzer
+   │                              (model pinned per analyzer + tier)
+   ▼                                         │
+Findings ◄───── Aggregator (dedupe, rank, confidence gate) ◄┘
+   │
+   ▼
+Publisher → inline comments / summary / exit code
+```
+
+Analyzers only ever see **profile + diff + tier budget**. A new language means no core
+change — language support lives in the profile scan and per-language tool detection,
+not in the reviewer.
+
+### Five extension points
+
+1. **Analyzer plugin** — declares `id`, applicable `tiers`, `languages`/globs, prompt,
+   tools, output schema. Dropped in `.pragent/analyzers/` or installed from npm.
+2. **Forge adapter** — `getDiff / getContext / publish / setStatus`. Gitea day 1;
+   GitLab + Jenkins next; `local` adapter for the dev loop.
+3. **Tier policy** — rule file per repo/org; overrides thresholds and the path risk map.
+4. **Profile enrichers** — extend `pragent init` (e.g. service-catalog ownership lookup).
+5. **Emitter sinks** — JSONL default; add OTel/Datadog/whatever.
+
+### Config layering
+
+`org defaults → repo .pragent/config.yml → PR labels/commands → CLI flags`
+
+Org policy can mark keys **locked**, so a repo cannot downgrade e.g. the security analyzer.
+
+## 2. Tiering and cost model
+
+Priced at Anthropic rates: Opus 5 $5/$25 per MTok, Sonnet 5 $3/$15, Haiku 4.5 $1/$5;
+cache read 0.1×, cache write 1.25×, Batch API −50%.
+
+**Key lever:** every analyzer in a PR shares one prompt prefix (profile + diff + pulled
+context). The first analyzer pays the cache write; the rest pay 0.1×. This is what makes
+fan-out affordable, and it is an architectural constraint, not an optimization — the
+prefix must be byte-identical across analyzer calls, verified in CI.
+
+| Tier | Trigger | Engine | Cost/PR |
+|---|---|---|---|
+| trivial | lockfile-only, generated, docs typo, <20 LOC off risk paths | rules only (or 1 Haiku triage) | ~$0.005 |
+| lite | <150 LOC, no risk paths | 1 Sonnet pass | ~$0.08 |
+| full | default; risk paths or >150 LOC | 4–5 Opus analyzers, shared cached prefix | ~$0.80 cold, ~$2.00 with agentic file reads |
+| oversized | >2k LOC or >50 files | structural summary + deep review on hot subset, hard `task_budget` | ~$5 ceiling |
+
+Full-tier breakdown: 53k prefix × 1.25 write = $0.33 → 3 further analyzers × 53k × 0.1 =
+$0.08 → unique prompts $0.04 → 12k output @ $25 = $0.30 → aggregator $0.07 ≈ **$0.82**.
+
+**Team of 20 devs, ~350 PRs/month, 1.7 review runs per PR:** ~$510/month ≈ **$25/dev**,
+plus a one-off $5–20 per repo for `pragent init`. Comparable to CodeRabbit ($30/dev) and
+Qodo ($19/seat), with ownership of the analyzers, the data, and the analytics.
+
+**Cost knobs in config:** tier→model map; `effort` per analyzer; 1h cache TTL on busy
+repos; Batch API (−50%) for nightly re-scans, analytics backfills, and eval runs (never
+for the blocking review path); hard per-PR spend ceiling plus `task_budget`, failing open
+with a "budget exceeded, partial review" comment.
+
+**Main variance driver:** agentic file reading ($0.80 → $2.00). Tool-call budget per
+analyzer is a first-class config key.
+
+## 3. Repo profile and analyzers
+
+### `.pragent/profile.yml` (written by `pragent init`, committed)
+
+```yaml
+version: 1
+generated_at: 2026-08-04T00:00:00Z
+generator: pragent 0.1.0
+languages:
+  - { name: typescript, share: 0.72, roots: [src/, packages/] }
+  - { name: go,         share: 0.21, roots: [services/ingest/] }
+build:
+  install: pnpm install --frozen-lockfile
+  test:    pnpm test
+  lint:    pnpm lint
+  typecheck: pnpm tsc --noEmit
+modules:
+  - { path: packages/auth,    role: security-critical, owners: ["@platform"] }
+  - { path: packages/billing, role: money-path,        owners: ["@payments"] }
+conventions:
+  error_handling: "Result<T,E> in TS; wrapped errors in Go"
+  test_layout: "co-located *.test.ts; Go table tests"
+  docs: { adr: docs/adr/, api: docs/api/, runbooks: docs/runbooks/ }
+risk_paths: ["**/auth/**", "**/migrations/**", "infra/**", "**/*.tf"]
+existing_tooling: { linters: [eslint, golangci-lint], sast: [semgrep], ci: woodpecker }
+```
+
+Refresh is a PR (`pragent init --refresh`), so profile drift is reviewable. A staleness
+warning fires when the profile predates N commits or a new language appears.
+
+### Analyzer contract
+
+```yaml
+# .pragent/analyzers/security.yml
+id: security
+tiers: [lite, full, oversized]
+languages: ["*"]
+paths: ["**"]
+model: { tier_full: claude-opus-5, tier_lite: claude-sonnet-5, effort: high }
+tools: [read_file, grep, list_deps]
+tool_budget: 12
+prompt: ./prompts/security.md
+output_schema: finding[]   # {file, line, severity, confidence, category, claim, fix}
+```
+
+Ships with: `security`, `code-quality`, `docs-consistency` (does the change contradict
+the docs it touches, and do public API changes update docs?), `performance`, `tests`
+(coverage of the changed behaviour, not line coverage). Teams add their own the same way
+— there is no privileged built-in path.
+
+**Aggregator:** dedupes across analyzers by `(file, line-window, category)`, drops
+findings below the repo's confidence gate, ranks by severity × confidence, caps comment
+count, and routes the remainder into the run record so nothing is silently lost.
+
+## 4. Analytics and traceability
+
+Every run emits one JSONL record, plus one per finding:
+
+```json
+{"run_id":"...","repo":"...","pr":412,"commit":"...","tier":"full",
+ "tier_reason":"rule:risk_path(**/auth/**)","analyzers":["security","tests"],
+ "tokens":{"in":54200,"cached":41000,"out":11800},"cost_usd":0.83,
+ "latency_ms":48200,"findings":{"total":7,"posted":4,"suppressed":3},
+ "verdict":"changes_requested","models":{"security":"claude-opus-5"},
+ "pragent_version":"0.1.0","profile_hash":"sha256:..."}
+```
+
+Traceability: `tier_reason` names the exact rule or triage call that chose the tier;
+`pragent explain <run_id>` reconstructs the decision path; `pragent replay <run_id>`
+re-runs the same inputs against a new prompt or model — the eval loop.
+
+Same event schema feeds OTel spans (one per analyzer) for teams with a collector. A
+dashboard is a later, optional consumer of the same schema — it is not in scope now.
+
+**Feedback signal:** a resolved/👎 reaction on a posted comment writes back a
+`finding_outcome` record. That is what makes false-positive rate measurable per analyzer,
+which is what makes the confidence gates tunable rather than guessed.
+
+## 5. Rollout phases
+
+1. **Walking skeleton** — `local` forge adapter, one analyzer, rules-only tiering, JSONL
+   emitter. Runs on a local diff, prints findings. No network beyond the model API.
+2. **Gitea end-to-end** — Gitea adapter, Woodpecker step, PR comments, status checks.
+3. **Profile + full tier** — `pragent init`, shared-prefix caching, analyzer fan-out,
+   aggregator.
+4. **Extensibility hardening** — plugin loading from `.pragent/analyzers/` and npm,
+   config layering with locked org keys, `explain` / `replay`.
+5. **Second forge** — GitLab adapter, Jenkins runner recipe. Proves the abstraction.
+6. **Analytics maturity** — OTel export, feedback loop, per-analyzer eval harness.
+
+## Non-goals (for now)
+
+- Central webhook service (CLI-only until a team actually needs zero-setup onboarding)
+- Vector/graph codebase index (profile first; index only as an opt-in plugin)
+- Auto-fix commits (findings and suggestions only — writing to branches is a later,
+  separately-gated decision)
+- Web dashboard (event schema first; dashboard is a downstream consumer)