commit 1c0c576e404eeb04e3c44ce011b02dad721c3088 Author: Marcos Date: Tue Aug 4 16:35:35 2026 +0000 docs: initial pragent design and README Extensible PR review framework: CLI-in-CI delivery, pi SDK agent loop, deterministic-plus-LLM tiering, committed repo profile, JSONL/OTel analytics. Design doc records the decisions, cost model, and rollout phases. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_011Ye1KNFMkkUtmzTypHXkoK diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..5941895 --- /dev/null +++ b/.gitignore @@ -0,0 +1,6 @@ +node_modules/ +dist/ +*.token +.pragent/cache/ +*.jsonl +.env diff --git a/README.md b/README.md new file mode 100644 index 0000000..70347fe --- /dev/null +++ b/README.md @@ -0,0 +1,85 @@ +# pragent + +An extensible, forge-agnostic PR review framework. Not a product — a toolkit that teams +extend with their own review dimensions. + +**Status:** design approved, implementation not started. See +[`docs/plans/2026-08-04-pragent-design.md`](docs/plans/2026-08-04-pragent-design.md). + +## What it is + +`pragent` runs as a CI step. It reads a pull request, decides how much attention the +change deserves, runs the analyzers that apply, and posts ranked findings back to the +forge. + +``` +pragent init # one-time repo scan → .pragent/profile.yml (committed, reviewable) +pragent review # the CI step: tier → analyze → aggregate → publish +pragent explain # why did this PR get this tier / these findings? +pragent replay # re-run a past PR against a new prompt or model (the eval loop) +pragent doctor # config, credentials, and adapter health +``` + +## Why not CodeRabbit / Greptile / Qodo + +Those are good products with fixed review dimensions and per-seat pricing. `pragent` +targets the case where a platform team needs to **add its own dimensions** — an internal +compliance rule, a service-catalog ownership check, a house performance idiom — without +forking a vendor's reviewer. Cost lands in the same range (~$25/dev/month at 350 PRs/mo +for 20 devs), but the analyzers, the data, and the analytics are yours. + +## Attention tiers + +Every PR is classified before any expensive work happens. Deterministic rules decide +first; an ambiguous case gets one cheap model call as tie-breaker. + +| Tier | What it means | Cost/PR | +|---|---|---| +| `trivial` | lockfile bumps, generated code, docs typos | ~$0.005 | +| `lite` | small change, no risk paths | ~$0.08 | +| `full` | the default for real changes | ~$0.80–2.00 | +| `oversized` | too big to review whole; structural summary + deep pass on the hot subset | ~$5 ceiling | + +Every tier decision records *why*, so a surprising outcome is explainable rather than +mysterious. + +## Extension points + +Five, all documented in the design doc. Teams override or add; nobody forks. + +1. **Analyzers** — drop a YAML + prompt in `.pragent/analyzers/`, or install from npm +2. **Forge adapters** — Gitea, GitLab, GitHub, local diff +3. **Tier policy** — thresholds and the path risk map, per repo or per org +4. **Profile enrichers** — extend what `pragent init` learns about a repo +5. **Emitter sinks** — JSONL by default, OpenTelemetry, or your own + +Org config can lock keys, so a repo cannot quietly disable the security analyzer. + +## Design principles + +- **Polyglot by construction.** Language knowledge lives in the repo profile, not in the + reviewer. A new language is a profile change, not a core change. +- **One shared prompt prefix.** All analyzers for a PR share a byte-identical cached + prefix. This is what makes fan-out affordable; it is enforced, not hoped for. +- **Everything is traceable.** Tier reasons, token counts, cost, latency, and finding + outcomes are recorded per run. False-positive rate is measurable per analyzer. +- **Fail open.** A budget ceiling or an analyzer crash yields a partial review with a + clear note, never a blocked pipeline with no explanation. + +## Stack + +TypeScript + Node, built on the [`pi`](https://github.com/badlogic/pi-mono) agent SDK. +Shipped as an npm package and an OCI image, so CI runners need no local Node install. + +## Roadmap + +1. Walking skeleton — local diff, one analyzer, rules-only tiering +2. Gitea end to end — adapter, Woodpecker step, PR comments, status checks +3. Profile + full tier — `pragent init`, shared-prefix caching, analyzer fan-out +4. Extensibility hardening — plugin loading, config layering, `explain` / `replay` +5. Second forge — GitLab adapter, Jenkins recipe +6. Analytics maturity — OTel export, feedback loop, eval harness + +## License + +TBD. diff --git a/docs/plans/2026-08-04-pragent-design.md b/docs/plans/2026-08-04-pragent-design.md new file mode 100644 index 0000000..0f25d70 --- /dev/null +++ b/docs/plans/2026-08-04-pragent-design.md @@ -0,0 +1,201 @@ +# pragent — Design + +**Date:** 2026-08-04 +**Status:** Approved (brainstorming complete, ready for implementation planning) + +## Problem + +Existing AI PR reviewers (CodeRabbit, Greptile, Qodo) are closed products with fixed +review dimensions and per-seat pricing. We need a **framework**, not a product: +enterprise-grade, polyglot, forge-agnostic, and extensible by product teams who want +to add their own review dimensions without forking the tool. + +## Decisions + +| Question | Decision | Why | +|---|---|---| +| Forge | Gitea first, GitLab (GitLab CI or Jenkins) next | Matches current infra; forge adapter interface keeps the second one cheap | +| Delivery | CLI binary run as a CI step | Stateless, no HA service to run, secrets stay in the runner, portable across forges | +| Agent harness | `pi` SDK (`earendil-works/pi`) | Agent loop as a library; we own orchestration, tiering, analytics | +| Runtime | TypeScript + Node; ship as npm package **and** OCI image | `pi` is TS; the image means project runners need no Node | +| Tiering | Deterministic rules first, cheap LLM triage as tie-breaker | Repeatable and auditable; adapts on semantic risk | +| Onboarding | `pragent init` writes a committed `.pragent/profile.yml` | Reviewable in a PR, no external index to run or keep fresh | +| Analytics | Structured JSONL per run + optional OpenTelemetry export | Vendor-neutral; no service required to start | + +## 1. Architecture + +``` +┌─ pragent CLI (npm pkg + OCI image) ─────────────────────┐ +│ commands: init | review | explain | replay | doctor │ +└──────────────────────┬──────────────────────────────────┘ + │ + ┌───────────────────┼────────────────────┬─────────────┐ + │ │ │ │ +Forge Adapter Triage Engine Analyzer Bus Emitter +(gitea|gitlab| (rules → cheap (plugins) (JSONL+OTel) + github|jenkins| LLM tiebreak) │ + local-diff) │ │ + │ ▼ ▼ + │ tier: trivial security / quality / + │ lite docs-consistency / + │ full performance / custom + │ oversized │ + │ ▼ + │ pi SDK AgentSession per analyzer + │ (model pinned per analyzer + tier) + ▼ │ +Findings ◄───── Aggregator (dedupe, rank, confidence gate) ◄┘ + │ + ▼ +Publisher → inline comments / summary / exit code +``` + +Analyzers only ever see **profile + diff + tier budget**. A new language means no core +change — language support lives in the profile scan and per-language tool detection, +not in the reviewer. + +### Five extension points + +1. **Analyzer plugin** — declares `id`, applicable `tiers`, `languages`/globs, prompt, + tools, output schema. Dropped in `.pragent/analyzers/` or installed from npm. +2. **Forge adapter** — `getDiff / getContext / publish / setStatus`. Gitea day 1; + GitLab + Jenkins next; `local` adapter for the dev loop. +3. **Tier policy** — rule file per repo/org; overrides thresholds and the path risk map. +4. **Profile enrichers** — extend `pragent init` (e.g. service-catalog ownership lookup). +5. **Emitter sinks** — JSONL default; add OTel/Datadog/whatever. + +### Config layering + +`org defaults → repo .pragent/config.yml → PR labels/commands → CLI flags` + +Org policy can mark keys **locked**, so a repo cannot downgrade e.g. the security analyzer. + +## 2. Tiering and cost model + +Priced at Anthropic rates: Opus 5 $5/$25 per MTok, Sonnet 5 $3/$15, Haiku 4.5 $1/$5; +cache read 0.1×, cache write 1.25×, Batch API −50%. + +**Key lever:** every analyzer in a PR shares one prompt prefix (profile + diff + pulled +context). The first analyzer pays the cache write; the rest pay 0.1×. This is what makes +fan-out affordable, and it is an architectural constraint, not an optimization — the +prefix must be byte-identical across analyzer calls, verified in CI. + +| Tier | Trigger | Engine | Cost/PR | +|---|---|---|---| +| trivial | lockfile-only, generated, docs typo, <20 LOC off risk paths | rules only (or 1 Haiku triage) | ~$0.005 | +| lite | <150 LOC, no risk paths | 1 Sonnet pass | ~$0.08 | +| full | default; risk paths or >150 LOC | 4–5 Opus analyzers, shared cached prefix | ~$0.80 cold, ~$2.00 with agentic file reads | +| oversized | >2k LOC or >50 files | structural summary + deep review on hot subset, hard `task_budget` | ~$5 ceiling | + +Full-tier breakdown: 53k prefix × 1.25 write = $0.33 → 3 further analyzers × 53k × 0.1 = +$0.08 → unique prompts $0.04 → 12k output @ $25 = $0.30 → aggregator $0.07 ≈ **$0.82**. + +**Team of 20 devs, ~350 PRs/month, 1.7 review runs per PR:** ~$510/month ≈ **$25/dev**, +plus a one-off $5–20 per repo for `pragent init`. Comparable to CodeRabbit ($30/dev) and +Qodo ($19/seat), with ownership of the analyzers, the data, and the analytics. + +**Cost knobs in config:** tier→model map; `effort` per analyzer; 1h cache TTL on busy +repos; Batch API (−50%) for nightly re-scans, analytics backfills, and eval runs (never +for the blocking review path); hard per-PR spend ceiling plus `task_budget`, failing open +with a "budget exceeded, partial review" comment. + +**Main variance driver:** agentic file reading ($0.80 → $2.00). Tool-call budget per +analyzer is a first-class config key. + +## 3. Repo profile and analyzers + +### `.pragent/profile.yml` (written by `pragent init`, committed) + +```yaml +version: 1 +generated_at: 2026-08-04T00:00:00Z +generator: pragent 0.1.0 +languages: + - { name: typescript, share: 0.72, roots: [src/, packages/] } + - { name: go, share: 0.21, roots: [services/ingest/] } +build: + install: pnpm install --frozen-lockfile + test: pnpm test + lint: pnpm lint + typecheck: pnpm tsc --noEmit +modules: + - { path: packages/auth, role: security-critical, owners: ["@platform"] } + - { path: packages/billing, role: money-path, owners: ["@payments"] } +conventions: + error_handling: "Result in TS; wrapped errors in Go" + test_layout: "co-located *.test.ts; Go table tests" + docs: { adr: docs/adr/, api: docs/api/, runbooks: docs/runbooks/ } +risk_paths: ["**/auth/**", "**/migrations/**", "infra/**", "**/*.tf"] +existing_tooling: { linters: [eslint, golangci-lint], sast: [semgrep], ci: woodpecker } +``` + +Refresh is a PR (`pragent init --refresh`), so profile drift is reviewable. A staleness +warning fires when the profile predates N commits or a new language appears. + +### Analyzer contract + +```yaml +# .pragent/analyzers/security.yml +id: security +tiers: [lite, full, oversized] +languages: ["*"] +paths: ["**"] +model: { tier_full: claude-opus-5, tier_lite: claude-sonnet-5, effort: high } +tools: [read_file, grep, list_deps] +tool_budget: 12 +prompt: ./prompts/security.md +output_schema: finding[] # {file, line, severity, confidence, category, claim, fix} +``` + +Ships with: `security`, `code-quality`, `docs-consistency` (does the change contradict +the docs it touches, and do public API changes update docs?), `performance`, `tests` +(coverage of the changed behaviour, not line coverage). Teams add their own the same way +— there is no privileged built-in path. + +**Aggregator:** dedupes across analyzers by `(file, line-window, category)`, drops +findings below the repo's confidence gate, ranks by severity × confidence, caps comment +count, and routes the remainder into the run record so nothing is silently lost. + +## 4. Analytics and traceability + +Every run emits one JSONL record, plus one per finding: + +```json +{"run_id":"...","repo":"...","pr":412,"commit":"...","tier":"full", + "tier_reason":"rule:risk_path(**/auth/**)","analyzers":["security","tests"], + "tokens":{"in":54200,"cached":41000,"out":11800},"cost_usd":0.83, + "latency_ms":48200,"findings":{"total":7,"posted":4,"suppressed":3}, + "verdict":"changes_requested","models":{"security":"claude-opus-5"}, + "pragent_version":"0.1.0","profile_hash":"sha256:..."} +``` + +Traceability: `tier_reason` names the exact rule or triage call that chose the tier; +`pragent explain ` reconstructs the decision path; `pragent replay ` +re-runs the same inputs against a new prompt or model — the eval loop. + +Same event schema feeds OTel spans (one per analyzer) for teams with a collector. A +dashboard is a later, optional consumer of the same schema — it is not in scope now. + +**Feedback signal:** a resolved/👎 reaction on a posted comment writes back a +`finding_outcome` record. That is what makes false-positive rate measurable per analyzer, +which is what makes the confidence gates tunable rather than guessed. + +## 5. Rollout phases + +1. **Walking skeleton** — `local` forge adapter, one analyzer, rules-only tiering, JSONL + emitter. Runs on a local diff, prints findings. No network beyond the model API. +2. **Gitea end-to-end** — Gitea adapter, Woodpecker step, PR comments, status checks. +3. **Profile + full tier** — `pragent init`, shared-prefix caching, analyzer fan-out, + aggregator. +4. **Extensibility hardening** — plugin loading from `.pragent/analyzers/` and npm, + config layering with locked org keys, `explain` / `replay`. +5. **Second forge** — GitLab adapter, Jenkins runner recipe. Proves the abstraction. +6. **Analytics maturity** — OTel export, feedback loop, per-analyzer eval harness. + +## Non-goals (for now) + +- Central webhook service (CLI-only until a team actually needs zero-setup onboarding) +- Vector/graph codebase index (profile first; index only as an opt-in plugin) +- Auto-fix commits (findings and suggestions only — writing to branches is a later, + separately-gated decision) +- Web dashboard (event schema first; dashboard is a downstream consumer)