Marcos
f59b906395
Audited the working tree and all 26 commits of history for credentials: none
found. No API keys, no private keys, no tokens — the live bot token, webhook
secret and admin token appear nowhere in the repo or its history.
What was there was infrastructure disclosure, which is recon material rather
than a leak, but has no business in a public repo:
- Tailnet addresses and cluster-internal hostnames in code, docs and the CI
template. The model endpoint is now supplied at runtime via
PRAGENT_MODEL_BASE_URL and patched into opencode.json by install_config();
the committed config carries a placeholder, guarded by a test.
- A host path (/home/marcos) as the default rtk directory — now unset.
- Real usernames in the onboarding docs — now alice/acme.
- A standing list of one-time setup tokens that were never revoked, named
individually. Removed. Note that removing the list does not revoke the
tokens: they should still be revoked in the Gitea admin UI.
The substitution happens in Python rather than via opencode's {env:VAR} config
templating, because the reviewer subprocess runs with an allow-listed
environment — resolving it before the process starts keeps that allow-list from
having to grow.
README rewritten for a reader who has never seen the project: what it does and
what that output looks like, honest status (pilot works, framework designed but
unbuilt), the security model up front given what this thing is, and the measured
cost numbers including the two effects that make naive estimates wrong.
History still contains the old addresses. They are tailnet-only and not
credentials, so no rewrite.
Tests: 131 -> 137.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B11e8TZZxJyzHW7jj7KWUN
178 lines
7.3 KiB
Markdown
178 lines
7.3 KiB
Markdown
# pragent
|
|
|
|
An AI pull-request reviewer for Gitea that posts **inline comments with suggested
|
|
fixes**, not a wall of prose — and reports what each review cost.
|
|
|
|
Label a PR `AI-REVIEW`. A webhook wakes a service that checks the repo out at the
|
|
PR's head commit, reads the changed files *and the code around them*, runs the
|
|
repo's own linters, and posts a review anchored to real lines.
|
|
|
|
**See it work:** [`pragent-demo` PR #1](../pragent-demo/pulls/1) — a PR with
|
|
planted defects, and the review it drew: 9 findings, 3 critical, all anchored
|
|
inline.
|
|
|
|
```
|
|
**[CRITICAL]** search_notes builds its SQL by string concatenation: owner and
|
|
term come from request['query'] and are spliced directly, so a term like
|
|
`' OR 1=1 --` reads every row in the table.
|
|
|
|
Fix: Parameterise owner and term with placeholders and a real LIKE pattern.
|
|
|
|
🪙 ~362 tok (11% · attributed output)
|
|
```
|
|
|
|
## Why this exists
|
|
|
|
CodeRabbit, Greptile and Qodo are good products with fixed review dimensions and
|
|
per-seat pricing. `pragent` targets the case where a platform team needs to **add
|
|
its own dimensions** — an internal compliance rule, a service-catalog ownership
|
|
check, a house performance idiom — without forking a vendor's reviewer. The
|
|
analyzers, the data, and the analytics are yours.
|
|
|
|
It is also self-hosted end to end: the model endpoint is a config value, so the
|
|
code never has to leave your network.
|
|
|
|
## Status
|
|
|
|
A **pilot** is live and reviewing real PRs. The full framework (`pragent init`,
|
|
tiering as code, analyzer fan-out, `explain` / `replay`) is designed but not
|
|
built — see [`docs/plans/`](docs/plans/).
|
|
|
|
What works today:
|
|
|
|
- a central webhook service, so onboarding a repo is *add the bot + add the label*
|
|
- whole-repo context: the reviewer reads callers and types, not just the hunk
|
|
- inline comments with language-highlighted suggested fixes, anchored to
|
|
post-change lines and validated in Python before posting
|
|
- per-commit dedupe, and prior reviews fed back so a re-push synthesises rather
|
|
than repeats
|
|
- `.pr-review.json` for per-repo focus and house rules
|
|
- optional token/cost reporting via an `AI-USAGE` label
|
|
- containment against hostile PR content (see [Security](#security))
|
|
|
|
Not yet: status checks, fail-close, attention tiering enforced in code (it is
|
|
currently a skill the agent follows), multi-model routing.
|
|
|
|
## How a review runs
|
|
|
|
```
|
|
PR labelled AI-REVIEW
|
|
│ Gitea webhook (HMAC-verified, body-capped, concurrency-bounded)
|
|
▼
|
|
review_pr()
|
|
1. dedupe already reviewed this exact sha? stop.
|
|
2. fetch diff + .pr-review.json from the BASE branch
|
|
3. checkout repo archive at head sha → temp workdir
|
|
4. sanitize delete author-controlled agent-instruction files
|
|
5. brief .pragent/brief.md, untrusted parts explicitly fenced
|
|
6. review opencode agent: read code, run linters, emit findings JSON
|
|
7. anchor validate every line against the diff's post-change lines
|
|
8. post inline comments + summary, as pragent-bot
|
|
```
|
|
|
|
Steps 1, 2, 7 and 8 are deterministic Python. The model's only job is step 6 —
|
|
producing correct findings. It never talks to Gitea, and a finding whose line
|
|
does not validate becomes a summary bullet rather than a misplaced comment.
|
|
|
|
## Setup
|
|
|
|
Onboarding a repo, once the service is running for that owner:
|
|
|
|
1. add `pragent-bot` as a **Write** collaborator
|
|
2. create the `AI-REVIEW` label
|
|
3. label a PR
|
|
|
|
Standing up the service itself — the webhook, the image, the Gitea SSRF
|
|
allow-list, the per-owner webhook registration — is in
|
|
[`pilot/README-webhook.md`](pilot/README-webhook.md). A legacy per-repo CI-step
|
|
path is in [`pilot/README.md`](pilot/README.md).
|
|
|
|
The model endpoint is supplied at runtime via `PRAGENT_MODEL_BASE_URL`; the
|
|
committed `opencode.json` carries a placeholder.
|
|
|
|
## Extending it
|
|
|
|
The review "factory" is [`.opencode/`](.opencode/README.md) — agent definitions
|
|
and skills as plain Markdown. Adding a review dimension is dropping a file in,
|
|
not writing code:
|
|
|
|
| Add | How |
|
|
|---|---|
|
|
| A review lens | `.opencode/agents/<name>.md` + one allow-list line in `pragent.md` |
|
|
| Domain knowledge | `.opencode/skills/<name>/SKILL.md`, referenced from the load table |
|
|
| Per-repo rules | `.pr-review.json` in the repo being reviewed |
|
|
|
|
Shipped skills: `attention-tiering` (the cost governor), `review-methodology`,
|
|
`findings-schema`, `linter-playbook`, `security-lens`, `malicious-change`,
|
|
`comment-craft`.
|
|
|
|
## Security
|
|
|
|
The reviewer runs an autonomous agent with shell access over a checkout of **the
|
|
PR author's branch**, and its bot account holds a Write credential. Anyone who
|
|
can open a PR can therefore put arbitrary text in front of the model and
|
|
arbitrary files on its disk — the setup exploited in the [April 2026 disclosures
|
|
against Claude Code Security Review, Gemini CLI Action and Copilot Agent][csa].
|
|
|
|
Four controls, none of which rely on the model behaving:
|
|
|
|
1. **No credentials in the agent's environment.** The subprocess environment is
|
|
built from an allow-list, not inherited. There is nothing to exfiltrate.
|
|
2. **No author-controlled instruction files on disk.** Nested `AGENTS.md`,
|
|
`CLAUDE.md`, `.cursorrules`, a repo `opencode.json` — all deleted before the
|
|
agent starts, so a PR cannot ship its own system prompt. They are still
|
|
*reviewed*, as data.
|
|
3. **Untrusted-data framing.** PR text and diffs are fenced; the agent reports
|
|
injection attempts as `critical` findings instead of following them.
|
|
4. **Reviewer config comes from the base branch**, so a PR cannot rewrite the
|
|
rules it is judged by.
|
|
|
|
Plus: tar-slip guards on the archive, a non-root container, and bounded
|
|
concurrency. Full threat model and residual risks: `pilot/README-webhook.md`.
|
|
|
|
[csa]: https://labs.cloudsecurityalliance.org/research/csa-research-note-comment-control-github-prompt-injection-20/
|
|
|
|
## What it costs
|
|
|
|
The pilot runs against a self-hosted model and bills nothing per token, but the
|
|
token *work* is real. `pilot/cost_model.py` prices it against published API
|
|
rates, calibrated against runs measured through the `AI-USAGE` label
|
|
(`OBSERVED_RUNS` in that file — append to it, don't guess).
|
|
|
|
Two measured reviews of a ~1100-line PR in this repo: 28 and 31 agent steps,
|
|
~2.1M input tokens each, **zero cache reads or writes**. The demo repo's PR, same
|
|
tier: 126K tokens.
|
|
|
|
| Model | this repo, ~1100-line PR | demo repo PR |
|
|
|---|---:|---:|
|
|
| Claude Opus 5 | ~$10.79 | ~$0.71 |
|
|
| Claude Sonnet 5 | ~$4.32 | ~$0.28 |
|
|
| Claude Haiku 4.5 | ~$2.16 | ~$0.14 |
|
|
|
|
Three things that estimate wrong if you skip them:
|
|
|
|
1. **The loop resends its context every step.** Cost is roughly quadratic in step
|
|
count, not linear in diff size. This is what `attention-tiering` exists to cap.
|
|
2. **Repository size dominates diff size.** The 16x gap above is the same
|
|
reviewer on the same tier — the difference is how much repo there was to read.
|
|
3. **Prompt caching is worth about a third of the bill** and is not currently
|
|
happening on this stack. Check `cache_read` before budgeting.
|
|
|
|
```bash
|
|
python3 pilot/cost_model.py --help # other mixes, volumes, models
|
|
```
|
|
|
|
## Development
|
|
|
|
```bash
|
|
python3 -m pytest tests -q # 137 tests, stdlib only, no network
|
|
```
|
|
|
|
The pilot is stdlib-only Python by design — it runs from a bare `python:slim`
|
|
image with the scripts mounted, and has no dependency resolution to go wrong at
|
|
review time.
|
|
|
|
## License
|
|
|
|
Not yet chosen. Until one is added, no reuse rights are granted.
|