gitea_admin/pragent
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

chore: scrub private infrastructure for a public repo, rewrite README

Browse Source 
Audited the working tree and all 26 commits of history for credentials: none
found. No API keys, no private keys, no tokens — the live bot token, webhook
secret and admin token appear nowhere in the repo or its history.

What was there was infrastructure disclosure, which is recon material rather
than a leak, but has no business in a public repo:

- Tailnet addresses and cluster-internal hostnames in code, docs and the CI
  template. The model endpoint is now supplied at runtime via
  PRAGENT_MODEL_BASE_URL and patched into opencode.json by install_config();
  the committed config carries a placeholder, guarded by a test.
- A host path (/home/marcos) as the default rtk directory — now unset.
- Real usernames in the onboarding docs — now alice/acme.
- A standing list of one-time setup tokens that were never revoked, named
  individually. Removed. Note that removing the list does not revoke the
  tokens: they should still be revoked in the Gitea admin UI.

The substitution happens in Python rather than via opencode's {env:VAR} config
templating, because the reviewer subprocess runs with an allow-listed
environment — resolving it before the process starts keeps that allow-list from
having to grow.

README rewritten for a reader who has never seen the project: what it does and
what that output looks like, honest status (pilot works, framework designed but
unbuilt), the security model up front given what this thing is, and the measured
cost numbers including the two effects that make naive estimates wrong.

History still contains the old addresses. They are tailnet-only and not
credentials, so no rewrite.

Tests: 131 -> 137.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B11e8TZZxJyzHW7jj7KWUN
This commit is contained in:
Marcos
2026-08-18 12:59:52 +00:00
parent 46513585ae
commit f59b906395
11 changed files with 292 additions and 164 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+150 -116
View File
@@ -1,143 +1,177 @@
# pragent
An extensible, forge-agnostic PR review framework. Not a product — a toolkit that teams
extend with their own review dimensions.
An AI pull-request reviewer for Gitea that posts **inline comments with suggested
fixes**, not a wall of prose — and reports what each review cost.
**Status:** design approved; framework build deferred. A **pilot** is live on
`glm-5.2:cloud` with two delivery paths:
Label a PR `AI-REVIEW`. A webhook wakes a service that checks the repo out at the
PR's head commit, reads the changed files *and the code around them*, runs the
repo's own linters, and posts a review anchored to real lines.
- **Central webhook service** (preferred, least per-repo setup): a Gitea
user-level webhook posts PR events to an always-on in-cluster service that
gates on the `AI-REVIEW` label. Onboarding a repo = add `pragent-bot`
collaborator + create the label + label a PR. See
[`pilot/README-webhook.md`](pilot/README-webhook.md).
- **CI-step** (legacy): a per-repo Gitea Action fetches the reviewer script at
runtime. See [`pilot/README.md`](pilot/README.md).
The framework design remains at
[`docs/plans/2026-08-04-pragent-design.md`](docs/plans/2026-08-04-pragent-design.md);
the pilot is its bootstrap and will be superseded by `pragent review` when the
framework build resumes.
## What it is
`pragent` runs as a CI step. It reads a pull request, decides how much attention the
change deserves, runs the analyzers that apply, and posts ranked findings back to the
forge.
**See it work:** [`pragent-demo` PR #1](../pragent-demo/pulls/1) — a PR with
planted defects, and the review it drew: 9 findings, 3 critical, all anchored
inline.
```
pragent init # one-time repo scan → .pragent/profile.yml (committed, reviewable)
pragent review # the CI step: tier → analyze → aggregate → publish
pragent explain # why did this PR get this tier / these findings?
pragent replay # re-run a past PR against a new prompt or model (the eval loop)
pragent doctor # config, credentials, and adapter health
**[CRITICAL]** search_notes builds its SQL by string concatenation: owner and
term come from request['query'] and are spliced directly, so a term like
`' OR 1=1 --` reads every row in the table.
Fix: Parameterise owner and term with placeholders and a real LIKE pattern.
🪙 ~362 tok (11% · attributed output)
```
## Why not CodeRabbit / Greptile / Qodo
## Why this exists
Those are good products with fixed review dimensions and per-seat pricing. `pragent`
targets the case where a platform team needs to **add its own dimensions** — an internal
compliance rule, a service-catalog ownership check, a house performance idiom — without
forking a vendor's reviewer. Cost lands in the same range (~$25/dev/month at 350 PRs/mo
for 20 devs), but the analyzers, the data, and the analytics are yours.
CodeRabbit, Greptile and Qodo are good products with fixed review dimensions and
per-seat pricing. `pragent` targets the case where a platform team needs to **add
its own dimensions** — an internal compliance rule, a service-catalog ownership
check, a house performance idiom — without forking a vendor's reviewer. The
analyzers, the data, and the analytics are yours.
## Attention tiers
It is also self-hosted end to end: the model endpoint is a config value, so the
code never has to leave your network.
Every PR is classified before any expensive work happens. Deterministic rules decide
first; an ambiguous case gets one cheap model call as tie-breaker.
## Status
| Tier | What it means | Cost/PR |
|---|---|---|
| `trivial` | lockfile bumps, generated code, docs typos | ~$0.005 |
| `lite` | small change, no risk paths | ~$0.08 |
| `full` | the default for real changes | ~$0.80–2.00 |
| `oversized` | too big to review whole; structural summary + deep pass on the hot subset | ~$5 ceiling |
A **pilot** is live and reviewing real PRs. The full framework (`pragent init`,
tiering as code, analyzer fan-out, `explain` / `replay`) is designed but not
built — see [`docs/plans/`](docs/plans/).
Every tier decision records *why*, so a surprising outcome is explainable rather than
mysterious.
What works today:
- a central webhook service, so onboarding a repo is *add the bot + add the label*
- whole-repo context: the reviewer reads callers and types, not just the hunk
- inline comments with language-highlighted suggested fixes, anchored to
post-change lines and validated in Python before posting
- per-commit dedupe, and prior reviews fed back so a re-push synthesises rather
than repeats
- `.pr-review.json` for per-repo focus and house rules
- optional token/cost reporting via an `AI-USAGE` label
- containment against hostile PR content (see [Security](#security))
Not yet: status checks, fail-close, attention tiering enforced in code (it is
currently a skill the agent follows), multi-model routing.
## How a review runs
```
PR labelled AI-REVIEW
│ Gitea webhook (HMAC-verified, body-capped, concurrency-bounded)
▼
review_pr()
1. dedupe already reviewed this exact sha? stop.
2. fetch diff + .pr-review.json from the BASE branch
3. checkout repo archive at head sha → temp workdir
4. sanitize delete author-controlled agent-instruction files
5. brief .pragent/brief.md, untrusted parts explicitly fenced
6. review opencode agent: read code, run linters, emit findings JSON
7. anchor validate every line against the diff's post-change lines
8. post inline comments + summary, as pragent-bot
```
Steps 1, 2, 7 and 8 are deterministic Python. The model's only job is step 6 —
producing correct findings. It never talks to Gitea, and a finding whose line
does not validate becomes a summary bullet rather than a misplaced comment.
## Setup
Onboarding a repo, once the service is running for that owner:
1. add `pragent-bot` as a **Write** collaborator
2. create the `AI-REVIEW` label
3. label a PR
Standing up the service itself — the webhook, the image, the Gitea SSRF
allow-list, the per-owner webhook registration — is in
[`pilot/README-webhook.md`](pilot/README-webhook.md). A legacy per-repo CI-step
path is in [`pilot/README.md`](pilot/README.md).
The model endpoint is supplied at runtime via `PRAGENT_MODEL_BASE_URL`; the
committed `opencode.json` carries a placeholder.
## Extending it
The review "factory" is [`.opencode/`](.opencode/README.md) — agent definitions
and skills as plain Markdown. Adding a review dimension is dropping a file in,
not writing code:
| Add | How |
|---|---|
| A review lens | `.opencode/agents/<name>.md` + one allow-list line in `pragent.md` |
| Domain knowledge | `.opencode/skills/<name>/SKILL.md`, referenced from the load table |
| Per-repo rules | `.pr-review.json` in the repo being reviewed |
Shipped skills: `attention-tiering` (the cost governor), `review-methodology`,
`findings-schema`, `linter-playbook`, `security-lens`, `malicious-change`,
`comment-craft`.
## Security
The reviewer runs an autonomous agent with shell access over a checkout of **the
PR author's branch**, and its bot account holds a Write credential. Anyone who
can open a PR can therefore put arbitrary text in front of the model and
arbitrary files on its disk — the setup exploited in the [April 2026 disclosures
against Claude Code Security Review, Gemini CLI Action and Copilot Agent][csa].
Four controls, none of which rely on the model behaving:
1. **No credentials in the agent's environment.** The subprocess environment is
built from an allow-list, not inherited. There is nothing to exfiltrate.
2. **No author-controlled instruction files on disk.** Nested `AGENTS.md`,
`CLAUDE.md`, `.cursorrules`, a repo `opencode.json` — all deleted before the
agent starts, so a PR cannot ship its own system prompt. They are still
*reviewed*, as data.
3. **Untrusted-data framing.** PR text and diffs are fenced; the agent reports
injection attempts as `critical` findings instead of following them.
4. **Reviewer config comes from the base branch**, so a PR cannot rewrite the
rules it is judged by.
Plus: tar-slip guards on the archive, a non-root container, and bounded
concurrency. Full threat model and residual risks: `pilot/README-webhook.md`.
[csa]: https://labs.cloudsecurityalliance.org/research/csa-research-note-comment-control-github-prompt-injection-20/
## What it costs
The pilot runs on `glm-5.2:cloud` through the on-network headroom proxy, so today it
bills nothing per token — but the token *work* is real, and `pilot/cost_model.py`
prices it against published API rates. Factory prompt sizes are measured from the
files in this repo; the per-tier workloads are calibrated against runs actually
measured through the `AI-USAGE` label (`OBSERVED_RUNS` in that file).
The pilot runs against a self-hosted model and bills nothing per token, but the
token *work* is real. `pilot/cost_model.py` prices it against published API
rates, calibrated against runs measured through the `AI-USAGE` label
(`OBSERVED_RUNS` in that file — append to it, don't guess).
**The measured anchor.** The hardening PR (`#7`, 16 files / ~1100 changed lines,
tier `full`) took 28 agent steps and 348s, and consumed **2,071,025 input** and
**17,303 output** tokens — with **zero cache reads or writes**, because the current
headroom/glm path does no prompt caching. Priced elsewhere, that single review is:
Two measured reviews of a ~1100-line PR in this repo: 28 and 31 agent steps,
~2.1M input tokens each, **zero cache reads or writes**. The demo repo's PR, same
tier: 126K tokens.
| Model | that review | blended per PR | 350 PRs/month |
|---|---:|---:|---:|
| Claude Opus 5 | $10.79 | ~$4.97 | ~$1,740 |
| GPT-5.6 Sol | $10.87 | ~$5.02 | ~$1,755 |
| Claude Sonnet 5 | $4.32 | ~$1.99 | ~$696 |
| GPT-5.6 Terra | $4.35 | ~$2.01 | ~$702 |
| Claude Haiku 4.5 | $2.16 | ~$0.99 | ~$348 |
| GPT-5.6 Luna | $0.43 | ~$0.20 | ~$70 |
| Model | this repo, ~1100-line PR | demo repo PR |
|---|---:|---:|
| Claude Opus 5 | ~$10.79 | ~$0.71 |
| Claude Sonnet 5 | ~$4.32 | ~$0.28 |
| Claude Haiku 4.5 | ~$2.16 | ~$0.14 |
Blended figures use a 5/35/55/5 tier mix with caching off, matching what is
actually observed. Run `python3 pilot/cost_model.py --help` for other mixes and
volumes.
Three things that estimate wrong if you skip them:
Two things dominate, and neither is the diff:
1. **The loop resends its context every step.** Cost is roughly quadratic in step
count, not linear in diff size. This is what `attention-tiering` exists to cap.
2. **Repository size dominates diff size.** The 16x gap above is the same
reviewer on the same tier — the difference is how much repo there was to read.
3. **Prompt caching is worth about a third of the bill** and is not currently
happening on this stack. Check `cache_read` before budgeting.
1. **The loop resends its context every step.** 28 steps over a ~17k-token diff
produced 2M input tokens. Cost is roughly quadratic in step count, which is why
`attention-tiering` caps steps, file reads and subagent fan-out per tier.
2. **Prompt caching is worth about a third of the bill** and is currently not
happening. Any move to a paid provider should confirm the `cache_read` column
goes nonzero before budgeting.
```bash
python3 pilot/cost_model.py --help # other mixes, volumes, models
```
An earlier version of this model assumed 12 steps and caching on, and was ~15x
low. The lesson is in the file: budget from `OBSERVED_RUNS`, not from the tier
table, and append a row every time a real review reports usage.
## Development
## Extension points
```bash
python3 -m pytest tests -q # 137 tests, stdlib only, no network
```
Five, all documented in the design doc. Teams override or add; nobody forks.
1. **Analyzers** — drop a YAML + prompt in `.pragent/analyzers/`, or install from npm
2. **Forge adapters** — Gitea, GitLab, GitHub, local diff
3. **Tier policy** — thresholds and the path risk map, per repo or per org
4. **Profile enrichers** — extend what `pragent init` learns about a repo
5. **Emitter sinks** — JSONL by default, OpenTelemetry, or your own
Org config can lock keys, so a repo cannot quietly disable the security analyzer.
## Design principles
- **Polyglot by construction.** Language knowledge lives in the repo profile, not in the
reviewer. A new language is a profile change, not a core change.
- **One shared prompt prefix.** All analyzers for a PR share a byte-identical cached
prefix. This is what makes fan-out affordable; it is enforced, not hoped for.
- **Everything is traceable.** Tier reasons, token counts, cost, latency, and finding
outcomes are recorded per run. False-positive rate is measurable per analyzer.
- **Fail open.** A budget ceiling or an analyzer crash yields a partial review with a
clear note, never a blocked pipeline with no explanation.
- **The reviewed code is untrusted input.** The reviewer runs an agent over a
branch anyone with PR access can write. So it holds no credentials in its
environment, the checkout is stripped of files an agent runtime would load as
instructions, PR-authored text is fenced as data, and reviewer config is read
from the base branch. See "Threat model" in
[`pilot/README-webhook.md`](pilot/README-webhook.md).
## Stack
TypeScript + Node, built on the [`pi`](https://github.com/badlogic/pi-mono) agent SDK.
Shipped as an npm package and an OCI image, so CI runners need no local Node install.
## Roadmap
1. Walking skeleton — local diff, one analyzer, rules-only tiering
2. Gitea end to end — adapter, Woodpecker step, PR comments, status checks
3. Profile + full tier — `pragent init`, shared-prefix caching, analyzer fan-out
4. Extensibility hardening — plugin loading, config layering, `explain` / `replay`
5. Second forge — GitLab adapter, Jenkins recipe
6. Analytics maturity — OTel export, feedback loop, eval harness
The pilot is stdlib-only Python by design — it runs from a bare `python:slim`
image with the scripts mounted, and has no dependency resolution to go wrong at
review time.
## License
TBD.
Not yet chosen. Until one is added, no reuse rights are granted.