harden the pilot against hostile PR content, add review skills + a cost model #7

2026-08-18T04:59:46Z

gitea_admin commented

2026-08-18 04:59:46 +00:00

Two commits.

harden(pilot) — the reviewer runs an opencode agent with bash: "*": allow over a checkout of the PR author's branch, and the pod holds a Gitea Write credential. There was no wall between those two facts.

_build_env now allow-lists the subprocess environment instead of inheriting it, so PRAGENT_BOT_TOKEN and WEBHOOK_SECRET never reach the agent.
sanitize_workdir deletes author-controlled agent-instruction files (nested AGENTS.md, CLAUDE.md, .cursorrules, a repo opencode.json/.opencode) before opencode starts.
The brief fences PR title/body/diff in --- UNTRUSTED --- markers; the agent, the three lens subagents and review-methodology now report injection attempts as a critical finding instead of obeying them.
.pr-review.json is read from the base branch, not the PR head.
Untar rejects escaping symlinks, traversal, and writes through a planted symlink.
The image runs as uid 10001.
Bounded review concurrency + an in-flight (repo, index, sha) claim that closes the check-then-act race in the sha dedupe; request bodies capped.
Correctness: blank context lines no longer desync the diff anchor counter; the body-only fallback no longer drops every finding; the files-endpoint diff fallback emits real a//b/ prefixes; the CI template pins PRAGENT_ENGINE=ollama.

feat(factory) — five conditionally-loaded skills (attention-tiering, linter-playbook, security-lens, malicious-change, comment-craft) and pilot/cost_model.py, which prices a review against published Claude and OpenAI rates from measured factory prompt sizes.

Tests: 68 -> 122.

Labelled AI-REVIEW + AI-USAGE to exercise the newly deployed image end to end.

Two commits. **`harden(pilot)`** — the reviewer runs an opencode agent with `bash: "*": allow` over a checkout of the PR author's branch, and the pod holds a Gitea Write credential. There was no wall between those two facts. - `_build_env` now allow-lists the subprocess environment instead of inheriting it, so `PRAGENT_BOT_TOKEN` and `WEBHOOK_SECRET` never reach the agent. - `sanitize_workdir` deletes author-controlled agent-instruction files (nested `AGENTS.md`, `CLAUDE.md`, `.cursorrules`, a repo `opencode.json`/`.opencode`) before opencode starts. - The brief fences PR title/body/diff in `--- UNTRUSTED ---` markers; the agent, the three lens subagents and `review-methodology` now report injection attempts as a `critical` finding instead of obeying them. - `.pr-review.json` is read from the base branch, not the PR head. - Untar rejects escaping symlinks, traversal, and writes through a planted symlink. - The image runs as uid 10001. - Bounded review concurrency + an in-flight `(repo, index, sha)` claim that closes the check-then-act race in the sha dedupe; request bodies capped. - Correctness: blank context lines no longer desync the diff anchor counter; the body-only fallback no longer drops every finding; the files-endpoint diff fallback emits real `a/`/`b/` prefixes; the CI template pins `PRAGENT_ENGINE=ollama`. **`feat(factory)`** — five conditionally-loaded skills (`attention-tiering`, `linter-playbook`, `security-lens`, `malicious-change`, `comment-craft`) and `pilot/cost_model.py`, which prices a review against published Claude and OpenAI rates from measured factory prompt sizes. Tests: 68 -> 122. Labelled `AI-REVIEW` + `AI-USAGE` to exercise the newly deployed image end to end.

gitea_admin added 2 commits 2026-08-18 04:59:46 +00:00

harden(pilot): contain hostile PR content, bound the webhook, fix anchoring 8c491a7626

The reviewer runs an opencode agent with `bash: "*": allow` over a checkout of
the PR author's branch, and the pod holds a Gitea Write credential. Those two
facts had no wall between them.

Security
- _build_env now allow-lists the subprocess environment instead of inheriting
  it, so PRAGENT_BOT_TOKEN and WEBHOOK_SECRET never reach the agent. This was
  the live hole: a PR body or an AGENTS.md could ask the agent to `curl` the
  token out, and it had both the value and the tool.
- sanitize_workdir deletes author-controlled agent-instruction files from the
  checkout before opencode starts (AGENTS.md at any depth, CLAUDE.md,
  .cursorrules, a repo opencode.json/.opencode, copilot-instructions.md).
  opencode loads nested AGENTS.md as instructions, so a PR could otherwise ship
  its own system prompt. They are still reviewed, as data.
- The brief fences PR title/body and diff in --- UNTRUSTED --- markers under a
  trust-boundary preamble; the pragent agent, the three lens subagents and the
  review-methodology skill now treat injection attempts as a critical finding
  to report rather than an instruction to obey.
- .pr-review.json is read from the PR's base branch, not the head sha. Its
  `instructions` field is spliced into the reviewer's prompt, so head-ref
  reading let any author rewrite the reviewer's rules. Fields are length-capped.
- Untar rejects escaping symlinks, parent traversal, and writes through a
  planted symlink (tar-slip).
- The image runs as uid 10001 instead of root.

Robustness
- Bounded review concurrency (PRAGENT_MAX_CONCURRENT_REVIEWS, default 2). Each
  review forks an opencode process; a thread per delivery was a fork bomb on a
  burst of labels or Gitea retries.
- An in-flight (repo, index, sha) claim closes the check-then-act race in the
  sha-marker dedupe, where two deliveries a second apart both read "not yet
  reviewed" and both posted.
- Request bodies are capped before being read into memory.

Correctness
- parse_diff_anchors counts a whitespace-stripped blank context line. Skipping
  it desynced the new-line counter for the rest of the hunk and silently
  misplaced every later inline comment in that file.
- post_inline_review's body-only fallback folds the anchored findings into the
  body. It previously posted a summary saying "N inline comment(s) below" with
  no comments and no findings — losing them all on the one path that matters.
- fetch_pr_diff's files-endpoint fallback emits real a// b/ prefixes (so
  changed_files and the anchor parser work on it) and reports both HTTP statuses
  in its error instead of the same one twice.
- The CI workflow template pins PRAGENT_ENGINE=ollama; review_pr defaults to
  opencode, which does not exist on a Gitea Actions runner.

Tests: 68 -> 101, covering each of the above.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B11e8TZZxJyzHW7jj7KWUN

feat(factory): five review skills + a per-review cost model 30d2a3d7da

Skills — the primary now loads conditionally (each one is input tokens), per a
load table in pragent.md:

- attention-tiering: classify every PR trivial/lite/full/oversized BEFORE
  reading anything, and cap file reads, linter runs and subagent fan-out per
  tier. This is the cost governor; the other skills defer to its budget.
- linter-playbook: per-ecosystem detect-and-run commands scoped to changed
  files, the never-install rule, and how to turn a diagnostic into a finding
  instead of pasting tool output.
- security-lens: the inline security checklist for when @security isn't worth
  delegating, built around a source -> sink test each finding must pass.
- malicious-change: hostile-PR detection — injection aimed at the reviewer,
  install/CI-time hooks, obfuscated payloads, dependency confusion, logic
  backdoors. Complements the runtime containment added in the previous commit:
  that stops the agent being hijacked, this makes it report the attempt.
- comment-craft: how to write problem/fix/suggestion so a maintainer can act in
  one read, and what to cut.

pilot/cost_model.py — prices a review against published Claude and OpenAI rates
(fetched 2026-08-18). Prompt sizes are measured from the factory files rather
than guessed; per-tier workloads come from the tiering budgets. The model is
explicit about the thing that actually dominates an agent loop: the whole
conversation is resent every step, so caching moves ~2.3x of the bill.

Blended over a 5/35/55/5 mix with caching on: ~$0.61/PR on Opus 5 or GPT-5.6
Sol, ~$0.24 on Sonnet 5 or Terra, ~$0.12 on Haiku 4.5, ~$0.02 on Luna. At 350
PRs/month that's ~$212 / ~$85 / ~$43 / ~$8.50.

Tests: 101 -> 122.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B11e8TZZxJyzHW7jj7KWUN

pragent-bot bot added the AI-REVIEW AI-USAGE labels 2026-08-18 05:00:04 +00:00

pragent-bot bot reviewed 2026-08-18 05:05:55 +00:00

pragent-bot bot left a comment

🤖 AI Review · pragent pilot · glm-5.2:cloud · 30d2a3d7

Tier: full (~1200 changed lines, security-sensitive: subprocess env, file I/O, tar extraction, CI, secrets). This PR hardens the pragent pilot against hostile PR content — allow-listed subprocess env (no bot token/webhook secret reach the agent), sanitize_workdir strips author-controlled agent-instruction files before opencode starts, _extract_tar_strip_one blocks tar-slip via escaping symlinks/traversal/symlink-through-writes, .pr-review.json is read from the base branch, the container runs as uid 10001, review concurrency is bounded, and an in-flight (repo,index,sha) claim closes the sha-dedupe check-then-act race; plus five conditionally-loaded review skills and a per-review cost model. Verified the tar guards against chained-symlink, parent-replacement and hardlink escape attempts (all blocked), confirmed _ENV_ALLOW drops every secret, confirmed sanitize runs before drop_factory, ran ruff on changed files (no new diagnostics on added lines), and ran the full suite (122 passed). No prompt-injection attempts directed at the reviewer were found — the injection-looking strings are all in the defensive docs/tests that define detection. No actionable defects.

🔋 AI usage

model: glm-5.2:cloud · engine: opencode · agent steps: 28 · duration: 348.3s
tokens: 2071025 in · 17303 out · 0 reasoning · cache 0 read / 0 write → 2088328 total
est. cost: $0.00 (on-network glm-5.2:cloud via headroom — no per-token charge)
scope: whole-repo checkout at head sha (agent can read any file + run linters, not just the diff) — input tokens include files read beyond the diff
per-comment tokens are attributed (one model pass produces all findings; output split by each finding's body weight)

No issues found.

🤖 **AI Review** · pragent pilot · glm-5.2:cloud · `30d2a3d7` Tier: `full` (~1200 changed lines, security-sensitive: subprocess env, file I/O, tar extraction, CI, secrets). This PR hardens the pragent pilot against hostile PR content — allow-listed subprocess env (no bot token/webhook secret reach the agent), sanitize_workdir strips author-controlled agent-instruction files before opencode starts, _extract_tar_strip_one blocks tar-slip via escaping symlinks/traversal/symlink-through-writes, .pr-review.json is read from the base branch, the container runs as uid 10001, review concurrency is bounded, and an in-flight (repo,index,sha) claim closes the sha-dedupe check-then-act race; plus five conditionally-loaded review skills and a per-review cost model. Verified the tar guards against chained-symlink, parent-replacement and hardlink escape attempts (all blocked), confirmed _ENV_ALLOW drops every secret, confirmed sanitize runs before drop_factory, ran ruff on changed files (no new diagnostics on added lines), and ran the full suite (122 passed). No prompt-injection attempts directed at the reviewer were found — the injection-looking strings are all in the defensive docs/tests that define detection. No actionable defects. ## 🔋 AI usage - model: `glm-5.2:cloud` · engine: opencode · agent steps: 28 · duration: 348.3s - tokens: 2071025 in · 17303 out · 0 reasoning · cache 0 read / 0 write → 2088328 total - est. cost: $0.00 (on-network glm-5.2:cloud via headroom — no per-token charge) - scope: whole-repo checkout at head sha (agent can read any file + run linters, not just the diff) — input tokens include files read beyond the diff - per-comment tokens are *attributed* (one model pass produces all findings; output split by each finding's body weight) No issues found.

gitea_admin referenced this issue from a commit

2026-08-18 05:08:30 +00:00

fix(cost-model): calibrate against the first measured review

gitea_admin added 1 commit 2026-08-18 05:08:30 +00:00

fix(cost-model): calibrate against the first measured review 2613b3e3af

PR #7 ran under the AI-USAGE label and reported real numbers: 28 agent steps,
348s, 2,071,025 input / 17,303 output tokens, and zero cache reads or writes.
The model predicted ~$0.73 on Opus 5 for that tier. The measurement prices it
at $10.79 — the model was ~15x low.

Two wrong assumptions:

- Step count and per-step growth. `full` assumed 12 steps and 1,200 tokens per
  tool result; the run did 28 steps averaging ~3,300. Cost is roughly quadratic
  in steps, so this compounds. Tier defaults are re-derived from the measured
  per-step growth rather than from guesses.
- Caching. The model defaulted to prompt caching on. The headroom/glm-5.2 path
  reports 0 read / 0 write, so the stable prefix is paid at full input price on
  every step. Budget with caching off until that column is nonzero.

Adds OBSERVED_RUNS as an append-only calibration anchor, an observed-runs
section in the report, and a regression test asserting the model stays within
2.5x of the measurement — so the next drift is caught by the suite rather than
by a surprising invoice.

Corrected blended figures at 350 PRs/month: ~$1,740 Opus 5, ~$1,755 GPT-5.6
Sol, ~$696 Sonnet 5, ~$348 Haiku 4.5, ~$70 GPT-5.6 Luna.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B11e8TZZxJyzHW7jj7KWUN

pragent-bot bot reviewed 2026-08-18 05:14:01 +00:00

pragent-bot bot left a comment

🤖 AI Review · pragent pilot · glm-5.2:cloud · 2613b3e3

AI review produced no parseable output.

🤖 **AI Review** · pragent pilot · glm-5.2:cloud · `2613b3e3` AI review produced no parseable output.

gitea_admin added 1 commit 2026-08-18 05:18:10 +00:00

fix(review): salvage the prose when the findings JSON is unparseable 5d28f97080

Found by running the reviewer against this branch. The second review of PR #7
ran a full agent loop — 330s of a 540s budget, no timeout — and ended without a
parseable ```json block. The code discarded the entire run and posted "AI review
produced no parseable output.", losing minutes of work and millions of tokens
for a message that tells the maintainer nothing and gives me nothing to debug.

Three changes on that path:

- salvage_summary() keeps the agent's prose (fenced blocks stripped, tail kept
  because the conclusion is written last) and posts it under an explicit banner
  saying it is unstructured and its line numbers were never validated against
  the diff. A partial review honestly labelled beats no review.
- The raw output's length and last 600 chars go to stderr, so the next
  occurrence is diagnosable from pod logs instead of invisible.
- The AI-USAGE section is still rendered. The label asked for it and the tokens
  were spent either way; dropping the measurement on the failure path is how the
  cost model stops getting calibration data exactly when it is most interesting.

Not fixed here: why the agent went off-format. The likely cause is the 40-step
cap in the agent definition being reached on a larger diff (the successful run
used 28), which wants either a higher cap or a step-budget warning in the
prompt. Needs the next occurrence's stderr to confirm rather than guess.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B11e8TZZxJyzHW7jj7KWUN

gitea_admin referenced this issue from a commit

2026-08-18 05:18:10 +00:00

fix(review): salvage the prose when the findings JSON is unparseable

pragent-bot bot reviewed 2026-08-18 05:21:22 +00:00

pragent-bot bot left a comment

🤖 AI Review · pragent pilot · glm-5.2:cloud · 5d28f970

Tier: full. This PR hardens the pragent pilot against hostile PR content: allow-listed subprocess env (no bot token/webhook secret reach the agent), sanitize_workdir strips author-controlled agent-instruction files before opencode starts, _extract_tar_strip_one blocks tar-slip via escaping symlinks/traversal/writes-through-symlinks, .pr-review.json is read from the base branch (URL-encoded), the container runs as uid 10001, review concurrency is bounded, and an in-flight (repo,index,sha) claim closes the sha-dedupe race; plus five conditionally-loaded review skills and a per-review cost model. Independently verified: tar guards block hardlink/dir-symlink/chained-symlink escapes, _build_env leaks no secrets, README cost figures reproduce exactly from cost_model.py, ruff/pyright errors are all on pre-existing untouched lines, and all 131 tests pass. No prompt-injection attempts directed at the reviewer. No actionable defects.

🔋 AI usage

model: glm-5.2:cloud · engine: opencode · agent steps: 31 · duration: 189.8s
tokens: 2213077 in · 9058 out · 0 reasoning · cache 0 read / 0 write → 2222135 total
est. cost: $0.00 (on-network glm-5.2:cloud via headroom — no per-token charge)
scope: whole-repo checkout at head sha (agent can read any file + run linters, not just the diff) — input tokens include files read beyond the diff
per-comment tokens are attributed (one model pass produces all findings; output split by each finding's body weight)

No issues found.

🤖 **AI Review** · pragent pilot · glm-5.2:cloud · `5d28f970` Tier: `full`. This PR hardens the pragent pilot against hostile PR content: allow-listed subprocess env (no bot token/webhook secret reach the agent), sanitize_workdir strips author-controlled agent-instruction files before opencode starts, _extract_tar_strip_one blocks tar-slip via escaping symlinks/traversal/writes-through-symlinks, .pr-review.json is read from the base branch (URL-encoded), the container runs as uid 10001, review concurrency is bounded, and an in-flight (repo,index,sha) claim closes the sha-dedupe race; plus five conditionally-loaded review skills and a per-review cost model. Independently verified: tar guards block hardlink/dir-symlink/chained-symlink escapes, _build_env leaks no secrets, README cost figures reproduce exactly from cost_model.py, ruff/pyright errors are all on pre-existing untouched lines, and all 131 tests pass. No prompt-injection attempts directed at the reviewer. No actionable defects. ## 🔋 AI usage - model: `glm-5.2:cloud` · engine: opencode · agent steps: 31 · duration: 189.8s - tokens: 2213077 in · 9058 out · 0 reasoning · cache 0 read / 0 write → 2222135 total - est. cost: $0.00 (on-network glm-5.2:cloud via headroom — no per-token charge) - scope: whole-repo checkout at head sha (agent can read any file + run linters, not just the diff) — input tokens include files read beyond the diff - per-comment tokens are *attributed* (one model pass produces all findings; output split by each finding's body weight) No issues found.

gitea_admin added 1 commit 2026-08-18 05:22:05 +00:00

chore(cost-model): record the second measured review 80501840e2

31 steps, 190s, 2,213,077 in / 9,058 out, cache 0/0 — within 7% of the first
run's input on the same tier, which is the first evidence the recalibrated tier
defaults hold rather than fitting a single point.

Also weakens the step-cap hypothesis from the previous commit: this run used 31
steps (more than the 28 that succeeded, and more than the run that failed) and
parsed fine, so hitting `steps: 40` is not on its own what breaks the output
format. Leaving the cap alone until the stderr logging catches a real failure.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B11e8TZZxJyzHW7jj7KWUN

pragent-bot bot reviewed 2026-08-18 05:25:39 +00:00

pragent-bot bot left a comment

🤖 AI Review · pragent pilot · glm-5.2:cloud · 80501840

Tier: full (~1200 changed lines, 23 files, security-sensitive: subprocess env, tar extraction, Dockerfile, webhook HMAC, CI). This PR hardens the pragent pilot against hostile PR content — _build_env allow-lists the subprocess env (PRAGENT_BOT_TOKEN/WEBHOOK_SECRET never reach the agent), sanitize_workdir strips author-controlled agent-instruction files before opencode starts, _extract_tar_strip_one blocks tar-slip via escaping symlinks/traversal/writes-through-symlinks, .pr-review.json is read from the base branch, the container runs as uid 10001, review concurrency is bounded, and an in-flight (repo,index,sha) claim closes the sha-dedupe race — plus five conditionally-loaded review skills and a calibrated per-review cost model. Independently verified: ruff (16 diagnostics, all on pre-existing untouched lines), pyright (2 errors, both pre-existing), 131 tests pass, the tar-slip defense holds against chained/escaping/dangling-symlink variants (escaping links are rejected at creation so a later file's parent symlink can only point inside dest), the env allow-list drops every secret, and the cost model reproduces the README's $10.79/review and ~$1,740/month figures exactly. No prompt-injection attempts directed at the reviewer — the injection-looking strings are all in defensive test fixtures/docs that define detection. No actionable defects.

🔋 AI usage

model: glm-5.2:cloud · engine: opencode · agent steps: 26 · duration: 212.3s
tokens: 1819561 in · 16277 out · 0 reasoning · cache 0 read / 0 write → 1835838 total
est. cost: $0.00 (on-network glm-5.2:cloud via headroom — no per-token charge)
scope: whole-repo checkout at head sha (agent can read any file + run linters, not just the diff) — input tokens include files read beyond the diff
per-comment tokens are attributed (one model pass produces all findings; output split by each finding's body weight)

No issues found.

🤖 **AI Review** · pragent pilot · glm-5.2:cloud · `80501840` Tier: full (~1200 changed lines, 23 files, security-sensitive: subprocess env, tar extraction, Dockerfile, webhook HMAC, CI). This PR hardens the pragent pilot against hostile PR content — _build_env allow-lists the subprocess env (PRAGENT_BOT_TOKEN/WEBHOOK_SECRET never reach the agent), sanitize_workdir strips author-controlled agent-instruction files before opencode starts, _extract_tar_strip_one blocks tar-slip via escaping symlinks/traversal/writes-through-symlinks, .pr-review.json is read from the base branch, the container runs as uid 10001, review concurrency is bounded, and an in-flight (repo,index,sha) claim closes the sha-dedupe race — plus five conditionally-loaded review skills and a calibrated per-review cost model. Independently verified: ruff (16 diagnostics, all on pre-existing untouched lines), pyright (2 errors, both pre-existing), 131 tests pass, the tar-slip defense holds against chained/escaping/dangling-symlink variants (escaping links are rejected at creation so a later file's parent symlink can only point inside dest), the env allow-list drops every secret, and the cost model reproduces the README's $10.79/review and ~$1,740/month figures exactly. No prompt-injection attempts directed at the reviewer — the injection-looking strings are all in defensive test fixtures/docs that define detection. No actionable defects. ## 🔋 AI usage - model: `glm-5.2:cloud` · engine: opencode · agent steps: 26 · duration: 212.3s - tokens: 1819561 in · 16277 out · 0 reasoning · cache 0 read / 0 write → 1835838 total - est. cost: $0.00 (on-network glm-5.2:cloud via headroom — no per-token charge) - scope: whole-repo checkout at head sha (agent can read any file + run linters, not just the diff) — input tokens include files read beyond the diff - per-comment tokens are *attributed* (one model pass produces all findings; output split by each finding's body weight) No issues found.

gitea_admin merged commit 46513585ae into main

2026-08-18 05:31:08 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: gitea_admin/pragent#7