Remove the obsolete dashboard now that Langfuse is the analytics surface.\nIntroduce focused transport, model, and configuration modules while preserving the ai_review facade, and document the current runtime architecture.
Ship token spend, latency and equivalent cost for every review to the
self-hosted Langfuse so per-model behaviour is queryable as a trend rather
than one PR comment at a time.
langfuse_trace.py is stdlib-only and emits via the public ingestion API.
Traces split into `ollama` and `claude` environments keyed off the bare model
name, not the provider: both paths go through the same headroom proxy, so the
provider prefix says nothing about which spend story a review belongs to. The
pilot's own path bills $0, so the reported cost is the equivalent price from
cost_model.PRICES.
ai_review.py calls _emit_langfuse on both token-spending exit paths (the
normal post and the salvage path). Import and emission are wrapped in a
blanket except: with no LANGFUSE_HOST or key pair the whole thing is a silent
no-op, and a telemetry failure must never fail a review.
These files were previously deployed only by way of the image build's
`COPY . /app`, so a clean checkout would have silently dropped tracing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Audited the working tree and all 26 commits of history for credentials: none
found. No API keys, no private keys, no tokens — the live bot token, webhook
secret and admin token appear nowhere in the repo or its history.
What was there was infrastructure disclosure, which is recon material rather
than a leak, but has no business in a public repo:
- Tailnet addresses and cluster-internal hostnames in code, docs and the CI
template. The model endpoint is now supplied at runtime via
PRAGENT_MODEL_BASE_URL and patched into opencode.json by install_config();
the committed config carries a placeholder, guarded by a test.
- A host path (/home/marcos) as the default rtk directory — now unset.
- Real usernames in the onboarding docs — now alice/acme.
- A standing list of one-time setup tokens that were never revoked, named
individually. Removed. Note that removing the list does not revoke the
tokens: they should still be revoked in the Gitea admin UI.
The substitution happens in Python rather than via opencode's {env:VAR} config
templating, because the reviewer subprocess runs with an allow-listed
environment — resolving it before the process starts keeps that allow-list from
having to grow.
README rewritten for a reader who has never seen the project: what it does and
what that output looks like, honest status (pilot works, framework designed but
unbuilt), the security model up front given what this thing is, and the measured
cost numbers including the two effects that make naive estimates wrong.
History still contains the old addresses. They are tailnet-only and not
credentials, so no rewrite.
Tests: 131 -> 137.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B11e8TZZxJyzHW7jj7KWUN
PR #7 ran under the AI-USAGE label and reported real numbers: 28 agent steps,
348s, 2,071,025 input / 17,303 output tokens, and zero cache reads or writes.
The model predicted ~$0.73 on Opus 5 for that tier. The measurement prices it
at $10.79 — the model was ~15x low.
Two wrong assumptions:
- Step count and per-step growth. `full` assumed 12 steps and 1,200 tokens per
tool result; the run did 28 steps averaging ~3,300. Cost is roughly quadratic
in steps, so this compounds. Tier defaults are re-derived from the measured
per-step growth rather than from guesses.
- Caching. The model defaulted to prompt caching on. The headroom/glm-5.2 path
reports 0 read / 0 write, so the stable prefix is paid at full input price on
every step. Budget with caching off until that column is nonzero.
Adds OBSERVED_RUNS as an append-only calibration anchor, an observed-runs
section in the report, and a regression test asserting the model stays within
2.5x of the measurement — so the next drift is caught by the suite rather than
by a surprising invoice.
Corrected blended figures at 350 PRs/month: ~$1,740 Opus 5, ~$1,755 GPT-5.6
Sol, ~$696 Sonnet 5, ~$348 Haiku 4.5, ~$70 GPT-5.6 Luna.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B11e8TZZxJyzHW7jj7KWUN
Skills — the primary now loads conditionally (each one is input tokens), per a
load table in pragent.md:
- attention-tiering: classify every PR trivial/lite/full/oversized BEFORE
reading anything, and cap file reads, linter runs and subagent fan-out per
tier. This is the cost governor; the other skills defer to its budget.
- linter-playbook: per-ecosystem detect-and-run commands scoped to changed
files, the never-install rule, and how to turn a diagnostic into a finding
instead of pasting tool output.
- security-lens: the inline security checklist for when @security isn't worth
delegating, built around a source -> sink test each finding must pass.
- malicious-change: hostile-PR detection — injection aimed at the reviewer,
install/CI-time hooks, obfuscated payloads, dependency confusion, logic
backdoors. Complements the runtime containment added in the previous commit:
that stops the agent being hijacked, this makes it report the attempt.
- comment-craft: how to write problem/fix/suggestion so a maintainer can act in
one read, and what to cut.
pilot/cost_model.py — prices a review against published Claude and OpenAI rates
(fetched 2026-08-18). Prompt sizes are measured from the factory files rather
than guessed; per-tier workloads come from the tiering budgets. The model is
explicit about the thing that actually dominates an agent loop: the whole
conversation is resent every step, so caching moves ~2.3x of the bill.
Blended over a 5/35/55/5 mix with caching on: ~$0.61/PR on Opus 5 or GPT-5.6
Sol, ~$0.24 on Sonnet 5 or Terra, ~$0.12 on Haiku 4.5, ~$0.02 on Luna. At 350
PRs/month that's ~$212 / ~$85 / ~$43 / ~$8.50.
Tests: 101 -> 122.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B11e8TZZxJyzHW7jj7KWUN
The reviewer runs an opencode agent with `bash: "*": allow` over a checkout of
the PR author's branch, and the pod holds a Gitea Write credential. Those two
facts had no wall between them.
Security
- _build_env now allow-lists the subprocess environment instead of inheriting
it, so PRAGENT_BOT_TOKEN and WEBHOOK_SECRET never reach the agent. This was
the live hole: a PR body or an AGENTS.md could ask the agent to `curl` the
token out, and it had both the value and the tool.
- sanitize_workdir deletes author-controlled agent-instruction files from the
checkout before opencode starts (AGENTS.md at any depth, CLAUDE.md,
.cursorrules, a repo opencode.json/.opencode, copilot-instructions.md).
opencode loads nested AGENTS.md as instructions, so a PR could otherwise ship
its own system prompt. They are still reviewed, as data.
- The brief fences PR title/body and diff in --- UNTRUSTED --- markers under a
trust-boundary preamble; the pragent agent, the three lens subagents and the
review-methodology skill now treat injection attempts as a critical finding
to report rather than an instruction to obey.
- .pr-review.json is read from the PR's base branch, not the head sha. Its
`instructions` field is spliced into the reviewer's prompt, so head-ref
reading let any author rewrite the reviewer's rules. Fields are length-capped.
- Untar rejects escaping symlinks, parent traversal, and writes through a
planted symlink (tar-slip).
- The image runs as uid 10001 instead of root.
Robustness
- Bounded review concurrency (PRAGENT_MAX_CONCURRENT_REVIEWS, default 2). Each
review forks an opencode process; a thread per delivery was a fork bomb on a
burst of labels or Gitea retries.
- An in-flight (repo, index, sha) claim closes the check-then-act race in the
sha-marker dedupe, where two deliveries a second apart both read "not yet
reviewed" and both posted.
- Request bodies are capped before being read into memory.
Correctness
- parse_diff_anchors counts a whitespace-stripped blank context line. Skipping
it desynced the new-line counter for the rest of the hunk and silently
misplaced every later inline comment in that file.
- post_inline_review's body-only fallback folds the anchored findings into the
body. It previously posted a summary saying "N inline comment(s) below" with
no comments and no findings — losing them all on the one path that matters.
- fetch_pr_diff's files-endpoint fallback emits real a// b/ prefixes (so
changed_files and the anchor parser work on it) and reports both HTTP statuses
in its error instead of the same one twice.
- The CI workflow template pins PRAGENT_ENGINE=ollama; review_pr defaults to
opencode, which does not exist on a Gitea Actions runner.
Tests: 68 -> 101, covering each of the above.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B11e8TZZxJyzHW7jj7KWUN
- pilot/webhook_server.py: stdlib HTTP receiver. HMAC-verifies X-Gitea-Signature,
gates on pull_request action + AI-REVIEW label, runs review_pr in a background
thread (responds 202 immediately so Gitea's delivery timeout never fires).
Accepts both GitHub-style (labeled/synchronize) and Gitea event-type-style
(label_updated/synchronized) action names.
- pilot/ai_review.py: extract review_pr() core so both the CI run() and the
webhook server share one review path. run() is now an env-driven wrapper.
- pilot/README-webhook.md: architecture, onboarding, one-time per-owner
user-webhook setup, the Gitea 1.26.1 system-webhook bug, the SSRF
ALLOWED_HOST_LIST change, K8s deploy + script-update recipe.
- README.md + design doc: note the webhook service as the preferred delivery
path (partially reverses 'central webhook = non-goal', pilot only).
Gitea 1.26.1 system webhooks broken (POST /admin/hooks -> 201 but never
persists); user-level webhooks (one per repo-owner) are the working fallback.
Gitea SSRF allow-list blocks in-cluster webhook targets by default; required a
scoped [webhook] ALLOWED_HOST_LIST addition + gitea restart.
E2E verified 2026-08-17: pragent-bot reviewed gitea_admin/pragent PR #2 and
masi/portfolio PR #3 via the webhook service (glm-5.2:cloud).
Co-Authored-By: Claude <noreply@anthropic.com>
Ships a working pragent pilot ahead of the framework build (design doc
deferred). Single stdlib-only reviewer script fetched at runtime by a per-repo
Gitea Action; reviews fire only on PRs with the AI-REVIEW label; model is
glm-5.2:cloud via the on-network headroom proxy; fail-open.
- pilot/ai_review.py: fetch PR diff, call model, post review as pragent-bot
- pilot/workflow-template.yml: per-repo Gitea Action gated on AI-REVIEW label
- pilot/README.md: onboarding (bot collaborator + secret + workflow + label)
- tests/pilot/test_ai_review.py: 15 unit tests for pure helpers (no network)
- README/design doc: note pilot is the bootstrap; framework build deferred
Co-Authored-By: Claude <noreply@anthropic.com>
Claude