gitea_admin/pragent
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

refactor: organize pilot modules and tests #16

Merged
masi merged 4 commits from refactor/split-pilot-modules into main 2026-09-01 01:22:12 +00:00
Conversation 2 Commits 4 Files Changed 71
+8174 -8039
66 changed files with 8174 additions and 8039 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Showing only changes of commit 7a510a926d - Show all commits
docs/architecture.md
+5 -5
View File
@@ -12,7 +12,7 @@ webhook_server ── trusted base config ──► review_config
│ bounded worker
▼
review_pr facade/orchestrator
├── gitea_client fetch diff, reviews, config; publish review
├── entrypoints/gitea fetch diff, reviews, config; publish review
├── diff_compress reduce prompt context
├── opencode_review isolated checkout + agent execution
│ └── model / repo factory (.opencode)
@@ -24,20 +24,20 @@ review_pr facade/orchestrator
## Seams and responsibilities
The external seam is `ai_review.review_pr(...)`: one call represents one review
attempt and returns success/skip status. The module is retained as a facade for
the CI and webhook callers that already import it.
attempt and returns success/skip status. The top-level module is a compatibility
shim; the implementation lives in `review/ai_review.py`.
The internal seams are deliberately narrower:
- `review_config.repo_enabled(get, ...)` owns the security-sensitive opt-in
decision. It receives a transport function, so malformed configuration and
failure behavior are deterministic in tests.
- `gitea_client.request()` and `GiteaClient` own HTTP authentication, JSON
- `entrypoints/gitea.request()` and `GiteaClient` own HTTP authentication, JSON
request encoding, timeout, and Gitea URL construction.
- `model_client.complete()` owns the legacy Anthropic-compatible request shape.
`opencode_review` is the preferred agent adapter and keeps Gitea I/O out of
the autonomous process.
- `diff_compress`, finding parsing, config filtering, and rendering remain
- `review/diff`, finding parsing, config filtering, and rendering remain
pure transformations. Their callers do not need to know how model or Gitea
transport works.
- `langfuse_trace` is an optional sink. It is fail-open and cannot change the
pilot/README.md
+17 -15
View File
@@ -23,21 +23,22 @@ as a PR comment and does not block CI.
| Module | Responsibility |
|---|---|
| `webhook_server.py` | HTTP ingress, signature verification, opt-in gate, concurrency |
| `review_config.py` | Trusted base-branch opt-in policy; transport injected for tests |
| `gitea_client.py` | HTTP transport adapter and repository-scoped client |
| `ai_review.py` | Compatibility facade and review orchestration |
| `model_client.py` | Anthropic-compatible model adapter and response text extraction |
| `opencode_review.py` | Hostile-checkout containment and agent execution |
| `diff_compress.py` | Diff compression and prior-review extraction |
| `feedback*.py` | Feedback persistence, harvesting, analysis, and Langfuse scores |
| `langfuse_trace.py` | Fail-open Langfuse ingestion and cost metadata |
| `cost_model.py` | Provider price catalog and equivalent-cost calculations |
| `eval_*.py` | Dataset bootstrap, evaluators, and behavioral scoring |
| `entrypoints/webhook.py` | HTTP ingress, signature verification, opt-in gate, concurrency |
| `review/config.py` | Trusted base-branch opt-in policy; transport injected for tests |
| `entrypoints/gitea.py` | HTTP transport adapter and repository-scoped client |
| `review/ai_review.py` | Review orchestration implementation |
| `ai_review.py` | Compatibility shim for existing imports and CI execution |
| `review/model.py` | Anthropic-compatible model adapter and response text extraction |
| `review/opencode.py` | Hostile-checkout containment and agent execution |
| `review/diff.py` | Diff compression and prior-review extraction |
| `feedback/*.py` | Feedback persistence, harvesting, analysis, and Langfuse scores |
| `observability/langfuse.py` | Fail-open Langfuse ingestion and cost metadata |
| `observability/cost.py` | Provider price catalog and equivalent-cost calculations |
| `evaluation/*.py` | Dataset bootstrap, evaluators, and behavioral scoring |
`ai_review.py` remains the stable import surface for existing workflow and
webhook deployments. New code should put policy, adapters, and pure transforms
in the focused modules above rather than adding unrelated functions there.
The top-level `.py` files are intentionally thin compatibility shims. They keep
existing workflow commands and imports stable while the implementations live in
the focused packages above. New code belongs in those packages, not in a shim.
## Onboard a repository
@@ -72,5 +73,6 @@ security, and webhook registration details.
python3 -m pytest tests -q
```
Tests use mocked transports and local fixtures. They do not require Gitea,
Tests are grouped under `tests/pilot/*_tests/`, matching the source domains.
They use mocked transports and local fixtures and do not require Gitea,
Langfuse, a model endpoint, or network access.
pilot/ai_review.py
+5 -2374
View File
File diff suppressed because it is too large Load Diff
pilot/cost_model.py
+6 -433
View File
@@ -1,434 +1,7 @@
#!/usr/bin/env python3
"""pragent pilot — per-review cost model.
Answers "what would this cost on a paid API?" for the pilot's agent loop. The
pilot currently runs on `glm-5.2:cloud` through the on-network headroom proxy at
no per-token charge, so every review's measured usage is *free but real*: it
tells us exactly what the same work would bill on Claude or GPT.
The model is deliberately explicit rather than a single fudge factor, because
the dominant cost in an agent loop is not the diff — it is **resending the
conversation on every step**. A 12-step review re-reads its own prefix 12 times.
Prompt caching is what makes that affordable, and whether caching is on changes
the answer by ~3x, so it's a parameter, not an assumption.
Token accounting per review:
step 1 input = prefix + brief
step k input = prefix + brief + (tool results accumulated through k-1)
total input = sum over steps
cached = the prefix + brief part of steps 2..n (stable, byte-identical)
uncached = step 1 in full + the growing tool-result tail
`prefix` = system + tool schemas + agent definition + the skills this tier loads.
Those sizes are MEASURED from the files in this repo (see `measure_factory`),
not guessed. Diff size, file reads, and step count are per-tier assumptions from
the `attention-tiering` skill's budgets — override them on the CLI to fit your
own repos.
Prices are per million tokens, from the providers' published pricing pages
(fetched 2026-08-18 — re-check before quoting):
https://platform.claude.com/docs/en/about-claude/pricing
https://developers.openai.com/api/docs/pricing
Usage:
python3 pilot/cost_model.py # all tiers, all models
python3 pilot/cost_model.py --prs-per-month 350
python3 pilot/cost_model.py --mix 5,35,55,5 # trivial,lite,full,oversized %
python3 pilot/cost_model.py --no-cache # what caching is worth
"""
from __future__ import annotations
import argparse
import os
from dataclasses import dataclass, field
CHARS_PER_TOKEN = 4 # English prose/code rule of thumb; ±15% is normal
# ---------------------------------------------------------------------------
# Prices — USD per million tokens
# ---------------------------------------------------------------------------
@dataclass(frozen=True)
class Price:
"""Per-MTok prices. `cache_write` and `cache_read` are absolute rates, not
multipliers, so providers with different cache economics stay comparable.
`provider` is the opencode provider name (`headroom`, `vllm-qwen38`, ...). It
doubles as the dispatch key for `.pr-review.json:model` overrides — when
a per-repo override is set, `_resolve_display_model` returns
`f"{provider}/{key}"` so the opencode subprocess routes correctly.
Default `headroom` preserved for the existing roster."""
name: str
input: float
output: float
cache_write: float
cache_read: float
provider: str = "headroom"
@property
def batch_input(self) -> float:
return self.input / 2
@property
def batch_output(self) -> float:
return self.output / 2
# Anthropic: cache write = 1.25x input (5-minute TTL), cache read = 0.1x input.
# OpenAI: cached input is a published rate (0.1x input); there is no separate
# cache-write charge — writes are billed as ordinary input.
PRICES: dict[str, Price] = {
"claude-opus-5": Price("Claude Opus 5", 5.00, 25.00, 6.25, 0.50),
"claude-sonnet-5": Price("Claude Sonnet 5", 2.00, 10.00, 2.50, 0.20),
"claude-haiku-4-5": Price("Claude Haiku 4.5", 1.00, 5.00, 1.25, 0.10),
"gpt-5.6-sol": Price("GPT-5.6 Sol", 5.00, 30.00, 5.00, 0.50),
"gpt-5.6-terra": Price("GPT-5.6 Terra", 2.00, 12.00, 2.00, 0.20),
"gpt-5.6-luna": Price("GPT-5.6 Luna", 0.20, 1.20, 0.20, 0.02),
# OpenAI — cached_input 0.1x, no separate cache_write
"gpt-5": Price("GPT-5", 1.25, 10.00, 1.25, 0.125),
"gpt-5-mini": Price("GPT-5 mini", 0.25, 2.00, 0.25, 0.025),
# Google Gemini — cache_write = input
"gemini-2.5-pro": Price("Gemini 2.5 Pro", 1.875, 12.50, 1.875, 0.1875),
"gemini-2.5-flash": Price("Gemini 2.5 Flash", 0.30, 2.50, 0.30, 0.03),
# xAI Grok — cache_write = input
"grok-4.5": Price("Grok 4.5", 2.00, 6.00, 2.00, 0.30),
"grok-4.3": Price("Grok 4.3", 1.25, 2.50, 1.25, 0.20),
# Self-hosted — AI workstation RTX 3090, vLLM + DFlash2 spec-decode, no
# per-token charge. provider="vllm-qwen38" so the opencode subprocess
# routes via the matching provider block in opencode.json
# (baseURL=http://192.168.1.79:18020/v1). Equivalent-cost column reads $0
# — the cost-comparison signal is that the same work would bill $X on a
# paid model.
"qwen3.8-27b": Price("Qwen3.8-27B (vLLM, MTP, 150k ctx)", 0.0, 0.0, 0.0, 0.0, provider="vllm-qwen38"),
}
# ---------------------------------------------------------------------------
# Factory footprint — measured from this repo
# ---------------------------------------------------------------------------
_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
# Skills the primary always loads, and the conditional ones per tier. Mirrors
# the load table in .opencode/agents/pragent.md.
ALWAYS_SKILLS = ("review-methodology", "findings-schema", "attention-tiering")
TIER_SKILLS: dict[str, tuple[str, ...]] = {
"trivial": (),
"lite": ("comment-craft",),
"full": ("linter-playbook", "security-lens", "comment-craft"),
"oversized": ("linter-playbook", "security-lens", "comment-craft", "malicious-change"),
}
# opencode's own system prompt + the JSON tool schemas it sends (read, grep,
# glob, bash, webfetch, skill, task, …). Not in this repo, so this is the one
# component that is an estimate rather than a measurement.
HARNESS_TOKENS = 3500
def _tok(path: str) -> int:
try:
with open(path, "rb") as f:
return len(f.read()) // CHARS_PER_TOKEN
except OSError:
return 0
def measure_factory(root: str = _ROOT) -> dict[str, int]:
"""Token size of each prompt component, measured from the files on disk."""
out = {"agent": _tok(os.path.join(root, ".opencode", "agents", "pragent.md"))}
skills_dir = os.path.join(root, ".opencode", "skills")
if os.path.isdir(skills_dir):
for name in sorted(os.listdir(skills_dir)):
p = os.path.join(skills_dir, name, "SKILL.md")
if os.path.isfile(p):
out[f"skill:{name}"] = _tok(p)
for lens in ("security", "tests", "perf"):
out[f"subagent:{lens}"] = _tok(os.path.join(root, ".opencode", "agents", f"{lens}.md"))
return out
def prefix_tokens(tier: str, factory: dict[str, int]) -> int:
"""Stable per-step prefix: harness + agent definition + loaded skills."""
total = HARNESS_TOKENS + factory.get("agent", 0)
for s in ALWAYS_SKILLS + TIER_SKILLS.get(tier, ()):
total += factory.get(f"skill:{s}", 0)
return total
# ---------------------------------------------------------------------------
# Per-tier workload assumptions
# ---------------------------------------------------------------------------
@dataclass
class Tier:
"""One tier's workload. Defaults follow the `attention-tiering` budgets."""
name: str
diff_tokens: int # the diff as it lands in the brief
steps: int # model turns in the agent loop
file_reads: int # files read from the checkout
tokens_per_read: int # avg tokens returned per read/grep/linter result
output_tokens: int # assistant output across all steps (incl. reasoning)
subagents: int = 0 # lens subagents spawned
brief_fixed: int = 600 # brief template + PR meta + prior reviews
share: float = 0.0 # fraction of PRs at this tier (for the monthly mix)
_factory: dict = field(default_factory=dict, repr=False)
DEFAULT_TIERS = [
# diff_tok steps reads tok/read output subs share
Tier("trivial", 400, 2, 0, 0, 600, 0, share=0.05),
Tier("lite", 1500, 6, 4, 2000, 2500, 0, share=0.35),
Tier("full", 6000, 24, 20, 3300, 12000, 0, share=0.55),
Tier("oversized", 25000, 35, 30, 3500, 20000, 2, share=0.05),
]
# ---------------------------------------------------------------------------
# Observed runs — the calibration anchor
# ---------------------------------------------------------------------------
# Real usage reported by opencode's step_finish events. Keep this list
# events. Keep this list append-only: it is the only thing separating this model
# from a guess, and the first entry corrected the tier assumptions by ~15x.
OBSERVED_RUNS: list[dict] = [
{
"label": "internal/hardening-PR (16 files, 1020 insertions / 91 deletions)",
"date": "2026-08-18",
"tier": "full",
"diff_tokens": 17_600, # 16 files, 1020 insertions / 91 deletions
"steps": 28,
"duration_s": 348.3,
"input": 2_071_025,
"output": 17_303,
"cache_read": 0,
"cache_write": 0,
"subagents": 0,
},
{
"label": "internal/hardening-PR (same PR, two commits later)",
"date": "2026-08-18",
"tier": "full",
"diff_tokens": 21_000, # same PR, two commits later
"steps": 31,
"duration_s": 189.8,
"input": 2_213_077,
"output": 9_058,
"cache_read": 0,
"cache_write": 0,
"subagents": 0,
},
# A third run of the same PR (sha 2613b3e, 31 steps' worth of work in 330s)
# ended without a parseable findings block and so reported no usage at all —
# the reason `salvage_summary` now keeps the usage section on that path.
]
def observed_usage(run: dict) -> Usage:
return Usage(
uncached_input=run["input"] - run.get("cache_read", 0),
cached_input=run.get("cache_read", 0),
cache_writes=run.get("cache_write", 0),
output=run["output"],
)
@dataclass
class Usage:
uncached_input: int = 0
cached_input: int = 0
cache_writes: int = 0
output: int = 0
@property
def total_input(self) -> int:
return self.uncached_input + self.cached_input
def tier_usage(tier: Tier, factory: dict[str, int], caching: bool = True) -> Usage:
"""Token usage for one review at this tier.
The agent loop resends the whole conversation each step. The prefix + brief
are byte-identical across steps, so with caching they are written once and
read back on every later step; the tool-result tail grows and is charged as
ordinary input. Without caching every step pays full input price for
everything it has accumulated — which is the quadratic term that makes an
uncached agent loop expensive.
"""
prefix = prefix_tokens(tier.name, factory)
stable = prefix + tier.brief_fixed + tier.diff_tokens
# Tool results arrive one per step, after the first.
result_steps = max(0, min(tier.file_reads, tier.steps - 1))
per_result = tier.tokens_per_read
u = Usage(output=tier.output_tokens)
if caching:
u.cache_writes = stable
u.cached_input = stable * max(0, tier.steps - 1)
u.uncached_input = 0
else:
u.uncached_input = stable * tier.steps
# The growing tail of tool results: a result produced at step i is resent on
# every step after it, so it is counted (steps - i) times.
tail = 0
for i in range(1, result_steps + 1):
tail += per_result * (tier.steps - i)
u.uncached_input += tail
# Each lens subagent is its own loop: its own prefix, the diff, a few reads.
for _ in range(tier.subagents):
sub_prefix = HARNESS_TOKENS + factory.get("subagent:security", 600)
sub_stable = sub_prefix + tier.diff_tokens
sub_steps = 6
if caching:
u.cache_writes += sub_stable
u.cached_input += sub_stable * (sub_steps - 1)
else:
u.uncached_input += sub_stable * sub_steps
for i in range(1, 4):
u.uncached_input += per_result * (sub_steps - i)
u.output += 1500
return u
def cost(u: Usage, price: Price, batch: bool = False) -> float:
"""USD for one review's usage at these prices."""
inp = price.batch_input if batch else price.input
out = price.batch_output if batch else price.output
cw = price.cache_write / 2 if batch else price.cache_write
cr = price.cache_read / 2 if batch else price.cache_read
return (
u.uncached_input * inp
+ u.cached_input * cr
+ u.cache_writes * cw
+ u.output * out
) / 1_000_000
# ---------------------------------------------------------------------------
# Reporting
# ---------------------------------------------------------------------------
def blended_cost(tiers: list[Tier], factory: dict, price: Price, caching: bool) -> float:
"""Weighted cost of one average PR across the tier mix."""
total_share = sum(t.share for t in tiers) or 1.0
return sum(
cost(tier_usage(t, factory, caching), price) * (t.share / total_share)
for t in tiers
)
def report(tiers: list[Tier], prs_per_month: int, caching: bool, models: list[str]) -> str:
factory = measure_factory()
lines: list[str] = []
lines.append(f"Factory footprint (measured, {CHARS_PER_TOKEN} chars/token):")
for k, v in sorted(factory.items()):
lines.append(f" {k:<34} {v:>6,} tok")
lines.append(f" {'harness (opencode + tool schemas, est.)':<34} {HARNESS_TOKENS:>6,} tok")
lines.append("")
lines.append(f"Per-review tokens (prompt caching: {'on' if caching else 'OFF'})")
lines.append(f" {'tier':<11} {'prefix':>8} {'uncached':>10} {'cached':>10} {'cwrite':>8} {'output':>8}")
for t in tiers:
u = tier_usage(t, factory, caching)
lines.append(
f" {t.name:<11} {prefix_tokens(t.name, factory):>8,} {u.uncached_input:>10,} "
f"{u.cached_input:>10,} {u.cache_writes:>8,} {u.output:>8,}"
)
lines.append("")
lines.append("Cost per review (USD)")
header = f" {'model':<18}" + "".join(f"{t.name:>12}" for t in tiers) + f"{'blended':>12}"
lines.append(header)
for key in models:
p = PRICES[key]
row = f" {p.name:<18}"
for t in tiers:
row += f"{cost(tier_usage(t, factory, caching), p):>12.4f}"
row += f"{blended_cost(tiers, factory, p, caching):>12.4f}"
lines.append(row)
lines.append("")
mix = ", ".join(f"{t.name} {t.share:.0%}" for t in tiers)
lines.append(f"Monthly at {prs_per_month} PRs/month (mix: {mix})")
lines.append(f" {'model':<18} {'per PR':>10} {'per month':>12} {'batch -50%':>12}")
for key in models:
p = PRICES[key]
per_pr = blended_cost(tiers, factory, p, caching)
lines.append(
f" {p.name:<18} {per_pr:>10.4f} {per_pr * prs_per_month:>12.2f}"
f" {per_pr * prs_per_month / 2:>12.2f}"
)
lines.append("")
lines.append("Batch column applies the 50% async discount; it is shown for scale only —")
lines.append("PR review is latency-sensitive and a stateful agent loop is not batchable.")
lines.append("")
lines.append(observed_report(models))
return "\n".join(lines)
def observed_report(models: list[str]) -> str:
"""Price the runs actually measured through the opencode usage telemetry."""
if not OBSERVED_RUNS:
return "No observed runs recorded yet."
lines = ["Observed runs (measured via opencode step_finish events)"]
for run in OBSERVED_RUNS:
u = observed_usage(run)
lines.append(
f" {run['label']} — tier {run['tier']}, {run['steps']} steps, "
f"{run['duration_s']:.0f}s, {run['input']:,} in / {run['output']:,} out, "
f"cache {run['cache_read']:,} read / {run['cache_write']:,} write"
)
row = " "
for key in models:
p = PRICES[key]
row += f" {p.name}: ${cost(u, p):.2f} "
lines.append(row)
lines.append("")
lines.append(" NOTE: the pilot's headroom/glm-5.2 path reports zero cache read and zero")
lines.append(" cache write, i.e. prompt caching is NOT in play today. On a provider where")
lines.append(" it is, the stable prefix (agent + skills + brief + diff, resent every step)")
lines.append(" drops to 0.1x — worth roughly a third of the bill on a run like the one")
lines.append(" above. Budget with caching OFF until the measured cache columns are nonzero.")
return "\n".join(lines)
def main(argv: list[str] | None = None) -> int:
ap = argparse.ArgumentParser(description="pragent per-review cost model")
ap.add_argument("--prs-per-month", type=int, default=350)
ap.add_argument("--mix", default="", help="trivial,lite,full,oversized as percentages")
ap.add_argument("--no-cache", action="store_true", help="model without prompt caching")
ap.add_argument("--models", default=",".join(PRICES))
args = ap.parse_args(argv)
tiers = DEFAULT_TIERS
if args.mix:
shares = [float(x) for x in args.mix.split(",")]
if len(shares) != len(tiers):
ap.error(f"--mix needs {len(tiers)} comma-separated values")
for t, s in zip(tiers, shares):
t.share = s / 100.0
models = [m.strip() for m in args.models.split(",") if m.strip()]
unknown = [m for m in models if m not in PRICES]
if unknown:
ap.error(f"unknown model(s): {', '.join(unknown)}")
print(report(tiers, args.prs_per_month, not args.no_cache, models))
return 0
"""Compatibility import for the cost catalog."""
import importlib
import sys
_module = importlib.import_module("observability.cost")
sys.modules[__name__] = _module
if __name__ == "__main__":
raise SystemExit(main())
raise SystemExit(_module.main())
pilot/diff_compress.py
+5 -254
View File
@@ -1,254 +1,5 @@
#!/usr/bin/env python3
r"""pragent pilot — diff compression + prior-review compaction.
Two pure helpers that shrink what lands in the model prompt without losing
signal:
* ``compress_diff(diff, *, context=2)`` — re-renders a unified diff so each
hunk keeps only ``context`` unchanged lines on either side of its +/- lines.
The default 2 matches what most reviewers see on GitHub/Gitea, and is
enough to anchor every ``+``/``-`` line and give the reviewer the enclosing
statement. Wider context = more reading; narrower = less. Set
``context=0`` for +/- only, ``context=-1`` to disable entirely.
Elided context is not merely deleted: each surviving run of lines is
re-emitted as its *own* ``@@ -a,b +c,d @@`` hunk with recomputed line
numbers, so the output stays a valid unified diff whose line numbers
still describe the post-change file. ``parse_diff_anchors`` (and the
model) therefore read the same line numbers before and after compression.
* ``extract_finding_bullets(review_body)`` — pulls the lines of a prior
review that look like a pragent finding (``- 🔴 [HIGH] `path:line` — …``,
or the older ``- **[HIGH]** …`` form) and drops everything else. The model
already has the diff — repeating the prose ("this PR adds eval() — risky")
is just token burn. Bullet-only priors cut ~75% off prior-review bytes on
a typical 4-finding review.
Stdlib only. No I/O. Tolerant of malformed input — never raises.
"""
from __future__ import annotations
import re
# A real hunk header: `@@ -old[,count] +new[,count] @@[ trailing section]`.
# Captures both starts, both counts, and the trailing function-context text.
# Matching the full shape (not just a `@@` prefix) matters: a *removed* line
# whose content begins with `@@` is body, not a header.
_HUNK_RE = re.compile(
r"^@@\s+-(\d+)(?:,(\d+))?\s+\+(\d+)(?:,(\d+))?\s+@@(.*)$"
)
# Match a pragent summary-bullet line, in any of the shapes the renderer has
# emitted: `- 🔴 [HIGH] \`path:line\` — …` (current, `_severity_badge`),
# `- **[HIGH]** …` (bold, pre-badge), `- [high] …` (plain, oldest).
# Anything between the bullet marker and `[SEV]` (emoji, bold markers,
# whitespace) is tolerated — it is decoration, not signal.
_FINDING_BULLET_RE = re.compile(
r"^\s*[-*]\s*[^\w\[]*\[(?P<sev>critical|high|medium|low)\]",
re.IGNORECASE,
)
def compress_diff(diff: str, *, context: int = 2) -> tuple[str, int, int]:
"""Re-render `diff` keeping at most `context` unchanged lines around +/-.
Args:
diff: unified-diff text (what `gitea .../pulls/{n}.diff` returns).
context: max unchanged lines to keep on each side of a hunk. Use 0
for +/- only, -1 to disable compression (raw passthrough).
Returns:
`(text, original_chars, kept_chars)`. `original_chars` is the character
length of `diff` as given; `kept_chars` is the character length of
`text`. Every emitted hunk header is recomputed to match the lines
under it, so the result is a valid unified diff. Lines that are not
part of a hunk (`diff --git`, `index …`, `Binary files differ`, mode
changes) pass through verbatim.
"""
if not diff:
return diff or "", len(diff or ""), len(diff or "")
if context < 0:
return diff, len(diff), len(diff)
orig = len(diff)
lines = diff.splitlines()
out: list[str] = []
i = 0
n = len(lines)
while i < n:
m = _HUNK_RE.match(lines[i])
if m is None:
# File header, index line, binary marker, mode change, prose —
# anything outside a hunk body. Copy verbatim.
out.append(lines[i])
i += 1
continue
i += 1
body_start = i
while i < n and _is_body_line(lines[i]):
i += 1
body = lines[body_start:i]
out.extend(
_render_hunk(
body,
old_start=int(m.group(1)),
new_start=int(m.group(3)),
section=m.group(5) or "",
context=context,
)
)
text = "\n".join(out) + ("\n" if diff.endswith("\n") else "")
if not text.strip():
# Nothing survived (or the input was nothing but newlines); fall back
# to the original so the worst case is no improvement, not data loss.
return diff, orig, orig
if len(text) >= orig:
# Re-emitted hunk headers can outweigh the context they replace on a
# small, densely-changed diff. Never hand back something longer than
# what we were given.
return diff, orig, orig
return text, orig, len(text)
def _is_body_line(line: str) -> bool:
r"""True if `line` belongs to the current hunk body.
Hunk bodies contain only ` `/`+`/`-` prefixed lines and `\ No newline at
end of file`. An empty line is a context line whose trailing space was
stripped (common in mail-formatted diffs), so it counts as body too.
The check is prefix-based *and* header-aware: a removed line reading
`---` or an added line reading `+++` (YAML document separators, setext
underlines, `--` SQL comments) is body, not a file header — the previous
implementation misread those and silently dropped the rest of the hunk.
A new file section always opens with `diff --git`, which ends the body.
"""
if line == "":
return True
if line.startswith("diff --git ") or line.startswith("Index: "):
return False
if _HUNK_RE.match(line):
return False
return line[0] in " +-\\"
def _render_hunk(
body: list[str],
*,
old_start: int,
new_start: int,
section: str,
context: int,
) -> list[str]:
r"""Trim `body` to `context` unchanged lines around its +/- lines.
Each surviving run of consecutive lines is emitted as a standalone hunk
with a recomputed ``@@ -a,b +c,d @@`` header, so post-change line numbers
stay truthful. A hunk with no +/- lines at all (pure context) is dropped
entirely; ``\ No newline at end of file`` markers are dropped as noise.
Returns the rendered lines (headers included), or [] if nothing survived.
"""
# Number every body line on both sides before anything is dropped.
numbered: list[tuple[str, int, int]] = [] # (line, old_no, new_no)
old_no, new_no = old_start, new_start
for ln in body:
if ln.startswith("\\"):
continue # `\ No newline at end of file` — no signal, no numbering
kind = ln[0] if ln else " "
if kind == "+":
numbered.append((ln, -1, new_no))
new_no += 1
elif kind == "-":
numbered.append((ln, old_no, -1))
old_no += 1
else:
numbered.append((ln, old_no, new_no))
old_no += 1
new_no += 1
changed = [j for j, (ln, _, _) in enumerate(numbered) if ln[:1] in ("+", "-")]
if not changed:
return []
keep: set[int] = set()
for k in changed:
for j in range(max(0, k - context), min(len(numbered) - 1, k + context) + 1):
keep.add(j)
out: list[str] = []
for run in _consecutive_runs(sorted(keep)):
chunk = [numbered[j] for j in run]
old_count = sum(1 for ln, _, _ in chunk if ln[:1] != "+")
new_count = sum(1 for ln, _, _ in chunk if ln[:1] != "-")
# A run's start is the first line that exists on that side. When a
# side has no lines at all (pure addition / pure deletion), unified
# diff convention is `start = line before, count = 0`.
old_first = next((o for ln, o, _ in chunk if o >= 0), None)
new_first = next((nw for ln, _, nw in chunk if nw >= 0), None)
old_hdr = old_first if old_first is not None else max(chunk[0][1], 0)
new_hdr = new_first if new_first is not None else max(chunk[0][2], 0)
if old_count == 0:
old_hdr = _side_start_before(numbered, run[0], side=1)
if new_count == 0:
new_hdr = _side_start_before(numbered, run[0], side=2)
out.append(
f"@@ -{old_hdr},{old_count} +{new_hdr},{new_count} @@{section}"
)
out.extend(ln for ln, _, _ in chunk)
return out
def _side_start_before(
numbered: list[tuple[str, int, int]], idx: int, *, side: int
) -> int:
"""Line number on `side` (1=old, 2=new) just before body index `idx`.
Used for the zero-count header form (`@@ -7,0 +8,3 @@`), where unified
diff names the line the change is inserted *after*.
"""
for j in range(idx - 1, -1, -1):
no = numbered[j][side]
if no >= 0:
return no
# Nothing before it: derive from the first numbered line on that side.
for _, old_no, new_no in numbered:
no = old_no if side == 1 else new_no
if no >= 0:
return max(no - 1, 0)
return 0
def _consecutive_runs(indices: list[int]) -> list[list[int]]:
"""Group a sorted index list into runs of consecutive integers."""
runs: list[list[int]] = []
for j in indices:
if runs and j == runs[-1][-1] + 1:
runs[-1].append(j)
else:
runs.append([j])
return runs
def extract_finding_bullets(review_body: str) -> list[str]:
"""Pull the finding-bullet lines out of a prior review body.
Returns the matching lines stripped of surrounding whitespace, preserving
the rendered ``[SEV] `path:line` — problem`` shape (badge emoji and bold
markers included, whichever the renderer used). Lines that look like
bullets but carry no severity tag are dropped — the reviewer synthesizes
from the matched ones. Continuation lines (` - **Fix:** …`) are not
finding lines and are dropped with the rest of the prose.
"""
if not review_body:
return []
out = []
for line in review_body.splitlines():
if _FINDING_BULLET_RE.match(line):
out.append(line.strip())
return out
"""Compatibility import for diff transforms."""
import importlib
import sys
_module = importlib.import_module("review.diff")
sys.modules[__name__] = _module
pilot/entrypoints/__init__.py
+1
View File
@@ -0,0 +1 @@
"""Executable integration entry points."""
pilot/entrypoints/gitea.py
+46
View File
@@ -0,0 +1,46 @@
"""Gitea transport adapter.
This module owns HTTP mechanics only. Review policy, parsing, and publishing
decisions stay in the review layer so they can be tested without a network.
"""
from __future__ import annotations
import json
import urllib.error
import urllib.request
def request(
method: str,
url: str,
token: str,
body: dict | None = None,
accept: str = "application/json",
) -> tuple[int, bytes]:
headers = {"Authorization": f"token {token}", "Accept": accept}
data = None
if body is not None:
data = json.dumps(body).encode()
headers["Content-Type"] = "application/json"
req = urllib.request.Request(url, data=data, headers=headers, method=method)
try:
with urllib.request.urlopen(req, timeout=180) as response:
return response.status, response.read()
except urllib.error.HTTPError as exc:
return exc.code, exc.read()
except urllib.error.URLError as exc:
raise RuntimeError(f"network error: {exc.reason}") from exc
class GiteaClient:
"""Small adapter for repository-scoped Gitea calls."""
def __init__(self, api: str, token: str):
self.api = api.rstrip("/")
self.token = token
def get(self, path: str, accept: str = "application/json") -> tuple[int, bytes]:
return request("GET", f"{self.api}/api/v1/repos/{path}", self.token, accept=accept)
def post(self, path: str, body: dict) -> tuple[int, bytes]:
return request("POST", f"{self.api}/api/v1/repos/{path}", self.token, body)
pilot/entrypoints/webhook.py
+306
View File
@@ -0,0 +1,306 @@
#!/usr/bin/env python3
"""pragent pilot — central webhook receiver.
A stdlib-only HTTP server that Gitea posts user-webhook events to. It gates on
the PR's base ref having `.pr-review.json` with `"enabled": true`, then runs
the same review core (`ai_review.review_pr`) the CI-step pilot uses, posting
findings back as `pragent-bot`.
Per-owner setup: one Gitea **user-level webhook** per repo-owner fires for every
repo that owner has; this service filters to opted-in PRs. (Gitea 1.26.1 system
webhooks are broken — see pilot/README-webhook.md.) Onboarding a repo = add the
bot as a Write collaborator + commit a `.pr-review.json` with `"enabled": true`
on the base ref.
Stdlib only — no pip install, runs on python:3-slim with the scripts mounted.
Endpoints:
POST /webhook Gitea webhook delivery (HMAC-verified)
GET /health liveness probe
Env:
WEBHOOK_SECRET shared secret used to register the Gitea webhook (HMAC)
GITEA_API in-cluster Gitea base URL
PRAGENT_BOT_TOKEN pragent-bot access token (non-admin; must be a Write
collaborator on each reviewed repo)
OLLAMA_URL headroom proxy URL, e.g. http://model-proxy.internal:8789
OLLAMA_MODEL model id, e.g. glm-5.2:cloud
OLLAMA_MAX_TOKENS (optional) output cap, default 6000
DIFF_MAX_CHARS (optional) diff truncation cap, default 150000
WEBHOOK_PORT (optional) listen port, default 8080
PRAGENT_MAX_CONCURRENT_REVIEWS
(optional) how many reviews may run at once, default 2.
Each review forks an opencode process that checks out a
repo and runs linters, so this is the real resource knob.
PRAGENT_MAX_BODY_BYTES
(optional) request-body cap, default 10 MiB
"""
import base64
import hashlib
import hmac
import json
import os
import threading
import urllib.parse
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from ai_review import gitea_get, review_pr
from review_config import repo_enabled
try:
import feedback_harvest # optional — absent in CI-step pod, present in
# central webhook service. Harvesting is the
# collection side of the feedback loop.
except ImportError:
feedback_harvest = None
# Pull-request webhook `action` values. We fire on EVERY pull_request action
# except `closed` (no point reviewing a closed/merged PR) — the
# `.pr-review.json:enabled` gate + sha dedupe downstream make broadening safe:
# a same-sha re-fire (title edit, assignee, milestone, label toggle…) is
# skipped by `review_pr`'s dedupe. Gitea emits GitHub-style `action` names
# (`labeled`, `synchronize`) even though the `X-Gitea-Event-Type` header uses
# `label_updated` / `synchronized`.
SKIP_ACTIONS = {"closed"}
GITEA_API = os.environ.get("GITEA_API", "http://gitea-http.gitea.svc.cluster.local:3000")
BOT_TOKEN = os.environ.get("PRAGENT_BOT_TOKEN", "")
OLLAMA_URL = os.environ.get("OLLAMA_URL", "http://model-proxy.internal:8789")
OLLAMA_MODEL = os.environ.get("OLLAMA_MODEL", "glm-5.2:cloud")
OLLAMA_MAX_TOKENS = int(os.environ.get("OLLAMA_MAX_TOKENS", "8000"))
DIFF_MAX_CHARS = int(os.environ.get("DIFF_MAX_CHARS", "150000"))
WEBHOOK_SECRET = os.environ.get("WEBHOOK_SECRET", "").encode()
PORT = int(os.environ.get("WEBHOOK_PORT", "8080"))
MAX_CONCURRENT = max(1, int(os.environ.get("PRAGENT_MAX_CONCURRENT_REVIEWS", "2")))
MAX_BODY_BYTES = int(os.environ.get("PRAGENT_MAX_BODY_BYTES", str(10 * 1024 * 1024)))
# Feedback DB — SQLite mounted at PRAGENT_FEEDBACK_DB. Empty / unset =
# feedback collection disabled (CI-step path doesn't have it).
FEEDBACK_DB = os.environ.get("PRAGENT_FEEDBACK_DB", "")
# Bound on reviews running at once. Every review forks an opencode process that
# untars a repo, reads files and shells out to linters, so an unbounded thread
# per delivery is a self-inflicted fork bomb the first time someone labels ten
# PRs (or Gitea retries a burst). Queued deliveries wait here rather than pile
# onto the box; the handler has already returned 202, so nothing times out.
_review_slots = threading.Semaphore(MAX_CONCURRENT)
# Reviews currently accepted or running, keyed (repo, index, sha). The
# sha-marker dedupe in `review_pr` reads Gitea *before* posting, so two
# deliveries for the same commit in flight together both see "not yet reviewed"
# and both post — the classic check-then-act race. Common triggers are Gitea
# retries after a slow 202 response and bursty re-fires from a rapid title /
# assign / label toggle. This set closes the window inside one process.
_inflight: set[tuple[str, str, str]] = set()
_inflight_lock = threading.Lock()
def is_repo_enabled(api: str, repo: str, ref: str, token: str) -> bool:
"""True iff `.pr-review.json` on `ref` has `"enabled": true`.
Reads from the given ref (typically the PR's base ref). False on any
failure: 404, parse error, missing file, missing `enabled`, wrong type.
The bool-coerce of `.get("enabled") is True` rejects the common
gotchas (`null`, `1`, `"yes"`, missing field all yield False).
"""
return repo_enabled(gitea_get, api, repo, ref, token)
def _verify_signature(raw_body: bytes, headers) -> bool:
if not WEBHOOK_SECRET:
return False # refuse to run without a configured secret
sig_header = headers.get("X-Gitea-Signature") or headers.get("X-Forgejo-Signature")
if not sig_header:
return False
mac = hmac.new(WEBHOOK_SECRET, raw_body, hashlib.sha256).hexdigest()
return hmac.compare_digest(mac, sig_header)
def _handle_pull_request(payload: dict) -> tuple[int, str]:
"""Decide whether to review; if so, kick it off in a background thread.
Returns (status, message) to Gitea immediately — the review itself runs
async so Gitea's delivery timeout never fires and causes a retry.
"""
action = payload.get("action", "")
pr = payload.get("pull_request") or {}
repo_obj = payload.get("repository") or {}
repo = repo_obj.get("full_name") or ""
if action in SKIP_ACTIONS:
return 200, f"ignore action={action}"
if not repo:
return 400, "no repository.full_name"
index = pr.get("number")
if index is None:
return 400, "no pull_request.number"
title = pr.get("title", "") or ""
body = pr.get("body", "") or ""
head = pr.get("head") or {}
sha = head.get("sha", "") or ""
base_ref = (pr.get("base") or {}).get("ref", "") or ""
if not is_repo_enabled(GITEA_API, repo, base_ref or "", BOT_TOKEN):
return 200, f"skip (repo not opted in) action={action}"
if not BOT_TOKEN:
return 500, "PRAGENT_BOT_TOKEN not set"
key = (repo, str(index), sha)
if not _claim(key):
return 200, f"ignore (already in flight) {repo}#{index} sha={sha[:8]}"
threading.Thread(
target=_run_review,
args=(key, title, body, base_ref),
daemon=True,
).start()
return 202, f"reviewing {repo}#{index} action={action} sha={sha[:8]}"
def _claim(key: tuple[str, str, str]) -> bool:
"""Reserve (repo, index, sha) for review. False if already claimed."""
with _inflight_lock:
if key in _inflight:
return False
_inflight.add(key)
return True
def _release(key: tuple[str, str, str]) -> None:
with _inflight_lock:
_inflight.discard(key)
def _run_review(
key: tuple[str, str, str], title: str, body: str, base_ref: str
) -> None:
repo, index, sha = key
# Harvest reactions on PRIOR bot comments on this PR (best-effort —
# piggy-backs the webhook path so we don't need a separate cron).
# Disabled if feedback_harvest isn't importable (CI-step image) or
# FEEDBACK_DB isn't set.
if FEEDBACK_DB and feedback_harvest is not None:
try:
hstats = feedback_harvest.harvest_for_pr(
api=GITEA_API, token=BOT_TOKEN,
repo=repo, pr_index=int(index), db_path=FEEDBACK_DB,
)
print(
f"pragent-webhook: harvested {repo}#{index} "
f"reviews={hstats['reviews_seen']} "
f"findings={hstats['findings_seen']} "
f"reactions={hstats['reactions_recorded']}",
flush=True,
)
except Exception as e:
# Harvest must never abort a review.
print(f"pragent-webhook: harvest failed for {repo}#{index}: {e}", flush=True)
try:
with _review_slots:
ok = review_pr(
api=GITEA_API,
repo=repo,
index=index,
title=title,
body=body,
sha=sha,
token=BOT_TOKEN,
ollama_url=OLLAMA_URL,
model=OLLAMA_MODEL,
max_tokens=OLLAMA_MAX_TOKENS,
max_chars=DIFF_MAX_CHARS,
base_ref=base_ref,
)
print(f"pragent-webhook: reviewed {repo}#{index} sha={sha[:8]} ok={ok}", flush=True)
except Exception as e: # review_pr is fail-open, but guard the thread anyway
print(f"pragent-webhook: thread crashed for {repo}#{index}: {e}", flush=True)
finally:
_release(key)
class Handler(BaseHTTPRequestHandler):
def _send(self, status: int, body: str) -> None:
data = body.encode()
self.send_response(status)
self.send_header("Content-Type", "text/plain")
self.send_header("Content-Length", str(len(data)))
self.end_headers()
self.wfile.write(data)
def do_GET(self):
if self.path == "/health":
with _inflight_lock:
n = len(_inflight)
self._send(200, f"ok inflight={n} max_concurrent={MAX_CONCURRENT}")
else:
self._send(404, "not found")
def do_POST(self):
if self.path != "/webhook":
self._send(404, "not found")
return
try:
length = int(self.headers.get("Content-Length", "0") or "0")
except ValueError:
self._send(400, "bad content-length")
return
# Cap before reading: the body is read whole into memory, so an
# unbounded Content-Length is a one-request OOM.
if length < 0 or length > MAX_BODY_BYTES:
self._send(413, "payload too large")
return
raw = self.rfile.read(length) if length else b""
if len(raw) != length:
self._send(400, "truncated body")
return
if not _verify_signature(raw, self.headers):
self._send(401, "invalid signature")
return
try:
payload = json.loads(raw)
except json.JSONDecodeError:
self._send(400, "invalid json")
return
event = self.headers.get("X-Gitea-Event") or payload.get("action") or ""
if event != "pull_request":
self._send(200, f"ignore event={event}")
return
repo_full = (payload.get("repository") or {}).get("full_name")
print(
f"pragent-webhook: pull_request action={payload.get('action')} repo={repo_full}",
flush=True,
)
status, msg = _handle_pull_request(payload)
self._send(status, msg)
def log_message(self, fmt, *args):
# Keep k8s logs to our own lines (see _run_review / _send paths).
print(f"pragent-webhook: {self.address_string()} {fmt % args}", flush=True)
def main() -> int:
if not WEBHOOK_SECRET:
print("pragent-webhook: FATAL: WEBHOOK_SECRET not set", flush=True)
return 1
if not BOT_TOKEN:
print("pragent-webhook: FATAL: PRAGENT_BOT_TOKEN not set", flush=True)
return 1
server = ThreadingHTTPServer(("0.0.0.0", PORT), Handler)
print(f"pragent-webhook: listening on :{PORT} (model={OLLAMA_MODEL})", flush=True)
try:
server.serve_forever()
except KeyboardInterrupt:
pass
return 0
if __name__ == "__main__":
raise SystemExit(main())
pilot/eval_bootstrap.py
+5 -348
View File
@@ -1,350 +1,7 @@
#!/usr/bin/env python3
"""pragent pilot — one-time Langfuse project setup for evaluation.
Three jobs, each idempotent so it can be re-run after any change:
1. **Score configs.** Registers the schema for every score pragent emits
(`eval_scores.SCORE_CONFIGS` + `feedback_scores.SCORE_CONFIGS`). Without
these the scores still ingest, but nothing stops a later scorer writing
`severity_max="HIGH"` beside today's `"high"` and quietly splitting one
series into two. Configs are immutable in Langfuse — a name that already
exists is left alone rather than updated.
2. **Dataset.** Seeds `pragent-reviews` from `feedback.db`: one item per PR
the reviewer has actually run on, carrying the repo/PR/sha as input and
the findings it posted as `expectedOutput`.
Read `expectedOutput` here as "what the reviewer said last time", not "what
is correct" — no human has labelled any of it. It is a regression baseline:
re-run a candidate model over these PRs and the diff against this column is
the behaviour change. Promoting an item to real ground truth means a human
editing it after reviewing the PR, which is what the dataset view is for.
3. **Trace backfill** (`--backfill-traces`). Scores only ride along with new
reviews, so without this the charts stay empty until the next PR lands.
Every trace `langfuse_trace` has ever written already carries the finding
count, the severity histogram and the cost in its metadata, which is
everything four of the five scorers need. `dropped_findings` is absent from
historical traces and is left unscored rather than backfilled as zero.
4. **Reports** what it found, so the gap between "reviews recorded" and
"reviews with human feedback" is visible rather than assumed.
Usage:
LANGFUSE_HOST=... LANGFUSE_PUBLIC_KEY=... LANGFUSE_SECRET_KEY=... \\
python3 eval_bootstrap.py --db /data/feedback.db
"""
from __future__ import annotations
import argparse
import base64
import json
import os
import sqlite3
"""Compatibility import for evaluation bootstrap."""
import importlib
import sys
import urllib.error
import urllib.request
from datetime import datetime, timezone
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import eval_scores # noqa: E402
import feedback_scores # noqa: E402
DATASET_NAME = "pragent-reviews"
def _conf() -> tuple[str, str, str]:
host = (os.environ.get("LANGFUSE_HOST") or "").strip().rstrip("/")
pk = (os.environ.get("LANGFUSE_PUBLIC_KEY") or "").strip()
sk = (os.environ.get("LANGFUSE_SECRET_KEY") or "").strip()
if not host or not pk or not sk:
raise SystemExit("LANGFUSE_HOST / LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY must be set")
return host, pk, sk
def _call(method: str, path: str, body: dict | None = None, timeout: float = 20.0):
host, pk, sk = _conf()
auth = base64.b64encode(f"{pk}:{sk}".encode()).decode("ascii")
data = json.dumps(body).encode() if body is not None else None
req = urllib.request.Request(
host + path,
data=data,
headers={
"Content-Type": "application/json",
"Authorization": f"Basic {auth}",
"User-Agent": "pragent-pilot/1.0",
},
method=method,
)
try:
with urllib.request.urlopen(req, timeout=timeout) as resp:
raw = resp.read()
return resp.status, (json.loads(raw) if raw else None)
except urllib.error.HTTPError as e:
return e.code, e.read()[:400].decode("utf-8", "replace")
# ---------------------------------------------------------------------------
# 1. Score configs
# ---------------------------------------------------------------------------
def ensure_score_configs() -> dict:
status, existing = _call("GET", "/api/public/score-configs?limit=100")
have = set()
if status == 200 and isinstance(existing, dict):
have = {c.get("name") for c in existing.get("data", [])}
created, skipped, failed = [], [], []
for cfg in list(eval_scores.SCORE_CONFIGS) + list(feedback_scores.SCORE_CONFIGS):
if cfg["name"] in have:
skipped.append(cfg["name"])
continue
st, resp = _call("POST", "/api/public/score-configs", cfg)
if st in (200, 201):
created.append(cfg["name"])
else:
failed.append({"name": cfg["name"], "status": st, "error": resp})
return {"created": created, "already_present": skipped, "failed": failed}
# ---------------------------------------------------------------------------
# 2. Dataset from recorded reviews
# ---------------------------------------------------------------------------
def item_id(repo: str, pr) -> str:
"""A dataset-item id that survives being put in a URL path.
The obvious `{repo}#{pr}` is unusable: the UI routes items as
`/datasets/{id}/items/{item_id}`, so the `/` in `owner/repo` splits into
extra path segments and everything after the `#` is a fragment the browser
never sends. The item is created fine and then 404s when opened.
Session ids elsewhere keep the `{repo}#{pr}` form — those are never path
segments, and `feedback_scores` depends on that shape.
"""
return f"{repo.replace('/', '__')}__pr{pr}"
def _item_metadata(*, repo, pr, head_sha, reviews_run, last_seen, findings) -> dict:
"""Filterable facets for one dataset item.
Kept flat and primitive: the filter bar matches a metadata key against a
literal, so a nested object or a list is not reachable from the UI.
"""
owner, _, repo_name = str(repo).partition("/")
sevs = [str(f["severity"] or "").lower() for f in findings]
ranked = [s for s in sevs if s in eval_scores.SEVERITY_RANK]
return {
"repo": repo,
"owner": owner or repo,
"repo_name": repo_name or repo,
"pr": int(pr),
"head_sha": head_sha,
"reviews_run": reviews_run,
"last_reviewed_at": last_seen,
"last_reviewed_iso": datetime.fromtimestamp(last_seen, timezone.utc).isoformat(),
"finding_count": len(findings),
"has_findings": bool(findings),
# "none" rather than omitting the key: a filter for silent reviews needs
# something to match, and an absent key matches nothing.
"max_severity": (
max(ranked, key=lambda s: eval_scores.SEVERITY_RANK[s]) if ranked else "none"
),
# Flags that this row is the reviewer's own past output, not a human
# judgement. Filter on it before anyone treats the dataset as truth.
"labelled_by_human": False,
}
def read_review_items(db_path: str) -> list[dict]:
"""One dataset item per (repo, pr) the reviewer has run on.
Keyed on the PR rather than on each individual review row: the same PR is
re-reviewed on every push, and 113 rows over 26 PRs would make a benchmark
that is 4x redundant and weighted towards whichever PR churned most.
"""
conn = sqlite3.connect(db_path)
conn.row_factory = sqlite3.Row
try:
prs = conn.execute(
"""
SELECT repo, pr, MAX(posted_at) AS last_seen, COUNT(*) AS reviews,
MAX(head_sha) AS head_sha
FROM review GROUP BY repo, pr ORDER BY repo, pr
"""
).fetchall()
items = []
for row in prs:
findings = conn.execute(
"""
SELECT path, line, severity, problem, fix
FROM inline_finding WHERE repo = ? AND pr = ?
ORDER BY path, line
""",
(row["repo"], row["pr"]),
).fetchall()
items.append(
{
"id": item_id(row["repo"], row["pr"]),
"input": {
"repo": row["repo"],
"pr": int(row["pr"]),
"head_sha": row["head_sha"],
},
"expectedOutput": {
"findings": [dict(f) for f in findings],
"finding_count": len(findings),
},
# The UI's filter bar reads metadata and nothing else, so
# anything worth slicing on is a top-level key here even
# where it duplicates `input`. `owner` and `repo_name` are
# split out because a filter on the joined `repo` can only
# match one repo at a time, never a whole org.
"metadata": _item_metadata(
repo=row["repo"],
pr=row["pr"],
head_sha=row["head_sha"],
reviews_run=int(row["reviews"]),
last_seen=int(row["last_seen"]),
findings=findings,
),
}
)
return items
finally:
conn.close()
def ensure_dataset(items: list[dict], name: str = DATASET_NAME) -> dict:
st, _ = _call(
"POST",
"/api/public/datasets",
{
"name": name,
"description": (
"PRs the pragent pilot has reviewed, seeded from feedback.db. "
"expectedOutput is the reviewer's own prior output — a regression "
"baseline, not human-verified ground truth."
),
"metadata": {"source": "feedback.db", "seeded_by": "eval_bootstrap.py"},
},
)
# A duplicate name is fine: the dataset already exists from an earlier run.
dataset_ok = st in (200, 201, 409)
created, failed = 0, []
for item in items:
body = {
"datasetName": name,
"id": item["id"], # idempotent: same PR updates rather than duplicates
"input": item["input"],
"expectedOutput": item["expectedOutput"],
"metadata": item["metadata"],
}
ist, resp = _call("POST", "/api/public/dataset-items", body)
if ist in (200, 201):
created += 1
else:
failed.append({"item": item["id"], "status": ist, "error": resp})
return {"dataset": name, "dataset_created": dataset_ok, "items_upserted": created, "failed": failed}
# ---------------------------------------------------------------------------
# 3. Backfill scores onto traces that predate the scorers
# ---------------------------------------------------------------------------
def _synth_findings(severities: dict) -> list[dict]:
"""Rebuild a findings list from a trace's severity histogram.
Only severity matters to the scorers, and that is all the histogram kept.
Reconstructing placeholders is honest here because every scorer being
backfilled reads nothing else off a finding.
"""
out = []
for sev, count in (severities or {}).items():
out.extend({"severity": sev} for _ in range(int(count)))
return out
def backfill_traces(limit_pages: int = 20) -> dict:
import eval_scores as es
scored, skipped, events = 0, 0, []
page = 1
while page <= limit_pages:
st, resp = _call("GET", f"/api/public/traces?limit=50&page={page}&name=pr-review")
if st != 200 or not isinstance(resp, dict):
break
rows = resp.get("data") or []
if not rows:
break
for tr in rows:
meta = tr.get("metadata") or {}
severities = meta.get("severities") or {}
count = meta.get("findings")
if count is None:
skipped += 1
continue
findings = _synth_findings(severities)
# The histogram is authoritative when present; a trace that recorded
# a count but no histogram still scores its rate.
if not findings and count:
findings = [{"severity": "medium"} for _ in range(int(count))]
batch = es.build_scores(
trace_id=tr["id"],
findings=findings,
environment=tr.get("environment") or "default",
cost_usd=(tr.get("totalCost") or meta.get("provider_cost_usd")),
timestamp=tr.get("timestamp"),
comment="backfilled from trace metadata",
)
events.extend(batch)
scored += 1
page += 1
posted = False
status = None
if events:
import langfuse_trace
host, pk, sk = _conf()
# Chunked: one 2000-event POST is refused, and a partial backfill that
# reports success is worse than a slow one.
for i in range(0, len(events), 200):
status = langfuse_trace._post(host, pk, sk, events[i:i + 200], 30.0)
posted = status in (200, 201, 207)
if not posted:
break
return {"traces_scored": scored, "traces_skipped": skipped, "scores": len(events),
"posted": posted, "http_status": status}
def main() -> int:
ap = argparse.ArgumentParser(description="Bootstrap Langfuse evaluation for the pragent pilot")
ap.add_argument("--db", default=os.environ.get("PRAGENT_FEEDBACK_DB", "/data/feedback.db"))
ap.add_argument("--skip-dataset", action="store_true")
ap.add_argument("--skip-configs", action="store_true")
ap.add_argument("--backfill-traces", action="store_true",
help="score traces written before the scorers existed")
args = ap.parse_args()
out: dict = {}
if not args.skip_configs:
out["score_configs"] = ensure_score_configs()
if not args.skip_dataset:
items = read_review_items(args.db)
out["dataset"] = ensure_dataset(items)
out["dataset"]["items_read"] = len(items)
if args.backfill_traces:
out["trace_backfill"] = backfill_traces()
print(json.dumps(out, indent=2))
failed = (out.get("score_configs", {}).get("failed") or []) + (
out.get("dataset", {}).get("failed") or []
)
return 1 if failed else 0
_module = importlib.import_module("evaluation.bootstrap")
sys.modules[__name__] = _module
if __name__ == "__main__":
raise SystemExit(main())
raise SystemExit(_module.main())
pilot/eval_experiment.py
+5 -210
View File
@@ -1,212 +1,7 @@
#!/usr/bin/env python3
"""pragent pilot — populate the Experiments tab from reviews already traced.
An "experiment" in Langfuse is a dataset run: a set of (dataset item, trace)
links under one run name. The Experiments tab then shows one row per item with
its scores, and lets two runs be diffed side by side.
Nothing here re-runs the reviewer. Every PR in `pragent-reviews` has already
been reviewed, and each of those reviews left a trace carrying its findings,
cost and scores. This links what exists, which is what makes the tab useful on
day one instead of after the next N pushes.
Runs are grouped by **model** by default, because that is the comparison the
pilot actually needs to make: the same PRs reviewed by MiniMax vs whatever
replaces it, with `finding_rate` and `cost_per_finding` side by side. Group by
`none` for a single "all traces" run.
One trace per (run, item) — the most recent. A PR re-reviewed on every push has
many traces, and a dataset run is defined as one output per input; feeding it
the other five would make the per-run averages meaningless.
Note on the endpoint: `POST /api/public/dataset-run-items` is deprecated in
favour of the SDK experiment runner / OTel ingestion, and disappears in
Langfuse v4. This instance is self-hosted v3, which the deprecation notice
explicitly exempts from the cutoff date, and the pilot is stdlib-only by
design. Revisit when this deployment moves to v4.
Usage:
LANGFUSE_HOST=... LANGFUSE_PUBLIC_KEY=... LANGFUSE_SECRET_KEY=... \\
python3 eval_experiment.py --dry-run
"""
from __future__ import annotations
import argparse
import json
import os
"""Compatibility import for evaluation experiments."""
import importlib
import sys
import urllib.parse
from collections import defaultdict
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import eval_bootstrap as eb # noqa: E402
TRACE_NAME = "pr-review"
# ---------------------------------------------------------------------------
# Reading what already exists
# ---------------------------------------------------------------------------
def fetch_traces(name: str = TRACE_NAME, limit: int = 100, max_pages: int = 50) -> list[dict]:
"""Every review trace, newest first."""
out: list[dict] = []
for page in range(1, max_pages + 1):
q = urllib.parse.urlencode({"name": name, "limit": limit, "page": page})
st, body = eb._call("GET", f"/api/public/traces?{q}")
if st != 200 or not isinstance(body, dict):
raise SystemExit(f"listing traces failed: {st} {body}")
data = body.get("data") or []
out.extend(data)
meta = body.get("meta") or {}
if page * meta.get("limit", limit) >= meta.get("totalItems", 0):
break
return out
def fetch_item_ids(dataset: str) -> set[str]:
"""Ids present in the dataset, so runs never reference a missing item."""
ids: set[str] = set()
for page in range(1, 51):
q = urllib.parse.urlencode({"datasetName": dataset, "limit": 100, "page": page})
st, body = eb._call("GET", f"/api/public/dataset-items?{q}")
if st != 200 or not isinstance(body, dict):
raise SystemExit(f"listing dataset items failed: {st} {body}")
ids.update(i["id"] for i in body.get("data") or [])
meta = body.get("meta") or {}
if page * meta.get("limit", 100) >= meta.get("totalItems", 0):
break
return ids
# ---------------------------------------------------------------------------
# Grouping traces into runs
# ---------------------------------------------------------------------------
def trace_model(trace: dict) -> str:
"""The model that produced a review, from its `model:` tag."""
for tag in trace.get("tags") or []:
if tag.startswith("model:"):
return tag[len("model:"):] or "unknown"
return "unknown"
def trace_item_id(trace: dict) -> str | None:
"""The dataset item a trace belongs to, or None if it is not a PR review."""
md = trace.get("metadata") or {}
repo, pr = md.get("repo"), md.get("pr")
if not repo or pr in (None, ""):
return None
return eb.item_id(str(repo), pr)
def _sort_key(trace: dict):
return (trace.get("timestamp") or "", trace.get("id") or "")
def plan_runs(traces: list[dict], known_items: set[str], group_by: str = "model") -> dict:
"""Map run name -> {item id: trace}, keeping only the newest trace per item.
Traces whose PR is not in the dataset are dropped: `feedback.db` is the
source for both, but a review can be traced without its row landing (the
posting step can fail after the model ran), and a run item pointing at a
non-existent dataset item is rejected.
"""
runs: dict[str, dict[str, dict]] = defaultdict(dict)
skipped_no_item, skipped_unknown = 0, 0
for tr in traces:
iid = trace_item_id(tr)
if iid is None:
skipped_unknown += 1
continue
if iid not in known_items:
skipped_no_item += 1
continue
run = "all-traces" if group_by == "none" else trace_model(tr)
prev = runs[run].get(iid)
if prev is None or _sort_key(tr) > _sort_key(prev):
runs[run][iid] = tr
return {
"runs": dict(runs),
"skipped_not_in_dataset": skipped_no_item,
"skipped_not_a_review": skipped_unknown,
}
def run_name(prefix: str, key: str) -> str:
return f"{prefix}-{key}" if prefix else key
# ---------------------------------------------------------------------------
# Writing the runs
# ---------------------------------------------------------------------------
def create_run(name: str, items: dict[str, dict], description: str = "") -> dict:
"""Link each (item, trace) pair into the named run. Idempotent per pair."""
created, failed = 0, []
for iid, tr in sorted(items.items()):
md = tr.get("metadata") or {}
body = {
"runName": name,
"runDescription": description,
"datasetItemId": iid,
"traceId": tr["id"],
"metadata": {
"model": trace_model(tr),
"engine": md.get("engine"),
"findings": md.get("findings"),
"duration_s": md.get("duration_s"),
"cost_basis": md.get("cost_basis"),
"linked_by": "eval_experiment.py",
},
}
st, resp = eb._call("POST", "/api/public/dataset-run-items", body)
if st in (200, 201):
created += 1
else:
failed.append({"item": iid, "status": st, "error": resp})
return {"run": name, "items_linked": created, "failed": failed}
def main(argv: list[str] | None = None) -> int:
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--dataset", default=eb.DATASET_NAME)
ap.add_argument("--group-by", choices=("model", "none"), default="model")
ap.add_argument("--prefix", default="baseline",
help="run name prefix; '' for the bare group key")
ap.add_argument("--dry-run", action="store_true")
args = ap.parse_args(argv)
traces = fetch_traces()
items = fetch_item_ids(args.dataset)
plan = plan_runs(traces, items, group_by=args.group_by)
report = {
"traces_read": len(traces),
"dataset_items": len(items),
"skipped_not_in_dataset": plan["skipped_not_in_dataset"],
"skipped_not_a_review": plan["skipped_not_a_review"],
"runs": {},
}
for key, mapping in sorted(plan["runs"].items()):
name = run_name(args.prefix, key)
if args.dry_run:
report["runs"][name] = {"items_would_link": len(mapping)}
continue
report["runs"][name] = create_run(
name,
mapping,
description=(
"Reviews already run by the pilot, linked after the fact. "
"Scores come from the traces; expectedOutput is the reviewer's "
"own prior output, not human-verified ground truth."
),
)
report["dry_run"] = args.dry_run
print(json.dumps(report, indent=2))
return 0
_module = importlib.import_module("evaluation.experiment")
sys.modules[__name__] = _module
if __name__ == "__main__":
raise SystemExit(main())
raise SystemExit(_module.main())
pilot/eval_judges.py
+5 -312
View File
@@ -1,314 +1,7 @@
#!/usr/bin/env python3
"""pragent pilot — LLM-as-a-judge evaluators for the reviewer.
The deterministic scorers in `eval_scores.py` measure *behaviour*: how many
findings, how severe, how much they cost. None of them can say whether a
finding was any good. With no human labels in `feedback.db`, a judge is the
only thing that can — so these two ask the questions that need no ground truth,
only the review itself:
`finding_actionability` — is each finding concrete enough to act on? A
reviewer that says "consider improving error handling" at file level is
indistinguishable from a useful one by finding count alone. This is the
failure mode a cheap model degrades into first.
`review_self_consistency` — does the summary agree with the findings it
posted? Claiming "no issues found" above a list of two criticals, or
describing a problem in prose that never became a finding, is a defect the
reviewer can commit entirely on its own.
Neither judge is asked whether a finding is *correct*. That needs the diff,
which these traces do not carry, and a judge asked to rule on correctness from
a summary alone will confabulate. Accuracy stays an open question until humans
start labelling — which is what `feedback_scores.py` is there to capture.
**The judge is a different model from the reviewer.** The reviewer runs
MiniMax-M2.7; the judge runs kimi-k2.7-code through the same headroom hub. A
model grading its own output agrees with itself for reasons that have nothing
to do with quality.
Evaluators score *observations*, and their variable mapping reads the
observation's own input/output — which is why `langfuse_trace` now writes the
review onto the generation and not just onto the trace.
Usage:
LANGFUSE_HOST=... LANGFUSE_PUBLIC_KEY=... LANGFUSE_SECRET_KEY=... \\
python3 eval_judges.py --dry-run
"""
from __future__ import annotations
import argparse
import json
import os
"""Compatibility import for evaluation judges."""
import importlib
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import eval_bootstrap as eb # noqa: E402
# The headroom hub in front of the local Ollama, plus a small pass-through
# proxy (`judge-proxy` on 8802) that patches every `thinking` content block
# to carry the `signature` field Langfuse's Anthropic adapter requires. The
# underlying model is kimi-k2.7-code through the hub on 8790; the proxy fixes
# the shape so Mastra's Zod parse stops failing.
JUDGE_PROVIDER = "headroom-ollama"
JUDGE_BASE_URL = os.environ.get("PRAGENT_JUDGE_BASE_URL", "http://100.74.17.70:8802")
JUDGE_API_KEY = os.environ.get("PRAGENT_JUDGE_API_KEY", "ollama")
JUDGE_MODEL = os.environ.get("PRAGENT_JUDGE_MODEL", "kimi-k2.7-code:cloud")
# The trace names this project emits (`pr-review` on the trace, `opencode-review`
# on the generation). Filter on `traceName` rather than observation `name` — the
# observation-rule schema only exposes `traceName` as a stringOptions column, and
# every observation inside these traces is the review itself, so the narrowness
# is the same.
REVIEW_TRACE_NAMES = ["pr-review", "opencode-review"]
def _model_config() -> dict:
return {"provider": JUDGE_PROVIDER, "model": JUDGE_MODEL}
JUDGES = [
{
"name": "finding_actionability",
"prompt": (
"You are auditing the output of an automated code reviewer.\n\n"
"PR under review:\n{{input}}\n\n"
"What the reviewer produced:\n{{output}}\n\n"
"Rate how ACTIONABLE the findings are, from 0 to 1. A finding is "
"actionable when a developer could act on it without asking a "
"follow-up question: it points at a specific location, names a "
"concrete problem, and proposes a fix that could be applied.\n\n"
"Score 1.0 when every finding is specific and fixable. Score around "
"0.5 when findings identify a real area but leave the developer to "
"work out what to change. Score near 0.0 when findings are generic "
"advice that would apply to almost any pull request.\n\n"
"Judge only specificity and actionability. You cannot see the diff, "
"so do NOT attempt to judge whether a finding is factually correct, "
"and do not penalise a finding for being one you cannot verify.\n\n"
"If the reviewer reported no findings at all, return 1.0 and say in "
"your reasoning that there was nothing to judge — a silent review is "
"measured by finding_rate, not here."
),
"outputDefinition": {
"dataType": "NUMERIC",
"minValue": 0,
"maxValue": 1,
"reasoning": {
"description": (
"Name the least actionable finding and say what it would "
"need in order to be acted on."
)
},
"score": {"description": "0 = generic advice, 1 = every finding is specific and fixable."},
},
},
{
"name": "review_self_consistency",
"prompt": (
"You are auditing the output of an automated code reviewer.\n\n"
"PR under review:\n{{input}}\n\n"
"What the reviewer produced:\n{{output}}\n\n"
"The output contains a prose `summary` and a list of `findings`. "
"Decide whether the summary is CONSISTENT with the findings.\n\n"
"Inconsistent means, for example: the summary says no issues were "
"found while findings are listed; the summary describes a problem "
"that never became a finding; the summary characterises the severity "
"of the findings in a way the findings themselves contradict; or the "
"summary refers to files that appear in no finding and in no part of "
"the PR description.\n\n"
"A summary that adds context beyond the findings is NOT inconsistent "
"as long as nothing in it contradicts them. A review that found "
"nothing and says so is consistent.\n\n"
"You cannot see the diff. Judge the summary against the findings and "
"the PR title only — never against what you imagine the code does."
),
"outputDefinition": {
"dataType": "BOOLEAN",
"reasoning": {
"description": "Quote the part of the summary that conflicts with the findings, if any."
},
"score": {"description": "true = summary agrees with the findings, false = it contradicts them."},
},
},
]
# Both judges read the observation's own input/output.
MAPPING = [
{"variable": "input", "source": "input"},
{"variable": "output", "source": "output"},
]
# ---------------------------------------------------------------------------
# LLM connection
# ---------------------------------------------------------------------------
def ensure_llm_connection() -> dict:
"""Point the project at the judge model. Upserted on `provider`."""
body = {
"provider": JUDGE_PROVIDER,
"adapter": "anthropic",
"baseURL": JUDGE_BASE_URL,
"secretKey": JUDGE_API_KEY,
"customModels": [JUDGE_MODEL],
# The hub serves two local models and none of Anthropic's, so the
# default catalogue would be a list of models that all fail on use.
"withDefaultModels": False,
}
st, resp = eb._call("PUT", "/api/public/llm-connections", body)
return {"status": st, "ok": st in (200, 201), "provider": JUDGE_PROVIDER,
"error": None if st in (200, 201) else resp}
# ---------------------------------------------------------------------------
# Evaluators
# ---------------------------------------------------------------------------
def existing_evaluators() -> dict[str, str]:
"""name -> id for evaluators already in the project."""
out: dict[str, str] = {}
st, body = eb._call("GET", "/api/public/unstable/evaluators?limit=100")
if st == 200 and isinstance(body, dict):
for ev in body.get("data") or []:
out[ev.get("name")] = ev.get("id")
return out
def ensure_evaluators() -> dict:
"""Create each judge if no version exists for the name yet.
POST /evaluators with a name that already exists creates a new version, not
a no-op — re-running this script would pile up versions until the page
listing them is unreadable. Skip when an evaluator of that name is present.
"""
created, skipped, failed = {}, [], []
existing = set(existing_evaluators())
for judge in JUDGES:
if judge["name"] in existing:
skipped.append(judge["name"])
continue
body = {
"type": "llm_as_judge",
"name": judge["name"],
"prompt": judge["prompt"],
"outputDefinition": judge["outputDefinition"],
"modelConfig": _model_config(),
}
st, resp = eb._call("POST", "/api/public/unstable/evaluators", body, timeout=60.0)
if st in (200, 201) and isinstance(resp, dict):
created[judge["name"]] = resp.get("id")
else:
failed.append({"name": judge["name"], "status": st, "error": resp})
return {"created": created, "skipped": skipped, "failed": failed}
# ---------------------------------------------------------------------------
# Rules — what gets judged, and how often
# ---------------------------------------------------------------------------
def rule_body(name: str, judge_name: str, sampling: float) -> dict:
"""POST /evaluation-rules shape for an LLM-as-judge trace rule.
Target is `trace` rather than `observation` on purpose: the standard
`/api/public/ingestion` path that ships review traces here feeds only
the trace-upsert queue, and `evalService.createEvalJobs` only creates
jobs for `targetObject ∈ {TRACE, DATASET}`. Observation rules are
triggered exclusively from the OTel ingestion pipeline, which this
pilot does not use. A trace rule reads the trace's own input/output —
`langfuse_trace` already writes `_review_input`/`_review_output` onto
the trace body for exactly this reason.
Mapping is required at both the rule root (server validates it there)
and inside `evaluator` (the API echoes it back).
"""
return {
"name": name,
"enabled": True,
"target": "trace",
"sampling": sampling,
"filter": [
{"column": "traceName", "operator": "any of",
"value": REVIEW_TRACE_NAMES, "type": "stringOptions"},
],
"evaluator": {
"name": judge_name,
"scope": "project",
"variableMapping": MAPPING,
},
"mapping": MAPPING,
}
def ensure_rules(evaluator_ids: dict[str, str], sampling: float) -> dict:
"""Idempotent: existing rules with the same name are skipped, not duplicated.
The API has no `name`-keyed upsert; the convention is to POST once and
re-run the script to verify the response. A duplicate POST raises 409.
"""
created, failed, skipped = [], [], []
existing = existing_rule_names()
for name, eid in evaluator_ids.items():
if not eid:
continue
rule_name = f"{name}-on-reviews"
if rule_name in existing:
skipped.append(name)
continue
st, resp = eb._call(
"POST", "/api/public/unstable/evaluation-rules",
rule_body(rule_name, name, sampling), timeout=60.0,
)
if st in (200, 201):
created.append(name)
else:
failed.append({"rule": name, "status": st, "error": resp})
return {"created": created, "failed": failed, "skipped": skipped}
def existing_rule_names() -> set[str]:
"""Names of observation-target rules already in the project."""
out: set[str] = set()
st, body = eb._call("GET", "/api/public/unstable/evaluation-rules?limit=100")
if st == 200 and isinstance(body, dict):
for r in body.get("data") or []:
if r.get("target") == "observation":
out.add(r.get("name"))
return out
def main(argv: list[str] | None = None) -> int:
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--sampling", type=float, default=1.0,
help="fraction of matching observations to judge (default: all)")
ap.add_argument("--skip-connection", action="store_true")
ap.add_argument("--dry-run", action="store_true")
args = ap.parse_args(argv)
if args.dry_run:
print(json.dumps({
"would_connect": {"provider": JUDGE_PROVIDER, "baseURL": JUDGE_BASE_URL,
"model": JUDGE_MODEL},
"would_create": [j["name"] for j in JUDGES],
"existing_evaluators": sorted(existing_evaluators()),
"sampling": args.sampling,
}, indent=2))
return 0
report = {}
if not args.skip_connection:
report["llm_connection"] = ensure_llm_connection()
report["evaluators"] = ensure_evaluators()
ids = dict(report["evaluators"]["created"])
# Fall back to whatever is already registered, so a re-run still wires rules.
for name, eid in existing_evaluators().items():
ids.setdefault(name, eid)
report["rules"] = ensure_rules(
{j["name"]: ids.get(j["name"]) for j in JUDGES}, args.sampling
)
print(json.dumps(report, indent=2))
return 0
_module = importlib.import_module("evaluation.judges")
sys.modules[__name__] = _module
if __name__ == "__main__":
raise SystemExit(main())
raise SystemExit(_module.main())
pilot/eval_scores.py
+7 -233
View File
@@ -1,233 +1,7 @@
#!/usr/bin/env python3
"""pragent pilot — deterministic review scorers.
Four numbers computed from a review that already happened, shipped to Langfuse
as scores on the review's trace. All are derived from data the reviewer already
has in hand: no LLM judge, no ground truth, no extra token spend.
Why these four and not `helpfulness`/`quality`
----------------------------------------------
They come from what the recorded reviews actually did, not from a generic eval
checklist:
* `severity_info_ratio` — of the findings ever posted to a PR, effectively all
landed at `info`. Either the model will not commit to a severity or the
per-repo `severity_threshold` is filtering the rest out. Trending the ratio
per model says which.
* `finding_rate` — most reviews post nothing at all. Silence on clean code is
the goal; silence because the run degraded is a failure. Same output, two
causes, and only the rate over time separates them.
* `dropped_findings` — `ai_review.parse_findings` discards any finding whose
`path`/`line` is unusable. That happens silently, so a model that emits ten
findings at invalid locations is indistinguishable from one that found
nothing. This is the only signal here that measures the *model's* output
rather than the review's.
* `cost_per_finding` — the equivalent-cost number is already trended per
review; per finding is what actually compares two models, since a cheaper
model that finds nothing is not cheaper.
None of these say whether a finding was *correct*. That needs labels, and the
labels come from `feedback_scores.py` once maintainers start reacting to review
comments. Read these as behavioural drift detectors, not as accuracy.
Fail-open, like every other telemetry path here: a scorer that raises returns no
score rather than failing the review.
"""
from __future__ import annotations
import uuid
from datetime import datetime, timezone
# Mirrors ai_review.SEVERITY_RANK. Duplicated rather than imported because this
# module is also run standalone (backfill) where ai_review's import side effects
# are unwanted.
SEVERITY_RANK = {"info": -1, "trivial": 0, "low": 1, "medium": 2, "high": 3, "critical": 4}
# Findings at or below this rank are "the model declined to commit". `trivial`
# and `info` are advisory by the reviewer's own prompt contract.
_ADVISORY_MAX_RANK = 0
# Score names. Named for what is measured, not for the mechanism producing it —
# these land on every trace and become the axis of every chart.
FINDING_RATE = "finding_rate"
SEVERITY_INFO_RATIO = "severity_info_ratio"
SEVERITY_MAX = "severity_max"
DROPPED_FINDINGS = "dropped_findings"
COST_PER_FINDING = "cost_per_finding"
def _sev(f: dict) -> str:
return str(f.get("severity") or "medium").strip().lower()
def finding_rate(findings: list[dict] | None) -> float:
"""How many findings this review posted. 0.0 is the restraint case."""
return float(len(findings or []))
def severity_info_ratio(findings: list[dict] | None) -> float | None:
"""Share of findings the model rated advisory (`info`/`trivial`).
`None` for a review with no findings — a ratio over an empty set is not 0,
it is undefined, and charting it as 0 would read as "perfectly calibrated".
"""
fs = findings or []
if not fs:
return None
advisory = sum(1 for f in fs if SEVERITY_RANK.get(_sev(f), 2) <= _ADVISORY_MAX_RANK)
return round(advisory / len(fs), 4)
def severity_max(findings: list[dict] | None) -> str:
"""Highest severity present, or `none` when the review was silent.
Categorical on purpose: the useful question is "did this review ever surface
something serious", and an average of severity ranks answers nothing.
"""
fs = findings or []
if not fs:
return "none"
top = max(fs, key=lambda f: SEVERITY_RANK.get(_sev(f), 2))
sev = _sev(top)
return sev if sev in SEVERITY_RANK else "medium"
def dropped_findings(raw_count: int | None, kept_count: int | None) -> float | None:
"""Findings the model emitted that the parser could not use.
`raw_count` is what came back in the JSON; `kept_count` is what survived
`_normalize_finding`. `None` when the caller could not determine the raw
count — better no score than a fabricated zero.
"""
if raw_count is None or kept_count is None:
return None
return float(max(0, int(raw_count) - int(kept_count)))
def cost_per_finding(cost_usd: float | None, findings: list[dict] | None) -> float | None:
"""Equivalent USD spent per finding posted.
`None` when nothing could be priced. A silent review divides by one, not by
zero: the run still cost money, and attributing that whole cost to "found
nothing" is the honest reading.
"""
if cost_usd is None:
return None
try:
c = float(cost_usd)
except (TypeError, ValueError):
return None
return round(c / max(1, len(findings or [])), 6)
def build_scores(
*,
trace_id: str,
findings: list[dict] | None,
environment: str,
cost_usd: float | None = None,
dropped_count: float | None = None,
timestamp: str | None = None,
comment: str = "",
) -> list[dict]:
"""The `score-create` ingestion events for one review.
`dropped_count` must be measured at parse time, not here: by the time
`findings` reaches this function the per-repo config has already filtered it
by severity threshold and `max_findings`, and those drops are the config
working as intended, not the model emitting garbage.
Returns [] rather than raising if something is unscoreable — scores are
telemetry and must never cost a review.
"""
# The ingestion envelope requires a timestamp on every event; omitting it
# gets the whole batch rejected with an HTTP 207 whose per-event 400s are
# easy to mistake for success.
ts = timestamp or datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
out: list[dict] = []
def add(name: str, value, data_type: str) -> None:
if value is None:
return
body = {
"id": str(uuid.uuid4()),
"traceId": trace_id,
"name": name,
"dataType": data_type,
"environment": environment,
}
if data_type == "CATEGORICAL":
body["value"] = str(value)
else:
body["value"] = float(value)
if comment:
body["comment"] = comment
out.append(
{
"id": str(uuid.uuid4()),
"type": "score-create",
"timestamp": ts,
"body": body,
}
)
try:
add(FINDING_RATE, finding_rate(findings), "NUMERIC")
add(SEVERITY_INFO_RATIO, severity_info_ratio(findings), "NUMERIC")
add(SEVERITY_MAX, severity_max(findings), "CATEGORICAL")
add(DROPPED_FINDINGS, dropped_count, "NUMERIC")
add(COST_PER_FINDING, cost_per_finding(cost_usd, findings), "NUMERIC")
except Exception: # pragma: no cover - defensive
return out
return out
# ---------------------------------------------------------------------------
# Score configs — the schema these scores must comply with
# ---------------------------------------------------------------------------
# Registered once per project via `eval_bootstrap.py`. Without configs the
# scores still ingest, but nothing constrains a future scorer from writing
# `severity_max="HIGH"` next to today's `"high"` and silently splitting the
# series in two.
SCORE_CONFIGS = [
{
"name": FINDING_RATE,
"dataType": "NUMERIC",
"minValue": 0,
"description": "Findings posted by one review. 0 = the reviewer stayed silent.",
},
{
"name": SEVERITY_INFO_RATIO,
"dataType": "NUMERIC",
"minValue": 0,
"maxValue": 1,
"description": "Share of a review's findings rated info/trivial. High = the model is not committing to a severity.",
},
{
"name": SEVERITY_MAX,
"dataType": "CATEGORICAL",
"categories": [
{"label": "none", "value": 0},
{"label": "info", "value": 1},
{"label": "trivial", "value": 2},
{"label": "low", "value": 3},
{"label": "medium", "value": 4},
{"label": "high", "value": 5},
{"label": "critical", "value": 6},
],
"description": "Highest severity surfaced by one review; 'none' when it posted nothing.",
},
{
"name": DROPPED_FINDINGS,
"dataType": "NUMERIC",
"minValue": 0,
"description": "Findings the model emitted that the parser rejected for an unusable path/line.",
},
{
"name": COST_PER_FINDING,
"dataType": "NUMERIC",
"minValue": 0,
"description": "Equivalent USD per finding posted. Silent reviews divide by 1, not 0.",
},
]
"""Compatibility import for evaluation scores."""
import importlib
import sys
_module = importlib.import_module("evaluation.scores")
sys.modules[__name__] = _module
if __name__ == "__main__":
raise SystemExit(_module.main())
pilot/evaluation/__init__.py
+1
View File
@@ -0,0 +1 @@
"""Langfuse evaluation bootstrap, experiments, and scoring."""
pilot/evaluation/bootstrap.py
+350
View File
@@ -0,0 +1,350 @@
#!/usr/bin/env python3
"""pragent pilot — one-time Langfuse project setup for evaluation.
Three jobs, each idempotent so it can be re-run after any change:
1. **Score configs.** Registers the schema for every score pragent emits
(`eval_scores.SCORE_CONFIGS` + `feedback_scores.SCORE_CONFIGS`). Without
these the scores still ingest, but nothing stops a later scorer writing
`severity_max="HIGH"` beside today's `"high"` and quietly splitting one
series into two. Configs are immutable in Langfuse — a name that already
exists is left alone rather than updated.
2. **Dataset.** Seeds `pragent-reviews` from `feedback.db`: one item per PR
the reviewer has actually run on, carrying the repo/PR/sha as input and
the findings it posted as `expectedOutput`.
Read `expectedOutput` here as "what the reviewer said last time", not "what
is correct" — no human has labelled any of it. It is a regression baseline:
re-run a candidate model over these PRs and the diff against this column is
the behaviour change. Promoting an item to real ground truth means a human
editing it after reviewing the PR, which is what the dataset view is for.
3. **Trace backfill** (`--backfill-traces`). Scores only ride along with new
reviews, so without this the charts stay empty until the next PR lands.
Every trace `langfuse_trace` has ever written already carries the finding
count, the severity histogram and the cost in its metadata, which is
everything four of the five scorers need. `dropped_findings` is absent from
historical traces and is left unscored rather than backfilled as zero.
4. **Reports** what it found, so the gap between "reviews recorded" and
"reviews with human feedback" is visible rather than assumed.
Usage:
LANGFUSE_HOST=... LANGFUSE_PUBLIC_KEY=... LANGFUSE_SECRET_KEY=... \\
python3 eval_bootstrap.py --db /data/feedback.db
"""
from __future__ import annotations
import argparse
import base64
import json
import os
import sqlite3
import sys
import urllib.error
import urllib.request
from datetime import datetime, timezone
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import eval_scores # noqa: E402
import feedback_scores # noqa: E402
DATASET_NAME = "pragent-reviews"
def _conf() -> tuple[str, str, str]:
host = (os.environ.get("LANGFUSE_HOST") or "").strip().rstrip("/")
pk = (os.environ.get("LANGFUSE_PUBLIC_KEY") or "").strip()
sk = (os.environ.get("LANGFUSE_SECRET_KEY") or "").strip()
if not host or not pk or not sk:
raise SystemExit("LANGFUSE_HOST / LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY must be set")
return host, pk, sk
def _call(method: str, path: str, body: dict | None = None, timeout: float = 20.0):
host, pk, sk = _conf()
auth = base64.b64encode(f"{pk}:{sk}".encode()).decode("ascii")
data = json.dumps(body).encode() if body is not None else None
req = urllib.request.Request(
host + path,
data=data,
headers={
"Content-Type": "application/json",
"Authorization": f"Basic {auth}",
"User-Agent": "pragent-pilot/1.0",
},
method=method,
)
try:
with urllib.request.urlopen(req, timeout=timeout) as resp:
raw = resp.read()
return resp.status, (json.loads(raw) if raw else None)
except urllib.error.HTTPError as e:
return e.code, e.read()[:400].decode("utf-8", "replace")
# ---------------------------------------------------------------------------
# 1. Score configs
# ---------------------------------------------------------------------------
def ensure_score_configs() -> dict:
status, existing = _call("GET", "/api/public/score-configs?limit=100")
have = set()
if status == 200 and isinstance(existing, dict):
have = {c.get("name") for c in existing.get("data", [])}
created, skipped, failed = [], [], []
for cfg in list(eval_scores.SCORE_CONFIGS) + list(feedback_scores.SCORE_CONFIGS):
if cfg["name"] in have:
skipped.append(cfg["name"])
continue
st, resp = _call("POST", "/api/public/score-configs", cfg)
if st in (200, 201):
created.append(cfg["name"])
else:
failed.append({"name": cfg["name"], "status": st, "error": resp})
return {"created": created, "already_present": skipped, "failed": failed}
# ---------------------------------------------------------------------------
# 2. Dataset from recorded reviews
# ---------------------------------------------------------------------------
def item_id(repo: str, pr) -> str:
"""A dataset-item id that survives being put in a URL path.
The obvious `{repo}#{pr}` is unusable: the UI routes items as
`/datasets/{id}/items/{item_id}`, so the `/` in `owner/repo` splits into
extra path segments and everything after the `#` is a fragment the browser
never sends. The item is created fine and then 404s when opened.
Session ids elsewhere keep the `{repo}#{pr}` form — those are never path
segments, and `feedback_scores` depends on that shape.
"""
return f"{repo.replace('/', '__')}__pr{pr}"
def _item_metadata(*, repo, pr, head_sha, reviews_run, last_seen, findings) -> dict:
"""Filterable facets for one dataset item.
Kept flat and primitive: the filter bar matches a metadata key against a
literal, so a nested object or a list is not reachable from the UI.
"""
owner, _, repo_name = str(repo).partition("/")
sevs = [str(f["severity"] or "").lower() for f in findings]
ranked = [s for s in sevs if s in eval_scores.SEVERITY_RANK]
return {
"repo": repo,
"owner": owner or repo,
"repo_name": repo_name or repo,
"pr": int(pr),
"head_sha": head_sha,
"reviews_run": reviews_run,
"last_reviewed_at": last_seen,
"last_reviewed_iso": datetime.fromtimestamp(last_seen, timezone.utc).isoformat(),
"finding_count": len(findings),
"has_findings": bool(findings),
# "none" rather than omitting the key: a filter for silent reviews needs
# something to match, and an absent key matches nothing.
"max_severity": (
max(ranked, key=lambda s: eval_scores.SEVERITY_RANK[s]) if ranked else "none"
),
# Flags that this row is the reviewer's own past output, not a human
# judgement. Filter on it before anyone treats the dataset as truth.
"labelled_by_human": False,
}
def read_review_items(db_path: str) -> list[dict]:
"""One dataset item per (repo, pr) the reviewer has run on.
Keyed on the PR rather than on each individual review row: the same PR is
re-reviewed on every push, and 113 rows over 26 PRs would make a benchmark
that is 4x redundant and weighted towards whichever PR churned most.
"""
conn = sqlite3.connect(db_path)
conn.row_factory = sqlite3.Row
try:
prs = conn.execute(
"""
SELECT repo, pr, MAX(posted_at) AS last_seen, COUNT(*) AS reviews,
MAX(head_sha) AS head_sha
FROM review GROUP BY repo, pr ORDER BY repo, pr
"""
).fetchall()
items = []
for row in prs:
findings = conn.execute(
"""
SELECT path, line, severity, problem, fix
FROM inline_finding WHERE repo = ? AND pr = ?
ORDER BY path, line
""",
(row["repo"], row["pr"]),
).fetchall()
items.append(
{
"id": item_id(row["repo"], row["pr"]),
"input": {
"repo": row["repo"],
"pr": int(row["pr"]),
"head_sha": row["head_sha"],
},
"expectedOutput": {
"findings": [dict(f) for f in findings],
"finding_count": len(findings),
},
# The UI's filter bar reads metadata and nothing else, so
# anything worth slicing on is a top-level key here even
# where it duplicates `input`. `owner` and `repo_name` are
# split out because a filter on the joined `repo` can only
# match one repo at a time, never a whole org.
"metadata": _item_metadata(
repo=row["repo"],
pr=row["pr"],
head_sha=row["head_sha"],
reviews_run=int(row["reviews"]),
last_seen=int(row["last_seen"]),
findings=findings,
),
}
)
return items
finally:
conn.close()
def ensure_dataset(items: list[dict], name: str = DATASET_NAME) -> dict:
st, _ = _call(
"POST",
"/api/public/datasets",
{
"name": name,
"description": (
"PRs the pragent pilot has reviewed, seeded from feedback.db. "
"expectedOutput is the reviewer's own prior output — a regression "
"baseline, not human-verified ground truth."
),
"metadata": {"source": "feedback.db", "seeded_by": "eval_bootstrap.py"},
},
)
# A duplicate name is fine: the dataset already exists from an earlier run.
dataset_ok = st in (200, 201, 409)
created, failed = 0, []
for item in items:
body = {
"datasetName": name,
"id": item["id"], # idempotent: same PR updates rather than duplicates
"input": item["input"],
"expectedOutput": item["expectedOutput"],
"metadata": item["metadata"],
}
ist, resp = _call("POST", "/api/public/dataset-items", body)
if ist in (200, 201):
created += 1
else:
failed.append({"item": item["id"], "status": ist, "error": resp})
return {"dataset": name, "dataset_created": dataset_ok, "items_upserted": created, "failed": failed}
# ---------------------------------------------------------------------------
# 3. Backfill scores onto traces that predate the scorers
# ---------------------------------------------------------------------------
def _synth_findings(severities: dict) -> list[dict]:
"""Rebuild a findings list from a trace's severity histogram.
Only severity matters to the scorers, and that is all the histogram kept.
Reconstructing placeholders is honest here because every scorer being
backfilled reads nothing else off a finding.
"""
out = []
for sev, count in (severities or {}).items():
out.extend({"severity": sev} for _ in range(int(count)))
return out
def backfill_traces(limit_pages: int = 20) -> dict:
import eval_scores as es
scored, skipped, events = 0, 0, []
page = 1
while page <= limit_pages:
st, resp = _call("GET", f"/api/public/traces?limit=50&page={page}&name=pr-review")
if st != 200 or not isinstance(resp, dict):
break
rows = resp.get("data") or []
if not rows:
break
for tr in rows:
meta = tr.get("metadata") or {}
severities = meta.get("severities") or {}
count = meta.get("findings")
if count is None:
skipped += 1
continue
findings = _synth_findings(severities)
# The histogram is authoritative when present; a trace that recorded
# a count but no histogram still scores its rate.
if not findings and count:
findings = [{"severity": "medium"} for _ in range(int(count))]
batch = es.build_scores(
trace_id=tr["id"],
findings=findings,
environment=tr.get("environment") or "default",
cost_usd=(tr.get("totalCost") or meta.get("provider_cost_usd")),
timestamp=tr.get("timestamp"),
comment="backfilled from trace metadata",
)
events.extend(batch)
scored += 1
page += 1
posted = False
status = None
if events:
import langfuse_trace
host, pk, sk = _conf()
# Chunked: one 2000-event POST is refused, and a partial backfill that
# reports success is worse than a slow one.
for i in range(0, len(events), 200):
status = langfuse_trace._post(host, pk, sk, events[i:i + 200], 30.0)
posted = status in (200, 201, 207)
if not posted:
break
return {"traces_scored": scored, "traces_skipped": skipped, "scores": len(events),
"posted": posted, "http_status": status}
def main() -> int:
ap = argparse.ArgumentParser(description="Bootstrap Langfuse evaluation for the pragent pilot")
ap.add_argument("--db", default=os.environ.get("PRAGENT_FEEDBACK_DB", "/data/feedback.db"))
ap.add_argument("--skip-dataset", action="store_true")
ap.add_argument("--skip-configs", action="store_true")
ap.add_argument("--backfill-traces", action="store_true",
help="score traces written before the scorers existed")
args = ap.parse_args()
out: dict = {}
if not args.skip_configs:
out["score_configs"] = ensure_score_configs()
if not args.skip_dataset:
items = read_review_items(args.db)
out["dataset"] = ensure_dataset(items)
out["dataset"]["items_read"] = len(items)
if args.backfill_traces:
out["trace_backfill"] = backfill_traces()
print(json.dumps(out, indent=2))
failed = (out.get("score_configs", {}).get("failed") or []) + (
out.get("dataset", {}).get("failed") or []
)
return 1 if failed else 0
if __name__ == "__main__":
raise SystemExit(main())
pilot/evaluation/experiment.py
+212
View File
@@ -0,0 +1,212 @@
#!/usr/bin/env python3
"""pragent pilot — populate the Experiments tab from reviews already traced.
An "experiment" in Langfuse is a dataset run: a set of (dataset item, trace)
links under one run name. The Experiments tab then shows one row per item with
its scores, and lets two runs be diffed side by side.
Nothing here re-runs the reviewer. Every PR in `pragent-reviews` has already
been reviewed, and each of those reviews left a trace carrying its findings,
cost and scores. This links what exists, which is what makes the tab useful on
day one instead of after the next N pushes.
Runs are grouped by **model** by default, because that is the comparison the
pilot actually needs to make: the same PRs reviewed by MiniMax vs whatever
replaces it, with `finding_rate` and `cost_per_finding` side by side. Group by
`none` for a single "all traces" run.
One trace per (run, item) — the most recent. A PR re-reviewed on every push has
many traces, and a dataset run is defined as one output per input; feeding it
the other five would make the per-run averages meaningless.
Note on the endpoint: `POST /api/public/dataset-run-items` is deprecated in
favour of the SDK experiment runner / OTel ingestion, and disappears in
Langfuse v4. This instance is self-hosted v3, which the deprecation notice
explicitly exempts from the cutoff date, and the pilot is stdlib-only by
design. Revisit when this deployment moves to v4.
Usage:
LANGFUSE_HOST=... LANGFUSE_PUBLIC_KEY=... LANGFUSE_SECRET_KEY=... \\
python3 eval_experiment.py --dry-run
"""
from __future__ import annotations
import argparse
import json
import os
import sys
import urllib.parse
from collections import defaultdict
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import eval_bootstrap as eb # noqa: E402
TRACE_NAME = "pr-review"
# ---------------------------------------------------------------------------
# Reading what already exists
# ---------------------------------------------------------------------------
def fetch_traces(name: str = TRACE_NAME, limit: int = 100, max_pages: int = 50) -> list[dict]:
"""Every review trace, newest first."""
out: list[dict] = []
for page in range(1, max_pages + 1):
q = urllib.parse.urlencode({"name": name, "limit": limit, "page": page})
st, body = eb._call("GET", f"/api/public/traces?{q}")
if st != 200 or not isinstance(body, dict):
raise SystemExit(f"listing traces failed: {st} {body}")
data = body.get("data") or []
out.extend(data)
meta = body.get("meta") or {}
if page * meta.get("limit", limit) >= meta.get("totalItems", 0):
break
return out
def fetch_item_ids(dataset: str) -> set[str]:
"""Ids present in the dataset, so runs never reference a missing item."""
ids: set[str] = set()
for page in range(1, 51):
q = urllib.parse.urlencode({"datasetName": dataset, "limit": 100, "page": page})
st, body = eb._call("GET", f"/api/public/dataset-items?{q}")
if st != 200 or not isinstance(body, dict):
raise SystemExit(f"listing dataset items failed: {st} {body}")
ids.update(i["id"] for i in body.get("data") or [])
meta = body.get("meta") or {}
if page * meta.get("limit", 100) >= meta.get("totalItems", 0):
break
return ids
# ---------------------------------------------------------------------------
# Grouping traces into runs
# ---------------------------------------------------------------------------
def trace_model(trace: dict) -> str:
"""The model that produced a review, from its `model:` tag."""
for tag in trace.get("tags") or []:
if tag.startswith("model:"):
return tag[len("model:"):] or "unknown"
return "unknown"
def trace_item_id(trace: dict) -> str | None:
"""The dataset item a trace belongs to, or None if it is not a PR review."""
md = trace.get("metadata") or {}
repo, pr = md.get("repo"), md.get("pr")
if not repo or pr in (None, ""):
return None
return eb.item_id(str(repo), pr)
def _sort_key(trace: dict):
return (trace.get("timestamp") or "", trace.get("id") or "")
def plan_runs(traces: list[dict], known_items: set[str], group_by: str = "model") -> dict:
"""Map run name -> {item id: trace}, keeping only the newest trace per item.
Traces whose PR is not in the dataset are dropped: `feedback.db` is the
source for both, but a review can be traced without its row landing (the
posting step can fail after the model ran), and a run item pointing at a
non-existent dataset item is rejected.
"""
runs: dict[str, dict[str, dict]] = defaultdict(dict)
skipped_no_item, skipped_unknown = 0, 0
for tr in traces:
iid = trace_item_id(tr)
if iid is None:
skipped_unknown += 1
continue
if iid not in known_items:
skipped_no_item += 1
continue
run = "all-traces" if group_by == "none" else trace_model(tr)
prev = runs[run].get(iid)
if prev is None or _sort_key(tr) > _sort_key(prev):
runs[run][iid] = tr
return {
"runs": dict(runs),
"skipped_not_in_dataset": skipped_no_item,
"skipped_not_a_review": skipped_unknown,
}
def run_name(prefix: str, key: str) -> str:
return f"{prefix}-{key}" if prefix else key
# ---------------------------------------------------------------------------
# Writing the runs
# ---------------------------------------------------------------------------
def create_run(name: str, items: dict[str, dict], description: str = "") -> dict:
"""Link each (item, trace) pair into the named run. Idempotent per pair."""
created, failed = 0, []
for iid, tr in sorted(items.items()):
md = tr.get("metadata") or {}
body = {
"runName": name,
"runDescription": description,
"datasetItemId": iid,
"traceId": tr["id"],
"metadata": {
"model": trace_model(tr),
"engine": md.get("engine"),
"findings": md.get("findings"),
"duration_s": md.get("duration_s"),
"cost_basis": md.get("cost_basis"),
"linked_by": "eval_experiment.py",
},
}
st, resp = eb._call("POST", "/api/public/dataset-run-items", body)
if st in (200, 201):
created += 1
else:
failed.append({"item": iid, "status": st, "error": resp})
return {"run": name, "items_linked": created, "failed": failed}
def main(argv: list[str] | None = None) -> int:
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--dataset", default=eb.DATASET_NAME)
ap.add_argument("--group-by", choices=("model", "none"), default="model")
ap.add_argument("--prefix", default="baseline",
help="run name prefix; '' for the bare group key")
ap.add_argument("--dry-run", action="store_true")
args = ap.parse_args(argv)
traces = fetch_traces()
items = fetch_item_ids(args.dataset)
plan = plan_runs(traces, items, group_by=args.group_by)
report = {
"traces_read": len(traces),
"dataset_items": len(items),
"skipped_not_in_dataset": plan["skipped_not_in_dataset"],
"skipped_not_a_review": plan["skipped_not_a_review"],
"runs": {},
}
for key, mapping in sorted(plan["runs"].items()):
name = run_name(args.prefix, key)
if args.dry_run:
report["runs"][name] = {"items_would_link": len(mapping)}
continue
report["runs"][name] = create_run(
name,
mapping,
description=(
"Reviews already run by the pilot, linked after the fact. "
"Scores come from the traces; expectedOutput is the reviewer's "
"own prior output, not human-verified ground truth."
),
)
report["dry_run"] = args.dry_run
print(json.dumps(report, indent=2))
return 0
if __name__ == "__main__":
raise SystemExit(main())
pilot/evaluation/judges.py
+314
View File
@@ -0,0 +1,314 @@
#!/usr/bin/env python3
"""pragent pilot — LLM-as-a-judge evaluators for the reviewer.
The deterministic scorers in `eval_scores.py` measure *behaviour*: how many
findings, how severe, how much they cost. None of them can say whether a
finding was any good. With no human labels in `feedback.db`, a judge is the
only thing that can — so these two ask the questions that need no ground truth,
only the review itself:
`finding_actionability` — is each finding concrete enough to act on? A
reviewer that says "consider improving error handling" at file level is
indistinguishable from a useful one by finding count alone. This is the
failure mode a cheap model degrades into first.
`review_self_consistency` — does the summary agree with the findings it
posted? Claiming "no issues found" above a list of two criticals, or
describing a problem in prose that never became a finding, is a defect the
reviewer can commit entirely on its own.
Neither judge is asked whether a finding is *correct*. That needs the diff,
which these traces do not carry, and a judge asked to rule on correctness from
a summary alone will confabulate. Accuracy stays an open question until humans
start labelling — which is what `feedback_scores.py` is there to capture.
**The judge is a different model from the reviewer.** The reviewer runs
MiniMax-M2.7; the judge runs kimi-k2.7-code through the same headroom hub. A
model grading its own output agrees with itself for reasons that have nothing
to do with quality.
Evaluators score *observations*, and their variable mapping reads the
observation's own input/output — which is why `langfuse_trace` now writes the
review onto the generation and not just onto the trace.
Usage:
LANGFUSE_HOST=... LANGFUSE_PUBLIC_KEY=... LANGFUSE_SECRET_KEY=... \\
python3 eval_judges.py --dry-run
"""
from __future__ import annotations
import argparse
import json
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import eval_bootstrap as eb # noqa: E402
# The headroom hub in front of the local Ollama, plus a small pass-through
# proxy (`judge-proxy` on 8802) that patches every `thinking` content block
# to carry the `signature` field Langfuse's Anthropic adapter requires. The
# underlying model is kimi-k2.7-code through the hub on 8790; the proxy fixes
# the shape so Mastra's Zod parse stops failing.
JUDGE_PROVIDER = "headroom-ollama"
JUDGE_BASE_URL = os.environ.get("PRAGENT_JUDGE_BASE_URL", "http://100.74.17.70:8802")
JUDGE_API_KEY = os.environ.get("PRAGENT_JUDGE_API_KEY", "ollama")
JUDGE_MODEL = os.environ.get("PRAGENT_JUDGE_MODEL", "kimi-k2.7-code:cloud")
# The trace names this project emits (`pr-review` on the trace, `opencode-review`
# on the generation). Filter on `traceName` rather than observation `name` — the
# observation-rule schema only exposes `traceName` as a stringOptions column, and
# every observation inside these traces is the review itself, so the narrowness
# is the same.
REVIEW_TRACE_NAMES = ["pr-review", "opencode-review"]
def _model_config() -> dict:
return {"provider": JUDGE_PROVIDER, "model": JUDGE_MODEL}
JUDGES = [
{
"name": "finding_actionability",
"prompt": (
"You are auditing the output of an automated code reviewer.\n\n"
"PR under review:\n{{input}}\n\n"
"What the reviewer produced:\n{{output}}\n\n"
"Rate how ACTIONABLE the findings are, from 0 to 1. A finding is "
"actionable when a developer could act on it without asking a "
"follow-up question: it points at a specific location, names a "
"concrete problem, and proposes a fix that could be applied.\n\n"
"Score 1.0 when every finding is specific and fixable. Score around "
"0.5 when findings identify a real area but leave the developer to "
"work out what to change. Score near 0.0 when findings are generic "
"advice that would apply to almost any pull request.\n\n"
"Judge only specificity and actionability. You cannot see the diff, "
"so do NOT attempt to judge whether a finding is factually correct, "
"and do not penalise a finding for being one you cannot verify.\n\n"
"If the reviewer reported no findings at all, return 1.0 and say in "
"your reasoning that there was nothing to judge — a silent review is "
"measured by finding_rate, not here."
),
"outputDefinition": {
"dataType": "NUMERIC",
"minValue": 0,
"maxValue": 1,
"reasoning": {
"description": (
"Name the least actionable finding and say what it would "
"need in order to be acted on."
)
},
"score": {"description": "0 = generic advice, 1 = every finding is specific and fixable."},
},
},
{
"name": "review_self_consistency",
"prompt": (
"You are auditing the output of an automated code reviewer.\n\n"
"PR under review:\n{{input}}\n\n"
"What the reviewer produced:\n{{output}}\n\n"
"The output contains a prose `summary` and a list of `findings`. "
"Decide whether the summary is CONSISTENT with the findings.\n\n"
"Inconsistent means, for example: the summary says no issues were "
"found while findings are listed; the summary describes a problem "
"that never became a finding; the summary characterises the severity "
"of the findings in a way the findings themselves contradict; or the "
"summary refers to files that appear in no finding and in no part of "
"the PR description.\n\n"
"A summary that adds context beyond the findings is NOT inconsistent "
"as long as nothing in it contradicts them. A review that found "
"nothing and says so is consistent.\n\n"
"You cannot see the diff. Judge the summary against the findings and "
"the PR title only — never against what you imagine the code does."
),
"outputDefinition": {
"dataType": "BOOLEAN",
"reasoning": {
"description": "Quote the part of the summary that conflicts with the findings, if any."
},
"score": {"description": "true = summary agrees with the findings, false = it contradicts them."},
},
},
]
# Both judges read the observation's own input/output.
MAPPING = [
{"variable": "input", "source": "input"},
{"variable": "output", "source": "output"},
]
# ---------------------------------------------------------------------------
# LLM connection
# ---------------------------------------------------------------------------
def ensure_llm_connection() -> dict:
"""Point the project at the judge model. Upserted on `provider`."""
body = {
"provider": JUDGE_PROVIDER,
"adapter": "anthropic",
"baseURL": JUDGE_BASE_URL,
"secretKey": JUDGE_API_KEY,
"customModels": [JUDGE_MODEL],
# The hub serves two local models and none of Anthropic's, so the
# default catalogue would be a list of models that all fail on use.
"withDefaultModels": False,
}
st, resp = eb._call("PUT", "/api/public/llm-connections", body)
return {"status": st, "ok": st in (200, 201), "provider": JUDGE_PROVIDER,
"error": None if st in (200, 201) else resp}
# ---------------------------------------------------------------------------
# Evaluators
# ---------------------------------------------------------------------------
def existing_evaluators() -> dict[str, str]:
"""name -> id for evaluators already in the project."""
out: dict[str, str] = {}
st, body = eb._call("GET", "/api/public/unstable/evaluators?limit=100")
if st == 200 and isinstance(body, dict):
for ev in body.get("data") or []:
out[ev.get("name")] = ev.get("id")
return out
def ensure_evaluators() -> dict:
"""Create each judge if no version exists for the name yet.
POST /evaluators with a name that already exists creates a new version, not
a no-op — re-running this script would pile up versions until the page
listing them is unreadable. Skip when an evaluator of that name is present.
"""
created, skipped, failed = {}, [], []
existing = set(existing_evaluators())
for judge in JUDGES:
if judge["name"] in existing:
skipped.append(judge["name"])
continue
body = {
"type": "llm_as_judge",
"name": judge["name"],
"prompt": judge["prompt"],
"outputDefinition": judge["outputDefinition"],
"modelConfig": _model_config(),
}
st, resp = eb._call("POST", "/api/public/unstable/evaluators", body, timeout=60.0)
if st in (200, 201) and isinstance(resp, dict):
created[judge["name"]] = resp.get("id")
else:
failed.append({"name": judge["name"], "status": st, "error": resp})
return {"created": created, "skipped": skipped, "failed": failed}
# ---------------------------------------------------------------------------
# Rules — what gets judged, and how often
# ---------------------------------------------------------------------------
def rule_body(name: str, judge_name: str, sampling: float) -> dict:
"""POST /evaluation-rules shape for an LLM-as-judge trace rule.
Target is `trace` rather than `observation` on purpose: the standard
`/api/public/ingestion` path that ships review traces here feeds only
the trace-upsert queue, and `evalService.createEvalJobs` only creates
jobs for `targetObject ∈ {TRACE, DATASET}`. Observation rules are
triggered exclusively from the OTel ingestion pipeline, which this
pilot does not use. A trace rule reads the trace's own input/output —
`langfuse_trace` already writes `_review_input`/`_review_output` onto
the trace body for exactly this reason.
Mapping is required at both the rule root (server validates it there)
and inside `evaluator` (the API echoes it back).
"""
return {
"name": name,
"enabled": True,
"target": "trace",
"sampling": sampling,
"filter": [
{"column": "traceName", "operator": "any of",
"value": REVIEW_TRACE_NAMES, "type": "stringOptions"},
],
"evaluator": {
"name": judge_name,
"scope": "project",
"variableMapping": MAPPING,
},
"mapping": MAPPING,
}
def ensure_rules(evaluator_ids: dict[str, str], sampling: float) -> dict:
"""Idempotent: existing rules with the same name are skipped, not duplicated.
The API has no `name`-keyed upsert; the convention is to POST once and
re-run the script to verify the response. A duplicate POST raises 409.
"""
created, failed, skipped = [], [], []
existing = existing_rule_names()
for name, eid in evaluator_ids.items():
if not eid:
continue
rule_name = f"{name}-on-reviews"
if rule_name in existing:
skipped.append(name)
continue
st, resp = eb._call(
"POST", "/api/public/unstable/evaluation-rules",
rule_body(rule_name, name, sampling), timeout=60.0,
)
if st in (200, 201):
created.append(name)
else:
failed.append({"rule": name, "status": st, "error": resp})
return {"created": created, "failed": failed, "skipped": skipped}
def existing_rule_names() -> set[str]:
"""Names of observation-target rules already in the project."""
out: set[str] = set()
st, body = eb._call("GET", "/api/public/unstable/evaluation-rules?limit=100")
if st == 200 and isinstance(body, dict):
for r in body.get("data") or []:
if r.get("target") == "observation":
out.add(r.get("name"))
return out
def main(argv: list[str] | None = None) -> int:
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--sampling", type=float, default=1.0,
help="fraction of matching observations to judge (default: all)")
ap.add_argument("--skip-connection", action="store_true")
ap.add_argument("--dry-run", action="store_true")
args = ap.parse_args(argv)
if args.dry_run:
print(json.dumps({
"would_connect": {"provider": JUDGE_PROVIDER, "baseURL": JUDGE_BASE_URL,
"model": JUDGE_MODEL},
"would_create": [j["name"] for j in JUDGES],
"existing_evaluators": sorted(existing_evaluators()),
"sampling": args.sampling,
}, indent=2))
return 0
report = {}
if not args.skip_connection:
report["llm_connection"] = ensure_llm_connection()
report["evaluators"] = ensure_evaluators()
ids = dict(report["evaluators"]["created"])
# Fall back to whatever is already registered, so a re-run still wires rules.
for name, eid in existing_evaluators().items():
ids.setdefault(name, eid)
report["rules"] = ensure_rules(
{j["name"]: ids.get(j["name"]) for j in JUDGES}, args.sampling
)
print(json.dumps(report, indent=2))
return 0
if __name__ == "__main__":
raise SystemExit(main())
pilot/evaluation/scores.py
+233
View File
@@ -0,0 +1,233 @@
#!/usr/bin/env python3
"""pragent pilot — deterministic review scorers.
Four numbers computed from a review that already happened, shipped to Langfuse
as scores on the review's trace. All are derived from data the reviewer already
has in hand: no LLM judge, no ground truth, no extra token spend.
Why these four and not `helpfulness`/`quality`
----------------------------------------------
They come from what the recorded reviews actually did, not from a generic eval
checklist:
* `severity_info_ratio` — of the findings ever posted to a PR, effectively all
landed at `info`. Either the model will not commit to a severity or the
per-repo `severity_threshold` is filtering the rest out. Trending the ratio
per model says which.
* `finding_rate` — most reviews post nothing at all. Silence on clean code is
the goal; silence because the run degraded is a failure. Same output, two
causes, and only the rate over time separates them.
* `dropped_findings` — `ai_review.parse_findings` discards any finding whose
`path`/`line` is unusable. That happens silently, so a model that emits ten
findings at invalid locations is indistinguishable from one that found
nothing. This is the only signal here that measures the *model's* output
rather than the review's.
* `cost_per_finding` — the equivalent-cost number is already trended per
review; per finding is what actually compares two models, since a cheaper
model that finds nothing is not cheaper.
None of these say whether a finding was *correct*. That needs labels, and the
labels come from `feedback_scores.py` once maintainers start reacting to review
comments. Read these as behavioural drift detectors, not as accuracy.
Fail-open, like every other telemetry path here: a scorer that raises returns no
score rather than failing the review.
"""
from __future__ import annotations
import uuid
from datetime import datetime, timezone
# Mirrors ai_review.SEVERITY_RANK. Duplicated rather than imported because this
# module is also run standalone (backfill) where ai_review's import side effects
# are unwanted.
SEVERITY_RANK = {"info": -1, "trivial": 0, "low": 1, "medium": 2, "high": 3, "critical": 4}
# Findings at or below this rank are "the model declined to commit". `trivial`
# and `info` are advisory by the reviewer's own prompt contract.
_ADVISORY_MAX_RANK = 0
# Score names. Named for what is measured, not for the mechanism producing it —
# these land on every trace and become the axis of every chart.
FINDING_RATE = "finding_rate"
SEVERITY_INFO_RATIO = "severity_info_ratio"
SEVERITY_MAX = "severity_max"
DROPPED_FINDINGS = "dropped_findings"
COST_PER_FINDING = "cost_per_finding"
def _sev(f: dict) -> str:
return str(f.get("severity") or "medium").strip().lower()
def finding_rate(findings: list[dict] | None) -> float:
"""How many findings this review posted. 0.0 is the restraint case."""
return float(len(findings or []))
def severity_info_ratio(findings: list[dict] | None) -> float | None:
"""Share of findings the model rated advisory (`info`/`trivial`).
`None` for a review with no findings — a ratio over an empty set is not 0,
it is undefined, and charting it as 0 would read as "perfectly calibrated".
"""
fs = findings or []
if not fs:
return None
advisory = sum(1 for f in fs if SEVERITY_RANK.get(_sev(f), 2) <= _ADVISORY_MAX_RANK)
return round(advisory / len(fs), 4)
def severity_max(findings: list[dict] | None) -> str:
"""Highest severity present, or `none` when the review was silent.
Categorical on purpose: the useful question is "did this review ever surface
something serious", and an average of severity ranks answers nothing.
"""
fs = findings or []
if not fs:
return "none"
top = max(fs, key=lambda f: SEVERITY_RANK.get(_sev(f), 2))
sev = _sev(top)
return sev if sev in SEVERITY_RANK else "medium"
def dropped_findings(raw_count: int | None, kept_count: int | None) -> float | None:
"""Findings the model emitted that the parser could not use.
`raw_count` is what came back in the JSON; `kept_count` is what survived
`_normalize_finding`. `None` when the caller could not determine the raw
count — better no score than a fabricated zero.
"""
if raw_count is None or kept_count is None:
return None
return float(max(0, int(raw_count) - int(kept_count)))
def cost_per_finding(cost_usd: float | None, findings: list[dict] | None) -> float | None:
"""Equivalent USD spent per finding posted.
`None` when nothing could be priced. A silent review divides by one, not by
zero: the run still cost money, and attributing that whole cost to "found
nothing" is the honest reading.
"""
if cost_usd is None:
return None
try:
c = float(cost_usd)
except (TypeError, ValueError):
return None
return round(c / max(1, len(findings or [])), 6)
def build_scores(
*,
trace_id: str,
findings: list[dict] | None,
environment: str,
cost_usd: float | None = None,
dropped_count: float | None = None,
timestamp: str | None = None,
comment: str = "",
) -> list[dict]:
"""The `score-create` ingestion events for one review.
`dropped_count` must be measured at parse time, not here: by the time
`findings` reaches this function the per-repo config has already filtered it
by severity threshold and `max_findings`, and those drops are the config
working as intended, not the model emitting garbage.
Returns [] rather than raising if something is unscoreable — scores are
telemetry and must never cost a review.
"""
# The ingestion envelope requires a timestamp on every event; omitting it
# gets the whole batch rejected with an HTTP 207 whose per-event 400s are
# easy to mistake for success.
ts = timestamp or datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
out: list[dict] = []
def add(name: str, value, data_type: str) -> None:
if value is None:
return
body = {
"id": str(uuid.uuid4()),
"traceId": trace_id,
"name": name,
"dataType": data_type,
"environment": environment,
}
if data_type == "CATEGORICAL":
body["value"] = str(value)
else:
body["value"] = float(value)
if comment:
body["comment"] = comment
out.append(
{
"id": str(uuid.uuid4()),
"type": "score-create",
"timestamp": ts,
"body": body,
}
)
try:
add(FINDING_RATE, finding_rate(findings), "NUMERIC")
add(SEVERITY_INFO_RATIO, severity_info_ratio(findings), "NUMERIC")
add(SEVERITY_MAX, severity_max(findings), "CATEGORICAL")
add(DROPPED_FINDINGS, dropped_count, "NUMERIC")
add(COST_PER_FINDING, cost_per_finding(cost_usd, findings), "NUMERIC")
except Exception: # pragma: no cover - defensive
return out
return out
# ---------------------------------------------------------------------------
# Score configs — the schema these scores must comply with
# ---------------------------------------------------------------------------
# Registered once per project via `eval_bootstrap.py`. Without configs the
# scores still ingest, but nothing constrains a future scorer from writing
# `severity_max="HIGH"` next to today's `"high"` and silently splitting the
# series in two.
SCORE_CONFIGS = [
{
"name": FINDING_RATE,
"dataType": "NUMERIC",
"minValue": 0,
"description": "Findings posted by one review. 0 = the reviewer stayed silent.",
},
{
"name": SEVERITY_INFO_RATIO,
"dataType": "NUMERIC",
"minValue": 0,
"maxValue": 1,
"description": "Share of a review's findings rated info/trivial. High = the model is not committing to a severity.",
},
{
"name": SEVERITY_MAX,
"dataType": "CATEGORICAL",
"categories": [
{"label": "none", "value": 0},
{"label": "info", "value": 1},
{"label": "trivial", "value": 2},
{"label": "low", "value": 3},
{"label": "medium", "value": 4},
{"label": "high", "value": 5},
{"label": "critical", "value": 6},
],
"description": "Highest severity surfaced by one review; 'none' when it posted nothing.",
},
{
"name": DROPPED_FINDINGS,
"dataType": "NUMERIC",
"minValue": 0,
"description": "Findings the model emitted that the parser rejected for an unusable path/line.",
},
{
"name": COST_PER_FINDING,
"dataType": "NUMERIC",
"minValue": 0,
"description": "Equivalent USD per finding posted. Silent reviews divide by 1, not 0.",
},
]
pilot/feedback/__init__.py
+2
View File
@@ -0,0 +1,2 @@
"""Feedback persistence and human-signal processing."""
from .store import *
pilot/feedback/analyze.py
+419
View File
@@ -0,0 +1,419 @@
"""pragent pilot — daily feedback analyzer.
Reads `feedback.db` (written by `feedback_harvest.py`) and produces a
markdown report that:
1. Ranks inline findings by **net false-positive score** (downvotes +
unresolved + negation-phrase replies − upvotes − resolved). Top of
this list = "the bot has been wrong about this repeatedly". These
are the candidates that *might* belong in the per-repo
`.pr-review.json:instructions` addendum.
2. Ranks findings by **net acceptance** — repeated 👍 / resolution =
"the bot's framing here is genuinely useful". These can be promoted
to the shared `architecture.md` so they don't have to be re-derived
every PR.
3. Reports a **restraint metric** — for every PR where the bot posted
zero findings, count how often a human reviewer also posted zero
substantive review comments. When the bot is loud on clean code,
that's a false-positive rate we can act on (DoorDash lesson:
"excessive noise on clean code is its own failure mode").
4. Reports a **case-review queue** — every disagreement case (a
downvote, unresolved, or a reply matching `FALSE_POSITIVE_PHRASES`)
is listed in full so a human can re-read the original PR and decide
if the finding was right or wrong.
Output is plain markdown so it can be posted as a Gitea issue / comment
without rendering work. Designed to be reviewed by a human, not auto-
applied — per the DoorDash pattern, every material change to model /
prompt / context goes through a benchmark gate first; this report IS
that gate (or, more precisely, the queue feeding the gate).
Never raises. A bad DB / no data → returns a friendly empty-state report.
"""
from __future__ import annotations
import argparse
import json
import logging
import os
import sqlite3
from collections import defaultdict
from datetime import datetime, timezone
from typing import Optional
import feedback
from feedback_harvest import (
FALSE_POSITIVE_PHRASES,
classify_reaction,
_is_negation_reply, # noqa: F401 (re-exported for the test suite)
)
log = logging.getLogger("pragent.feedback.analyze")
# How many findings to surface in each top-list. Capped because the
# reports are read by humans; more than 20 per list and they skim.
TOP_N = 20
# Restraint threshold — fraction of "clean" PRs (zero findings) where
# the bot produced ANY findings. Above this we recommend `.pr-review.json:
# exclude_patterns` or a stricter `severity_threshold`.
RESTRAINT_NOISE_THRESHOLD = 0.25
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _net_score(row) -> tuple[int, int]:
"""Return (false_positive_score, acceptance_score) for one finding row.
FP signals: downvotes (+1), unresolved (+1), negation-phrase replies (+2).
Acceptance signals: upvotes (+1), resolved (+1).
"""
fp = 0
ac = 0
fp += int(row["downvotes"] or 0)
fp += 1 if row["resolved"] == 0 else 0 # 0/1/NULL; 0 = unresolved
ac += 1 if row["resolved"] == 1 else 0
ac += int(row["upvotes"] or 0)
if row["reply_bodies"] and _is_negation_reply(row["reply_bodies"]):
fp += 2
return fp, ac
def _short_problem(problem: str, n: int = 100) -> str:
s = (problem or "").strip().replace("\n", " ")
return s if len(s) <= n else s[: n - 1] + "…"
def _restraint_stats(conn: sqlite3.Connection) -> dict:
"""How often does the bot post findings on PRs that received zero
bot findings (= presumably clean)? Looks at `review.findings_total`
if present, otherwise counts `inline_finding` per PR.
NOTE: until `post_inline_review` records `findings_total`, this falls
back to "PRs with at least one finding row" which is an underestimate
(a bot review with zero findings leaves no row).
"""
total_prs_with_review = conn.execute(
"SELECT COUNT(DISTINCT repo || '#' || pr) FROM review"
).fetchone()[0]
prs_with_findings = conn.execute(
"SELECT COUNT(DISTINCT repo || '#' || pr) FROM inline_finding"
).fetchone()[0]
if total_prs_with_review == 0:
return {"total": 0, "noisy": 0, "ratio": 0.0}
# This is currently "PRs where the bot left at least one inline
# comment". A precise "findings_total per review" needs
# post_inline_review to record it (TODO in the wiring step). Until
# then, treat this as a floor: real noise is >= this.
return {
"total": total_prs_with_review,
"noisy": prs_with_findings,
"ratio": prs_with_findings / total_prs_with_review,
}
def _case_review_queue(conn: sqlite3.Connection, limit: int = 30) -> list[dict]:
"""Findings that humans pushed back on — for manual re-review."""
rows = feedback.findings_with_votes(conn)
cases = []
for r in rows:
fp_score, _ = _net_score(r)
if fp_score <= 0:
continue
cases.append({
"posthash": r["posthash"],
"repo": r["repo"],
"pr": r["pr"],
"path": r["path"],
"line": r["line"],
"severity": r["severity"],
"problem": _short_problem(r["problem"], 200),
"fp_score": fp_score,
"upvotes": r["upvotes"] or 0,
"downvotes": r["downvotes"] or 0,
"resolved": r["resolved"],
"reply_count": r["reply_count"] or 0,
"reply_excerpt": _short_problem(r["reply_bodies"] or "", 200),
})
cases.sort(key=lambda c: c["fp_score"], reverse=True)
return cases[:limit]
def _format_table(headers: list[str], rows: list[list[str]]) -> str:
if not rows:
return "_none yet_\n"
out = ["| " + " | ".join(headers) + " |",
"|" + "|".join(["---"] * len(headers)) + "|"]
for row in rows:
out.append("| " + " | ".join(row) + " |")
return "\n".join(out) + "\n"
def _md_escape(s: str) -> str:
"""Escape pipes + newlines so the value stays in one table cell."""
return (s or "").replace("|", "\\|").replace("\n", " ").strip()
# ---------------------------------------------------------------------------
# Main report builder
# ---------------------------------------------------------------------------
def analyze(db_path: str, *, since_ts: Optional[int] = None,
as_json: bool = False) -> str:
"""Build the daily report. Returns a markdown string by default;
`as_json=True` returns a structured dict (for tests + automation)."""
conn = feedback.init(db_path)
try:
findings = list(feedback.findings_with_votes(conn, since_ts=since_ts))
total_findings = len(findings)
repo_set = {f["repo"] for f in findings}
case_queue = _case_review_queue(conn)
restraint = _restraint_stats(conn)
# Compute scores
scored: list[tuple[int, int, sqlite3.Row]] = []
for f in findings:
fp, ac = _net_score(f)
scored.append((fp, ac, f))
# Top false-positive patterns (sorted by fp score, deduped by posthash).
# `occurrences` comes from the inline_finding row — posthash UNIQUE
# means a single row can carry a count > 1 (set by record_inline_finding's
# ON CONFLICT DO UPDATE).
fp_by_hash: dict[str, dict] = {}
for fp, ac, f in scored:
if fp <= 0:
continue
ph = f["posthash"]
entry = fp_by_hash.setdefault(ph, {
"posthash": ph, "fp_score": 0, "ac_score": 0,
"repo": f["repo"], "path": f["path"], "line": f["line"],
"severity": f["severity"], "problem": f["problem"],
"occurrences": f["occurrences"], "upvs": 0, "downs": 0,
"resolved_true": 0, "resolved_false": 0,
})
entry["fp_score"] += fp
entry["ac_score"] += ac
entry["upvs"] += f["upvotes"] or 0
entry["downs"] += f["downvotes"] or 0
if f["resolved"] == 1:
entry["resolved_true"] += 1
elif f["resolved"] == 0:
entry["resolved_false"] += 1
fp_sorted = sorted(
fp_by_hash.values(), key=lambda e: e["fp_score"], reverse=True,
)[:TOP_N]
# Top accepted patterns
ac_by_hash: dict[str, dict] = {}
for fp, ac, f in scored:
if ac <= 0:
continue
ph = f["posthash"]
entry = ac_by_hash.setdefault(ph, {
"posthash": ph, "ac_score": 0, "fp_score": 0,
"repo": f["repo"], "path": f["path"], "line": f["line"],
"severity": f["severity"], "problem": f["problem"],
"occurrences": f["occurrences"], "upvs": 0, "downs": 0,
"resolved_true": 0,
})
entry["ac_score"] += ac
entry["fp_score"] += fp
entry["upvs"] += f["upvotes"] or 0
entry["downs"] += f["downvotes"] or 0
if f["resolved"] == 1:
entry["resolved_true"] += 1
ac_sorted = sorted(
ac_by_hash.values(), key=lambda e: e["ac_score"], reverse=True,
)[:TOP_N]
# Restraint recommendation
if restraint["ratio"] > RESTRAINT_NOISE_THRESHOLD:
restraint_msg = (
f"⚠️ Bot posted findings on **{restraint['ratio']:.0%}** of "
f"reviewed PRs ({restraint['noisy']} / {restraint['total']}). "
f"Above the {RESTRAINT_NOISE_THRESHOLD:.0%} threshold — "
"consider raising `.pr-review.json:severity_threshold` to "
"`medium` or `high` for noisy repos, or adding "
"`patterns.deny` to skip stylistic-only findings."
)
else:
restraint_msg = (
f"✅ Bot stayed quiet on **{1 - restraint['ratio']:.0%}** of "
f"reviewed PRs ({restraint['total'] - restraint['noisy']} / "
f"{restraint['total']}). Restraint OK."
)
if as_json:
return json.dumps({
"total_findings": total_findings,
"repos_seen": sorted(repo_set),
"restraint": restraint,
"top_false_positive": fp_sorted,
"top_accepted": ac_sorted,
"case_review_queue": case_queue,
"restraint_msg": restraint_msg,
}, indent=2)
# Markdown
ts_str = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC")
out = [f"# pragent feedback report — {ts_str}", ""]
out.append(f"- **findings analyzed**: {total_findings}")
out.append(f"- **repos with feedback**: {len(repo_set)} "
f"({', '.join(sorted(repo_set))})")
out.append(f"- **case-review queue**: {len(case_queue)} disagreement(s)")
out.append("")
out.append("## Restraint")
out.append("")
out.append(restraint_msg)
out.append("")
out.append("> DoorDash rule (2026-07-06): *excessive noise on clean "
"code is its own failure mode*. `severity_threshold` + "
"`patterns.deny` are the knobs that dial restraint.")
out.append("")
out.append(f"## Top {len(fp_sorted)} false-positive candidates")
out.append("")
out.append("Aggregated by `posthash` (path:line:severity:problem). "
"Sort key = downvotes + unresolved + negation-phrase replies "
"− upvotes − resolved.")
out.append("")
rows = []
for e in fp_sorted:
rows.append([
str(e["fp_score"]),
f"`{_md_escape(e['repo'])}`",
f"`{_md_escape(e['path'])}:{e['line']}`",
e["severity"],
_md_escape(_short_problem(e["problem"])),
f"👍{e['upvs']} 👎{e['downs']}",
f"✅{e['resolved_true']} ❌{e['resolved_false']}",
str(e["occurrences"]),
])
out.append(_format_table(
["FP", "repo", "path:line", "sev", "problem",
"votes", "resolved", "seen"],
rows,
))
out.append("")
out.append("_Review each row before adding it to "
"`.pr-review.json:instructions`. Human reactions are NOT "
"ground truth (DoorDash, 2026-07-06: authors accept/reject "
"for workflow reasons) — re-read the PR before acting._")
out.append("")
out.append(f"## Top {len(ac_sorted)} accepted patterns")
out.append("")
out.append("Aggregated by posthash. Sort key = upvotes + resolved − "
"downvotes − unresolved − negation-phrase replies.")
out.append("")
rows = []
for e in ac_sorted:
rows.append([
str(e["ac_score"]),
f"`{_md_escape(e['repo'])}`",
f"`{_md_escape(e['path'])}:{e['line']}`",
e["severity"],
_md_escape(_short_problem(e["problem"])),
f"👍{e['upvs']} 👎{e['downs']}",
f"✅{e['resolved_true']}",
str(e["occurrences"]),
])
out.append(_format_table(
["AC", "repo", "path:line", "sev", "problem",
"votes", "resolved", "seen"],
rows,
))
out.append("")
out.append("_Promote widely-accepted patterns into the shared "
"`architecture.md` on Nexus raw-hosted (or the per-repo "
"`additional_context_urls`). These become part of the "
"prompt-cached prefix → ~0 marginal cost on step 2+._")
out.append("")
out.append(f"## Case-review queue ({len(case_queue)})")
out.append("")
if not case_queue:
out.append("_No disagreements recorded yet. Once humans start "
"reacting 👎 / leaving replies / not resolving bot "
"comments, cases will appear here._")
else:
out.append("Each row needs a human to re-read the original PR and "
"decide: was the bot right? If not, draft an "
"`instructions` addendum or a `patterns.deny` rule.")
out.append("")
for c in case_queue:
url = (
f"https://gitea.marcospaulo.dev.br/{c['repo']}/pulls/"
f"{c['pr']}/files#r{c['posthash']}"
)
out.append(f"### FP={c['fp_score']} · {c['repo']}#{c['pr']}")
out.append(
f"- file: `{_md_escape(c['path'])}:{c['line']}` · "
f"severity: `{c['severity']}`",
)
out.append(f"- problem: {_md_escape(c['problem'])}")
out.append(
f"- signals: 👍{c['upvotes']} 👎{c['downvotes']} · "
f"resolved={c['resolved']} · replies={c['reply_count']}",
)
if c["reply_excerpt"]:
out.append(
f"- last reply: {_md_escape(c['reply_excerpt'])}",
)
out.append(f"- posthash: `{c['posthash']}`")
out.append("")
out.append("## Where this report goes")
out.append("")
out.append("- **Per-repo actions** (`.pr-review.json:instructions`, "
"`patterns.deny`, `severity_threshold`): edit the file on "
"`main` via a regular PR. The next PR review picks up the "
"change automatically.")
out.append("- **Cross-repo actions** (shared house-rules): update the "
"`PRAGENT_ADDITIONAL_CONTEXT_URL` document on Nexus "
"raw-hosted (`canalhandia/architecture.md` etc).")
out.append("- **Benchmark gate** (DoorDash pattern): before changing "
"the model / prompt / context window, replay this report "
"against the labeled `posthash` corpus. If a candidate "
"addendum flips ≥ 1 currently-accepted finding into "
"false-positive, drop it.")
out.append("")
out.append(f"_Generated from `{db_path}` by `feedback_analyze.py`._")
return "\n".join(out)
finally:
conn.close()
# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------
def main() -> int:
p = argparse.ArgumentParser(description="Build the daily feedback report.")
p.add_argument("--db", default=os.environ.get(
"PRAGENT_FEEDBACK_DB", "/data/feedback.db",
))
p.add_argument("--since", type=int, default=None,
help="Unix timestamp; only include findings posted since")
p.add_argument("--json", action="store_true",
help="Emit structured JSON instead of markdown")
p.add_argument("--out", default="-",
help="Write to this path instead of stdout ('-' = stdout)")
args = p.parse_args()
out = analyze(args.db, since_ts=args.since, as_json=args.json)
if args.out == "-":
print(out)
else:
with open(args.out, "w") as f:
f.write(out)
print(f"wrote {args.out}", file=sys.stderr)
return 0
if __name__ == "__main__":
import sys
raise SystemExit(main())
pilot/feedback/harvest.py
+394
View File
@@ -0,0 +1,394 @@
"""pragent pilot — feedback harvester.
For each PR the webhook server is about to review, walk back through the
Gitea-side state of every bot comment from every prior review on that PR
and record:
- reactions on the review body + on each inline comment
- thread-resolved state (Gitea's `resolver` field; non-empty = resolved)
- replies (issue-comments with `review_comment_id` matching ours)
- the bot's own findings_count + inline_count per review (for the
restraint metric)
Everything is best-effort. A single 404 or 5xx is logged and skipped — we
must never abort a review because the feedback DB had a hiccup.
The harvester is intentionally separate from `review_pr` so it can be
called independently (e.g. by the daily analyzer's "backfill" mode) and
tested in isolation against a mocked Gitea client.
"""
from __future__ import annotations
import json
import logging
import re
import time
import urllib.parse
import urllib.request
from typing import Optional
import ai_review # used as ai_review.gitea_get(...) so test mocks land on the binding
from feedback import (
init,
record_inline_finding,
record_reaction,
record_reply,
record_review,
record_thread_state,
posthash,
)
log = logging.getLogger("pragent.feedback.harvest")
# Reviewer identity — only collect feedback on comments authored by us.
# Avoids harvesting reactions on human comments (which we never want to
# count toward "bot usefulness").
REVIEWER_LOGIN = "pragent-bot"
# Reactions content tokens Gitea uses. We track +1 / -1 explicitly; the
# others are stored as-is so the analyzer can mine them (👀 eyes,
# laugh, hooray, confused, heart, rocket, …) without hardcoding a list
# that drifts across Gitea versions.
POSITIVE_REACTIONS = {"+1", "heart", "hooray", "laugh", "rocket"}
NEGATIVE_REACTIONS = {"-1", "confused"}
# Note: Gitea's `eyes` reaction (👀) means "I'm watching" — not approval
# or disapproval. Treated as neutral by the analyzer.
# Phrases that, in a reply, indicate the author thinks the bot's finding
# was wrong. Casing + punctuation ignored; substring match is good enough
# (false positives in the analyzer cost a human minute; false negatives
# hide regressions).
FALSE_POSITIVE_PHRASES = (
"false positive", "not actually", "this is fine", "this is intentional",
"not a bug", "intentional", "wrong here", "isn't actually",
"is not actually", "don't think this is", "i disagree", "this isn't right",
"this is correct", "this is expected", "by design", "this is by design",
)
# Gitea review-comment payload includes a 'body' field that may carry our
# sha marker + severity header. We extract severity + path/line from it
# as a fallback when the finding wasn't already seeded at post-time (old
# reviews before feedback.py existed).
SEV_RE = re.compile(r"\*\*\[(CRITICAL|HIGH|MEDIUM|LOW|INFO)\]\*\*", re.IGNORECASE)
PATH_LINE_RE = re.compile(r"`([^?:\n]+?):(\d+)`")
SHA_MARKER_RE = re.compile(r"<!--\s*pragent:sha=([0-9a-f]+)\s*-->", re.IGNORECASE)
# ---------------------------------------------------------------------------
# Low-level HTTP — tolerant JSON parse (Gitea sometimes returns `null` where
# we expect `[]`, e.g. reactions on a fresh comment)
# ---------------------------------------------------------------------------
def _gitea_get_json(api: str, repo: str, path: str, token: str) -> tuple[int, object]:
status, raw = ai_review.gitea_get(api, repo, path, token)
if status != 200:
return status, None
try:
return status, json.loads(raw.decode("utf-8", errors="replace"))
except (json.JSONDecodeError, ValueError):
return status, None
# ---------------------------------------------------------------------------
# Parse helpers
# ---------------------------------------------------------------------------
def _parse_severity(body: str) -> str:
m = SEV_RE.search(body or "")
return m.group(1).upper() if m else "INFO"
def _parse_path_line(body: str) -> tuple[Optional[str], Optional[int]]:
m = PATH_LINE_RE.search(body or "")
if not m:
return None, None
path = m.group(1).strip()
try:
return path, int(m.group(2))
except ValueError:
return path, None
def _parse_sha(body: str) -> Optional[str]:
m = SHA_MARKER_RE.search(body or "")
return m.group(1) if m else None
def _is_negation_reply(body: str) -> bool:
if not body:
return False
norm = body.lower()
return any(p in norm for p in FALSE_POSITIVE_PHRASES)
# ---------------------------------------------------------------------------
# Reaction classification (cheap, used by the analyzer — not the harvester
# itself)
# ---------------------------------------------------------------------------
def classify_reaction(content: str) -> str:
"""Bucket a reaction into 'positive', 'negative', or 'neutral'."""
c = (content or "").strip().lower()
if c in POSITIVE_REACTIONS:
return "positive"
if c in NEGATIVE_REACTIONS:
return "negative"
return "neutral"
# ---------------------------------------------------------------------------
# Main harvest entry
# ---------------------------------------------------------------------------
def harvest_for_pr(
*,
api: str,
token: str,
repo: str,
pr_index: int,
db_path: str,
page_size: int = 50,
) -> dict:
"""Walk every bot-authored review on the given PR and record reactions
+ thread state + replies. Returns a stats dict for logging.
`db_path` is the SQLite file path (env: `PRAGENT_FEEDBACK_DB`,
typically `/data/feedback.db` mounted via the `feedback-data` PVC).
"""
conn = init(db_path)
stats = {
"reviews_seen": 0, "findings_seen": 0,
"reactions_recorded": 0, "thread_states_recorded": 0,
"replies_recorded": 0, "errors": 0,
}
try:
# 1. List every review on the PR (paginated, but PRs rarely have >page_size)
status, payload = _gitea_get_json(
api, repo, f"pulls/{pr_index}/reviews?per_page={page_size}", token,
)
if status != 200 or not isinstance(payload, list):
log.info("harvest: reviews list failed status=%d", status)
stats["errors"] += 1
return stats
for rev in payload:
user = (rev.get("user") or {}).get("login", "")
if user != REVIEWER_LOGIN:
continue
stats["reviews_seen"] += 1
review_id_gitea = rev.get("id")
head_sha = rev.get("commit_id", "")
review_body = rev.get("body", "") or ""
body_sha = _parse_sha(review_body)
# Trust the sha marker inside the body — Gitea's commit_id field is
# for the LAST commit, not necessarily the reviewed head. If we
# can't find a marker, fall back to commit_id.
effective_sha = body_sha or head_sha
created_at = _parse_iso_ts(rev.get("created_at", ""))
db_review_id = record_review(
conn, repo=repo, pr=pr_index, head_sha=effective_sha,
review_id_gitea=review_id_gitea,
posted_at=created_at,
)
# 2. Inline comments for this review
if review_id_gitea is None:
continue
rstatus, rpayload = _gitea_get_json(
api, repo, f"pulls/{pr_index}/reviews/{review_id_gitea}/comments",
token,
)
if rstatus != 200 or not isinstance(rpayload, list):
stats["errors"] += 1
continue
for ic in rpayload:
ic_id = ic.get("id")
if ic_id is None:
continue
ic_body = ic.get("body", "") or ""
ic_path = ic.get("path")
ic_line = ic.get("position") or ic.get("line")
ic_severity = _parse_severity(ic_body)
# Fall back to body parse when Gitea didn't echo path/line
if not ic_path or not ic_line:
bp, bl = _parse_path_line(ic_body)
ic_path = ic_path or bp
ic_line = ic_line or bl
if not ic_path or not ic_line:
log.info(
"harvest: inline %s missing path/line, skipping", ic_id,
)
continue
finding_id = record_inline_finding(
conn, review_id=db_review_id, repo=repo, pr=pr_index,
path=ic_path, line=ic_line, severity=ic_severity,
problem=_strip_severity_header(ic_body),
fix="", suggestion="",
comment_id=ic_id,
posted_at=created_at,
)
stats["findings_seen"] += 1
if finding_id is None:
continue
# 3. Reactions on the inline comment
react_status, react_payload = _gitea_get_json(
api, repo, f"issues/comments/{ic_id}/reactions", token,
)
if react_status == 200 and isinstance(react_payload, list):
for r in react_payload:
ruser = (r.get("user") or {}).get("login", "") or "?"
# Gitea has occasionally returned `content` as a
# dict on older versions; coerce to str defensively.
rcontent = str(r.get("content") or "").strip()
if not rcontent:
continue
if record_reaction(
conn, comment_id=ic_id, user=ruser,
content=rcontent,
created_at=_parse_iso_ts(r.get("created_at", "")),
):
stats["reactions_recorded"] += 1
# 4. Thread state (Gitea's `resolver` field on the inline
# comment). Some Gitea versions serialize this as a user
# object ({login, ...}) instead of a username string —
# coerce defensively before calling .strip().
resolver_raw = ic.get("resolver")
if isinstance(resolver_raw, dict):
resolver = (resolver_raw.get("login") or "").strip()
else:
resolver = str(resolver_raw or "").strip()
if resolver_raw is not None: # field present, even if ""
record_thread_state(
conn, finding_id=finding_id,
resolved=bool(resolver),
)
stats["thread_states_recorded"] += 1
# 5. Replies on this review (issue-comments whose
# `review_comment_id` points at one of our inline comments).
# Some Gitea versions don't expose `review_comment_id` on the
# issue-comment endpoint — in that case `replies` stays
# empty; we degrade gracefully.
try:
_harvest_replies(
api=api, repo=repo, token=token,
pr_index=pr_index, review_id=review_id_gitea,
inline_comments=rpayload, conn=conn,
stats=stats,
)
except Exception as e:
log.info("harvest: replies fetch failed: %s", e)
stats["errors"] += 1
finally:
conn.close()
return stats
def _harvest_replies(
*, api: str, repo: str, token: str, pr_index: int,
review_id: int, inline_comments: list, conn, stats: dict,
) -> None:
"""Fetch issue comments on this PR; record those whose
`review_comment_id` matches one of our inline comment IDs.
Gitea 1.26 doesn't include that field — we fall back to fetching each
inline comment individually via `issues/comments/{id}` (does include
the field) only if the bulk fetch is empty.
"""
inline_ids = {c.get("id") for c in inline_comments if c.get("id") is not None}
if not inline_ids:
return
status, payload = _gitea_get_json(
api, repo, f"issues/{pr_index}/comments?per_page=100", token,
)
if status != 200 or not isinstance(payload, list):
return
# Build mapping inline_id -> finding_id (one SELECT instead of N)
rows = conn.execute(
"SELECT comment_id, id FROM inline_finding WHERE comment_id IN ("
+ ",".join("?" * len(inline_ids)) + ")",
list(inline_ids),
).fetchall()
inline_to_finding = {r[0]: r[1] for r in rows}
for c in payload:
rcid = c.get("review_comment_id")
if not rcid or rcid not in inline_to_finding:
continue
author = (c.get("user") or {}).get("login", "") or "?"
body = c.get("body", "") or ""
ts = _parse_iso_ts(c.get("created_at", ""))
if record_reply(
conn, finding_id=inline_to_finding[rcid],
author=author, body=body, created_at=ts,
):
stats["replies_recorded"] += 1
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _strip_severity_header(body: str) -> str:
"""Drop the leading `**[SEVERITY]**` so the posthash captures the
substance, not the severity label."""
return SEV_RE.sub("", body or "", count=1).strip()
def _parse_iso_ts(s: str) -> int:
if not s:
return int(time.time())
try:
# Python 3.11+ fromisoformat tolerates the trailing 'Z'.
return int(__import__("datetime").datetime.fromisoformat(
s.replace("Z", "+00:00")
).timestamp())
except Exception:
return int(time.time())
# ---------------------------------------------------------------------------
# CLI for manual backfill / first-time seed
# ---------------------------------------------------------------------------
def main() -> int:
import argparse, os
p = argparse.ArgumentParser(
description="Harvest reactions/threads/replies on bot PR comments.",
)
p.add_argument("--api", default=os.environ.get(
"GITEA_API", "http://gitea-http.gitea.svc.cluster.local:3000",
))
p.add_argument("--token", default=os.environ.get("PRAGENT_BOT_TOKEN", ""))
p.add_argument("--repo", required=True, help="owner/name")
p.add_argument("--pr", type=int, required=True, help="PR index")
p.add_argument("--db", default=os.environ.get(
"PRAGENT_FEEDBACK_DB", "/data/feedback.db",
))
args = p.parse_args()
if not args.token:
print("PRAGENT_BOT_TOKEN required", flush=True)
return 2
logging.basicConfig(level=logging.INFO)
stats = harvest_for_pr(
api=args.api, token=args.token,
repo=args.repo, pr_index=args.pr, db_path=args.db,
)
print(json.dumps(stats), flush=True)
return 0
if __name__ == "__main__":
raise SystemExit(main())
pilot/feedback/post.py
+128
View File
@@ -0,0 +1,128 @@
"""pragent pilot — daily feedback report delivery.
Calls `feedback_analyze.analyze()` and posts the markdown report as a
comment on a single long-lived "feedback roll-up" issue in
`gitea_admin/pragent`. Comments are append-only history — one comment per
run, timestamped in the body. This keeps every report in one place, easy
to scroll, and avoids the issue-explosion of "one issue per day".
If the issue doesn't exist yet, create it. Subsequent runs just add a
new comment.
Designed for the daily K8s CronJob (`k8s/pragent-feedback-cronjob.yaml`)
but runnable from CLI for ad-hoc checks.
Env:
GITEA_API in-cluster Gitea base URL
PRAGENT_BOT_TOKEN bot token (Write collaborator on gitea_admin/pragent)
PRAGENT_FEEDBACK_DB path to SQLite (default /data/feedback.db)
PRAGENT_FEEDBACK_ISSUE_REPO default gitea_admin/pragent
PRAGENT_FEEDBACK_ISSUE_TITLE default "pragent feedback roll-up"
"""
from __future__ import annotations
import argparse
import json
import logging
import os
import sys
import ai_review
from feedback_analyze import analyze
log = logging.getLogger("pragent.feedback.post")
REPO_DEFAULT = "gitea_admin/pragent"
TITLE_DEFAULT = "pragent feedback roll-up"
def _find_or_create_issue(api: str, token: str, repo: str, title: str) -> int:
"""Locate the open issue with this title; create one if missing.
Gitea's issue search is via `GET /repos/{o}/{r}/issues?state=open&q=...`
(q matches title + body). We filter client-side for the exact title
to avoid query-text false matches.
"""
status, raw = ai_review.gitea_get(api, repo, "issues?state=open&per_page=50", token)
if status == 200:
try:
for issue in json.loads(raw):
if issue.get("title") == title:
# NB: the comment URL needs the per-repo `number`, not the
# global `id`. `id=60 num=8` for an early-N create; we want
# `num=8` for `/repos/o/r/issues/8/comments`.
return int(issue["number"])
except (json.JSONDecodeError, ValueError, KeyError):
pass
# Create
status, raw = ai_review.gitea_post(
api, repo, "issues", token,
{"title": title, "body": "pragent feedback roll-up — auto-created."},
)
if status not in (200, 201):
raise RuntimeError(f"issue create failed: HTTP {status} body={raw[:200]!r}")
return int(json.loads(raw)["number"])
def _post_comment(api: str, token: str, repo: str, issue_number: int, body: str) -> int:
status, raw = ai_review.gitea_post(
api, repo, f"issues/{issue_number}/comments", token, {"body": body},
)
if status not in (200, 201):
raise RuntimeError(f"comment post failed: HTTP {status} body={raw[:200]!r}")
return json.loads(raw)["id"]
def deliver(
*, api: str, token: str, db_path: str,
repo: str = REPO_DEFAULT, title: str = TITLE_DEFAULT,
since_ts: int | None = None,
) -> dict:
"""Build the report and post it as a comment. Returns a stats dict."""
report = analyze(db_path, since_ts=since_ts)
issue_id = _find_or_create_issue(api, token, repo, title)
comment_id = _post_comment(api, token, repo, issue_id, report)
return {
"repo": repo, "issue_id": issue_id, "comment_id": comment_id,
"report_bytes": len(report.encode()),
}
def main() -> int:
p = argparse.ArgumentParser(
description="Post the daily feedback report to Gitea.",
)
p.add_argument("--api", default=os.environ.get(
"GITEA_API", "http://gitea-http.gitea.svc.cluster.local:3000",
))
p.add_argument("--token", default=os.environ.get("PRAGENT_BOT_TOKEN", ""))
p.add_argument("--db", default=os.environ.get(
"PRAGENT_FEEDBACK_DB", "/data/feedback.db",
))
p.add_argument("--repo", default=os.environ.get(
"PRAGENT_FEEDBACK_ISSUE_REPO", REPO_DEFAULT,
))
p.add_argument("--title", default=os.environ.get(
"PRAGENT_FEEDBACK_ISSUE_TITLE", TITLE_DEFAULT,
))
p.add_argument("--since", type=int, default=None,
help="Unix timestamp; only include findings posted since")
args = p.parse_args()
if not args.token:
print("PRAGENT_BOT_TOKEN required", flush=True)
return 2
logging.basicConfig(level=logging.INFO)
stats = deliver(
api=args.api, token=args.token, db_path=args.db,
repo=args.repo, title=args.title, since_ts=args.since,
)
print(json.dumps(stats), flush=True)
return 0
if __name__ == "__main__":
raise SystemExit(main())
pilot/feedback/scores.py
+247
View File
@@ -0,0 +1,247 @@
#!/usr/bin/env python3
"""pragent pilot — feedback DB to Langfuse scores.
`feedback.db` already records every reaction, thread resolution and reply a
maintainer leaves on a bot comment. That is the only ground truth pragent has
about whether a finding was any good, and until now it went to a markdown report
nobody reads and nowhere else. This ships it to Langfuse as session-level
scores, so "was the reviewer right" sits on the same axis as "what did it cost".
Session, not trace
------------------
`langfuse_trace` sets `sessionId` to `"{repo}#{pr}"` and lets the trace id be a
fresh uuid per review. Feedback arrives days later against a PR, not against one
particular re-run of the reviewer, and nothing in `feedback.db` records which
trace produced which comment. Scoring the session is therefore both the
available join and the honest granularity: this is feedback on the review of
this PR, not on one invocation.
Two scores, deliberately separated
----------------------------------
* `review_engagement` — the share of a PR's findings that got any human
response at all. This is a signal about the *feedback loop*, not the
reviewer: at the time of writing it is 0.0 across all 113 recorded reviews,
which is exactly the fact that makes an accuracy metric impossible today.
It must be watched first, because every other quality number is vapour
until it moves.
* `review_acceptance` — net verdict over the findings that *did* get a
response: (upvotes + resolved) - (downvotes + negation replies), normalised
to -1..1. Computed only over engaged findings, so an ignored review scores
`None` rather than 0. Zero would read as "humans judged this exactly
neutral"; the truth is nobody looked.
Fail-open and idempotent. Score ids are derived from (repo, pr, name) so a
re-run overwrites rather than duplicates.
"""
from __future__ import annotations
import argparse
import json
import os
import sqlite3
import sys
import uuid
from datetime import datetime, timezone
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from feedback_harvest import classify_reaction, _is_negation_reply # noqa: E402
REVIEW_ENGAGEMENT = "review_engagement"
REVIEW_ACCEPTANCE = "review_acceptance"
# Stable namespace so the same (repo, pr, score) always produces the same score
# id — Langfuse treats a repeated id as an update, which is what a backfill of a
# still-accumulating PR should do.
_NS = uuid.UUID("6f1d9c2e-4a77-4f2a-9c1a-0d3b5e8a7c41")
def _score_id(repo: str, pr: int, name: str) -> str:
return str(uuid.uuid5(_NS, f"{repo}#{pr}#{name}"))
def collect_pr_feedback(conn: sqlite3.Connection, repo: str, pr: int) -> dict:
"""Tally one PR's findings and the human responses attached to them.
Returns counts only — the scoring maths lives in `score_pr` so it can be
tested without a database.
"""
rows = conn.execute(
"SELECT id, comment_id FROM inline_finding WHERE repo = ? AND pr = ?",
(repo, pr),
).fetchall()
total = len(rows)
engaged = 0
positive = 0
negative = 0
for row in rows:
fid = row["id"] if isinstance(row, sqlite3.Row) else row[0]
cid = row["comment_id"] if isinstance(row, sqlite3.Row) else row[1]
pos = neg = 0
if cid is not None:
for r in conn.execute(
"SELECT content FROM reaction WHERE comment_id = ?", (cid,)
):
kind = classify_reaction(r[0])
if kind == "positive":
pos += 1
elif kind == "negative":
neg += 1
for r in conn.execute(
"SELECT resolved FROM thread_state WHERE finding_id = ?", (fid,)
):
# A resolved thread means the maintainer acted on the finding.
if r[0]:
pos += 1
# A reply counts as engagement either way; only a negation phrase makes
# it a vote against. A neutral reply ("done", "good catch, but…") is
# deliberately not a positive vote — it says someone looked, not that
# they agreed.
replied = 0
for r in conn.execute(
"SELECT body FROM reply WHERE finding_id = ?", (fid,)
):
replied += 1
if _is_negation_reply(r[0]):
neg += 1
if pos or neg or replied:
engaged += 1
positive += pos
negative += neg
return {"total": total, "engaged": engaged, "positive": positive, "negative": negative}
def score_pr(tally: dict) -> dict:
"""Turn one PR's tally into score values.
`review_acceptance` is `None` when nothing was engaged — see the module
docstring on why that is not 0.
"""
total = int(tally.get("total") or 0)
engaged = int(tally.get("engaged") or 0)
pos = int(tally.get("positive") or 0)
neg = int(tally.get("negative") or 0)
engagement = round(engaged / total, 4) if total else None
acceptance = None
if pos or neg:
acceptance = round((pos - neg) / (pos + neg), 4)
return {REVIEW_ENGAGEMENT: engagement, REVIEW_ACCEPTANCE: acceptance}
def build_score_events(
repo: str, pr: int, values: dict, environment: str = "default",
timestamp: str | None = None,
) -> list[dict]:
"""`score-create` events for one PR's feedback.
Every event carries a timestamp: the ingestion endpoint rejects those that
do not, and it reports the rejection as a per-event 400 inside an HTTP 207,
which reads as success to a caller that only checks the status code.
"""
ts = timestamp or datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
events = []
for name, value in values.items():
if value is None:
continue
events.append(
{
"id": str(uuid.uuid4()),
"type": "score-create",
"timestamp": ts,
"body": {
"id": _score_id(repo, pr, name),
"sessionId": f"{repo}#{pr}",
"name": name,
"value": float(value),
"dataType": "NUMERIC",
"environment": environment,
"comment": f"from feedback.db · {repo}#{pr}",
},
}
)
return events
SCORE_CONFIGS = [
{
"name": REVIEW_ENGAGEMENT,
"dataType": "NUMERIC",
"minValue": 0,
"maxValue": 1,
"description": "Share of a PR's findings that drew any human reaction, resolution or reply. 0 = nobody engaged with the review.",
},
{
"name": REVIEW_ACCEPTANCE,
"dataType": "NUMERIC",
"minValue": -1,
"maxValue": 1,
"description": "Net human verdict over engaged findings: +1 all accepted, -1 all rejected. Absent when nothing was engaged.",
},
]
def iter_prs(conn: sqlite3.Connection):
for row in conn.execute(
"SELECT DISTINCT repo, pr FROM inline_finding ORDER BY repo, pr"
):
yield row[0], int(row[1])
def backfill(db_path: str, *, environment: str = "default", dry_run: bool = False) -> dict:
"""Score every PR in the feedback DB. Returns a summary dict."""
conn = sqlite3.connect(db_path)
conn.row_factory = sqlite3.Row
events: list[dict] = []
scanned = 0
engaged_prs = 0
try:
for repo, pr in iter_prs(conn):
scanned += 1
tally = collect_pr_feedback(conn, repo, pr)
values = score_pr(tally)
if (values.get(REVIEW_ENGAGEMENT) or 0) > 0:
engaged_prs += 1
events.extend(build_score_events(repo, pr, values, environment))
finally:
conn.close()
summary = {"prs_scanned": scanned, "prs_with_engagement": engaged_prs, "scores": len(events)}
if dry_run or not events:
summary["posted"] = False
return summary
import langfuse_trace
conf = langfuse_trace._enabled()
if conf is None:
summary["posted"] = False
summary["error"] = "Langfuse not configured (LANGFUSE_HOST / keys unset)"
return summary
host, pk, sk = conf
status = langfuse_trace._post(host, pk, sk, events, 15.0)
summary["posted"] = status in (200, 201, 207)
summary["http_status"] = status
return summary
def main() -> int:
ap = argparse.ArgumentParser(description="Ship feedback.db verdicts to Langfuse as scores")
ap.add_argument("--db", default=os.environ.get("PRAGENT_FEEDBACK_DB", "/data/feedback.db"))
ap.add_argument("--environment", default="default")
ap.add_argument("--dry-run", action="store_true")
args = ap.parse_args()
summary = backfill(args.db, environment=args.environment, dry_run=args.dry_run)
print(json.dumps(summary, indent=2))
return 0 if summary.get("posted") or args.dry_run else 1
if __name__ == "__main__":
raise SystemExit(main())
pilot/feedback.py → pilot/feedback/store.py
View File
pilot/feedback_analyze.py
+6 -418
View File
@@ -1,419 +1,7 @@
"""pragent pilot — daily feedback analyzer.
Reads `feedback.db` (written by `feedback_harvest.py`) and produces a
markdown report that:
1. Ranks inline findings by **net false-positive score** (downvotes +
unresolved + negation-phrase replies − upvotes − resolved). Top of
this list = "the bot has been wrong about this repeatedly". These
are the candidates that *might* belong in the per-repo
`.pr-review.json:instructions` addendum.
2. Ranks findings by **net acceptance** — repeated 👍 / resolution =
"the bot's framing here is genuinely useful". These can be promoted
to the shared `architecture.md` so they don't have to be re-derived
every PR.
3. Reports a **restraint metric** — for every PR where the bot posted
zero findings, count how often a human reviewer also posted zero
substantive review comments. When the bot is loud on clean code,
that's a false-positive rate we can act on (DoorDash lesson:
"excessive noise on clean code is its own failure mode").
4. Reports a **case-review queue** — every disagreement case (a
downvote, unresolved, or a reply matching `FALSE_POSITIVE_PHRASES`)
is listed in full so a human can re-read the original PR and decide
if the finding was right or wrong.
Output is plain markdown so it can be posted as a Gitea issue / comment
without rendering work. Designed to be reviewed by a human, not auto-
applied — per the DoorDash pattern, every material change to model /
prompt / context goes through a benchmark gate first; this report IS
that gate (or, more precisely, the queue feeding the gate).
Never raises. A bad DB / no data → returns a friendly empty-state report.
"""
from __future__ import annotations
import argparse
import json
import logging
import os
import sqlite3
from collections import defaultdict
from datetime import datetime, timezone
from typing import Optional
import feedback
from feedback_harvest import (
FALSE_POSITIVE_PHRASES,
classify_reaction,
_is_negation_reply, # noqa: F401 (re-exported for the test suite)
)
log = logging.getLogger("pragent.feedback.analyze")
# How many findings to surface in each top-list. Capped because the
# reports are read by humans; more than 20 per list and they skim.
TOP_N = 20
# Restraint threshold — fraction of "clean" PRs (zero findings) where
# the bot produced ANY findings. Above this we recommend `.pr-review.json:
# exclude_patterns` or a stricter `severity_threshold`.
RESTRAINT_NOISE_THRESHOLD = 0.25
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _net_score(row) -> tuple[int, int]:
"""Return (false_positive_score, acceptance_score) for one finding row.
FP signals: downvotes (+1), unresolved (+1), negation-phrase replies (+2).
Acceptance signals: upvotes (+1), resolved (+1).
"""
fp = 0
ac = 0
fp += int(row["downvotes"] or 0)
fp += 1 if row["resolved"] == 0 else 0 # 0/1/NULL; 0 = unresolved
ac += 1 if row["resolved"] == 1 else 0
ac += int(row["upvotes"] or 0)
if row["reply_bodies"] and _is_negation_reply(row["reply_bodies"]):
fp += 2
return fp, ac
def _short_problem(problem: str, n: int = 100) -> str:
s = (problem or "").strip().replace("\n", " ")
return s if len(s) <= n else s[: n - 1] + "…"
def _restraint_stats(conn: sqlite3.Connection) -> dict:
"""How often does the bot post findings on PRs that received zero
bot findings (= presumably clean)? Looks at `review.findings_total`
if present, otherwise counts `inline_finding` per PR.
NOTE: until `post_inline_review` records `findings_total`, this falls
back to "PRs with at least one finding row" which is an underestimate
(a bot review with zero findings leaves no row).
"""
total_prs_with_review = conn.execute(
"SELECT COUNT(DISTINCT repo || '#' || pr) FROM review"
).fetchone()[0]
prs_with_findings = conn.execute(
"SELECT COUNT(DISTINCT repo || '#' || pr) FROM inline_finding"
).fetchone()[0]
if total_prs_with_review == 0:
return {"total": 0, "noisy": 0, "ratio": 0.0}
# This is currently "PRs where the bot left at least one inline
# comment". A precise "findings_total per review" needs
# post_inline_review to record it (TODO in the wiring step). Until
# then, treat this as a floor: real noise is >= this.
return {
"total": total_prs_with_review,
"noisy": prs_with_findings,
"ratio": prs_with_findings / total_prs_with_review,
}
def _case_review_queue(conn: sqlite3.Connection, limit: int = 30) -> list[dict]:
"""Findings that humans pushed back on — for manual re-review."""
rows = feedback.findings_with_votes(conn)
cases = []
for r in rows:
fp_score, _ = _net_score(r)
if fp_score <= 0:
continue
cases.append({
"posthash": r["posthash"],
"repo": r["repo"],
"pr": r["pr"],
"path": r["path"],
"line": r["line"],
"severity": r["severity"],
"problem": _short_problem(r["problem"], 200),
"fp_score": fp_score,
"upvotes": r["upvotes"] or 0,
"downvotes": r["downvotes"] or 0,
"resolved": r["resolved"],
"reply_count": r["reply_count"] or 0,
"reply_excerpt": _short_problem(r["reply_bodies"] or "", 200),
})
cases.sort(key=lambda c: c["fp_score"], reverse=True)
return cases[:limit]
def _format_table(headers: list[str], rows: list[list[str]]) -> str:
if not rows:
return "_none yet_\n"
out = ["| " + " | ".join(headers) + " |",
"|" + "|".join(["---"] * len(headers)) + "|"]
for row in rows:
out.append("| " + " | ".join(row) + " |")
return "\n".join(out) + "\n"
def _md_escape(s: str) -> str:
"""Escape pipes + newlines so the value stays in one table cell."""
return (s or "").replace("|", "\\|").replace("\n", " ").strip()
# ---------------------------------------------------------------------------
# Main report builder
# ---------------------------------------------------------------------------
def analyze(db_path: str, *, since_ts: Optional[int] = None,
as_json: bool = False) -> str:
"""Build the daily report. Returns a markdown string by default;
`as_json=True` returns a structured dict (for tests + automation)."""
conn = feedback.init(db_path)
try:
findings = list(feedback.findings_with_votes(conn, since_ts=since_ts))
total_findings = len(findings)
repo_set = {f["repo"] for f in findings}
case_queue = _case_review_queue(conn)
restraint = _restraint_stats(conn)
# Compute scores
scored: list[tuple[int, int, sqlite3.Row]] = []
for f in findings:
fp, ac = _net_score(f)
scored.append((fp, ac, f))
# Top false-positive patterns (sorted by fp score, deduped by posthash).
# `occurrences` comes from the inline_finding row — posthash UNIQUE
# means a single row can carry a count > 1 (set by record_inline_finding's
# ON CONFLICT DO UPDATE).
fp_by_hash: dict[str, dict] = {}
for fp, ac, f in scored:
if fp <= 0:
continue
ph = f["posthash"]
entry = fp_by_hash.setdefault(ph, {
"posthash": ph, "fp_score": 0, "ac_score": 0,
"repo": f["repo"], "path": f["path"], "line": f["line"],
"severity": f["severity"], "problem": f["problem"],
"occurrences": f["occurrences"], "upvs": 0, "downs": 0,
"resolved_true": 0, "resolved_false": 0,
})
entry["fp_score"] += fp
entry["ac_score"] += ac
entry["upvs"] += f["upvotes"] or 0
entry["downs"] += f["downvotes"] or 0
if f["resolved"] == 1:
entry["resolved_true"] += 1
elif f["resolved"] == 0:
entry["resolved_false"] += 1
fp_sorted = sorted(
fp_by_hash.values(), key=lambda e: e["fp_score"], reverse=True,
)[:TOP_N]
# Top accepted patterns
ac_by_hash: dict[str, dict] = {}
for fp, ac, f in scored:
if ac <= 0:
continue
ph = f["posthash"]
entry = ac_by_hash.setdefault(ph, {
"posthash": ph, "ac_score": 0, "fp_score": 0,
"repo": f["repo"], "path": f["path"], "line": f["line"],
"severity": f["severity"], "problem": f["problem"],
"occurrences": f["occurrences"], "upvs": 0, "downs": 0,
"resolved_true": 0,
})
entry["ac_score"] += ac
entry["fp_score"] += fp
entry["upvs"] += f["upvotes"] or 0
entry["downs"] += f["downvotes"] or 0
if f["resolved"] == 1:
entry["resolved_true"] += 1
ac_sorted = sorted(
ac_by_hash.values(), key=lambda e: e["ac_score"], reverse=True,
)[:TOP_N]
# Restraint recommendation
if restraint["ratio"] > RESTRAINT_NOISE_THRESHOLD:
restraint_msg = (
f"⚠️ Bot posted findings on **{restraint['ratio']:.0%}** of "
f"reviewed PRs ({restraint['noisy']} / {restraint['total']}). "
f"Above the {RESTRAINT_NOISE_THRESHOLD:.0%} threshold — "
"consider raising `.pr-review.json:severity_threshold` to "
"`medium` or `high` for noisy repos, or adding "
"`patterns.deny` to skip stylistic-only findings."
)
else:
restraint_msg = (
f"✅ Bot stayed quiet on **{1 - restraint['ratio']:.0%}** of "
f"reviewed PRs ({restraint['total'] - restraint['noisy']} / "
f"{restraint['total']}). Restraint OK."
)
if as_json:
return json.dumps({
"total_findings": total_findings,
"repos_seen": sorted(repo_set),
"restraint": restraint,
"top_false_positive": fp_sorted,
"top_accepted": ac_sorted,
"case_review_queue": case_queue,
"restraint_msg": restraint_msg,
}, indent=2)
# Markdown
ts_str = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC")
out = [f"# pragent feedback report — {ts_str}", ""]
out.append(f"- **findings analyzed**: {total_findings}")
out.append(f"- **repos with feedback**: {len(repo_set)} "
f"({', '.join(sorted(repo_set))})")
out.append(f"- **case-review queue**: {len(case_queue)} disagreement(s)")
out.append("")
out.append("## Restraint")
out.append("")
out.append(restraint_msg)
out.append("")
out.append("> DoorDash rule (2026-07-06): *excessive noise on clean "
"code is its own failure mode*. `severity_threshold` + "
"`patterns.deny` are the knobs that dial restraint.")
out.append("")
out.append(f"## Top {len(fp_sorted)} false-positive candidates")
out.append("")
out.append("Aggregated by `posthash` (path:line:severity:problem). "
"Sort key = downvotes + unresolved + negation-phrase replies "
"− upvotes − resolved.")
out.append("")
rows = []
for e in fp_sorted:
rows.append([
str(e["fp_score"]),
f"`{_md_escape(e['repo'])}`",
f"`{_md_escape(e['path'])}:{e['line']}`",
e["severity"],
_md_escape(_short_problem(e["problem"])),
f"👍{e['upvs']} 👎{e['downs']}",
f"✅{e['resolved_true']} ❌{e['resolved_false']}",
str(e["occurrences"]),
])
out.append(_format_table(
["FP", "repo", "path:line", "sev", "problem",
"votes", "resolved", "seen"],
rows,
))
out.append("")
out.append("_Review each row before adding it to "
"`.pr-review.json:instructions`. Human reactions are NOT "
"ground truth (DoorDash, 2026-07-06: authors accept/reject "
"for workflow reasons) — re-read the PR before acting._")
out.append("")
out.append(f"## Top {len(ac_sorted)} accepted patterns")
out.append("")
out.append("Aggregated by posthash. Sort key = upvotes + resolved − "
"downvotes − unresolved − negation-phrase replies.")
out.append("")
rows = []
for e in ac_sorted:
rows.append([
str(e["ac_score"]),
f"`{_md_escape(e['repo'])}`",
f"`{_md_escape(e['path'])}:{e['line']}`",
e["severity"],
_md_escape(_short_problem(e["problem"])),
f"👍{e['upvs']} 👎{e['downs']}",
f"✅{e['resolved_true']}",
str(e["occurrences"]),
])
out.append(_format_table(
["AC", "repo", "path:line", "sev", "problem",
"votes", "resolved", "seen"],
rows,
))
out.append("")
out.append("_Promote widely-accepted patterns into the shared "
"`architecture.md` on Nexus raw-hosted (or the per-repo "
"`additional_context_urls`). These become part of the "
"prompt-cached prefix → ~0 marginal cost on step 2+._")
out.append("")
out.append(f"## Case-review queue ({len(case_queue)})")
out.append("")
if not case_queue:
out.append("_No disagreements recorded yet. Once humans start "
"reacting 👎 / leaving replies / not resolving bot "
"comments, cases will appear here._")
else:
out.append("Each row needs a human to re-read the original PR and "
"decide: was the bot right? If not, draft an "
"`instructions` addendum or a `patterns.deny` rule.")
out.append("")
for c in case_queue:
url = (
f"https://gitea.marcospaulo.dev.br/{c['repo']}/pulls/"
f"{c['pr']}/files#r{c['posthash']}"
)
out.append(f"### FP={c['fp_score']} · {c['repo']}#{c['pr']}")
out.append(
f"- file: `{_md_escape(c['path'])}:{c['line']}` · "
f"severity: `{c['severity']}`",
)
out.append(f"- problem: {_md_escape(c['problem'])}")
out.append(
f"- signals: 👍{c['upvotes']} 👎{c['downvotes']} · "
f"resolved={c['resolved']} · replies={c['reply_count']}",
)
if c["reply_excerpt"]:
out.append(
f"- last reply: {_md_escape(c['reply_excerpt'])}",
)
out.append(f"- posthash: `{c['posthash']}`")
out.append("")
out.append("## Where this report goes")
out.append("")
out.append("- **Per-repo actions** (`.pr-review.json:instructions`, "
"`patterns.deny`, `severity_threshold`): edit the file on "
"`main` via a regular PR. The next PR review picks up the "
"change automatically.")
out.append("- **Cross-repo actions** (shared house-rules): update the "
"`PRAGENT_ADDITIONAL_CONTEXT_URL` document on Nexus "
"raw-hosted (`canalhandia/architecture.md` etc).")
out.append("- **Benchmark gate** (DoorDash pattern): before changing "
"the model / prompt / context window, replay this report "
"against the labeled `posthash` corpus. If a candidate "
"addendum flips ≥ 1 currently-accepted finding into "
"false-positive, drop it.")
out.append("")
out.append(f"_Generated from `{db_path}` by `feedback_analyze.py`._")
return "\n".join(out)
finally:
conn.close()
# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------
def main() -> int:
p = argparse.ArgumentParser(description="Build the daily feedback report.")
p.add_argument("--db", default=os.environ.get(
"PRAGENT_FEEDBACK_DB", "/data/feedback.db",
))
p.add_argument("--since", type=int, default=None,
help="Unix timestamp; only include findings posted since")
p.add_argument("--json", action="store_true",
help="Emit structured JSON instead of markdown")
p.add_argument("--out", default="-",
help="Write to this path instead of stdout ('-' = stdout)")
args = p.parse_args()
out = analyze(args.db, since_ts=args.since, as_json=args.json)
if args.out == "-":
print(out)
else:
with open(args.out, "w") as f:
f.write(out)
print(f"wrote {args.out}", file=sys.stderr)
return 0
if __name__ == "__main__":
"""Compatibility import for feedback analysis."""
import importlib
import sys
raise SystemExit(main())
_module = importlib.import_module("feedback.analyze")
sys.modules[__name__] = _module
if __name__ == "__main__":
raise SystemExit(_module.main())
pilot/feedback_harvest.py
+6 -393
View File
@@ -1,394 +1,7 @@
"""pragent pilot — feedback harvester.
For each PR the webhook server is about to review, walk back through the
Gitea-side state of every bot comment from every prior review on that PR
and record:
- reactions on the review body + on each inline comment
- thread-resolved state (Gitea's `resolver` field; non-empty = resolved)
- replies (issue-comments with `review_comment_id` matching ours)
- the bot's own findings_count + inline_count per review (for the
restraint metric)
Everything is best-effort. A single 404 or 5xx is logged and skipped — we
must never abort a review because the feedback DB had a hiccup.
The harvester is intentionally separate from `review_pr` so it can be
called independently (e.g. by the daily analyzer's "backfill" mode) and
tested in isolation against a mocked Gitea client.
"""
from __future__ import annotations
import json
import logging
import re
import time
import urllib.parse
import urllib.request
from typing import Optional
import ai_review # used as ai_review.gitea_get(...) so test mocks land on the binding
from feedback import (
init,
record_inline_finding,
record_reaction,
record_reply,
record_review,
record_thread_state,
posthash,
)
log = logging.getLogger("pragent.feedback.harvest")
# Reviewer identity — only collect feedback on comments authored by us.
# Avoids harvesting reactions on human comments (which we never want to
# count toward "bot usefulness").
REVIEWER_LOGIN = "pragent-bot"
# Reactions content tokens Gitea uses. We track +1 / -1 explicitly; the
# others are stored as-is so the analyzer can mine them (👀 eyes,
# laugh, hooray, confused, heart, rocket, …) without hardcoding a list
# that drifts across Gitea versions.
POSITIVE_REACTIONS = {"+1", "heart", "hooray", "laugh", "rocket"}
NEGATIVE_REACTIONS = {"-1", "confused"}
# Note: Gitea's `eyes` reaction (👀) means "I'm watching" — not approval
# or disapproval. Treated as neutral by the analyzer.
# Phrases that, in a reply, indicate the author thinks the bot's finding
# was wrong. Casing + punctuation ignored; substring match is good enough
# (false positives in the analyzer cost a human minute; false negatives
# hide regressions).
FALSE_POSITIVE_PHRASES = (
"false positive", "not actually", "this is fine", "this is intentional",
"not a bug", "intentional", "wrong here", "isn't actually",
"is not actually", "don't think this is", "i disagree", "this isn't right",
"this is correct", "this is expected", "by design", "this is by design",
)
# Gitea review-comment payload includes a 'body' field that may carry our
# sha marker + severity header. We extract severity + path/line from it
# as a fallback when the finding wasn't already seeded at post-time (old
# reviews before feedback.py existed).
SEV_RE = re.compile(r"\*\*\[(CRITICAL|HIGH|MEDIUM|LOW|INFO)\]\*\*", re.IGNORECASE)
PATH_LINE_RE = re.compile(r"`([^?:\n]+?):(\d+)`")
SHA_MARKER_RE = re.compile(r"<!--\s*pragent:sha=([0-9a-f]+)\s*-->", re.IGNORECASE)
# ---------------------------------------------------------------------------
# Low-level HTTP — tolerant JSON parse (Gitea sometimes returns `null` where
# we expect `[]`, e.g. reactions on a fresh comment)
# ---------------------------------------------------------------------------
def _gitea_get_json(api: str, repo: str, path: str, token: str) -> tuple[int, object]:
status, raw = ai_review.gitea_get(api, repo, path, token)
if status != 200:
return status, None
try:
return status, json.loads(raw.decode("utf-8", errors="replace"))
except (json.JSONDecodeError, ValueError):
return status, None
# ---------------------------------------------------------------------------
# Parse helpers
# ---------------------------------------------------------------------------
def _parse_severity(body: str) -> str:
m = SEV_RE.search(body or "")
return m.group(1).upper() if m else "INFO"
def _parse_path_line(body: str) -> tuple[Optional[str], Optional[int]]:
m = PATH_LINE_RE.search(body or "")
if not m:
return None, None
path = m.group(1).strip()
try:
return path, int(m.group(2))
except ValueError:
return path, None
def _parse_sha(body: str) -> Optional[str]:
m = SHA_MARKER_RE.search(body or "")
return m.group(1) if m else None
def _is_negation_reply(body: str) -> bool:
if not body:
return False
norm = body.lower()
return any(p in norm for p in FALSE_POSITIVE_PHRASES)
# ---------------------------------------------------------------------------
# Reaction classification (cheap, used by the analyzer — not the harvester
# itself)
# ---------------------------------------------------------------------------
def classify_reaction(content: str) -> str:
"""Bucket a reaction into 'positive', 'negative', or 'neutral'."""
c = (content or "").strip().lower()
if c in POSITIVE_REACTIONS:
return "positive"
if c in NEGATIVE_REACTIONS:
return "negative"
return "neutral"
# ---------------------------------------------------------------------------
# Main harvest entry
# ---------------------------------------------------------------------------
def harvest_for_pr(
*,
api: str,
token: str,
repo: str,
pr_index: int,
db_path: str,
page_size: int = 50,
) -> dict:
"""Walk every bot-authored review on the given PR and record reactions
+ thread state + replies. Returns a stats dict for logging.
`db_path` is the SQLite file path (env: `PRAGENT_FEEDBACK_DB`,
typically `/data/feedback.db` mounted via the `feedback-data` PVC).
"""
conn = init(db_path)
stats = {
"reviews_seen": 0, "findings_seen": 0,
"reactions_recorded": 0, "thread_states_recorded": 0,
"replies_recorded": 0, "errors": 0,
}
try:
# 1. List every review on the PR (paginated, but PRs rarely have >page_size)
status, payload = _gitea_get_json(
api, repo, f"pulls/{pr_index}/reviews?per_page={page_size}", token,
)
if status != 200 or not isinstance(payload, list):
log.info("harvest: reviews list failed status=%d", status)
stats["errors"] += 1
return stats
for rev in payload:
user = (rev.get("user") or {}).get("login", "")
if user != REVIEWER_LOGIN:
continue
stats["reviews_seen"] += 1
review_id_gitea = rev.get("id")
head_sha = rev.get("commit_id", "")
review_body = rev.get("body", "") or ""
body_sha = _parse_sha(review_body)
# Trust the sha marker inside the body — Gitea's commit_id field is
# for the LAST commit, not necessarily the reviewed head. If we
# can't find a marker, fall back to commit_id.
effective_sha = body_sha or head_sha
created_at = _parse_iso_ts(rev.get("created_at", ""))
db_review_id = record_review(
conn, repo=repo, pr=pr_index, head_sha=effective_sha,
review_id_gitea=review_id_gitea,
posted_at=created_at,
)
# 2. Inline comments for this review
if review_id_gitea is None:
continue
rstatus, rpayload = _gitea_get_json(
api, repo, f"pulls/{pr_index}/reviews/{review_id_gitea}/comments",
token,
)
if rstatus != 200 or not isinstance(rpayload, list):
stats["errors"] += 1
continue
for ic in rpayload:
ic_id = ic.get("id")
if ic_id is None:
continue
ic_body = ic.get("body", "") or ""
ic_path = ic.get("path")
ic_line = ic.get("position") or ic.get("line")
ic_severity = _parse_severity(ic_body)
# Fall back to body parse when Gitea didn't echo path/line
if not ic_path or not ic_line:
bp, bl = _parse_path_line(ic_body)
ic_path = ic_path or bp
ic_line = ic_line or bl
if not ic_path or not ic_line:
log.info(
"harvest: inline %s missing path/line, skipping", ic_id,
)
continue
finding_id = record_inline_finding(
conn, review_id=db_review_id, repo=repo, pr=pr_index,
path=ic_path, line=ic_line, severity=ic_severity,
problem=_strip_severity_header(ic_body),
fix="", suggestion="",
comment_id=ic_id,
posted_at=created_at,
)
stats["findings_seen"] += 1
if finding_id is None:
continue
# 3. Reactions on the inline comment
react_status, react_payload = _gitea_get_json(
api, repo, f"issues/comments/{ic_id}/reactions", token,
)
if react_status == 200 and isinstance(react_payload, list):
for r in react_payload:
ruser = (r.get("user") or {}).get("login", "") or "?"
# Gitea has occasionally returned `content` as a
# dict on older versions; coerce to str defensively.
rcontent = str(r.get("content") or "").strip()
if not rcontent:
continue
if record_reaction(
conn, comment_id=ic_id, user=ruser,
content=rcontent,
created_at=_parse_iso_ts(r.get("created_at", "")),
):
stats["reactions_recorded"] += 1
# 4. Thread state (Gitea's `resolver` field on the inline
# comment). Some Gitea versions serialize this as a user
# object ({login, ...}) instead of a username string —
# coerce defensively before calling .strip().
resolver_raw = ic.get("resolver")
if isinstance(resolver_raw, dict):
resolver = (resolver_raw.get("login") or "").strip()
else:
resolver = str(resolver_raw or "").strip()
if resolver_raw is not None: # field present, even if ""
record_thread_state(
conn, finding_id=finding_id,
resolved=bool(resolver),
)
stats["thread_states_recorded"] += 1
# 5. Replies on this review (issue-comments whose
# `review_comment_id` points at one of our inline comments).
# Some Gitea versions don't expose `review_comment_id` on the
# issue-comment endpoint — in that case `replies` stays
# empty; we degrade gracefully.
try:
_harvest_replies(
api=api, repo=repo, token=token,
pr_index=pr_index, review_id=review_id_gitea,
inline_comments=rpayload, conn=conn,
stats=stats,
)
except Exception as e:
log.info("harvest: replies fetch failed: %s", e)
stats["errors"] += 1
finally:
conn.close()
return stats
def _harvest_replies(
*, api: str, repo: str, token: str, pr_index: int,
review_id: int, inline_comments: list, conn, stats: dict,
) -> None:
"""Fetch issue comments on this PR; record those whose
`review_comment_id` matches one of our inline comment IDs.
Gitea 1.26 doesn't include that field — we fall back to fetching each
inline comment individually via `issues/comments/{id}` (does include
the field) only if the bulk fetch is empty.
"""
inline_ids = {c.get("id") for c in inline_comments if c.get("id") is not None}
if not inline_ids:
return
status, payload = _gitea_get_json(
api, repo, f"issues/{pr_index}/comments?per_page=100", token,
)
if status != 200 or not isinstance(payload, list):
return
# Build mapping inline_id -> finding_id (one SELECT instead of N)
rows = conn.execute(
"SELECT comment_id, id FROM inline_finding WHERE comment_id IN ("
+ ",".join("?" * len(inline_ids)) + ")",
list(inline_ids),
).fetchall()
inline_to_finding = {r[0]: r[1] for r in rows}
for c in payload:
rcid = c.get("review_comment_id")
if not rcid or rcid not in inline_to_finding:
continue
author = (c.get("user") or {}).get("login", "") or "?"
body = c.get("body", "") or ""
ts = _parse_iso_ts(c.get("created_at", ""))
if record_reply(
conn, finding_id=inline_to_finding[rcid],
author=author, body=body, created_at=ts,
):
stats["replies_recorded"] += 1
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _strip_severity_header(body: str) -> str:
"""Drop the leading `**[SEVERITY]**` so the posthash captures the
substance, not the severity label."""
return SEV_RE.sub("", body or "", count=1).strip()
def _parse_iso_ts(s: str) -> int:
if not s:
return int(time.time())
try:
# Python 3.11+ fromisoformat tolerates the trailing 'Z'.
return int(__import__("datetime").datetime.fromisoformat(
s.replace("Z", "+00:00")
).timestamp())
except Exception:
return int(time.time())
# ---------------------------------------------------------------------------
# CLI for manual backfill / first-time seed
# ---------------------------------------------------------------------------
def main() -> int:
import argparse, os
p = argparse.ArgumentParser(
description="Harvest reactions/threads/replies on bot PR comments.",
)
p.add_argument("--api", default=os.environ.get(
"GITEA_API", "http://gitea-http.gitea.svc.cluster.local:3000",
))
p.add_argument("--token", default=os.environ.get("PRAGENT_BOT_TOKEN", ""))
p.add_argument("--repo", required=True, help="owner/name")
p.add_argument("--pr", type=int, required=True, help="PR index")
p.add_argument("--db", default=os.environ.get(
"PRAGENT_FEEDBACK_DB", "/data/feedback.db",
))
args = p.parse_args()
if not args.token:
print("PRAGENT_BOT_TOKEN required", flush=True)
return 2
logging.basicConfig(level=logging.INFO)
stats = harvest_for_pr(
api=args.api, token=args.token,
repo=args.repo, pr_index=args.pr, db_path=args.db,
)
print(json.dumps(stats), flush=True)
return 0
"""Compatibility import for feedback harvesting."""
import importlib
import sys
_module = importlib.import_module("feedback.harvest")
sys.modules[__name__] = _module
if __name__ == "__main__":
raise SystemExit(main())
raise SystemExit(_module.main())
pilot/feedback_post.py
+5 -126
View File
@@ -1,128 +1,7 @@
"""pragent pilot — daily feedback report delivery.
Calls `feedback_analyze.analyze()` and posts the markdown report as a
comment on a single long-lived "feedback roll-up" issue in
`gitea_admin/pragent`. Comments are append-only history — one comment per
run, timestamped in the body. This keeps every report in one place, easy
to scroll, and avoids the issue-explosion of "one issue per day".
If the issue doesn't exist yet, create it. Subsequent runs just add a
new comment.
Designed for the daily K8s CronJob (`k8s/pragent-feedback-cronjob.yaml`)
but runnable from CLI for ad-hoc checks.
Env:
GITEA_API in-cluster Gitea base URL
PRAGENT_BOT_TOKEN bot token (Write collaborator on gitea_admin/pragent)
PRAGENT_FEEDBACK_DB path to SQLite (default /data/feedback.db)
PRAGENT_FEEDBACK_ISSUE_REPO default gitea_admin/pragent
PRAGENT_FEEDBACK_ISSUE_TITLE default "pragent feedback roll-up"
"""
from __future__ import annotations
import argparse
import json
import logging
import os
"""Compatibility import for feedback posting."""
import importlib
import sys
import ai_review
from feedback_analyze import analyze
log = logging.getLogger("pragent.feedback.post")
REPO_DEFAULT = "gitea_admin/pragent"
TITLE_DEFAULT = "pragent feedback roll-up"
def _find_or_create_issue(api: str, token: str, repo: str, title: str) -> int:
"""Locate the open issue with this title; create one if missing.
Gitea's issue search is via `GET /repos/{o}/{r}/issues?state=open&q=...`
(q matches title + body). We filter client-side for the exact title
to avoid query-text false matches.
"""
status, raw = ai_review.gitea_get(api, repo, "issues?state=open&per_page=50", token)
if status == 200:
try:
for issue in json.loads(raw):
if issue.get("title") == title:
# NB: the comment URL needs the per-repo `number`, not the
# global `id`. `id=60 num=8` for an early-N create; we want
# `num=8` for `/repos/o/r/issues/8/comments`.
return int(issue["number"])
except (json.JSONDecodeError, ValueError, KeyError):
pass
# Create
status, raw = ai_review.gitea_post(
api, repo, "issues", token,
{"title": title, "body": "pragent feedback roll-up — auto-created."},
)
if status not in (200, 201):
raise RuntimeError(f"issue create failed: HTTP {status} body={raw[:200]!r}")
return int(json.loads(raw)["number"])
def _post_comment(api: str, token: str, repo: str, issue_number: int, body: str) -> int:
status, raw = ai_review.gitea_post(
api, repo, f"issues/{issue_number}/comments", token, {"body": body},
)
if status not in (200, 201):
raise RuntimeError(f"comment post failed: HTTP {status} body={raw[:200]!r}")
return json.loads(raw)["id"]
def deliver(
*, api: str, token: str, db_path: str,
repo: str = REPO_DEFAULT, title: str = TITLE_DEFAULT,
since_ts: int | None = None,
) -> dict:
"""Build the report and post it as a comment. Returns a stats dict."""
report = analyze(db_path, since_ts=since_ts)
issue_id = _find_or_create_issue(api, token, repo, title)
comment_id = _post_comment(api, token, repo, issue_id, report)
return {
"repo": repo, "issue_id": issue_id, "comment_id": comment_id,
"report_bytes": len(report.encode()),
}
def main() -> int:
p = argparse.ArgumentParser(
description="Post the daily feedback report to Gitea.",
)
p.add_argument("--api", default=os.environ.get(
"GITEA_API", "http://gitea-http.gitea.svc.cluster.local:3000",
))
p.add_argument("--token", default=os.environ.get("PRAGENT_BOT_TOKEN", ""))
p.add_argument("--db", default=os.environ.get(
"PRAGENT_FEEDBACK_DB", "/data/feedback.db",
))
p.add_argument("--repo", default=os.environ.get(
"PRAGENT_FEEDBACK_ISSUE_REPO", REPO_DEFAULT,
))
p.add_argument("--title", default=os.environ.get(
"PRAGENT_FEEDBACK_ISSUE_TITLE", TITLE_DEFAULT,
))
p.add_argument("--since", type=int, default=None,
help="Unix timestamp; only include findings posted since")
args = p.parse_args()
if not args.token:
print("PRAGENT_BOT_TOKEN required", flush=True)
return 2
logging.basicConfig(level=logging.INFO)
stats = deliver(
api=args.api, token=args.token, db_path=args.db,
repo=args.repo, title=args.title, since_ts=args.since,
)
print(json.dumps(stats), flush=True)
return 0
_module = importlib.import_module("feedback.post")
sys.modules[__name__] = _module
if __name__ == "__main__":
raise SystemExit(main())
raise SystemExit(_module.main())
pilot/feedback_scores.py
+5 -245
View File
@@ -1,247 +1,7 @@
#!/usr/bin/env python3
"""pragent pilot — feedback DB to Langfuse scores.
`feedback.db` already records every reaction, thread resolution and reply a
maintainer leaves on a bot comment. That is the only ground truth pragent has
about whether a finding was any good, and until now it went to a markdown report
nobody reads and nowhere else. This ships it to Langfuse as session-level
scores, so "was the reviewer right" sits on the same axis as "what did it cost".
Session, not trace
------------------
`langfuse_trace` sets `sessionId` to `"{repo}#{pr}"` and lets the trace id be a
fresh uuid per review. Feedback arrives days later against a PR, not against one
particular re-run of the reviewer, and nothing in `feedback.db` records which
trace produced which comment. Scoring the session is therefore both the
available join and the honest granularity: this is feedback on the review of
this PR, not on one invocation.
Two scores, deliberately separated
----------------------------------
* `review_engagement` — the share of a PR's findings that got any human
response at all. This is a signal about the *feedback loop*, not the
reviewer: at the time of writing it is 0.0 across all 113 recorded reviews,
which is exactly the fact that makes an accuracy metric impossible today.
It must be watched first, because every other quality number is vapour
until it moves.
* `review_acceptance` — net verdict over the findings that *did* get a
response: (upvotes + resolved) - (downvotes + negation replies), normalised
to -1..1. Computed only over engaged findings, so an ignored review scores
`None` rather than 0. Zero would read as "humans judged this exactly
neutral"; the truth is nobody looked.
Fail-open and idempotent. Score ids are derived from (repo, pr, name) so a
re-run overwrites rather than duplicates.
"""
from __future__ import annotations
import argparse
import json
import os
import sqlite3
"""Compatibility import for feedback scores."""
import importlib
import sys
import uuid
from datetime import datetime, timezone
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from feedback_harvest import classify_reaction, _is_negation_reply # noqa: E402
REVIEW_ENGAGEMENT = "review_engagement"
REVIEW_ACCEPTANCE = "review_acceptance"
# Stable namespace so the same (repo, pr, score) always produces the same score
# id — Langfuse treats a repeated id as an update, which is what a backfill of a
# still-accumulating PR should do.
_NS = uuid.UUID("6f1d9c2e-4a77-4f2a-9c1a-0d3b5e8a7c41")
def _score_id(repo: str, pr: int, name: str) -> str:
return str(uuid.uuid5(_NS, f"{repo}#{pr}#{name}"))
def collect_pr_feedback(conn: sqlite3.Connection, repo: str, pr: int) -> dict:
"""Tally one PR's findings and the human responses attached to them.
Returns counts only — the scoring maths lives in `score_pr` so it can be
tested without a database.
"""
rows = conn.execute(
"SELECT id, comment_id FROM inline_finding WHERE repo = ? AND pr = ?",
(repo, pr),
).fetchall()
total = len(rows)
engaged = 0
positive = 0
negative = 0
for row in rows:
fid = row["id"] if isinstance(row, sqlite3.Row) else row[0]
cid = row["comment_id"] if isinstance(row, sqlite3.Row) else row[1]
pos = neg = 0
if cid is not None:
for r in conn.execute(
"SELECT content FROM reaction WHERE comment_id = ?", (cid,)
):
kind = classify_reaction(r[0])
if kind == "positive":
pos += 1
elif kind == "negative":
neg += 1
for r in conn.execute(
"SELECT resolved FROM thread_state WHERE finding_id = ?", (fid,)
):
# A resolved thread means the maintainer acted on the finding.
if r[0]:
pos += 1
# A reply counts as engagement either way; only a negation phrase makes
# it a vote against. A neutral reply ("done", "good catch, but…") is
# deliberately not a positive vote — it says someone looked, not that
# they agreed.
replied = 0
for r in conn.execute(
"SELECT body FROM reply WHERE finding_id = ?", (fid,)
):
replied += 1
if _is_negation_reply(r[0]):
neg += 1
if pos or neg or replied:
engaged += 1
positive += pos
negative += neg
return {"total": total, "engaged": engaged, "positive": positive, "negative": negative}
def score_pr(tally: dict) -> dict:
"""Turn one PR's tally into score values.
`review_acceptance` is `None` when nothing was engaged — see the module
docstring on why that is not 0.
"""
total = int(tally.get("total") or 0)
engaged = int(tally.get("engaged") or 0)
pos = int(tally.get("positive") or 0)
neg = int(tally.get("negative") or 0)
engagement = round(engaged / total, 4) if total else None
acceptance = None
if pos or neg:
acceptance = round((pos - neg) / (pos + neg), 4)
return {REVIEW_ENGAGEMENT: engagement, REVIEW_ACCEPTANCE: acceptance}
def build_score_events(
repo: str, pr: int, values: dict, environment: str = "default",
timestamp: str | None = None,
) -> list[dict]:
"""`score-create` events for one PR's feedback.
Every event carries a timestamp: the ingestion endpoint rejects those that
do not, and it reports the rejection as a per-event 400 inside an HTTP 207,
which reads as success to a caller that only checks the status code.
"""
ts = timestamp or datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
events = []
for name, value in values.items():
if value is None:
continue
events.append(
{
"id": str(uuid.uuid4()),
"type": "score-create",
"timestamp": ts,
"body": {
"id": _score_id(repo, pr, name),
"sessionId": f"{repo}#{pr}",
"name": name,
"value": float(value),
"dataType": "NUMERIC",
"environment": environment,
"comment": f"from feedback.db · {repo}#{pr}",
},
}
)
return events
SCORE_CONFIGS = [
{
"name": REVIEW_ENGAGEMENT,
"dataType": "NUMERIC",
"minValue": 0,
"maxValue": 1,
"description": "Share of a PR's findings that drew any human reaction, resolution or reply. 0 = nobody engaged with the review.",
},
{
"name": REVIEW_ACCEPTANCE,
"dataType": "NUMERIC",
"minValue": -1,
"maxValue": 1,
"description": "Net human verdict over engaged findings: +1 all accepted, -1 all rejected. Absent when nothing was engaged.",
},
]
def iter_prs(conn: sqlite3.Connection):
for row in conn.execute(
"SELECT DISTINCT repo, pr FROM inline_finding ORDER BY repo, pr"
):
yield row[0], int(row[1])
def backfill(db_path: str, *, environment: str = "default", dry_run: bool = False) -> dict:
"""Score every PR in the feedback DB. Returns a summary dict."""
conn = sqlite3.connect(db_path)
conn.row_factory = sqlite3.Row
events: list[dict] = []
scanned = 0
engaged_prs = 0
try:
for repo, pr in iter_prs(conn):
scanned += 1
tally = collect_pr_feedback(conn, repo, pr)
values = score_pr(tally)
if (values.get(REVIEW_ENGAGEMENT) or 0) > 0:
engaged_prs += 1
events.extend(build_score_events(repo, pr, values, environment))
finally:
conn.close()
summary = {"prs_scanned": scanned, "prs_with_engagement": engaged_prs, "scores": len(events)}
if dry_run or not events:
summary["posted"] = False
return summary
import langfuse_trace
conf = langfuse_trace._enabled()
if conf is None:
summary["posted"] = False
summary["error"] = "Langfuse not configured (LANGFUSE_HOST / keys unset)"
return summary
host, pk, sk = conf
status = langfuse_trace._post(host, pk, sk, events, 15.0)
summary["posted"] = status in (200, 201, 207)
summary["http_status"] = status
return summary
def main() -> int:
ap = argparse.ArgumentParser(description="Ship feedback.db verdicts to Langfuse as scores")
ap.add_argument("--db", default=os.environ.get("PRAGENT_FEEDBACK_DB", "/data/feedback.db"))
ap.add_argument("--environment", default="default")
ap.add_argument("--dry-run", action="store_true")
args = ap.parse_args()
summary = backfill(args.db, environment=args.environment, dry_run=args.dry_run)
print(json.dumps(summary, indent=2))
return 0 if summary.get("posted") or args.dry_run else 1
_module = importlib.import_module("feedback.scores")
sys.modules[__name__] = _module
if __name__ == "__main__":
raise SystemExit(main())
raise SystemExit(_module.main())
pilot/gitea_client.py
+5 -46
View File
@@ -1,46 +1,5 @@
"""Gitea transport adapter.
This module owns HTTP mechanics only. Review policy, parsing, and publishing
decisions stay in the review layer so they can be tested without a network.
"""
from __future__ import annotations
import json
import urllib.error
import urllib.request
def request(
method: str,
url: str,
token: str,
body: dict | None = None,
accept: str = "application/json",
) -> tuple[int, bytes]:
headers = {"Authorization": f"token {token}", "Accept": accept}
data = None
if body is not None:
data = json.dumps(body).encode()
headers["Content-Type"] = "application/json"
req = urllib.request.Request(url, data=data, headers=headers, method=method)
try:
with urllib.request.urlopen(req, timeout=180) as response:
return response.status, response.read()
except urllib.error.HTTPError as exc:
return exc.code, exc.read()
except urllib.error.URLError as exc:
raise RuntimeError(f"network error: {exc.reason}") from exc
class GiteaClient:
"""Small adapter for repository-scoped Gitea calls."""
def __init__(self, api: str, token: str):
self.api = api.rstrip("/")
self.token = token
def get(self, path: str, accept: str = "application/json") -> tuple[int, bytes]:
return request("GET", f"{self.api}/api/v1/repos/{path}", self.token, accept=accept)
def post(self, path: str, body: dict) -> tuple[int, bytes]:
return request("POST", f"{self.api}/api/v1/repos/{path}", self.token, body)
"""Compatibility import for the Gitea adapter."""
import importlib
import sys
_module = importlib.import_module("entrypoints.gitea")
sys.modules[__name__] = _module
pilot/langfuse_trace.py
+4 -467
View File
@@ -1,468 +1,5 @@
#!/usr/bin/env python3
"""pragent pilot — Langfuse trace emission.
Ships one trace per PR review to a self-hosted Langfuse (v3) so the reviewer's
token spend, latency and per-model behaviour are queryable outside the review
body. The review body already renders a usage table; that table is per-PR and
disappears into Gitea. This is the same numbers, aggregated.
Why hand-rolled instead of the `langfuse` SDK: the pilot image is stdlib-only
(see pilot/Dockerfile — no requirements.txt anywhere in the repo), and the
ingestion API is a single authenticated POST of a JSON batch. Pulling an SDK
plus its otel dependency tree into a fail-open telemetry side-path is a bad
trade.
Provider split
--------------
`environment` on every trace is either `ollama` or `claude`, derived from the
resolved display model (`resolve_environment`). That is what keeps the two
spend stories separate in Langfuse: every view, filter and cost breakdown
takes an environment selector, so "what did the local/self-hosted path cost"
and "what did the Claude path cost" are two views of one project rather than
two projects with two key pairs to rotate. Tags carry the finer split
(`provider:headroom`, `model:...`, `engine:opencode`).
Cost
----
The pilot's own path bills $0 (headroom proxy, no per-token charge), so the
`cost` reported to Langfuse is the *equivalent* cost from `cost_model` — what
the same tokens would bill on the comparison model. That is the number worth
trending; a chart of $0.00 is not.
A model is "free" when `cost_model.PRICES` has no entry for it (MiniMax-M2.7,
glm-5.2:cloud) or when its entry is all zeros (the self-hosted vLLM qwen). In
both cases the reported cost is priced against the comparison target instead —
same precedence the review body uses: `.pr-review.json:cost_target` >
`PRAGENT_PRICE_TARGET` > `claude-sonnet-5`. A paid model is priced as itself.
Because a hypothetical and a real charge must never be read as the same
number, every trace is tagged `cost:actual` or `cost:equivalent:<target>`, and
the generation's metadata carries `cost_basis`.
Fail-open: every entry point swallows its own exceptions. Telemetry must never
cost a review.
Env:
LANGFUSE_HOST e.g. http://langfuse-web.langfuse.svc.cluster.local:3000
LANGFUSE_PUBLIC_KEY pk-lf-...
LANGFUSE_SECRET_KEY sk-lf-...
LANGFUSE_TIMEOUT seconds, default 5
LANGFUSE_DEBUG 1 to log ingestion failures to stderr
Disabled (silently) when host or either key is unset.
"""
from __future__ import annotations
import base64
import json
import os
"""Compatibility import for Langfuse telemetry."""
import importlib
import sys
import time
import urllib.error
import urllib.request
import uuid
from datetime import datetime, timezone
INGESTION_PATH = "/api/public/ingestion"
# Model-key prefixes that mean "this review ran against Anthropic-shaped
# billing". Everything else (glm, MiniMax, qwen, local vLLM) is the ollama /
# self-hosted side of the split.
_CLAUDE_PREFIXES = ("claude-", "anthropic/")
def _now_iso() -> str:
return datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
def _enabled() -> tuple[str, str, str] | None:
host = (os.environ.get("LANGFUSE_HOST") or "").strip().rstrip("/")
pk = (os.environ.get("LANGFUSE_PUBLIC_KEY") or "").strip()
sk = (os.environ.get("LANGFUSE_SECRET_KEY") or "").strip()
if not host or not pk or not sk:
return None
return host, pk, sk
def _debug(msg: str) -> None:
if os.environ.get("LANGFUSE_DEBUG"):
print(f"pragent/langfuse: {msg}", file=sys.stderr, flush=True)
def strip_provider(model: str) -> str:
"""`headroom/claude-sonnet-5` -> `claude-sonnet-5`. Bare names pass through."""
return model.split("/", 1)[1] if "/" in model else model
def provider_of(model: str) -> str:
"""The opencode provider block a display model routes through."""
return model.split("/", 1)[0] if "/" in model else "headroom"
def resolve_environment(model: str) -> str:
"""Which spend story this review belongs to: `claude` or `ollama`.
Keyed off the bare model name, not the provider, because both paths route
through the same `headroom` proxy — `headroom/claude-sonnet-5` is Claude
spend, `headroom/glm-5.2:cloud` is not.
"""
bare = strip_provider(model).lower()
return "claude" if bare.startswith(_CLAUDE_PREFIXES) else "ollama"
def _usage_details(usage: dict) -> dict:
"""opencode's usage dict -> Langfuse `usageDetails`.
Langfuse sums every key except the ones it knows are derived, so `input`
here is the *uncached* portion: reporting both `input` (which opencode
reports as the full input, cache included) and `cache_read_input_tokens`
would double-count.
"""
inp = int(usage.get("input") or 0)
cache_read = int(usage.get("cache_read") or 0)
cache_write = int(usage.get("cache_write") or 0)
details = {
"input": max(0, inp - cache_read),
"output": int(usage.get("output") or 0),
}
if cache_read:
details["cache_read_input_tokens"] = cache_read
if cache_write:
details["cache_write_input_tokens"] = cache_write
reasoning = int(usage.get("reasoning") or 0)
if reasoning:
details["reasoning"] = reasoning
return details
DEFAULT_PRICE_TARGET = "claude-sonnet-5"
def resolve_price_target(price_target: str | None = None) -> str:
"""The model to price free/unknown runs against.
Mirrors `ai_review._resolve_price_target`: an explicit target (which the
caller reads from `.pr-review.json:cost_target`) wins, then
`PRAGENT_PRICE_TARGET`, then Sonnet.
"""
if price_target and price_target.strip():
return price_target.strip()
env = os.environ.get("PRAGENT_PRICE_TARGET", "").strip()
return env or DEFAULT_PRICE_TARGET
def _is_free(price) -> bool:
"""A price entry that charges nothing — self-hosted or proxied at no cost."""
return price.input == 0 and price.output == 0
def _cost_details(usage: dict, model: str, price_target: str | None = None) -> tuple[dict, str]:
"""USD for this usage plus the basis it was computed on.
Returns `({"total": …}, basis)` where basis is `actual` for a model that
genuinely bills, or `equivalent:<target>` for one that does not. `({}, "")`
when nothing can be priced at all — better no number than a wrong one.
Local import + broad except: `cost_model` is only present on the opencode
path, and an unknown model key must not break telemetry.
"""
try:
from cost_model import PRICES, Usage, cost
bare = strip_provider(model)
price = PRICES.get(bare)
basis = "actual"
if price is None or _is_free(price):
# MiniMax / glm / self-hosted qwen: $0 through the proxy, so the
# useful number is what these tokens would have billed elsewhere.
target = resolve_price_target(price_target)
price = PRICES.get(target)
if price is None:
_debug(f"comparison target {target!r} not in PRICES")
return {}, ""
basis = f"equivalent:{target}"
u = Usage(
uncached_input=max(0, int(usage.get("input") or 0) - int(usage.get("cache_read") or 0)),
cached_input=int(usage.get("cache_read") or 0),
cache_writes=int(usage.get("cache_write") or 0),
output=int(usage.get("output") or 0),
)
return {"total": round(cost(u, price), 6)}, basis
except Exception as e: # pragma: no cover - defensive
_debug(f"cost lookup failed for {model!r}: {e}")
return {}, ""
def _severity_counts(findings: list[dict] | None) -> dict:
counts: dict[str, int] = {}
for f in findings or []:
sev = str(f.get("severity") or "unknown").lower()
counts[sev] = counts.get(sev, 0) + 1
return counts
def build_batch(
*,
repo: str,
index: str,
sha: str,
title: str,
model: str,
usage: dict | None,
findings: list[dict] | None = None,
summary: str = "",
engine: str = "opencode",
tier: str = "",
lenses: list[str] | None = None,
trace_id: str | None = None,
release: str = "",
price_target: str | None = None,
dropped_count: float | None = None,
) -> list[dict]:
"""The ingestion batch for one review: a trace, a generation, and scores.
Split out from `emit_review_trace` so the shape is testable without a
Langfuse to POST to.
`dropped_count` is how many findings the parser rejected for an unusable
`path`/`line`, measured where the model output was parsed. Passing it turns
on the `dropped_findings` score; leaving it `None` omits that score rather
than reporting a zero the caller never measured.
"""
usage = usage or {}
tid = trace_id or str(uuid.uuid4())
ts = _now_iso()
env = resolve_environment(model)
duration = float(usage.get("duration_s") or 0.0)
started = datetime.fromtimestamp(
time.time() - duration, tz=timezone.utc
).isoformat().replace("+00:00", "Z")
tags = [
f"provider:{provider_of(model)}",
f"model:{strip_provider(model)}",
f"engine:{engine}",
f"repo:{repo}",
]
if tier:
tags.append(f"tier:{tier}")
for lens in lenses or []:
tags.append(f"lens:{lens}")
costs, cost_basis = _cost_details(usage, model, price_target) if usage else ({}, "")
if cost_basis:
# Filterable in Langfuse, so an equivalent-cost chart can never be
# mistaken for money actually spent.
tags.append(f"cost:{cost_basis}")
metadata = {
"repo": repo,
"pr": index,
"sha": sha,
"engine": engine,
"steps": usage.get("steps"),
"duration_s": duration or None,
"findings": len(findings or []),
"severities": _severity_counts(findings),
"provider_cost_usd": usage.get("cost"),
"cost_basis": cost_basis or None,
}
if lenses:
metadata["lenses"] = lenses
if tier:
metadata["tier"] = tier
metadata = {k: v for k, v in metadata.items() if v not in (None, {}, [])}
trace_body = {
"id": tid,
"name": "pr-review",
"timestamp": ts,
"environment": env,
"sessionId": f"{repo}#{index}",
"input": _review_input(repo, index, sha, title),
"output": _review_output(summary, findings),
"metadata": metadata,
"tags": tags,
}
if release:
trace_body["release"] = release
events = [
{
"id": str(uuid.uuid4()),
"type": "trace-create",
"timestamp": ts,
"body": trace_body,
}
]
if usage:
gen_body = {
"id": str(uuid.uuid4()),
"traceId": tid,
"type": "GENERATION",
"name": f"{engine}-review",
"environment": env,
"startTime": started,
"endTime": ts,
"model": strip_provider(model),
"usageDetails": _usage_details(usage),
"metadata": metadata,
"level": "DEFAULT",
# Repeated from the trace on purpose: an evaluator's variable
# mapping reads the *observation's* input/output, so a generation
# left blank cannot be judged at all.
"input": _review_input(repo, index, sha, title),
"output": _review_output(summary, findings),
}
if costs:
gen_body["costDetails"] = costs
events.append(
{
"id": str(uuid.uuid4()),
"type": "generation-create",
"timestamp": ts,
"body": gen_body,
}
)
events.extend(
_score_events(
trace_id=tid,
findings=findings,
environment=env,
cost_usd=costs.get("total"),
dropped_count=dropped_count,
timestamp=ts,
cost_basis=cost_basis,
)
)
return events
MAX_JUDGED_FINDINGS = 25
_FIELD_CAP = 600
def _review_input(repo: str, index, sha: str, title: str) -> dict:
return {"repo": repo, "pr": index, "sha": sha, "title": title}
def _review_output(summary: str, findings) -> dict:
"""What the reviewer actually said, in a shape an evaluator can read.
The findings themselves are included, not just their count. A judge given
only `{"summary": ..., "findings": 3}` can say nothing about whether those
three findings are specific, actionable, or consistent with the summary —
which is the whole question worth asking of a reviewer that has no ground
truth to check against.
Capped rather than complete: this rides in every ingestion batch, and a
review with 80 findings would push the payload past what is reasonable to
store per trace. `finding_count` stays exact so nothing reading the count
is misled by the cap.
"""
items = list(findings or [])
return {
"summary": summary[:2000],
"finding_count": len(items),
"findings_truncated": len(items) > MAX_JUDGED_FINDINGS,
"findings": [
{
"path": f.get("path"),
"line": f.get("line"),
"severity": f.get("severity"),
"problem": str(f.get("problem") or "")[:_FIELD_CAP],
"fix": str(f.get("fix") or "")[:_FIELD_CAP],
}
for f in items[:MAX_JUDGED_FINDINGS]
],
}
def _score_events(*, cost_basis: str, **kwargs) -> list[dict]:
"""Deterministic scores for this review, or [] if the scorer is missing.
Local import + blanket except for the same reason the rest of this module
swallows: `eval_scores` is optional, and a scoring bug must not cost the
trace it was supposed to annotate.
"""
try:
import eval_scores
# The cost score is only meaningful next to its basis — a $/finding
# figure computed from an equivalent price is not money that was spent.
comment = f"cost basis: {cost_basis}" if cost_basis else ""
return eval_scores.build_scores(comment=comment, **kwargs)
except Exception as e: # pragma: no cover - defensive
_debug(f"scoring failed: {e}")
return []
def _post(host: str, pk: str, sk: str, batch: list[dict], timeout: float) -> int:
payload = json.dumps({"batch": batch}).encode("utf-8")
auth = base64.b64encode(f"{pk}:{sk}".encode("utf-8")).decode("ascii")
req = urllib.request.Request(
host + INGESTION_PATH,
data=payload,
headers={
"Content-Type": "application/json",
"Authorization": f"Basic {auth}",
"User-Agent": "pragent-pilot/1.0",
},
method="POST",
)
with urllib.request.urlopen(req, timeout=timeout) as resp:
_warn_on_rejected_events(resp.read())
return resp.status
def _warn_on_rejected_events(raw: bytes) -> None:
"""Surface per-event rejections hiding inside a 207.
The ingestion endpoint answers 207 Multi-Status when *some* events failed,
so a caller that only checks the status code reads a batch where every
single event was rejected as a success. That failure mode is invisible
exactly when it matters — the traces simply never appear.
"""
try:
body = json.loads(raw or b"{}")
errors = body.get("errors") or []
if errors:
first = errors[0]
_debug(
f"{len(errors)} event(s) rejected by ingestion; "
f"first: status={first.get('status')} {first.get('error')}"
)
except Exception: # pragma: no cover - never let logging break emission
pass
def emit_review_trace(**kwargs) -> bool:
"""Ship one review's trace. Returns True if Langfuse accepted it.
No-op (False) when Langfuse is unconfigured. Never raises — a telemetry
outage must not turn into a failed review.
"""
conf = _enabled()
if conf is None:
return False
host, pk, sk = conf
try:
timeout = float(os.environ.get("LANGFUSE_TIMEOUT", "5"))
except ValueError:
timeout = 5.0
try:
batch = build_batch(**kwargs)
status = _post(host, pk, sk, batch, timeout)
if status not in (200, 201, 207):
_debug(f"ingestion returned HTTP {status}")
return False
return True
except urllib.error.HTTPError as e:
_debug(f"ingestion HTTP {e.code}: {e.read()[:300]!r}")
except Exception as e:
_debug(f"ingestion failed: {e}")
return False
_module = importlib.import_module("observability.langfuse")
sys.modules[__name__] = _module
pilot/model_client.py
+5 -36
View File
@@ -1,36 +1,5 @@
"""Model-provider adapter for the legacy Anthropic-compatible endpoint."""
from __future__ import annotations
import json
try: # Works both as `python pilot/ai_review.py` and `import pilot.model_client`.
from .gitea_client import request
except ImportError: # pragma: no cover - script-style runtime
from gitea_client import request
def parse_text_blocks(content: object) -> str:
"""Return only text blocks from an Anthropic-style response."""
if not isinstance(content, list):
return ""
return "\n".join(
block["text"]
for block in content
if isinstance(block, dict)
and block.get("type") == "text"
and isinstance(block.get("text"), str)
).strip()
def complete(base_url: str, model: str, system: str, user: str, max_tokens: int) -> str:
payload = {
"model": model,
"max_tokens": max_tokens,
"system": system,
"messages": [{"role": "user", "content": user}],
}
status, raw = request("POST", f"{base_url.rstrip('/')}/v1/messages", "ollama", payload)
if status != 200:
detail = raw[:500].decode("utf-8", errors="replace")
raise RuntimeError(f"model call failed: HTTP {status}: {detail}")
return parse_text_blocks(json.loads(raw).get("content", []))
"""Compatibility import for the legacy model adapter."""
import importlib
import sys
_module = importlib.import_module("review.model")
sys.modules[__name__] = _module
pilot/observability/__init__.py
+1
View File
@@ -0,0 +1 @@
"""Cost modeling and Langfuse telemetry."""
pilot/observability/cost.py
+434
View File
@@ -0,0 +1,434 @@
#!/usr/bin/env python3
"""pragent pilot — per-review cost model.
Answers "what would this cost on a paid API?" for the pilot's agent loop. The
pilot currently runs on `glm-5.2:cloud` through the on-network headroom proxy at
no per-token charge, so every review's measured usage is *free but real*: it
tells us exactly what the same work would bill on Claude or GPT.
The model is deliberately explicit rather than a single fudge factor, because
the dominant cost in an agent loop is not the diff — it is **resending the
conversation on every step**. A 12-step review re-reads its own prefix 12 times.
Prompt caching is what makes that affordable, and whether caching is on changes
the answer by ~3x, so it's a parameter, not an assumption.
Token accounting per review:
step 1 input = prefix + brief
step k input = prefix + brief + (tool results accumulated through k-1)
total input = sum over steps
cached = the prefix + brief part of steps 2..n (stable, byte-identical)
uncached = step 1 in full + the growing tool-result tail
`prefix` = system + tool schemas + agent definition + the skills this tier loads.
Those sizes are MEASURED from the files in this repo (see `measure_factory`),
not guessed. Diff size, file reads, and step count are per-tier assumptions from
the `attention-tiering` skill's budgets — override them on the CLI to fit your
own repos.
Prices are per million tokens, from the providers' published pricing pages
(fetched 2026-08-18 — re-check before quoting):
https://platform.claude.com/docs/en/about-claude/pricing
https://developers.openai.com/api/docs/pricing
Usage:
python3 pilot/cost_model.py # all tiers, all models
python3 pilot/cost_model.py --prs-per-month 350
python3 pilot/cost_model.py --mix 5,35,55,5 # trivial,lite,full,oversized %
python3 pilot/cost_model.py --no-cache # what caching is worth
"""
from __future__ import annotations
import argparse
import os
from dataclasses import dataclass, field
CHARS_PER_TOKEN = 4 # English prose/code rule of thumb; ±15% is normal
# ---------------------------------------------------------------------------
# Prices — USD per million tokens
# ---------------------------------------------------------------------------
@dataclass(frozen=True)
class Price:
"""Per-MTok prices. `cache_write` and `cache_read` are absolute rates, not
multipliers, so providers with different cache economics stay comparable.
`provider` is the opencode provider name (`headroom`, `vllm-qwen38`, ...). It
doubles as the dispatch key for `.pr-review.json:model` overrides — when
a per-repo override is set, `_resolve_display_model` returns
`f"{provider}/{key}"` so the opencode subprocess routes correctly.
Default `headroom` preserved for the existing roster."""
name: str
input: float
output: float
cache_write: float
cache_read: float
provider: str = "headroom"
@property
def batch_input(self) -> float:
return self.input / 2
@property
def batch_output(self) -> float:
return self.output / 2
# Anthropic: cache write = 1.25x input (5-minute TTL), cache read = 0.1x input.
# OpenAI: cached input is a published rate (0.1x input); there is no separate
# cache-write charge — writes are billed as ordinary input.
PRICES: dict[str, Price] = {
"claude-opus-5": Price("Claude Opus 5", 5.00, 25.00, 6.25, 0.50),
"claude-sonnet-5": Price("Claude Sonnet 5", 2.00, 10.00, 2.50, 0.20),
"claude-haiku-4-5": Price("Claude Haiku 4.5", 1.00, 5.00, 1.25, 0.10),
"gpt-5.6-sol": Price("GPT-5.6 Sol", 5.00, 30.00, 5.00, 0.50),
"gpt-5.6-terra": Price("GPT-5.6 Terra", 2.00, 12.00, 2.00, 0.20),
"gpt-5.6-luna": Price("GPT-5.6 Luna", 0.20, 1.20, 0.20, 0.02),
# OpenAI — cached_input 0.1x, no separate cache_write
"gpt-5": Price("GPT-5", 1.25, 10.00, 1.25, 0.125),
"gpt-5-mini": Price("GPT-5 mini", 0.25, 2.00, 0.25, 0.025),
# Google Gemini — cache_write = input
"gemini-2.5-pro": Price("Gemini 2.5 Pro", 1.875, 12.50, 1.875, 0.1875),
"gemini-2.5-flash": Price("Gemini 2.5 Flash", 0.30, 2.50, 0.30, 0.03),
# xAI Grok — cache_write = input
"grok-4.5": Price("Grok 4.5", 2.00, 6.00, 2.00, 0.30),
"grok-4.3": Price("Grok 4.3", 1.25, 2.50, 1.25, 0.20),
# Self-hosted — AI workstation RTX 3090, vLLM + DFlash2 spec-decode, no
# per-token charge. provider="vllm-qwen38" so the opencode subprocess
# routes via the matching provider block in opencode.json
# (baseURL=http://192.168.1.79:18020/v1). Equivalent-cost column reads $0
# — the cost-comparison signal is that the same work would bill $X on a
# paid model.
"qwen3.8-27b": Price("Qwen3.8-27B (vLLM, MTP, 150k ctx)", 0.0, 0.0, 0.0, 0.0, provider="vllm-qwen38"),
}
# ---------------------------------------------------------------------------
# Factory footprint — measured from this repo
# ---------------------------------------------------------------------------
_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
# Skills the primary always loads, and the conditional ones per tier. Mirrors
# the load table in .opencode/agents/pragent.md.
ALWAYS_SKILLS = ("review-methodology", "findings-schema", "attention-tiering")
TIER_SKILLS: dict[str, tuple[str, ...]] = {
"trivial": (),
"lite": ("comment-craft",),
"full": ("linter-playbook", "security-lens", "comment-craft"),
"oversized": ("linter-playbook", "security-lens", "comment-craft", "malicious-change"),
}
# opencode's own system prompt + the JSON tool schemas it sends (read, grep,
# glob, bash, webfetch, skill, task, …). Not in this repo, so this is the one
# component that is an estimate rather than a measurement.
HARNESS_TOKENS = 3500
def _tok(path: str) -> int:
try:
with open(path, "rb") as f:
return len(f.read()) // CHARS_PER_TOKEN
except OSError:
return 0
def measure_factory(root: str = _ROOT) -> dict[str, int]:
"""Token size of each prompt component, measured from the files on disk."""
out = {"agent": _tok(os.path.join(root, ".opencode", "agents", "pragent.md"))}
skills_dir = os.path.join(root, ".opencode", "skills")
if os.path.isdir(skills_dir):
for name in sorted(os.listdir(skills_dir)):
p = os.path.join(skills_dir, name, "SKILL.md")
if os.path.isfile(p):
out[f"skill:{name}"] = _tok(p)
for lens in ("security", "tests", "perf"):
out[f"subagent:{lens}"] = _tok(os.path.join(root, ".opencode", "agents", f"{lens}.md"))
return out
def prefix_tokens(tier: str, factory: dict[str, int]) -> int:
"""Stable per-step prefix: harness + agent definition + loaded skills."""
total = HARNESS_TOKENS + factory.get("agent", 0)
for s in ALWAYS_SKILLS + TIER_SKILLS.get(tier, ()):
total += factory.get(f"skill:{s}", 0)
return total
# ---------------------------------------------------------------------------
# Per-tier workload assumptions
# ---------------------------------------------------------------------------
@dataclass
class Tier:
"""One tier's workload. Defaults follow the `attention-tiering` budgets."""
name: str
diff_tokens: int # the diff as it lands in the brief
steps: int # model turns in the agent loop
file_reads: int # files read from the checkout
tokens_per_read: int # avg tokens returned per read/grep/linter result
output_tokens: int # assistant output across all steps (incl. reasoning)
subagents: int = 0 # lens subagents spawned
brief_fixed: int = 600 # brief template + PR meta + prior reviews
share: float = 0.0 # fraction of PRs at this tier (for the monthly mix)
_factory: dict = field(default_factory=dict, repr=False)
DEFAULT_TIERS = [
# diff_tok steps reads tok/read output subs share
Tier("trivial", 400, 2, 0, 0, 600, 0, share=0.05),
Tier("lite", 1500, 6, 4, 2000, 2500, 0, share=0.35),
Tier("full", 6000, 24, 20, 3300, 12000, 0, share=0.55),
Tier("oversized", 25000, 35, 30, 3500, 20000, 2, share=0.05),
]
# ---------------------------------------------------------------------------
# Observed runs — the calibration anchor
# ---------------------------------------------------------------------------
# Real usage reported by opencode's step_finish events. Keep this list
# events. Keep this list append-only: it is the only thing separating this model
# from a guess, and the first entry corrected the tier assumptions by ~15x.
OBSERVED_RUNS: list[dict] = [
{
"label": "internal/hardening-PR (16 files, 1020 insertions / 91 deletions)",
"date": "2026-08-18",
"tier": "full",
"diff_tokens": 17_600, # 16 files, 1020 insertions / 91 deletions
"steps": 28,
"duration_s": 348.3,
"input": 2_071_025,
"output": 17_303,
"cache_read": 0,
"cache_write": 0,
"subagents": 0,
},
{
"label": "internal/hardening-PR (same PR, two commits later)",
"date": "2026-08-18",
"tier": "full",
"diff_tokens": 21_000, # same PR, two commits later
"steps": 31,
"duration_s": 189.8,
"input": 2_213_077,
"output": 9_058,
"cache_read": 0,
"cache_write": 0,
"subagents": 0,
},
# A third run of the same PR (sha 2613b3e, 31 steps' worth of work in 330s)
# ended without a parseable findings block and so reported no usage at all —
# the reason `salvage_summary` now keeps the usage section on that path.
]
def observed_usage(run: dict) -> Usage:
return Usage(
uncached_input=run["input"] - run.get("cache_read", 0),
cached_input=run.get("cache_read", 0),
cache_writes=run.get("cache_write", 0),
output=run["output"],
)
@dataclass
class Usage:
uncached_input: int = 0
cached_input: int = 0
cache_writes: int = 0
output: int = 0
@property
def total_input(self) -> int:
return self.uncached_input + self.cached_input
def tier_usage(tier: Tier, factory: dict[str, int], caching: bool = True) -> Usage:
"""Token usage for one review at this tier.
The agent loop resends the whole conversation each step. The prefix + brief
are byte-identical across steps, so with caching they are written once and
read back on every later step; the tool-result tail grows and is charged as
ordinary input. Without caching every step pays full input price for
everything it has accumulated — which is the quadratic term that makes an
uncached agent loop expensive.
"""
prefix = prefix_tokens(tier.name, factory)
stable = prefix + tier.brief_fixed + tier.diff_tokens
# Tool results arrive one per step, after the first.
result_steps = max(0, min(tier.file_reads, tier.steps - 1))
per_result = tier.tokens_per_read
u = Usage(output=tier.output_tokens)
if caching:
u.cache_writes = stable
u.cached_input = stable * max(0, tier.steps - 1)
u.uncached_input = 0
else:
u.uncached_input = stable * tier.steps
# The growing tail of tool results: a result produced at step i is resent on
# every step after it, so it is counted (steps - i) times.
tail = 0
for i in range(1, result_steps + 1):
tail += per_result * (tier.steps - i)
u.uncached_input += tail
# Each lens subagent is its own loop: its own prefix, the diff, a few reads.
for _ in range(tier.subagents):
sub_prefix = HARNESS_TOKENS + factory.get("subagent:security", 600)
sub_stable = sub_prefix + tier.diff_tokens
sub_steps = 6
if caching:
u.cache_writes += sub_stable
u.cached_input += sub_stable * (sub_steps - 1)
else:
u.uncached_input += sub_stable * sub_steps
for i in range(1, 4):
u.uncached_input += per_result * (sub_steps - i)
u.output += 1500
return u
def cost(u: Usage, price: Price, batch: bool = False) -> float:
"""USD for one review's usage at these prices."""
inp = price.batch_input if batch else price.input
out = price.batch_output if batch else price.output
cw = price.cache_write / 2 if batch else price.cache_write
cr = price.cache_read / 2 if batch else price.cache_read
return (
u.uncached_input * inp
+ u.cached_input * cr
+ u.cache_writes * cw
+ u.output * out
) / 1_000_000
# ---------------------------------------------------------------------------
# Reporting
# ---------------------------------------------------------------------------
def blended_cost(tiers: list[Tier], factory: dict, price: Price, caching: bool) -> float:
"""Weighted cost of one average PR across the tier mix."""
total_share = sum(t.share for t in tiers) or 1.0
return sum(
cost(tier_usage(t, factory, caching), price) * (t.share / total_share)
for t in tiers
)
def report(tiers: list[Tier], prs_per_month: int, caching: bool, models: list[str]) -> str:
factory = measure_factory()
lines: list[str] = []
lines.append(f"Factory footprint (measured, {CHARS_PER_TOKEN} chars/token):")
for k, v in sorted(factory.items()):
lines.append(f" {k:<34} {v:>6,} tok")
lines.append(f" {'harness (opencode + tool schemas, est.)':<34} {HARNESS_TOKENS:>6,} tok")
lines.append("")
lines.append(f"Per-review tokens (prompt caching: {'on' if caching else 'OFF'})")
lines.append(f" {'tier':<11} {'prefix':>8} {'uncached':>10} {'cached':>10} {'cwrite':>8} {'output':>8}")
for t in tiers:
u = tier_usage(t, factory, caching)
lines.append(
f" {t.name:<11} {prefix_tokens(t.name, factory):>8,} {u.uncached_input:>10,} "
f"{u.cached_input:>10,} {u.cache_writes:>8,} {u.output:>8,}"
)
lines.append("")
lines.append("Cost per review (USD)")
header = f" {'model':<18}" + "".join(f"{t.name:>12}" for t in tiers) + f"{'blended':>12}"
lines.append(header)
for key in models:
p = PRICES[key]
row = f" {p.name:<18}"
for t in tiers:
row += f"{cost(tier_usage(t, factory, caching), p):>12.4f}"
row += f"{blended_cost(tiers, factory, p, caching):>12.4f}"
lines.append(row)
lines.append("")
mix = ", ".join(f"{t.name} {t.share:.0%}" for t in tiers)
lines.append(f"Monthly at {prs_per_month} PRs/month (mix: {mix})")
lines.append(f" {'model':<18} {'per PR':>10} {'per month':>12} {'batch -50%':>12}")
for key in models:
p = PRICES[key]
per_pr = blended_cost(tiers, factory, p, caching)
lines.append(
f" {p.name:<18} {per_pr:>10.4f} {per_pr * prs_per_month:>12.2f}"
f" {per_pr * prs_per_month / 2:>12.2f}"
)
lines.append("")
lines.append("Batch column applies the 50% async discount; it is shown for scale only —")
lines.append("PR review is latency-sensitive and a stateful agent loop is not batchable.")
lines.append("")
lines.append(observed_report(models))
return "\n".join(lines)
def observed_report(models: list[str]) -> str:
"""Price the runs actually measured through the opencode usage telemetry."""
if not OBSERVED_RUNS:
return "No observed runs recorded yet."
lines = ["Observed runs (measured via opencode step_finish events)"]
for run in OBSERVED_RUNS:
u = observed_usage(run)
lines.append(
f" {run['label']} — tier {run['tier']}, {run['steps']} steps, "
f"{run['duration_s']:.0f}s, {run['input']:,} in / {run['output']:,} out, "
f"cache {run['cache_read']:,} read / {run['cache_write']:,} write"
)
row = " "
for key in models:
p = PRICES[key]
row += f" {p.name}: ${cost(u, p):.2f} "
lines.append(row)
lines.append("")
lines.append(" NOTE: the pilot's headroom/glm-5.2 path reports zero cache read and zero")
lines.append(" cache write, i.e. prompt caching is NOT in play today. On a provider where")
lines.append(" it is, the stable prefix (agent + skills + brief + diff, resent every step)")
lines.append(" drops to 0.1x — worth roughly a third of the bill on a run like the one")
lines.append(" above. Budget with caching OFF until the measured cache columns are nonzero.")
return "\n".join(lines)
def main(argv: list[str] | None = None) -> int:
ap = argparse.ArgumentParser(description="pragent per-review cost model")
ap.add_argument("--prs-per-month", type=int, default=350)
ap.add_argument("--mix", default="", help="trivial,lite,full,oversized as percentages")
ap.add_argument("--no-cache", action="store_true", help="model without prompt caching")
ap.add_argument("--models", default=",".join(PRICES))
args = ap.parse_args(argv)
tiers = DEFAULT_TIERS
if args.mix:
shares = [float(x) for x in args.mix.split(",")]
if len(shares) != len(tiers):
ap.error(f"--mix needs {len(tiers)} comma-separated values")
for t, s in zip(tiers, shares):
t.share = s / 100.0
models = [m.strip() for m in args.models.split(",") if m.strip()]
unknown = [m for m in models if m not in PRICES]
if unknown:
ap.error(f"unknown model(s): {', '.join(unknown)}")
print(report(tiers, args.prs_per_month, not args.no_cache, models))
return 0
if __name__ == "__main__":
raise SystemExit(main())
pilot/observability/langfuse.py
+468
View File
@@ -0,0 +1,468 @@
#!/usr/bin/env python3
"""pragent pilot — Langfuse trace emission.
Ships one trace per PR review to a self-hosted Langfuse (v3) so the reviewer's
token spend, latency and per-model behaviour are queryable outside the review
body. The review body already renders a usage table; that table is per-PR and
disappears into Gitea. This is the same numbers, aggregated.
Why hand-rolled instead of the `langfuse` SDK: the pilot image is stdlib-only
(see pilot/Dockerfile — no requirements.txt anywhere in the repo), and the
ingestion API is a single authenticated POST of a JSON batch. Pulling an SDK
plus its otel dependency tree into a fail-open telemetry side-path is a bad
trade.
Provider split
--------------
`environment` on every trace is either `ollama` or `claude`, derived from the
resolved display model (`resolve_environment`). That is what keeps the two
spend stories separate in Langfuse: every view, filter and cost breakdown
takes an environment selector, so "what did the local/self-hosted path cost"
and "what did the Claude path cost" are two views of one project rather than
two projects with two key pairs to rotate. Tags carry the finer split
(`provider:headroom`, `model:...`, `engine:opencode`).
Cost
----
The pilot's own path bills $0 (headroom proxy, no per-token charge), so the
`cost` reported to Langfuse is the *equivalent* cost from `cost_model` — what
the same tokens would bill on the comparison model. That is the number worth
trending; a chart of $0.00 is not.
A model is "free" when `cost_model.PRICES` has no entry for it (MiniMax-M2.7,
glm-5.2:cloud) or when its entry is all zeros (the self-hosted vLLM qwen). In
both cases the reported cost is priced against the comparison target instead —
same precedence the review body uses: `.pr-review.json:cost_target` >
`PRAGENT_PRICE_TARGET` > `claude-sonnet-5`. A paid model is priced as itself.
Because a hypothetical and a real charge must never be read as the same
number, every trace is tagged `cost:actual` or `cost:equivalent:<target>`, and
the generation's metadata carries `cost_basis`.
Fail-open: every entry point swallows its own exceptions. Telemetry must never
cost a review.
Env:
LANGFUSE_HOST e.g. http://langfuse-web.langfuse.svc.cluster.local:3000
LANGFUSE_PUBLIC_KEY pk-lf-...
LANGFUSE_SECRET_KEY sk-lf-...
LANGFUSE_TIMEOUT seconds, default 5
LANGFUSE_DEBUG 1 to log ingestion failures to stderr
Disabled (silently) when host or either key is unset.
"""
from __future__ import annotations
import base64
import json
import os
import sys
import time
import urllib.error
import urllib.request
import uuid
from datetime import datetime, timezone
INGESTION_PATH = "/api/public/ingestion"
# Model-key prefixes that mean "this review ran against Anthropic-shaped
# billing". Everything else (glm, MiniMax, qwen, local vLLM) is the ollama /
# self-hosted side of the split.
_CLAUDE_PREFIXES = ("claude-", "anthropic/")
def _now_iso() -> str:
return datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
def _enabled() -> tuple[str, str, str] | None:
host = (os.environ.get("LANGFUSE_HOST") or "").strip().rstrip("/")
pk = (os.environ.get("LANGFUSE_PUBLIC_KEY") or "").strip()
sk = (os.environ.get("LANGFUSE_SECRET_KEY") or "").strip()
if not host or not pk or not sk:
return None
return host, pk, sk
def _debug(msg: str) -> None:
if os.environ.get("LANGFUSE_DEBUG"):
print(f"pragent/langfuse: {msg}", file=sys.stderr, flush=True)
def strip_provider(model: str) -> str:
"""`headroom/claude-sonnet-5` -> `claude-sonnet-5`. Bare names pass through."""
return model.split("/", 1)[1] if "/" in model else model
def provider_of(model: str) -> str:
"""The opencode provider block a display model routes through."""
return model.split("/", 1)[0] if "/" in model else "headroom"
def resolve_environment(model: str) -> str:
"""Which spend story this review belongs to: `claude` or `ollama`.
Keyed off the bare model name, not the provider, because both paths route
through the same `headroom` proxy — `headroom/claude-sonnet-5` is Claude
spend, `headroom/glm-5.2:cloud` is not.
"""
bare = strip_provider(model).lower()
return "claude" if bare.startswith(_CLAUDE_PREFIXES) else "ollama"
def _usage_details(usage: dict) -> dict:
"""opencode's usage dict -> Langfuse `usageDetails`.
Langfuse sums every key except the ones it knows are derived, so `input`
here is the *uncached* portion: reporting both `input` (which opencode
reports as the full input, cache included) and `cache_read_input_tokens`
would double-count.
"""
inp = int(usage.get("input") or 0)
cache_read = int(usage.get("cache_read") or 0)
cache_write = int(usage.get("cache_write") or 0)
details = {
"input": max(0, inp - cache_read),
"output": int(usage.get("output") or 0),
}
if cache_read:
details["cache_read_input_tokens"] = cache_read
if cache_write:
details["cache_write_input_tokens"] = cache_write
reasoning = int(usage.get("reasoning") or 0)
if reasoning:
details["reasoning"] = reasoning
return details
DEFAULT_PRICE_TARGET = "claude-sonnet-5"
def resolve_price_target(price_target: str | None = None) -> str:
"""The model to price free/unknown runs against.
Mirrors `ai_review._resolve_price_target`: an explicit target (which the
caller reads from `.pr-review.json:cost_target`) wins, then
`PRAGENT_PRICE_TARGET`, then Sonnet.
"""
if price_target and price_target.strip():
return price_target.strip()
env = os.environ.get("PRAGENT_PRICE_TARGET", "").strip()
return env or DEFAULT_PRICE_TARGET
def _is_free(price) -> bool:
"""A price entry that charges nothing — self-hosted or proxied at no cost."""
return price.input == 0 and price.output == 0
def _cost_details(usage: dict, model: str, price_target: str | None = None) -> tuple[dict, str]:
"""USD for this usage plus the basis it was computed on.
Returns `({"total": …}, basis)` where basis is `actual` for a model that
genuinely bills, or `equivalent:<target>` for one that does not. `({}, "")`
when nothing can be priced at all — better no number than a wrong one.
Local import + broad except: `cost_model` is only present on the opencode
path, and an unknown model key must not break telemetry.
"""
try:
from cost_model import PRICES, Usage, cost
bare = strip_provider(model)
price = PRICES.get(bare)
basis = "actual"
if price is None or _is_free(price):
# MiniMax / glm / self-hosted qwen: $0 through the proxy, so the
# useful number is what these tokens would have billed elsewhere.
target = resolve_price_target(price_target)
price = PRICES.get(target)
if price is None:
_debug(f"comparison target {target!r} not in PRICES")
return {}, ""
basis = f"equivalent:{target}"
u = Usage(
uncached_input=max(0, int(usage.get("input") or 0) - int(usage.get("cache_read") or 0)),
cached_input=int(usage.get("cache_read") or 0),
cache_writes=int(usage.get("cache_write") or 0),
output=int(usage.get("output") or 0),
)
return {"total": round(cost(u, price), 6)}, basis
except Exception as e: # pragma: no cover - defensive
_debug(f"cost lookup failed for {model!r}: {e}")
return {}, ""
def _severity_counts(findings: list[dict] | None) -> dict:
counts: dict[str, int] = {}
for f in findings or []:
sev = str(f.get("severity") or "unknown").lower()
counts[sev] = counts.get(sev, 0) + 1
return counts
def build_batch(
*,
repo: str,
index: str,
sha: str,
title: str,
model: str,
usage: dict | None,
findings: list[dict] | None = None,
summary: str = "",
engine: str = "opencode",
tier: str = "",
lenses: list[str] | None = None,
trace_id: str | None = None,
release: str = "",
price_target: str | None = None,
dropped_count: float | None = None,
) -> list[dict]:
"""The ingestion batch for one review: a trace, a generation, and scores.
Split out from `emit_review_trace` so the shape is testable without a
Langfuse to POST to.
`dropped_count` is how many findings the parser rejected for an unusable
`path`/`line`, measured where the model output was parsed. Passing it turns
on the `dropped_findings` score; leaving it `None` omits that score rather
than reporting a zero the caller never measured.
"""
usage = usage or {}
tid = trace_id or str(uuid.uuid4())
ts = _now_iso()
env = resolve_environment(model)
duration = float(usage.get("duration_s") or 0.0)
started = datetime.fromtimestamp(
time.time() - duration, tz=timezone.utc
).isoformat().replace("+00:00", "Z")
tags = [
f"provider:{provider_of(model)}",
f"model:{strip_provider(model)}",
f"engine:{engine}",
f"repo:{repo}",
]
if tier:
tags.append(f"tier:{tier}")
for lens in lenses or []:
tags.append(f"lens:{lens}")
costs, cost_basis = _cost_details(usage, model, price_target) if usage else ({}, "")
if cost_basis:
# Filterable in Langfuse, so an equivalent-cost chart can never be
# mistaken for money actually spent.
tags.append(f"cost:{cost_basis}")
metadata = {
"repo": repo,
"pr": index,
"sha": sha,
"engine": engine,
"steps": usage.get("steps"),
"duration_s": duration or None,
"findings": len(findings or []),
"severities": _severity_counts(findings),
"provider_cost_usd": usage.get("cost"),
"cost_basis": cost_basis or None,
}
if lenses:
metadata["lenses"] = lenses
if tier:
metadata["tier"] = tier
metadata = {k: v for k, v in metadata.items() if v not in (None, {}, [])}
trace_body = {
"id": tid,
"name": "pr-review",
"timestamp": ts,
"environment": env,
"sessionId": f"{repo}#{index}",
"input": _review_input(repo, index, sha, title),
"output": _review_output(summary, findings),
"metadata": metadata,
"tags": tags,
}
if release:
trace_body["release"] = release
events = [
{
"id": str(uuid.uuid4()),
"type": "trace-create",
"timestamp": ts,
"body": trace_body,
}
]
if usage:
gen_body = {
"id": str(uuid.uuid4()),
"traceId": tid,
"type": "GENERATION",
"name": f"{engine}-review",
"environment": env,
"startTime": started,
"endTime": ts,
"model": strip_provider(model),
"usageDetails": _usage_details(usage),
"metadata": metadata,
"level": "DEFAULT",
# Repeated from the trace on purpose: an evaluator's variable
# mapping reads the *observation's* input/output, so a generation
# left blank cannot be judged at all.
"input": _review_input(repo, index, sha, title),
"output": _review_output(summary, findings),
}
if costs:
gen_body["costDetails"] = costs
events.append(
{
"id": str(uuid.uuid4()),
"type": "generation-create",
"timestamp": ts,
"body": gen_body,
}
)
events.extend(
_score_events(
trace_id=tid,
findings=findings,
environment=env,
cost_usd=costs.get("total"),
dropped_count=dropped_count,
timestamp=ts,
cost_basis=cost_basis,
)
)
return events
MAX_JUDGED_FINDINGS = 25
_FIELD_CAP = 600
def _review_input(repo: str, index, sha: str, title: str) -> dict:
return {"repo": repo, "pr": index, "sha": sha, "title": title}
def _review_output(summary: str, findings) -> dict:
"""What the reviewer actually said, in a shape an evaluator can read.
The findings themselves are included, not just their count. A judge given
only `{"summary": ..., "findings": 3}` can say nothing about whether those
three findings are specific, actionable, or consistent with the summary —
which is the whole question worth asking of a reviewer that has no ground
truth to check against.
Capped rather than complete: this rides in every ingestion batch, and a
review with 80 findings would push the payload past what is reasonable to
store per trace. `finding_count` stays exact so nothing reading the count
is misled by the cap.
"""
items = list(findings or [])
return {
"summary": summary[:2000],
"finding_count": len(items),
"findings_truncated": len(items) > MAX_JUDGED_FINDINGS,
"findings": [
{
"path": f.get("path"),
"line": f.get("line"),
"severity": f.get("severity"),
"problem": str(f.get("problem") or "")[:_FIELD_CAP],
"fix": str(f.get("fix") or "")[:_FIELD_CAP],
}
for f in items[:MAX_JUDGED_FINDINGS]
],
}
def _score_events(*, cost_basis: str, **kwargs) -> list[dict]:
"""Deterministic scores for this review, or [] if the scorer is missing.
Local import + blanket except for the same reason the rest of this module
swallows: `eval_scores` is optional, and a scoring bug must not cost the
trace it was supposed to annotate.
"""
try:
import eval_scores
# The cost score is only meaningful next to its basis — a $/finding
# figure computed from an equivalent price is not money that was spent.
comment = f"cost basis: {cost_basis}" if cost_basis else ""
return eval_scores.build_scores(comment=comment, **kwargs)
except Exception as e: # pragma: no cover - defensive
_debug(f"scoring failed: {e}")
return []
def _post(host: str, pk: str, sk: str, batch: list[dict], timeout: float) -> int:
payload = json.dumps({"batch": batch}).encode("utf-8")
auth = base64.b64encode(f"{pk}:{sk}".encode("utf-8")).decode("ascii")
req = urllib.request.Request(
host + INGESTION_PATH,
data=payload,
headers={
"Content-Type": "application/json",
"Authorization": f"Basic {auth}",
"User-Agent": "pragent-pilot/1.0",
},
method="POST",
)
with urllib.request.urlopen(req, timeout=timeout) as resp:
_warn_on_rejected_events(resp.read())
return resp.status
def _warn_on_rejected_events(raw: bytes) -> None:
"""Surface per-event rejections hiding inside a 207.
The ingestion endpoint answers 207 Multi-Status when *some* events failed,
so a caller that only checks the status code reads a batch where every
single event was rejected as a success. That failure mode is invisible
exactly when it matters — the traces simply never appear.
"""
try:
body = json.loads(raw or b"{}")
errors = body.get("errors") or []
if errors:
first = errors[0]
_debug(
f"{len(errors)} event(s) rejected by ingestion; "
f"first: status={first.get('status')} {first.get('error')}"
)
except Exception: # pragma: no cover - never let logging break emission
pass
def emit_review_trace(**kwargs) -> bool:
"""Ship one review's trace. Returns True if Langfuse accepted it.
No-op (False) when Langfuse is unconfigured. Never raises — a telemetry
outage must not turn into a failed review.
"""
conf = _enabled()
if conf is None:
return False
host, pk, sk = conf
try:
timeout = float(os.environ.get("LANGFUSE_TIMEOUT", "5"))
except ValueError:
timeout = 5.0
try:
batch = build_batch(**kwargs)
status = _post(host, pk, sk, batch, timeout)
if status not in (200, 201, 207):
_debug(f"ingestion returned HTTP {status}")
return False
return True
except urllib.error.HTTPError as e:
_debug(f"ingestion HTTP {e.code}: {e.read()[:300]!r}")
except Exception as e:
_debug(f"ingestion failed: {e}")
return False
pilot/opencode_review.py
+5 -1752
View File
File diff suppressed because it is too large Load Diff
pilot/review/__init__.py
+1
View File
@@ -0,0 +1 @@
"""Review pipeline modules: orchestration, model adapters, parsing, and diff work."""
pilot/review/ai_review.py
+2376
View File
File diff suppressed because it is too large Load Diff
pilot/review/config.py
+32
View File
@@ -0,0 +1,32 @@
"""Trusted repository configuration and opt-in policy."""
from __future__ import annotations
import base64
import json
import urllib.parse
from collections.abc import Callable
def repo_enabled(
get: Callable[..., tuple[int, bytes]],
api: str,
repo: str,
ref: str,
token: str,
) -> bool:
"""Read the opt-in flag from the trusted base branch.
The transport is injected so the policy is testable without a live Gitea.
Any missing, malformed, or non-boolean value disables review.
"""
path = "contents/.pr-review.json?ref=" + urllib.parse.quote(ref, safe="")
status, raw = get(api, repo, path, token)
if status != 200:
return False
try:
envelope = json.loads(raw)
encoded = envelope.get("content", "").replace("\n", "")
config = json.loads(base64.b64decode(encoded).decode("utf-8", errors="replace"))
except (AttributeError, TypeError, ValueError, json.JSONDecodeError):
return False
return isinstance(config, dict) and config.get("enabled") is True
pilot/review/diff.py
+254
View File
@@ -0,0 +1,254 @@
#!/usr/bin/env python3
r"""pragent pilot — diff compression + prior-review compaction.
Two pure helpers that shrink what lands in the model prompt without losing
signal:
* ``compress_diff(diff, *, context=2)`` — re-renders a unified diff so each
hunk keeps only ``context`` unchanged lines on either side of its +/- lines.
The default 2 matches what most reviewers see on GitHub/Gitea, and is
enough to anchor every ``+``/``-`` line and give the reviewer the enclosing
statement. Wider context = more reading; narrower = less. Set
``context=0`` for +/- only, ``context=-1`` to disable entirely.
Elided context is not merely deleted: each surviving run of lines is
re-emitted as its *own* ``@@ -a,b +c,d @@`` hunk with recomputed line
numbers, so the output stays a valid unified diff whose line numbers
still describe the post-change file. ``parse_diff_anchors`` (and the
model) therefore read the same line numbers before and after compression.
* ``extract_finding_bullets(review_body)`` — pulls the lines of a prior
review that look like a pragent finding (``- 🔴 [HIGH] `path:line` — …``,
or the older ``- **[HIGH]** …`` form) and drops everything else. The model
already has the diff — repeating the prose ("this PR adds eval() — risky")
is just token burn. Bullet-only priors cut ~75% off prior-review bytes on
a typical 4-finding review.
Stdlib only. No I/O. Tolerant of malformed input — never raises.
"""
from __future__ import annotations
import re
# A real hunk header: `@@ -old[,count] +new[,count] @@[ trailing section]`.
# Captures both starts, both counts, and the trailing function-context text.
# Matching the full shape (not just a `@@` prefix) matters: a *removed* line
# whose content begins with `@@` is body, not a header.
_HUNK_RE = re.compile(
r"^@@\s+-(\d+)(?:,(\d+))?\s+\+(\d+)(?:,(\d+))?\s+@@(.*)$"
)
# Match a pragent summary-bullet line, in any of the shapes the renderer has
# emitted: `- 🔴 [HIGH] \`path:line\` — …` (current, `_severity_badge`),
# `- **[HIGH]** …` (bold, pre-badge), `- [high] …` (plain, oldest).
# Anything between the bullet marker and `[SEV]` (emoji, bold markers,
# whitespace) is tolerated — it is decoration, not signal.
_FINDING_BULLET_RE = re.compile(
r"^\s*[-*]\s*[^\w\[]*\[(?P<sev>critical|high|medium|low)\]",
re.IGNORECASE,
)
def compress_diff(diff: str, *, context: int = 2) -> tuple[str, int, int]:
"""Re-render `diff` keeping at most `context` unchanged lines around +/-.
Args:
diff: unified-diff text (what `gitea .../pulls/{n}.diff` returns).
context: max unchanged lines to keep on each side of a hunk. Use 0
for +/- only, -1 to disable compression (raw passthrough).
Returns:
`(text, original_chars, kept_chars)`. `original_chars` is the character
length of `diff` as given; `kept_chars` is the character length of
`text`. Every emitted hunk header is recomputed to match the lines
under it, so the result is a valid unified diff. Lines that are not
part of a hunk (`diff --git`, `index …`, `Binary files differ`, mode
changes) pass through verbatim.
"""
if not diff:
return diff or "", len(diff or ""), len(diff or "")
if context < 0:
return diff, len(diff), len(diff)
orig = len(diff)
lines = diff.splitlines()
out: list[str] = []
i = 0
n = len(lines)
while i < n:
m = _HUNK_RE.match(lines[i])
if m is None:
# File header, index line, binary marker, mode change, prose —
# anything outside a hunk body. Copy verbatim.
out.append(lines[i])
i += 1
continue
i += 1
body_start = i
while i < n and _is_body_line(lines[i]):
i += 1
body = lines[body_start:i]
out.extend(
_render_hunk(
body,
old_start=int(m.group(1)),
new_start=int(m.group(3)),
section=m.group(5) or "",
context=context,
)
)
text = "\n".join(out) + ("\n" if diff.endswith("\n") else "")
if not text.strip():
# Nothing survived (or the input was nothing but newlines); fall back
# to the original so the worst case is no improvement, not data loss.
return diff, orig, orig
if len(text) >= orig:
# Re-emitted hunk headers can outweigh the context they replace on a
# small, densely-changed diff. Never hand back something longer than
# what we were given.
return diff, orig, orig
return text, orig, len(text)
def _is_body_line(line: str) -> bool:
r"""True if `line` belongs to the current hunk body.
Hunk bodies contain only ` `/`+`/`-` prefixed lines and `\ No newline at
end of file`. An empty line is a context line whose trailing space was
stripped (common in mail-formatted diffs), so it counts as body too.
The check is prefix-based *and* header-aware: a removed line reading
`---` or an added line reading `+++` (YAML document separators, setext
underlines, `--` SQL comments) is body, not a file header — the previous
implementation misread those and silently dropped the rest of the hunk.
A new file section always opens with `diff --git`, which ends the body.
"""
if line == "":
return True
if line.startswith("diff --git ") or line.startswith("Index: "):
return False
if _HUNK_RE.match(line):
return False
return line[0] in " +-\\"
def _render_hunk(
body: list[str],
*,
old_start: int,
new_start: int,
section: str,
context: int,
) -> list[str]:
r"""Trim `body` to `context` unchanged lines around its +/- lines.
Each surviving run of consecutive lines is emitted as a standalone hunk
with a recomputed ``@@ -a,b +c,d @@`` header, so post-change line numbers
stay truthful. A hunk with no +/- lines at all (pure context) is dropped
entirely; ``\ No newline at end of file`` markers are dropped as noise.
Returns the rendered lines (headers included), or [] if nothing survived.
"""
# Number every body line on both sides before anything is dropped.
numbered: list[tuple[str, int, int]] = [] # (line, old_no, new_no)
old_no, new_no = old_start, new_start
for ln in body:
if ln.startswith("\\"):
continue # `\ No newline at end of file` — no signal, no numbering
kind = ln[0] if ln else " "
if kind == "+":
numbered.append((ln, -1, new_no))
new_no += 1
elif kind == "-":
numbered.append((ln, old_no, -1))
old_no += 1
else:
numbered.append((ln, old_no, new_no))
old_no += 1
new_no += 1
changed = [j for j, (ln, _, _) in enumerate(numbered) if ln[:1] in ("+", "-")]
if not changed:
return []
keep: set[int] = set()
for k in changed:
for j in range(max(0, k - context), min(len(numbered) - 1, k + context) + 1):
keep.add(j)
out: list[str] = []
for run in _consecutive_runs(sorted(keep)):
chunk = [numbered[j] for j in run]
old_count = sum(1 for ln, _, _ in chunk if ln[:1] != "+")
new_count = sum(1 for ln, _, _ in chunk if ln[:1] != "-")
# A run's start is the first line that exists on that side. When a
# side has no lines at all (pure addition / pure deletion), unified
# diff convention is `start = line before, count = 0`.
old_first = next((o for ln, o, _ in chunk if o >= 0), None)
new_first = next((nw for ln, _, nw in chunk if nw >= 0), None)
old_hdr = old_first if old_first is not None else max(chunk[0][1], 0)
new_hdr = new_first if new_first is not None else max(chunk[0][2], 0)
if old_count == 0:
old_hdr = _side_start_before(numbered, run[0], side=1)
if new_count == 0:
new_hdr = _side_start_before(numbered, run[0], side=2)
out.append(
f"@@ -{old_hdr},{old_count} +{new_hdr},{new_count} @@{section}"
)
out.extend(ln for ln, _, _ in chunk)
return out
def _side_start_before(
numbered: list[tuple[str, int, int]], idx: int, *, side: int
) -> int:
"""Line number on `side` (1=old, 2=new) just before body index `idx`.
Used for the zero-count header form (`@@ -7,0 +8,3 @@`), where unified
diff names the line the change is inserted *after*.
"""
for j in range(idx - 1, -1, -1):
no = numbered[j][side]
if no >= 0:
return no
# Nothing before it: derive from the first numbered line on that side.
for _, old_no, new_no in numbered:
no = old_no if side == 1 else new_no
if no >= 0:
return max(no - 1, 0)
return 0
def _consecutive_runs(indices: list[int]) -> list[list[int]]:
"""Group a sorted index list into runs of consecutive integers."""
runs: list[list[int]] = []
for j in indices:
if runs and j == runs[-1][-1] + 1:
runs[-1].append(j)
else:
runs.append([j])
return runs
def extract_finding_bullets(review_body: str) -> list[str]:
"""Pull the finding-bullet lines out of a prior review body.
Returns the matching lines stripped of surrounding whitespace, preserving
the rendered ``[SEV] `path:line` — problem`` shape (badge emoji and bold
markers included, whichever the renderer used). Lines that look like
bullets but carry no severity tag are dropped — the reviewer synthesizes
from the matched ones. Continuation lines (` - **Fix:** …`) are not
finding lines and are dropped with the rest of the prose.
"""
if not review_body:
return []
out = []
for line in review_body.splitlines():
if _FINDING_BULLET_RE.match(line):
out.append(line.strip())
return out
pilot/review/model.py
+36
View File
@@ -0,0 +1,36 @@
"""Model-provider adapter for the legacy Anthropic-compatible endpoint."""
from __future__ import annotations
import json
try: # Works both as `python pilot/ai_review.py` and `import pilot.model_client`.
from .gitea_client import request
except ImportError: # pragma: no cover - script-style runtime
from gitea_client import request
def parse_text_blocks(content: object) -> str:
"""Return only text blocks from an Anthropic-style response."""
if not isinstance(content, list):
return ""
return "\n".join(
block["text"]
for block in content
if isinstance(block, dict)
and block.get("type") == "text"
and isinstance(block.get("text"), str)
).strip()
def complete(base_url: str, model: str, system: str, user: str, max_tokens: int) -> str:
payload = {
"model": model,
"max_tokens": max_tokens,
"system": system,
"messages": [{"role": "user", "content": user}],
}
status, raw = request("POST", f"{base_url.rstrip('/')}/v1/messages", "ollama", payload)
if status != 200:
detail = raw[:500].decode("utf-8", errors="replace")
raise RuntimeError(f"model call failed: HTTP {status}: {detail}")
return parse_text_blocks(json.loads(raw).get("content", []))
pilot/review/opencode.py
+1754
View File
File diff suppressed because it is too large Load Diff
pilot/review/ports.py
+17
View File
@@ -0,0 +1,17 @@
"""Stable interfaces shared by the review pipeline and its adapters."""
from __future__ import annotations
from typing import Protocol
class Forge(Protocol):
def get(self, path: str, accept: str = "application/json") -> tuple[int, bytes]: ...
def post(self, path: str, body: dict) -> tuple[int, bytes]: ...
class Reviewer(Protocol):
def review(self, system: str, user: str, max_tokens: int) -> str: ...
class Telemetry(Protocol):
def emit(self, **event: object) -> None: ...
pilot/review_config.py
+5 -32
View File
@@ -1,32 +1,5 @@
"""Trusted repository configuration and opt-in policy."""
from __future__ import annotations
import base64
import json
import urllib.parse
from collections.abc import Callable
def repo_enabled(
get: Callable[..., tuple[int, bytes]],
api: str,
repo: str,
ref: str,
token: str,
) -> bool:
"""Read the opt-in flag from the trusted base branch.
The transport is injected so the policy is testable without a live Gitea.
Any missing, malformed, or non-boolean value disables review.
"""
path = "contents/.pr-review.json?ref=" + urllib.parse.quote(ref, safe="")
status, raw = get(api, repo, path, token)
if status != 200:
return False
try:
envelope = json.loads(raw)
encoded = envelope.get("content", "").replace("\n", "")
config = json.loads(base64.b64decode(encoded).decode("utf-8", errors="replace"))
except (AttributeError, TypeError, ValueError, json.JSONDecodeError):
return False
return isinstance(config, dict) and config.get("enabled") is True
"""Compatibility import for trusted review configuration."""
import importlib
import sys
_module = importlib.import_module("review.config")
sys.modules[__name__] = _module
pilot/review_ports.py
+5 -17
View File
@@ -1,17 +1,5 @@
"""Stable interfaces shared by the review pipeline and its adapters."""
from __future__ import annotations
from typing import Protocol
class Forge(Protocol):
def get(self, path: str, accept: str = "application/json") -> tuple[int, bytes]: ...
def post(self, path: str, body: dict) -> tuple[int, bytes]: ...
class Reviewer(Protocol):
def review(self, system: str, user: str, max_tokens: int) -> str: ...
class Telemetry(Protocol):
def emit(self, **event: object) -> None: ...
"""Compatibility import for review ports."""
import importlib
import sys
_module = importlib.import_module("review.ports")
sys.modules[__name__] = _module
pilot/webhook_server.py
+6 -305
View File
@@ -1,306 +1,7 @@
#!/usr/bin/env python3
"""pragent pilot — central webhook receiver.
A stdlib-only HTTP server that Gitea posts user-webhook events to. It gates on
the PR's base ref having `.pr-review.json` with `"enabled": true`, then runs
the same review core (`ai_review.review_pr`) the CI-step pilot uses, posting
findings back as `pragent-bot`.
Per-owner setup: one Gitea **user-level webhook** per repo-owner fires for every
repo that owner has; this service filters to opted-in PRs. (Gitea 1.26.1 system
webhooks are broken — see pilot/README-webhook.md.) Onboarding a repo = add the
bot as a Write collaborator + commit a `.pr-review.json` with `"enabled": true`
on the base ref.
Stdlib only — no pip install, runs on python:3-slim with the scripts mounted.
Endpoints:
POST /webhook Gitea webhook delivery (HMAC-verified)
GET /health liveness probe
Env:
WEBHOOK_SECRET shared secret used to register the Gitea webhook (HMAC)
GITEA_API in-cluster Gitea base URL
PRAGENT_BOT_TOKEN pragent-bot access token (non-admin; must be a Write
collaborator on each reviewed repo)
OLLAMA_URL headroom proxy URL, e.g. http://model-proxy.internal:8789
OLLAMA_MODEL model id, e.g. glm-5.2:cloud
OLLAMA_MAX_TOKENS (optional) output cap, default 6000
DIFF_MAX_CHARS (optional) diff truncation cap, default 150000
WEBHOOK_PORT (optional) listen port, default 8080
PRAGENT_MAX_CONCURRENT_REVIEWS
(optional) how many reviews may run at once, default 2.
Each review forks an opencode process that checks out a
repo and runs linters, so this is the real resource knob.
PRAGENT_MAX_BODY_BYTES
(optional) request-body cap, default 10 MiB
"""
import base64
import hashlib
import hmac
import json
import os
import threading
import urllib.parse
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from ai_review import gitea_get, review_pr
from review_config import repo_enabled
try:
import feedback_harvest # optional — absent in CI-step pod, present in
# central webhook service. Harvesting is the
# collection side of the feedback loop.
except ImportError:
feedback_harvest = None
# Pull-request webhook `action` values. We fire on EVERY pull_request action
# except `closed` (no point reviewing a closed/merged PR) — the
# `.pr-review.json:enabled` gate + sha dedupe downstream make broadening safe:
# a same-sha re-fire (title edit, assignee, milestone, label toggle…) is
# skipped by `review_pr`'s dedupe. Gitea emits GitHub-style `action` names
# (`labeled`, `synchronize`) even though the `X-Gitea-Event-Type` header uses
# `label_updated` / `synchronized`.
SKIP_ACTIONS = {"closed"}
GITEA_API = os.environ.get("GITEA_API", "http://gitea-http.gitea.svc.cluster.local:3000")
BOT_TOKEN = os.environ.get("PRAGENT_BOT_TOKEN", "")
OLLAMA_URL = os.environ.get("OLLAMA_URL", "http://model-proxy.internal:8789")
OLLAMA_MODEL = os.environ.get("OLLAMA_MODEL", "glm-5.2:cloud")
OLLAMA_MAX_TOKENS = int(os.environ.get("OLLAMA_MAX_TOKENS", "8000"))
DIFF_MAX_CHARS = int(os.environ.get("DIFF_MAX_CHARS", "150000"))
WEBHOOK_SECRET = os.environ.get("WEBHOOK_SECRET", "").encode()
PORT = int(os.environ.get("WEBHOOK_PORT", "8080"))
MAX_CONCURRENT = max(1, int(os.environ.get("PRAGENT_MAX_CONCURRENT_REVIEWS", "2")))
MAX_BODY_BYTES = int(os.environ.get("PRAGENT_MAX_BODY_BYTES", str(10 * 1024 * 1024)))
# Feedback DB — SQLite mounted at PRAGENT_FEEDBACK_DB. Empty / unset =
# feedback collection disabled (CI-step path doesn't have it).
FEEDBACK_DB = os.environ.get("PRAGENT_FEEDBACK_DB", "")
# Bound on reviews running at once. Every review forks an opencode process that
# untars a repo, reads files and shells out to linters, so an unbounded thread
# per delivery is a self-inflicted fork bomb the first time someone labels ten
# PRs (or Gitea retries a burst). Queued deliveries wait here rather than pile
# onto the box; the handler has already returned 202, so nothing times out.
_review_slots = threading.Semaphore(MAX_CONCURRENT)
# Reviews currently accepted or running, keyed (repo, index, sha). The
# sha-marker dedupe in `review_pr` reads Gitea *before* posting, so two
# deliveries for the same commit in flight together both see "not yet reviewed"
# and both post — the classic check-then-act race. Common triggers are Gitea
# retries after a slow 202 response and bursty re-fires from a rapid title /
# assign / label toggle. This set closes the window inside one process.
_inflight: set[tuple[str, str, str]] = set()
_inflight_lock = threading.Lock()
def is_repo_enabled(api: str, repo: str, ref: str, token: str) -> bool:
"""True iff `.pr-review.json` on `ref` has `"enabled": true`.
Reads from the given ref (typically the PR's base ref). False on any
failure: 404, parse error, missing file, missing `enabled`, wrong type.
The bool-coerce of `.get("enabled") is True` rejects the common
gotchas (`null`, `1`, `"yes"`, missing field all yield False).
"""
return repo_enabled(gitea_get, api, repo, ref, token)
def _verify_signature(raw_body: bytes, headers) -> bool:
if not WEBHOOK_SECRET:
return False # refuse to run without a configured secret
sig_header = headers.get("X-Gitea-Signature") or headers.get("X-Forgejo-Signature")
if not sig_header:
return False
mac = hmac.new(WEBHOOK_SECRET, raw_body, hashlib.sha256).hexdigest()
return hmac.compare_digest(mac, sig_header)
def _handle_pull_request(payload: dict) -> tuple[int, str]:
"""Decide whether to review; if so, kick it off in a background thread.
Returns (status, message) to Gitea immediately — the review itself runs
async so Gitea's delivery timeout never fires and causes a retry.
"""
action = payload.get("action", "")
pr = payload.get("pull_request") or {}
repo_obj = payload.get("repository") or {}
repo = repo_obj.get("full_name") or ""
if action in SKIP_ACTIONS:
return 200, f"ignore action={action}"
if not repo:
return 400, "no repository.full_name"
index = pr.get("number")
if index is None:
return 400, "no pull_request.number"
title = pr.get("title", "") or ""
body = pr.get("body", "") or ""
head = pr.get("head") or {}
sha = head.get("sha", "") or ""
base_ref = (pr.get("base") or {}).get("ref", "") or ""
if not is_repo_enabled(GITEA_API, repo, base_ref or "", BOT_TOKEN):
return 200, f"skip (repo not opted in) action={action}"
if not BOT_TOKEN:
return 500, "PRAGENT_BOT_TOKEN not set"
key = (repo, str(index), sha)
if not _claim(key):
return 200, f"ignore (already in flight) {repo}#{index} sha={sha[:8]}"
threading.Thread(
target=_run_review,
args=(key, title, body, base_ref),
daemon=True,
).start()
return 202, f"reviewing {repo}#{index} action={action} sha={sha[:8]}"
def _claim(key: tuple[str, str, str]) -> bool:
"""Reserve (repo, index, sha) for review. False if already claimed."""
with _inflight_lock:
if key in _inflight:
return False
_inflight.add(key)
return True
def _release(key: tuple[str, str, str]) -> None:
with _inflight_lock:
_inflight.discard(key)
def _run_review(
key: tuple[str, str, str], title: str, body: str, base_ref: str
) -> None:
repo, index, sha = key
# Harvest reactions on PRIOR bot comments on this PR (best-effort —
# piggy-backs the webhook path so we don't need a separate cron).
# Disabled if feedback_harvest isn't importable (CI-step image) or
# FEEDBACK_DB isn't set.
if FEEDBACK_DB and feedback_harvest is not None:
try:
hstats = feedback_harvest.harvest_for_pr(
api=GITEA_API, token=BOT_TOKEN,
repo=repo, pr_index=int(index), db_path=FEEDBACK_DB,
)
print(
f"pragent-webhook: harvested {repo}#{index} "
f"reviews={hstats['reviews_seen']} "
f"findings={hstats['findings_seen']} "
f"reactions={hstats['reactions_recorded']}",
flush=True,
)
except Exception as e:
# Harvest must never abort a review.
print(f"pragent-webhook: harvest failed for {repo}#{index}: {e}", flush=True)
try:
with _review_slots:
ok = review_pr(
api=GITEA_API,
repo=repo,
index=index,
title=title,
body=body,
sha=sha,
token=BOT_TOKEN,
ollama_url=OLLAMA_URL,
model=OLLAMA_MODEL,
max_tokens=OLLAMA_MAX_TOKENS,
max_chars=DIFF_MAX_CHARS,
base_ref=base_ref,
)
print(f"pragent-webhook: reviewed {repo}#{index} sha={sha[:8]} ok={ok}", flush=True)
except Exception as e: # review_pr is fail-open, but guard the thread anyway
print(f"pragent-webhook: thread crashed for {repo}#{index}: {e}", flush=True)
finally:
_release(key)
class Handler(BaseHTTPRequestHandler):
def _send(self, status: int, body: str) -> None:
data = body.encode()
self.send_response(status)
self.send_header("Content-Type", "text/plain")
self.send_header("Content-Length", str(len(data)))
self.end_headers()
self.wfile.write(data)
def do_GET(self):
if self.path == "/health":
with _inflight_lock:
n = len(_inflight)
self._send(200, f"ok inflight={n} max_concurrent={MAX_CONCURRENT}")
else:
self._send(404, "not found")
def do_POST(self):
if self.path != "/webhook":
self._send(404, "not found")
return
try:
length = int(self.headers.get("Content-Length", "0") or "0")
except ValueError:
self._send(400, "bad content-length")
return
# Cap before reading: the body is read whole into memory, so an
# unbounded Content-Length is a one-request OOM.
if length < 0 or length > MAX_BODY_BYTES:
self._send(413, "payload too large")
return
raw = self.rfile.read(length) if length else b""
if len(raw) != length:
self._send(400, "truncated body")
return
if not _verify_signature(raw, self.headers):
self._send(401, "invalid signature")
return
try:
payload = json.loads(raw)
except json.JSONDecodeError:
self._send(400, "invalid json")
return
event = self.headers.get("X-Gitea-Event") or payload.get("action") or ""
if event != "pull_request":
self._send(200, f"ignore event={event}")
return
repo_full = (payload.get("repository") or {}).get("full_name")
print(
f"pragent-webhook: pull_request action={payload.get('action')} repo={repo_full}",
flush=True,
)
status, msg = _handle_pull_request(payload)
self._send(status, msg)
def log_message(self, fmt, *args):
# Keep k8s logs to our own lines (see _run_review / _send paths).
print(f"pragent-webhook: {self.address_string()} {fmt % args}", flush=True)
def main() -> int:
if not WEBHOOK_SECRET:
print("pragent-webhook: FATAL: WEBHOOK_SECRET not set", flush=True)
return 1
if not BOT_TOKEN:
print("pragent-webhook: FATAL: PRAGENT_BOT_TOKEN not set", flush=True)
return 1
server = ThreadingHTTPServer(("0.0.0.0", PORT), Handler)
print(f"pragent-webhook: listening on :{PORT} (model={OLLAMA_MODEL})", flush=True)
try:
server.serve_forever()
except KeyboardInterrupt:
pass
return 0
"""Compatibility import for the webhook entry point."""
import importlib
import sys
_module = importlib.import_module("entrypoints.webhook")
sys.modules[__name__] = _module
if __name__ == "__main__":
raise SystemExit(main())
raise SystemExit(_module.main())
tests/pilot/conftest.py
+9
View File
@@ -0,0 +1,9 @@
"""Make the pilot package roots available to every categorized test."""
from __future__ import annotations
import os
import sys
PILOT_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..", "pilot"))
if PILOT_ROOT not in sys.path:
sys.path.insert(0, PILOT_ROOT)
tests/pilot/entrypoint_tests/__init__.py
+1
View File
@@ -0,0 +1 @@
"""Entrypoint tests."""
tests/pilot/test_webhook_server.py → tests/pilot/entrypoint_tests/webhook_test.py
+1 -1
View File
@@ -5,7 +5,7 @@ import sys
import threading
HERE = os.path.dirname(os.path.abspath(__file__))
ROOT = os.path.abspath(os.path.join(HERE, "..", ".."))
ROOT = os.path.abspath(os.path.join(HERE, "..", "..", ".."))
sys.path.insert(0, os.path.join(ROOT, "pilot"))
import webhook_server as ws # noqa: E402
tests/pilot/evaluation_tests/__init__.py
+1
View File
@@ -0,0 +1 @@
"""Evaluation tests."""
tests/pilot/test_eval_bootstrap.py → tests/pilot/evaluation_tests/bootstrap_test.py
+1 -1
View File
@@ -4,7 +4,7 @@ import sqlite3
import sys
import urllib.parse
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "pilot"))
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "..", "pilot"))
import eval_bootstrap as eb # noqa: E402
tests/pilot/test_eval_experiment.py → tests/pilot/evaluation_tests/experiment_test.py
+1 -1
View File
@@ -2,7 +2,7 @@
import os
import sys
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "pilot"))
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "..", "pilot"))
import eval_experiment as ex # noqa: E402
tests/pilot/test_eval_judges.py → tests/pilot/evaluation_tests/judges_test.py
View File
tests/pilot/test_eval_scores.py → tests/pilot/evaluation_tests/scores_test.py
+1 -1
View File
@@ -4,7 +4,7 @@ import sys
import pytest
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "pilot"))
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "..", "pilot"))
import eval_scores as es # noqa: E402
tests/pilot/feedback_tests/__init__.py
+1
View File
@@ -0,0 +1 @@
"""Feedback tests."""
tests/pilot/test_feedback_analyze.py → tests/pilot/feedback_tests/analyze_test.py
+1 -1
View File
@@ -16,7 +16,7 @@ import tempfile
import unittest
HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, os.path.join(HERE, "..", "..", "pilot"))
sys.path.insert(0, os.path.join(HERE, "..", "..", "..", "pilot"))
import feedback # noqa: E402
import feedback_analyze # noqa: E402
tests/pilot/test_feedback_harvest.py → tests/pilot/feedback_tests/harvest_test.py
+1 -1
View File
@@ -17,7 +17,7 @@ import unittest
from unittest.mock import patch
HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, os.path.join(HERE, "..", "..", "pilot"))
sys.path.insert(0, os.path.join(HERE, "..", "..", "..", "pilot"))
import ai_review # noqa: E402
import feedback # noqa: E402
tests/pilot/test_feedback_post.py → tests/pilot/feedback_tests/post_test.py
+1 -1
View File
@@ -11,7 +11,7 @@ import unittest
from unittest.mock import patch
HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, os.path.join(HERE, "..", "..", "pilot"))
sys.path.insert(0, os.path.join(HERE, "..", "..", "..", "pilot"))
import feedback # noqa: E402
import feedback_analyze # noqa: E402
tests/pilot/test_feedback_scores.py → tests/pilot/feedback_tests/scores_test.py
+1 -1
View File
@@ -4,7 +4,7 @@ import sys
import pytest
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "pilot"))
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "..", "pilot"))
import feedback # noqa: E402
import feedback_scores as fs # noqa: E402
tests/pilot/test_feedback.py → tests/pilot/feedback_tests/store_test.py
View File
tests/pilot/observability_tests/__init__.py
+1
View File
@@ -0,0 +1 @@
"""Observability tests."""
tests/pilot/test_cost_model.py → tests/pilot/observability_tests/cost_test.py
+1 -1
View File
@@ -3,7 +3,7 @@ import os
import sys
HERE = os.path.dirname(os.path.abspath(__file__))
ROOT = os.path.abspath(os.path.join(HERE, "..", ".."))
ROOT = os.path.abspath(os.path.join(HERE, "..", "..", ".."))
sys.path.insert(0, os.path.join(ROOT, "pilot"))
import cost_model as cm # noqa: E402
tests/pilot/test_langfuse_trace.py → tests/pilot/observability_tests/langfuse_test.py
+1 -1
View File
@@ -8,7 +8,7 @@ import os
import sys
HERE = os.path.dirname(os.path.abspath(__file__))
ROOT = os.path.abspath(os.path.join(HERE, "..", ".."))
ROOT = os.path.abspath(os.path.join(HERE, "..", "..", ".."))
sys.path.insert(0, os.path.join(ROOT, "pilot"))
import langfuse_trace as lt # noqa: E402
tests/pilot/review_tests/__init__.py
+1
View File
@@ -0,0 +1 @@
"""Review tests."""
tests/pilot/test_ai_review.py → tests/pilot/review_tests/ai_review_test.py
+1 -2
View File
@@ -6,7 +6,7 @@ import sys
# Allow running without install: add repo root to path.
HERE = os.path.dirname(os.path.abspath(__file__))
ROOT = os.path.abspath(os.path.join(HERE, "..", ".."))
ROOT = os.path.abspath(os.path.join(HERE, "..", "..", ".."))
sys.path.insert(0, os.path.join(ROOT, "pilot"))
import ai_review # noqa: E402
@@ -2149,4 +2149,3 @@ def test_format_review_body_confidence_clamps_out_of_range():
body_lo = format_review_body("- x", "glm-5.2:cloud", "abcdef1234567890", confidence=0)
assert "Merge confidence: 1/5 🔴" in body_lo
tests/pilot/test_diff_compress.py → tests/pilot/review_tests/diff_test.py
+1 -1
View File
@@ -4,7 +4,7 @@ import re
import sys
HERE = os.path.dirname(os.path.abspath(__file__))
ROOT = os.path.abspath(os.path.join(HERE, "..", ".."))
ROOT = os.path.abspath(os.path.join(HERE, "..", "..", ".."))
sys.path.insert(0, os.path.join(ROOT, "pilot"))
import diff_compress # noqa: E402
tests/pilot/test_opencode_review.py → tests/pilot/review_tests/opencode_test.py
+1 -1
View File
@@ -6,7 +6,7 @@ import sys
import tarfile
HERE = os.path.dirname(os.path.abspath(__file__))
ROOT = os.path.abspath(os.path.join(HERE, "..", ".."))
ROOT = os.path.abspath(os.path.join(HERE, "..", "..", ".."))
sys.path.insert(0, os.path.join(ROOT, "pilot"))
import opencode_review as oc # noqa: E402
tests/pilot/test_parse_dropped.py → tests/pilot/review_tests/parse_dropped_test.py
+1 -1
View File
@@ -9,7 +9,7 @@ import os
import sys
HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, os.path.abspath(os.path.join(HERE, "..", "..", "pilot")))
sys.path.insert(0, os.path.abspath(os.path.join(HERE, "..", "..", "..", "pilot")))
import ai_review # noqa: E402