The filter bar matches on metadata only — not on input and not on the item id —
so a dataset seeded with repo/pr in `input` alone could not be sliced by repo
at all. Every facet worth filtering on is now a flat primitive in `metadata`:
repo, owner, repo_name, pr, head_sha, finding_count, has_findings,
max_severity, reviews_run and the review timestamp both ways. `owner` is split
out because a filter on the joined repo matches one repo, never a whole org,
and `max_severity` is "none" rather than absent because an absent key matches
no filter.
`eval_experiment.py` links reviews that already ran into a dataset run, one run
per model, so the Experiments tab is populated without re-running the reviewer.
One trace per (run, item), the most recent: a PR re-reviewed on every push has
many traces and a run is one output per input.
It posts to the deprecated /api/public/dataset-run-items — the notice exempts
self-hosted v3 from the cutoff date and the pilot is stdlib-only by design.
Revisit at v4.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Items were keyed `{repo}#{pr}`, e.g. `netcracker/interview#29`. Both
characters break the UI's item route: the `/` in `owner/repo` splits into
extra path segments, and everything after the `#` is a fragment the browser
never sends. Items were created successfully and then 404'd when opened.
Ids are now `{owner}__{repo}__pr{n}`, which needs no percent-encoding. The
real repo and pr stay intact in `input`, so nothing downstream reads the id
back apart. Session ids elsewhere keep the `{repo}#{pr}` form — those are
never path segments and feedback_scores depends on that shape.
The 28 existing items were unusable and are regenerable from feedback.db;
they were deleted and recreated under the new ids.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds the evaluation layer on top of the review traces: five deterministic
scores describing how the reviewer behaved, a bridge that turns human reactions
into ground truth, and a dataset seeded from the reviews already run.
The two are kept apart on purpose. feedback.db has recorded 113 reviews and
zero reactions, resolutions or replies — nobody has ever responded to a bot
comment — so an accuracy metric cannot be built yet. The scorers therefore
measure behaviour, which is computable from data in hand, and feedback_scores
turns verdicts into scores the moment any arrive.
eval_scores.py emits finding_rate, severity_info_ratio, severity_max,
dropped_findings and cost_per_finding into the same ingestion batch as the
trace. Undefined values are omitted rather than reported as zero: an info ratio
over a silent review is undefined, and charting it as 0 would read as perfect
calibration.
dropped_findings needed a parser change. Both parsers silently discard findings
with an unusable path/line, which made a model emitting garbage locations
indistinguishable from one that found nothing. last_parse_dropped() exposes the
delta, read at parse time — after apply_repo_config the drops are the config
working as intended, not the model misbehaving.
feedback_scores.py scores the session ("{repo}#{pr}"), because feedback arrives
days later against a PR and nothing records which re-run produced which
comment. review_acceptance is absent rather than 0 when nothing was engaged.
eval_bootstrap.py registers the score configs, seeds the pragent-reviews
dataset, and can backfill scores onto traces that predate the scorers.
expectedOutput is the reviewer's own prior output, flagged
labelled_by_human: false — a regression baseline, not verified truth.
Also fixes a silent telemetry failure: the ingestion endpoint answers 207 when
only some events succeed, so a batch with every event rejected still looked
like success. Score events were missing the required per-event timestamp and
ingested nothing while reporting 207. _warn_on_rejected_events now logs the
per-event errors under LANGFUSE_DEBUG.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ship token spend, latency and equivalent cost for every review to the
self-hosted Langfuse so per-model behaviour is queryable as a trend rather
than one PR comment at a time.
langfuse_trace.py is stdlib-only and emits via the public ingestion API.
Traces split into `ollama` and `claude` environments keyed off the bare model
name, not the provider: both paths go through the same headroom proxy, so the
provider prefix says nothing about which spend story a review belongs to. The
pilot's own path bills $0, so the reported cost is the equivalent price from
cost_model.PRICES.
ai_review.py calls _emit_langfuse on both token-spending exit paths (the
normal post and the salvage path). Import and emission are wrapped in a
blanket except: with no LANGFUSE_HOST or key pair the whole thing is a silent
no-op, and a telemetry failure must never fail a review.
These files were previously deployed only by way of the image build's
`COPY . /app`, so a clean checkout would have silently dropped tracing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- opencode_review.install_config: opencode 1.3.10 rejects factory
opencode.json$'\''s top-level "$schema" key with 'Unrecognized key'
at parse time, causing every fresh review to fail as the misleading
'opencode empty text (rc=0)'. Add a small drop-list of known-bad
top-level keys + a sanitizer applied to both the fast (no-env) and
slow (env-substituted) write paths.
- feedback_harvest: Gitea versions occasionally serialize reaction
content / inline-comment resolver as a dict instead of a string,
crashing harvester with 'dict object has no attribute strip'. Coerce
both via str() with a brief comment documenting the WHY.
Affected reviews: techspark/suaspark-dashboard #11 (fixed),
PRAgent reviews on PR #9 era (recovered).
Verified: opencode 1.3.10 emits real '### Summary of Changes' body.
POST /r/<owner>/<name>/edit reads the current .pr-review.json via the
Gitea contents API (404 → start from {}), updates static_message and
model, then PUTs the file back as the bot identity. Static message is
stripped and capped at 400 chars; model is validated against
cost_model.PRICES (unknown → flash + redirect, no save). CSRF token is
the dashboard auth token (same value, simplest binding).
Helpers (_http) moved to urllib.request at module top so tests can patch
the entry point; the helper returns the Gitea status code so 404 (file
missing) cleanly omits sha on the create-PUT path.
6 tests under tests/pilot/test_dashboard_edit.py covering the happy
path, the 400-char cap, the unknown-model rejection, auth + CSRF gates,
and the file-missing creation path.
Mirrors webhook_server.py's BaseHTTPRequestHandler + ThreadingHTTPServer
shape. Pure stdlib, no pip deps. Routes:
GET / overview (totals + 7-day sparkline + top repos)
GET /r/<owner>/<name> repo summary (severity histogram + top findings)
GET /r/<owner>/<name>/<index> one PR's findings
GET /r/<owner>/<name>/<index>/raw raw Markdown body (via Gitea contents API)
GET /static/style.css dark-mode stylesheet
GET /login login form
POST /login compare token, set HttpOnly+SameSite cookie
POST /r/<owner>/<name>/edit (Tasks C+D)
Auth: when PRAGENT_DASHBOARD_TOKEN is set, every route except /login and
/static/* requires Cookie: pragent_dash=<token>. Unset → tailnet-only.
All HTML rendered via string.Template; every dynamic value is escaped
with html.escape(..., quote=True). No .format, no f-string templates.
12 tests under tests/pilot/test_dashboard.py.
Three pure functions — overview / repo_summary / pr_summary — that open
the feedback SQLite via feedback.init, run their queries, and return
plain dicts/lists. All three tolerate a missing or empty DB by returning
a zero-shaped dict.
Cost is hardcoded 0.0: per-review usage:cost isn't stored, only the raw
review rows are. Surfacing a rolled-up dollar figure without telemetry
would be guessing, so we don't.
17 new tests under tests/pilot/test_dashboard_data.py.
Repos that need to pin the review engine (e.g. 'this project requires
claude-sonnet-5 for the budget line' or 'route everything through
gpt-5.6-luna for now') can declare a top-level 'model' string in
.pr-review.json. The parser validates the value against cost_model.PRICES
(lazy import — ollama path stays dep-free) and silently drops unknown
values with a stderr pointer to the valid key set so a typo in the
config file surfaces in the logs instead of silently falling back.
The orchestrator gains a small _resolve_display_model(base, config)
helper that implements a 3-way precedence:
1. OPENCODE_MODEL env (operator override, used verbatim)
2. config['model'] (per-repo override)
3. f'headroom/{base}' (default)
review_pr resolves once (lazy fallback before config is loaded) and
re-resolves after .pr-review.json is fetched, then threads the result
into the opencode subprocess, REVIEW_HEADER, and the cost-line
parenthetical. Same single value everywhere — no more mix of bare
OLLAMA_MODEL id in the header and a stale free-tier literal in the cost
line.
Tests cover parsing acceptance, parsing rejection (capsys stderr),
type validation, precedence in all 4 (env×config) combinations, and an
end-to-end sanity check that format_review_body shows the override and
not the base id.
Repos can pin a free-text notice (e.g. 'this repo is in maintenance mode',
or 'reviewers: focus on the public API only for this quarter') in
.pr-review.json:static_message. The string is stripped and capped at 400
chars (mirror of the existing instructions cap), then rendered as a
Markdown blockquote (> {msg}) directly under the REVIEW_HEADER so it
surfaces on every review without scrolling.
Plumbing: parse_repo_config exposes 'static_message'; format_review_body
accepts a static_message kwarg and inserts the blockquote before the
'### Summary of Changes' section; review_pr threads
config.get('static_message') to both call sites (salvage path + happy
path). Empty / non-string values are silently dropped, mirroring the
parser's 'ignore blank' handling for every other text field.
The webhook pod now routes through headroom's MiniMax-M2.7 endpoint, but the
cost line in the AI-usage collapsible still read 'headroom glm-5.2:cloud'.
Resolve a single display_model at the top of review_pr (OPENCODE_MODEL env
wins, else headroom/{OLLAMA_MODEL}) and pass it to:
* the opencode subprocess (was already doing this on the same line, now
sharing the value)
* format_review_body so REVIEW_HEADER also reflects the actual run
* _render_collapsible_usage so the parenthetical reads
'({display_model} — free tier)' or '({display_model} — billed)'.
Test additions in tests/pilot/test_ai_review.py cover:
* the parenthetical picks up the passed-in model verbatim
* the full provider prefix survives (headroom/<id>) for the opencode path
* nonzero cost flips the inner clause from 'free tier' to 'billed'
* the existing nonzero-cost assertion flips to assert 'billed' instead
of dropping the parenthetical entirely
opencode's @ai-sdk/anthropic provider sends the config's apiKey as the
x-api-key header. The committed opencode.json carries apiKey="ollama"
(placeholder) so the repo can be public. When the headroom upstream
switches to an auth-gated provider (e.g. MiniMax), the placeholder
returns 'No credentials found'. install_config now also patches
options.apiKey from PRAGENT_MODEL_API_KEY when set, mirroring the
existing baseURL patching.
Co-Authored-By: Claude <noreply@anthropic.com>
The previous two commits put the label re-read in the webhook, at review
start. That is too early to help: the review claims on the AI-REVIEW event
and the re-read runs milliseconds later, while the reviewer's second click
(AI-USAGE) is still a second or two away. It would have kept 404ing quietly
if the path fix hadn't landed, and even fixed it caught nothing.
Move the check to where the decision is actually used — just before the
usage block is rendered, after the model has run. That is a minute or more
after the trigger, by which time the label is there. Attribution is computed
in the same branch, so a late opt-in still gets its per-comment token lines.
`pr_has_label` goes through the existing gitea_get helper, which owns the
/api/v1 prefix, so the path can't drift again. Any failure returns False and
the payload's verdict stands: a review is never lost over a usage section.
Reverts the webhook-side re-read from 4ef62f2 and aedea97.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B11e8TZZxJyzHW7jj7KWUN
GITEA_API is the host with no version prefix — ai_review._get / _post append
/api/v1 per call. The re-read helper didn't, so it 404'd on every review and
silently fell back to the payload's verdict: "could not re-read labels for
gitea_admin/pragent#10: HTTP Error 404" in the pod log, usage block still
missing. Caught by labelling PR #10 AI-REVIEW then AI-USAGE, which is the
exact sequence the fix exists to handle.
Test asserts the composed URL, since a wrong path here fails silently by
design (the helper swallows errors so a review is never lost over a usage
section).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B11e8TZZxJyzHW7jj7KWUN
Labelling a PR is two webhook events. AI-REVIEW arrives first, the review
claims (repo, index, sha) and starts; the AI-USAGE event that follows a
moment later hits the in-flight dedupe and is dropped. `report_usage` was
snapshotted from the first payload, which had not seen AI-USAGE yet, so the
review posted without its usage block even though the label was on the PR by
the time it finished — observed on gitea_admin/pragent#9 (`usage=False` in
the pod log, no <details> section in the posted review).
Re-read the labels from the API at review start and upgrade the flag. The
read is best-effort: any failure logs and returns [], leaving the payload's
verdict intact, because a usage section is not worth failing a review over.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B11e8TZZxJyzHW7jj7KWUN
Four defects, all found reviewing PR #9 (two of them by pragent-bot's own
review of that PR, which the anchoring bug then misplaced):
* compress_diff dropped context lines but copied the original `@@` hunk
header verbatim, so the header no longer described the lines beneath it.
parse_diff_anchors then walked stale headers and produced anchor sets
shifted by the number of elided lines, misplacing inline comments or
demoting them to bullets. Each surviving run of lines is now re-emitted as
its own hunk with a recomputed `@@ -a,b +c,d @@`, so the output stays a
valid unified diff whose numbers describe the real post-change file. The
pseudo-marker `@@ … N context line(s) omitted … @@` is gone; it parsed as
a hunk header and reset the anchor counter to 0. Anchoring additionally
runs on the raw diff now, so the prompt window can never shrink the
anchorable set.
* compress_diff's `_FILE_HEADER` regex matched diff *body* lines: a removed
YAML `---` separator or an added `++` line was read as a file header,
truncating the hunk and dropping its `@@` header with it. Body detection
is now prefix-based, with a full-shape hunk-header regex.
* extract_finding_bullets could not match the bullets pragent itself posts:
summary_bullets renders an emoji severity badge between the `-` and the
`[SEV]` tag, which the regex rejected, so compact_prior_reviews always
returned [] and every re-review repeated its previous findings.
* triage returning `{"lenses":[]}` — documented in .opencode/agents/triage.md
as "no lens has surface, skip the fan-out" — ran every lens instead, since
_intersect_with_triage mapped an empty selection to "all" and the call site
had a second `or reviewers` fallback. `[]` and None are now distinct
outcomes: `[]` skips, None fails open. A roster naming only unknown lens
ids now fails open rather than silencing the review. The skip path returns
a well-formed empty-findings response instead of "", which had landed in
ai_review's unparseable-output branch and posted "AI review produced no
parseable output" — a malfunction message for a normal verdict.
Also: non-URL references (a CVE id, a doc title) rendered as
`[CVE-2024-1234](CVE-2024-1234)`, a broken relative link in Gitea — now
plain text. PRAGENT_DIFF_CONTEXT and friends parse through _int_env, so a
typo logs and falls back instead of killing a review mid-flight. Removed
format_usage_section, dead since the collapsible usage block replaced it and
carrying a duplicate copy of the price-target logic.
Tests: 290 -> 301. New coverage for hunk-header fidelity before/after
compression, header-shaped content lines, the bullet round-trip against the
real renderer, and triage's three outcomes (previously untested).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B11e8TZZxJyzHW7jj7KWUN
Marcos