pragent/pilot/README-eval.md

Evaluation — scorers, ground truth, and the dataset

Langfuse already receives one trace per review (README-langfuse.md). This is the layer on top: numbers attached to those traces that say how the reviewer behaved, and the beginnings of a ground-truth signal that says whether it was right.

Those two things are deliberately kept apart, because only one of them exists yet.

What could and could not be built

feedback.db has recorded 113 reviews across 4 repos. It has recorded zero reactions, zero thread resolutions and zero replies. The harvester, the schema and the daily analyzer are all working; nobody has ever reacted to a bot comment.

That rules out an accuracy metric today. Correctness needs labels, and a judge scored against no labels is theatre. So the scorers here measure behaviour, which is computable from data already in hand, and a separate bridge exists to turn human reactions into scores the moment any arrive.

The five behavioural scores

Emitted with every review by eval_scores.py, folded into the same ingestion batch as the trace so they cost no extra request.

score	type	what a change in it means
finding_rate	NUMERIC	Findings posted. 0 is the restraint case — good on clean code, a failure when the run degraded. Only the rate over time separates those.
severity_info_ratio	NUMERIC 0–1	Share of findings the model rated info/trivial. Rising = the model is hedging rather than committing. None when the review was silent: a ratio over an empty set is undefined, and charting it as 0 would read as perfect calibration.
severity_max	CATEGORICAL	Highest severity surfaced, none when silent. Categorical because "did this ever surface something serious" is the real question, and a mean of severity ranks answers nothing.
dropped_findings	NUMERIC	Findings the model emitted that the parser rejected for an unusable path/line. This is the only score here that measures the model's raw output.
cost_per_finding	NUMERIC	Equivalent USD per finding. A cheaper model that finds nothing is not cheaper.

Why dropped_findings needed a change to the parser

parse_findings and parse_review_output discard any finding with a missing or unusable location. That happens silently, so a model emitting ten findings at invalid locations was indistinguishable from a model that found nothing — both produce an empty list. ai_review.last_parse_dropped() exposes the delta, recorded at parse time.

It must be read at parse time specifically: by the time findings reach _emit_langfuse, apply_repo_config has already filtered them by severity_threshold and max_findings, and those drops are the config working as intended, not the model misbehaving.

Ground truth: feedback_scores.py

Turns feedback.db into two session-level scores, keyed on "{repo}#{pr}" (which is what langfuse_trace already sets as sessionId).

score	meaning
review_engagement	Share of a PR's findings that drew any human reaction, resolution or reply. Watch this first — every quality number is vapour until it moves off 0.
review_acceptance	Net verdict over engaged findings, −1 to +1. Absent, not 0, when nothing was engaged: zero would claim humans judged the review neutral, when the truth is nobody looked.

Session-level rather than trace-level because feedback arrives days later against a PR, and nothing in feedback.db records which re-run of the reviewer produced which comment. The session is both the available join and the honest granularity.

Score ids are uuid5(namespace, repo#pr#name), so the daily backfill updates rather than duplicates.

The dataset

pragent-reviews, one item per PR the reviewer has run on, seeded by eval_bootstrap.py from feedback.db.

expectedOutput is the reviewer's own prior output, not human-verified truth — every item carries metadata.labelled_by_human: false. Read it as a regression baseline: re-run a candidate model over these PRs and the diff against this column is the behaviour change. Promoting an item to real ground truth means a human editing it in the dataset view after re-reading the PR.

Item ids

{owner}__{repo}__pr{n}. The obvious {repo}#{pr} cannot be used: items are routed as /datasets/{id}/items/{item_id}, so the / in owner/repo splits into extra path segments and everything after # is a fragment the browser never sends — the item is created fine by the API and then 404s when opened. Session ids elsewhere keep {repo}#{pr}; those are never path segments.

Filterable metadata

The filter bar matches on metadata only — not on input, and not on the item id — so every facet worth slicing on is a flat, primitive key in metadata even where it duplicates input:

key	why it is there
repo, owner, repo_name	owner exists because a filter on the joined repo matches one repo, never a whole org
pr, head_sha	jump from a filtered row back to the actual PR
finding_count, has_findings	isolate the silent reviews, which are the interesting negatives
max_severity	"none" rather than absent — an absent key matches no filter
reviews_run	how churny the PR was; high values skew per-item averages
last_reviewed_at / _iso	epoch sorts, ISO reads
labelled_by_human	false everywhere today; the flag to filter on before trusting any of it

Nested objects and lists are deliberately absent: the filter bar cannot reach into them.

max_severity is derived from feedback.db, whose severity column is re-parsed out of the rendered comment by feedback_harvest._parse_severity and defaults to INFO when its regex misses the badge. Trust the severity_max score (read from the model's structured output) over this facet.

Experiments

eval_experiment.py links reviews that already ran into a dataset run, so the Experiments tab is populated without re-running anything. Runs are grouped by model — the comparison the pilot actually needs is the same PRs under a candidate model with finding_rate and cost_per_finding side by side. A new model produces a new run automatically on the next invocation.

One trace per (run, item), the most recent: a PR re-reviewed on every push has many traces, and a run is one output per input.

It uses POST /api/public/dataset-run-items, which is deprecated in favour of the SDK experiment runner and disappears in Langfuse v4. The deprecation notice exempts self-hosted v3 from the cutoff date, and this pilot is stdlib-only by design. Revisit when this deployment moves to v4.

Coverage is bounded by the dataset, not by the traces: items only exist for PRs with a row in feedback.db, and a review that posted no comment leaves a trace but no row. That is why a run links fewer items than there are traces.

Running it

# once per project: score configs + dataset (+ score historical traces)
python3 pilot/eval_bootstrap.py --db /data/feedback.db --backfill-traces

# ship feedback verdicts (runs daily from the feedback CronJob)
python3 pilot/feedback_scores.py --db /data/feedback.db

# link already-traced reviews into a dataset run per model
python3 pilot/eval_experiment.py --dry-run
python3 pilot/eval_experiment.py

Both need LANGFUSE_HOST, LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY. In cluster they come from the pragent-langfuse Secret and point at the ClusterIP — never the NodePort, whose oauth2-proxy 302s ingestion to Logto and drops it.

Gotcha: HTTP 207 is not success

The ingestion endpoint answers 207 Multi-Status when some events failed, so a batch where every event was rejected still returns 207. An early version of these scorers omitted the required per-event timestamp and silently ingested nothing while reporting success. langfuse_trace._warn_on_rejected_events now logs the per-event errors under LANGFUSE_DEBUG=1. If scores are missing, check that before anything else.

What the first run showed

Backfilled over 42 existing traces and 13 PRs:

cost_per_finding     n=42  mean=0.3133  min=0.0880  max=0.9042
finding_rate         n=42  mean=0.4762  min=0.0000  max=4.0000
severity_info_ratio  n=14  mean=0.0000
review_engagement    n=14  mean=0.0000
severity_max         {none: 28, medium: 11, high: 1, critical: 2}

Two things worth keeping:

• The reviewer is not info-heavy. feedback.db shows 61 of 62 findings at INFO, which looked like a badly calibrated model. It is not: severity_max reads medium/high/critical on every trace that found anything, and severity_info_ratio is flat 0. The INFO in the DB comes from feedback_harvest._parse_severity, which defaults to INFO when its regex misses the severity badge in the rendered comment. The DB severity is a re-parse artifact; the score reads the model's structured output directly.
• 28 of 42 reviews found nothing (67%), and engagement is flat zero. The first is not yet interpretable without the second.