pragent/pilot/README-eval.md

Evaluation — scorers, ground truth, and the dataset

Langfuse already receives one trace per review (README-langfuse.md). This is the layer on top: numbers attached to those traces that say how the reviewer behaved, and the beginnings of a ground-truth signal that says whether it was right.

Those two things are deliberately kept apart, because only one of them exists yet.

What could and could not be built

feedback.db has recorded 113 reviews across 4 repos. It has recorded zero reactions, zero thread resolutions and zero replies. The harvester, the schema and the daily analyzer are all working; nobody has ever reacted to a bot comment.

That rules out an accuracy metric today. Correctness needs labels, and a judge scored against no labels is theatre. So the scorers here measure behaviour, which is computable from data already in hand, and a separate bridge exists to turn human reactions into scores the moment any arrive.

The five behavioural scores

Emitted with every review by eval_scores.py, folded into the same ingestion batch as the trace so they cost no extra request.

score	type	what a change in it means
finding_rate	NUMERIC	Findings posted. 0 is the restraint case — good on clean code, a failure when the run degraded. Only the rate over time separates those.
severity_info_ratio	NUMERIC 0–1	Share of findings the model rated info/trivial. Rising = the model is hedging rather than committing. None when the review was silent: a ratio over an empty set is undefined, and charting it as 0 would read as perfect calibration.
severity_max	CATEGORICAL	Highest severity surfaced, none when silent. Categorical because "did this ever surface something serious" is the real question, and a mean of severity ranks answers nothing.
dropped_findings	NUMERIC	Findings the model emitted that the parser rejected for an unusable path/line. This is the only score here that measures the model's raw output.
cost_per_finding	NUMERIC	Equivalent USD per finding. A cheaper model that finds nothing is not cheaper.

Why dropped_findings needed a change to the parser

parse_findings and parse_review_output discard any finding with a missing or unusable location. That happens silently, so a model emitting ten findings at invalid locations was indistinguishable from a model that found nothing — both produce an empty list. ai_review.last_parse_dropped() exposes the delta, recorded at parse time.

It must be read at parse time specifically: by the time findings reach _emit_langfuse, apply_repo_config has already filtered them by severity_threshold and max_findings, and those drops are the config working as intended, not the model misbehaving.

Ground truth: feedback_scores.py

Turns feedback.db into two session-level scores, keyed on "{repo}#{pr}" (which is what langfuse_trace already sets as sessionId).

score	meaning
review_engagement	Share of a PR's findings that drew any human reaction, resolution or reply. Watch this first — every quality number is vapour until it moves off 0.
review_acceptance	Net verdict over engaged findings, −1 to +1. Absent, not 0, when nothing was engaged: zero would claim humans judged the review neutral, when the truth is nobody looked.

Session-level rather than trace-level because feedback arrives days later against a PR, and nothing in feedback.db records which re-run of the reviewer produced which comment. The session is both the available join and the honest granularity.

Score ids are uuid5(namespace, repo#pr#name), so the daily backfill updates rather than duplicates.

The dataset

pragent-reviews, one item per PR the reviewer has run on, seeded by eval_bootstrap.py from feedback.db.

expectedOutput is the reviewer's own prior output, not human-verified truth — every item carries metadata.labelled_by_human: false. Read it as a regression baseline: re-run a candidate model over these PRs and the diff against this column is the behaviour change. Promoting an item to real ground truth means a human editing it in the dataset view after re-reading the PR.

Running it

# once per project: score configs + dataset (+ score historical traces)
python3 pilot/eval_bootstrap.py --db /data/feedback.db --backfill-traces

# ship feedback verdicts (runs daily from the feedback CronJob)
python3 pilot/feedback_scores.py --db /data/feedback.db

Both need LANGFUSE_HOST, LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY. In cluster they come from the pragent-langfuse Secret and point at the ClusterIP — never the NodePort, whose oauth2-proxy 302s ingestion to Logto and drops it.

Gotcha: HTTP 207 is not success

The ingestion endpoint answers 207 Multi-Status when some events failed, so a batch where every event was rejected still returns 207. An early version of these scorers omitted the required per-event timestamp and silently ingested nothing while reporting success. langfuse_trace._warn_on_rejected_events now logs the per-event errors under LANGFUSE_DEBUG=1. If scores are missing, check that before anything else.

What the first run showed

Backfilled over 42 existing traces and 13 PRs:

cost_per_finding     n=42  mean=0.3133  min=0.0880  max=0.9042
finding_rate         n=42  mean=0.4762  min=0.0000  max=4.0000
severity_info_ratio  n=14  mean=0.0000
review_engagement    n=14  mean=0.0000
severity_max         {none: 28, medium: 11, high: 1, critical: 2}

Two things worth keeping:

• The reviewer is not info-heavy. feedback.db shows 61 of 62 findings at INFO, which looked like a badly calibrated model. It is not: severity_max reads medium/high/critical on every trace that found anything, and severity_info_ratio is flat 0. The INFO in the DB comes from feedback_harvest._parse_severity, which defaults to INFO when its regex misses the severity badge in the rendered comment. The DB severity is a re-parse artifact; the score reads the model's structured output directly.
• 28 of 42 reviews found nothing (67%), and engagement is flat zero. The first is not yet interpretable without the second.