feat(pilot): behavioural scorers, feedback ground truth, and an eval dataset #12

2026-08-31T14:23:18Z

gitea_admin commented

2026-08-31 14:23:18 +00:00

Evaluation layer on top of the review traces.

Why behaviour and not accuracy. feedback.db has 113 reviews and zero reactions, resolutions or replies. No labels exist, so a correctness judge would be scoring against nothing. The five scorers here measure behaviour (computable today); feedback_scores.py turns human verdicts into scores the moment any arrive.

Scores (eval_scores.py, folded into the existing ingestion batch — no extra request): finding_rate, severity_info_ratio, severity_max, dropped_findings, cost_per_finding. Undefined values are omitted, never reported as 0.

Parser change. Both parsers silently dropped findings with an unusable path/line, making a model that emits garbage locations indistinguishable from one that found nothing. last_parse_dropped() exposes the delta at parse time — after apply_repo_config the drops are the config working as intended.

Ground truth (feedback_scores.py): review_engagement and review_acceptance, scored on the session ({repo}#{pr}). Acceptance is absent, not 0, when nothing was engaged.

Dataset (eval_bootstrap.py): pragent-reviews, 28 items. expectedOutput is the reviewer's own prior output, flagged labelled_by_human: false — a regression baseline, not verified truth.

Bug fixed: ingestion answers 207 when only some events succeed, so a batch with every event rejected read as success. Score events lacked the required per-event timestamp and silently ingested nothing. _warn_on_rejected_events now logs per-event errors under LANGFUSE_DEBUG.

First run, over 42 traces and 13 PRs:

cost_per_finding     n=42  mean=0.3133  min=0.0880  max=0.9042
finding_rate         n=42  mean=0.4762  min=0.0000  max=4.0000
severity_info_ratio  n=14  mean=0.0000
review_engagement    n=14  mean=0.0000
severity_max         {none: 28, medium: 11, high: 1, critical: 2}

This corrected a wrong read of the data: the 61/62 INFO findings in feedback.db looked like a miscalibrated model, but severity_info_ratio is flat 0 and severity_max reads medium/high/critical. The INFO comes from feedback_harvest._parse_severity defaulting when its regex misses the rendered badge.

498 tests pass.

🤖 Generated with Claude Code

Evaluation layer on top of the review traces. **Why behaviour and not accuracy.** `feedback.db` has 113 reviews and **zero** reactions, resolutions or replies. No labels exist, so a correctness judge would be scoring against nothing. The five scorers here measure behaviour (computable today); `feedback_scores.py` turns human verdicts into scores the moment any arrive. **Scores** (`eval_scores.py`, folded into the existing ingestion batch — no extra request): `finding_rate`, `severity_info_ratio`, `severity_max`, `dropped_findings`, `cost_per_finding`. Undefined values are omitted, never reported as 0. **Parser change.** Both parsers silently dropped findings with an unusable `path`/`line`, making a model that emits garbage locations indistinguishable from one that found nothing. `last_parse_dropped()` exposes the delta at parse time — after `apply_repo_config` the drops are the config working as intended. **Ground truth** (`feedback_scores.py`): `review_engagement` and `review_acceptance`, scored on the session (`{repo}#{pr}`). Acceptance is *absent*, not 0, when nothing was engaged. **Dataset** (`eval_bootstrap.py`): `pragent-reviews`, 28 items. `expectedOutput` is the reviewer's own prior output, flagged `labelled_by_human: false` — a regression baseline, not verified truth. **Bug fixed:** ingestion answers 207 when only some events succeed, so a batch with *every* event rejected read as success. Score events lacked the required per-event `timestamp` and silently ingested nothing. `_warn_on_rejected_events` now logs per-event errors under `LANGFUSE_DEBUG`. **First run, over 42 traces and 13 PRs:** ``` cost_per_finding n=42 mean=0.3133 min=0.0880 max=0.9042 finding_rate n=42 mean=0.4762 min=0.0000 max=4.0000 severity_info_ratio n=14 mean=0.0000 review_engagement n=14 mean=0.0000 severity_max {none: 28, medium: 11, high: 1, critical: 2} ``` This corrected a wrong read of the data: the 61/62 `INFO` findings in `feedback.db` looked like a miscalibrated model, but `severity_info_ratio` is flat 0 and `severity_max` reads medium/high/critical. The `INFO` comes from `feedback_harvest._parse_severity` defaulting when its regex misses the rendered badge. 498 tests pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

gitea_admin added 1 commit 2026-08-31 14:23:19 +00:00

feat(pilot): behavioural scorers, feedback ground truth, and an eval dataset 2f96e66aab

Adds the evaluation layer on top of the review traces: five deterministic
scores describing how the reviewer behaved, a bridge that turns human reactions
into ground truth, and a dataset seeded from the reviews already run.

The two are kept apart on purpose. feedback.db has recorded 113 reviews and
zero reactions, resolutions or replies — nobody has ever responded to a bot
comment — so an accuracy metric cannot be built yet. The scorers therefore
measure behaviour, which is computable from data in hand, and feedback_scores
turns verdicts into scores the moment any arrive.

eval_scores.py emits finding_rate, severity_info_ratio, severity_max,
dropped_findings and cost_per_finding into the same ingestion batch as the
trace. Undefined values are omitted rather than reported as zero: an info ratio
over a silent review is undefined, and charting it as 0 would read as perfect
calibration.

dropped_findings needed a parser change. Both parsers silently discard findings
with an unusable path/line, which made a model emitting garbage locations
indistinguishable from one that found nothing. last_parse_dropped() exposes the
delta, read at parse time — after apply_repo_config the drops are the config
working as intended, not the model misbehaving.

feedback_scores.py scores the session ("{repo}#{pr}"), because feedback arrives
days later against a PR and nothing records which re-run produced which
comment. review_acceptance is absent rather than 0 when nothing was engaged.

eval_bootstrap.py registers the score configs, seeds the pragent-reviews
dataset, and can backfill scores onto traces that predate the scorers.
expectedOutput is the reviewer's own prior output, flagged
labelled_by_human: false — a regression baseline, not verified truth.

Also fixes a silent telemetry failure: the ingestion endpoint answers 207 when
only some events succeed, so a batch with every event rejected still looked
like success. Score events were missing the required per-event timestamp and
ingested nothing while reporting 207. _warn_on_rejected_events now logs the
per-event errors under LANGFUSE_DEBUG.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

gitea_admin merged commit d9eb4d9822 into main

2026-08-31 14:48:06 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: gitea_admin/pragent#12