Claude
2f96e66aab
Adds the evaluation layer on top of the review traces: five deterministic
scores describing how the reviewer behaved, a bridge that turns human reactions
into ground truth, and a dataset seeded from the reviews already run.
The two are kept apart on purpose. feedback.db has recorded 113 reviews and
zero reactions, resolutions or replies — nobody has ever responded to a bot
comment — so an accuracy metric cannot be built yet. The scorers therefore
measure behaviour, which is computable from data in hand, and feedback_scores
turns verdicts into scores the moment any arrive.
eval_scores.py emits finding_rate, severity_info_ratio, severity_max,
dropped_findings and cost_per_finding into the same ingestion batch as the
trace. Undefined values are omitted rather than reported as zero: an info ratio
over a silent review is undefined, and charting it as 0 would read as perfect
calibration.
dropped_findings needed a parser change. Both parsers silently discard findings
with an unusable path/line, which made a model emitting garbage locations
indistinguishable from one that found nothing. last_parse_dropped() exposes the
delta, read at parse time — after apply_repo_config the drops are the config
working as intended, not the model misbehaving.
feedback_scores.py scores the session ("{repo}#{pr}"), because feedback arrives
days later against a PR and nothing records which re-run produced which
comment. review_acceptance is absent rather than 0 when nothing was engaged.
eval_bootstrap.py registers the score configs, seeds the pragent-reviews
dataset, and can backfill scores onto traces that predate the scorers.
expectedOutput is the reviewer's own prior output, flagged
labelled_by_human: false — a regression baseline, not verified truth.
Also fixes a silent telemetry failure: the ingestion endpoint answers 207 when
only some events succeed, so a batch with every event rejected still looked
like success. Score events were missing the required per-event timestamp and
ingested nothing while reporting 207. _warn_on_rejected_events now logs the
per-event errors under LANGFUSE_DEBUG.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
91 lines
3.2 KiB
Python
91 lines
3.2 KiB
Python
"""The parse-time drop counter feeding the `dropped_findings` score.
|
|
|
|
A model that emits findings at unusable locations produces an empty findings
|
|
list, exactly like a model that found nothing. These tests pin the signal that
|
|
tells the two apart.
|
|
"""
|
|
import json
|
|
import os
|
|
import sys
|
|
|
|
HERE = os.path.dirname(os.path.abspath(__file__))
|
|
sys.path.insert(0, os.path.abspath(os.path.join(HERE, "..", "..", "pilot")))
|
|
|
|
import ai_review # noqa: E402
|
|
|
|
|
|
def _payload(findings):
|
|
return "```json\n" + json.dumps({"summary": "s", "findings": findings}) + "\n```"
|
|
|
|
|
|
GOOD = {"severity": "high", "path": "a.py", "line": 3, "problem": "p", "fix": "f"}
|
|
NO_PATH = {"severity": "high", "line": 3, "problem": "p"}
|
|
NO_LINE = {"severity": "high", "path": "a.py", "problem": "p"}
|
|
BAD_LINE = {"severity": "high", "path": "a.py", "line": 0, "problem": "p"}
|
|
|
|
|
|
def test_no_drops_on_clean_output():
|
|
_, findings, *_ = ai_review.parse_review_output(_payload([GOOD, GOOD]))
|
|
assert len(findings) == 2
|
|
assert ai_review.last_parse_dropped() == 0
|
|
|
|
|
|
def test_counts_findings_missing_path():
|
|
_, findings, *_ = ai_review.parse_review_output(_payload([GOOD, NO_PATH]))
|
|
assert len(findings) == 1
|
|
assert ai_review.last_parse_dropped() == 1
|
|
|
|
|
|
def test_counts_findings_missing_line():
|
|
_, findings, *_ = ai_review.parse_review_output(_payload([NO_LINE, NO_LINE]))
|
|
assert findings == []
|
|
assert ai_review.last_parse_dropped() == 2
|
|
|
|
|
|
def test_counts_findings_with_unusable_line():
|
|
_, findings, *_ = ai_review.parse_review_output(_payload([BAD_LINE]))
|
|
assert findings == []
|
|
assert ai_review.last_parse_dropped() == 1
|
|
|
|
|
|
def test_all_dropped_is_distinguishable_from_found_nothing():
|
|
ai_review.parse_review_output(_payload([NO_PATH, NO_PATH, NO_PATH]))
|
|
all_dropped = ai_review.last_parse_dropped()
|
|
ai_review.parse_review_output(_payload([]))
|
|
found_nothing = ai_review.last_parse_dropped()
|
|
assert all_dropped == 3 and found_nothing == 0
|
|
|
|
|
|
def test_counter_resets_on_unparseable_output():
|
|
# Otherwise a salvage-path review inherits the previous review's count.
|
|
ai_review.parse_review_output(_payload([NO_PATH, NO_PATH]))
|
|
assert ai_review.last_parse_dropped() == 2
|
|
ai_review.parse_review_output("no json here at all")
|
|
assert ai_review.last_parse_dropped() == 0
|
|
|
|
|
|
def test_counter_resets_on_malformed_json():
|
|
ai_review.parse_review_output(_payload([NO_PATH]))
|
|
ai_review.parse_review_output("```json\n{not valid json,,,}\n```")
|
|
assert ai_review.last_parse_dropped() == 0
|
|
|
|
|
|
def test_parse_findings_tracks_drops_too():
|
|
# The non-opencode path must be scored on the same basis.
|
|
findings = ai_review.parse_findings(json.dumps({"findings": [GOOD, NO_PATH]}))
|
|
assert len(findings) == 1
|
|
assert ai_review.last_parse_dropped() == 1
|
|
|
|
|
|
def test_parse_findings_resets_on_garbage():
|
|
ai_review.parse_findings(json.dumps({"findings": [NO_PATH]}))
|
|
assert ai_review.last_parse_dropped() == 1
|
|
ai_review.parse_findings("not json")
|
|
assert ai_review.last_parse_dropped() == 0
|
|
|
|
|
|
def test_bare_array_output_is_counted():
|
|
_, findings, *_ = ai_review.parse_review_output("```json\n" + json.dumps([GOOD, NO_PATH]) + "\n```")
|
|
assert len(findings) == 1
|
|
assert ai_review.last_parse_dropped() == 1
|