feat(pilot): behavioural scorers, feedback ground truth, and an eval dataset

Adds the evaluation layer on top of the review traces: five deterministic
scores describing how the reviewer behaved, a bridge that turns human reactions
into ground truth, and a dataset seeded from the reviews already run.

The two are kept apart on purpose. feedback.db has recorded 113 reviews and
zero reactions, resolutions or replies — nobody has ever responded to a bot
comment — so an accuracy metric cannot be built yet. The scorers therefore
measure behaviour, which is computable from data in hand, and feedback_scores
turns verdicts into scores the moment any arrive.

eval_scores.py emits finding_rate, severity_info_ratio, severity_max,
dropped_findings and cost_per_finding into the same ingestion batch as the
trace. Undefined values are omitted rather than reported as zero: an info ratio
over a silent review is undefined, and charting it as 0 would read as perfect
calibration.

dropped_findings needed a parser change. Both parsers silently discard findings
with an unusable path/line, which made a model emitting garbage locations
indistinguishable from one that found nothing. last_parse_dropped() exposes the
delta, read at parse time — after apply_repo_config the drops are the config
working as intended, not the model misbehaving.

feedback_scores.py scores the session ("{repo}#{pr}"), because feedback arrives
days later against a PR and nothing records which re-run produced which
comment. review_acceptance is absent rather than 0 when nothing was engaged.

eval_bootstrap.py registers the score configs, seeds the pragent-reviews
dataset, and can backfill scores onto traces that predate the scorers.
expectedOutput is the reviewer's own prior output, flagged
labelled_by_human: false — a regression baseline, not verified truth.

Also fixes a silent telemetry failure: the ingestion endpoint answers 207 when
only some events succeed, so a batch with every event rejected still looked
like success. Score events were missing the required per-event timestamp and
ingested nothing while reporting 207. _warn_on_rejected_events now logs the
per-event errors under LANGFUSE_DEBUG.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

pilot/ai_review.py

@@ -678,6 +678,19 @@ def _strip_path_prefix(p: str) -> str:
 # ---------------------------------------------------------------------------
 
 
+# How many raw findings the last `parse_review_output` / `parse_findings` call
+# rejected for an unusable path/line. A side channel rather than a return value
+# because both parsers already return fixed-width tuples that several callers
+# and their tests unpack positionally; widening them to carry a telemetry
+# number would be a breaking change for a fail-open signal.
+_LAST_PARSE_DROPPED: dict[str, int] = {"n": 0}
+
+
+def last_parse_dropped() -> int:
+    """Findings the last parse discarded. Read it immediately after parsing."""
+    return int(_LAST_PARSE_DROPPED.get("n") or 0)
+
+
 def _normalize_finding(f: dict) -> dict | None:
     """Validate + normalize one raw finding dict. Returns None if it's unusable
     (missing path/line). Normalises severity, keeps `reference` (default "")."""
@@ -754,6 +767,7 @@ def parse_findings(text: str) -> list[dict]:
     Also accepts a bare JSON array as the outer value: ``[{...}, {...}]`` —
     some agents skip the ``{"summary":..., "findings":[...]}`` wrapper.
     """
+    _LAST_PARSE_DROPPED["n"] = 0
     data = _parse_json_tolerant(text)
     if isinstance(data, dict):
         findings = data.get("findings")
@@ -768,6 +782,7 @@ def parse_findings(text: str) -> list[dict]:
         n = _normalize_finding(f)
         if n is not None:
             out.append(n)
+    _LAST_PARSE_DROPPED["n"] = len(findings) - len(out)
     return out
 
 
@@ -823,6 +838,7 @@ def parse_review_output(
     block), with a tolerant fallback that scans for the last balanced
     object/array in the prose tail. Never raises.
     """
+    _LAST_PARSE_DROPPED["n"] = 0
     blob = _last_json_block(text)
     if blob is None:
         return "", [], [], [], [], "", ""
@@ -856,6 +872,12 @@ def parse_review_output(
             n = _normalize_finding(f)
             if n is not None:
                 out.append(n)
+        # A model that emits findings at unusable locations is indistinguishable
+        # from one that found nothing, because both end up with an empty `out`.
+        # Stash the delta so the caller can score it (see `eval_scores`).
+        _LAST_PARSE_DROPPED["n"] = len(findings_raw) - len(out)
+    else:
+        _LAST_PARSE_DROPPED["n"] = 0
     return summary, out, summary_changes, risks, walkthrough, risk_verdict, test_coverage
 
 
@@ -2080,6 +2102,7 @@ def _emit_langfuse(
     summary: str,
     engine: str,
     config: dict | None = None,
+    dropped_count: float | None = None,
 ) -> None:
     """Ship this review's usage to Langfuse, if one is configured.
 
@@ -2105,7 +2128,7 @@ def _emit_langfuse(
             repo=repo, index=index, sha=sha, title=title, model=model,
             usage=usage, findings=findings, summary=summary or "",
             engine=engine, lenses=(usage or {}).get("lenses"),
-            price_target=price_target,
+            price_target=price_target, dropped_count=dropped_count,
         )
     except Exception as e:
         print(f"pragent: langfuse emit skipped: {e}", file=sys.stderr)
@@ -2235,6 +2258,7 @@ def review_pr(
                     additional_context=additional_context,
                 )
             review_summary, findings, summary_changes, risks, _walkthrough, _risk_verdict, _test_coverage = parse_review_output(stdout)
+            parse_dropped = last_parse_dropped()
             if not findings and not review_summary:
                 # The findings JSON was missing or malformed. Don't discard the
                 # run: salvage the prose, keep the usage report (the tokens were
@@ -2255,12 +2279,14 @@ def review_pr(
                     repo=repo, index=index, sha=sha, title=title,
                     model=display_model, usage=usage, findings=[],
                     summary=salvaged, engine=engine, config=config,
+                    dropped_count=parse_dropped,
                 )
                 return True
         else:
             user_prompt = build_user_prompt(title, body + compression_note, diff, config, prior, additional_context)
             raw_findings = call_model(ollama_url, model, SYSTEM_PROMPT, user_prompt, max_tokens)
             findings = parse_findings(raw_findings)
+            parse_dropped = last_parse_dropped()
             usage = None
 
         # Filter / cap findings per `.pr-review.json` (style, threshold, max,
@@ -2343,6 +2369,7 @@ def review_pr(
             repo=repo, index=index, sha=sha, title=title,
             model=display_model, usage=usage, findings=findings,
             summary=review_summary, engine=engine, config=config,
+            dropped_count=parse_dropped,
         )
         print(
             f"pragent: reviewed {repo}#{index} sha={sha[:8]} "