feat(eval): filterable item metadata and dataset runs for the Experiments tab

The filter bar matches on metadata only — not on input and not on the item id —
so a dataset seeded with repo/pr in `input` alone could not be sliced by repo
at all. Every facet worth filtering on is now a flat primitive in `metadata`:
repo, owner, repo_name, pr, head_sha, finding_count, has_findings,
max_severity, reviews_run and the review timestamp both ways. `owner` is split
out because a filter on the joined repo matches one repo, never a whole org,
and `max_severity` is "none" rather than absent because an absent key matches
no filter.

`eval_experiment.py` links reviews that already ran into a dataset run, one run
per model, so the Experiments tab is populated without re-running the reviewer.
One trace per (run, item), the most recent: a PR re-reviewed on every push has
many traces and a run is one output per input.

It posts to the deprecated /api/public/dataset-run-items — the notice exempts
self-hosted v3 from the cutoff date and the pilot is stdlib-only by design.
Revisit at v4.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

pilot/README-eval.md

@@ -75,6 +75,58 @@ regression baseline: re-run a candidate model over these PRs and the diff
 against this column is the behaviour change. Promoting an item to real ground
 truth means a human editing it in the dataset view after re-reading the PR.
 
+### Item ids
+
+`{owner}__{repo}__pr{n}`. The obvious `{repo}#{pr}` cannot be used: items are
+routed as `/datasets/{id}/items/{item_id}`, so the `/` in `owner/repo` splits
+into extra path segments and everything after `#` is a fragment the browser
+never sends — the item is created fine by the API and then 404s when opened.
+Session ids elsewhere keep `{repo}#{pr}`; those are never path segments.
+
+### Filterable metadata
+
+The filter bar matches on `metadata` only — not on `input`, and not on the item
+id — so every facet worth slicing on is a flat, primitive key in `metadata`
+even where it duplicates `input`:
+
+| key | why it is there |
+| --- | --- |
+| `repo`, `owner`, `repo_name` | `owner` exists because a filter on the joined `repo` matches one repo, never a whole org |
+| `pr`, `head_sha` | jump from a filtered row back to the actual PR |
+| `finding_count`, `has_findings` | isolate the silent reviews, which are the interesting negatives |
+| `max_severity` | `"none"` rather than absent — an absent key matches no filter |
+| `reviews_run` | how churny the PR was; high values skew per-item averages |
+| `last_reviewed_at` / `_iso` | epoch sorts, ISO reads |
+| `labelled_by_human` | `false` everywhere today; the flag to filter on before trusting any of it |
+
+Nested objects and lists are deliberately absent: the filter bar cannot reach
+into them.
+
+`max_severity` is derived from `feedback.db`, whose `severity` column is
+re-parsed out of the rendered comment by `feedback_harvest._parse_severity` and
+defaults to `INFO` when its regex misses the badge. Trust the `severity_max`
+**score** (read from the model's structured output) over this facet.
+
+## Experiments
+
+`eval_experiment.py` links reviews that already ran into a dataset run, so the
+Experiments tab is populated without re-running anything. Runs are grouped by
+model — the comparison the pilot actually needs is the same PRs under a
+candidate model with `finding_rate` and `cost_per_finding` side by side. A new
+model produces a new run automatically on the next invocation.
+
+One trace per (run, item), the most recent: a PR re-reviewed on every push has
+many traces, and a run is one output per input.
+
+It uses `POST /api/public/dataset-run-items`, which is deprecated in favour of
+the SDK experiment runner and disappears in Langfuse v4. The deprecation notice
+exempts self-hosted v3 from the cutoff date, and this pilot is stdlib-only by
+design. Revisit when this deployment moves to v4.
+
+Coverage is bounded by the dataset, not by the traces: items only exist for PRs
+with a row in `feedback.db`, and a review that posted no comment leaves a trace
+but no row. That is why a run links fewer items than there are traces.
+
 ## Running it
 
 ```bash
@@ -83,6 +135,10 @@ python3 pilot/eval_bootstrap.py --db /data/feedback.db --backfill-traces
 
 # ship feedback verdicts (runs daily from the feedback CronJob)
 python3 pilot/feedback_scores.py --db /data/feedback.db
+
+# link already-traced reviews into a dataset run per model
+python3 pilot/eval_experiment.py --dry-run
+python3 pilot/eval_experiment.py
 ```
 
 Both need `LANGFUSE_HOST`, `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`. In