feat(eval): LLM-as-judge evaluators, dataset item fixes, and Experiments runs #13

2026-08-31T15:39:05Z

gitea_admin commented

2026-08-31 15:39:05 +00:00

Three layered changes for the Langfuse eval side of pragent:

1. Dataset item ids that survive a URL path
Items were keyed {repo}#{pr} — both characters break the UI item route /datasets/{id}/items/{item_id}. Now {owner}__{repo}__pr{n}, which round-trips through quote(safe=) unchanged. The real repo and pr stay in input. 28 broken items were deleted and recreated (29 now).

2. Filterable metadata + Experiments runs
The filter bar matches on metadata only, so every facet worth slicing on is a flat primitive key: repo, owner, repo_name, pr, head_sha, finding_count, has_findings, max_severity, reviews_run, last_reviewed_at/_iso, labelled_by_human. Nested objects and lists are deliberately absent: the bar cannot reach into them.

eval_experiment.py links already-traced reviews into dataset runs grouped by model. One trace per (run, item), the newest. Re-review churn no longer duplicates results.

3. LLM-as-judge evaluators
Two llm_as_judge evaluators score the review generation: NUMERIC 0–1 on finding actionability, BOOLEAN on whether the summary agrees with the findings. Both run on every observation whose trace name is pr-review or opencode-review. The judge is kimi-k2.7-code through the headroom hub, routed via a new judge-proxy pod that patches the signature field local Ollama drops from thinking blocks.

Trace + generation output now includes the findings themselves (capped at 25) rather than just the count, so a judge has something to grade. Generation input/output mirrors the trace so an observation-level evaluator can read them.

Idempotent on re-run: evaluators and rules are skipped, not duplicated. The connection is upserted on provider.

🤖 Generated with Claude Code

Three layered changes for the Langfuse eval side of pragent: **1. Dataset item ids that survive a URL path** Items were keyed `{repo}#{pr}` — both characters break the UI item route `/datasets/{id}/items/{item_id}`. Now `{owner}__{repo}__pr{n}`, which round-trips through `quote(safe=)` unchanged. The real repo and pr stay in `input`. 28 broken items were deleted and recreated (29 now). **2. Filterable metadata + Experiments runs** The filter bar matches on `metadata` only, so every facet worth slicing on is a flat primitive key: `repo`, `owner`, `repo_name`, `pr`, `head_sha`, `finding_count`, `has_findings`, `max_severity`, `reviews_run`, `last_reviewed_at`/`_iso`, `labelled_by_human`. Nested objects and lists are deliberately absent: the bar cannot reach into them. `eval_experiment.py` links already-traced reviews into dataset runs grouped by model. One trace per (run, item), the newest. Re-review churn no longer duplicates results. **3. LLM-as-judge evaluators** Two `llm_as_judge` evaluators score the review generation: NUMERIC 0–1 on finding actionability, BOOLEAN on whether the summary agrees with the findings. Both run on every observation whose trace name is `pr-review` or `opencode-review`. The judge is `kimi-k2.7-code` through the headroom hub, routed via a new `judge-proxy` pod that patches the `signature` field local Ollama drops from thinking blocks. Trace + generation output now includes the findings themselves (capped at 25) rather than just the count, so a judge has something to grade. Generation input/output mirrors the trace so an observation-level evaluator can read them. Idempotent on re-run: evaluators and rules are skipped, not duplicated. The connection is upserted on provider. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

gitea_admin added 1 commit 2026-08-31 15:39:06 +00:00

fix(eval): dataset item ids that survive a URL path 1534a99630

Items were keyed `{repo}#{pr}`, e.g. `netcracker/interview#29`. Both
characters break the UI's item route: the `/` in `owner/repo` splits into
extra path segments, and everything after the `#` is a fragment the browser
never sends. Items were created successfully and then 404'd when opened.

Ids are now `{owner}__{repo}__pr{n}`, which needs no percent-encoding. The
real repo and pr stay intact in `input`, so nothing downstream reads the id
back apart. Session ids elsewhere keep the `{repo}#{pr}` form — those are
never path segments and feedback_scores depends on that shape.

The 28 existing items were unusable and are regenerable from feedback.db;
they were deleted and recreated under the new ids.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

gitea_admin changed title from ~~fix(eval): dataset item ids that survive a URL path~~ to fix(eval): url-safe dataset item ids, filterable metadata, and Experiments runs

2026-08-31 15:53:44 +00:00

gitea_admin added 1 commit 2026-08-31 15:53:44 +00:00

feat(eval): filterable item metadata and dataset runs for the Experiments tab 2e1ad817e7

The filter bar matches on metadata only — not on input and not on the item id —
so a dataset seeded with repo/pr in `input` alone could not be sliced by repo
at all. Every facet worth filtering on is now a flat primitive in `metadata`:
repo, owner, repo_name, pr, head_sha, finding_count, has_findings,
max_severity, reviews_run and the review timestamp both ways. `owner` is split
out because a filter on the joined repo matches one repo, never a whole org,
and `max_severity` is "none" rather than absent because an absent key matches
no filter.

`eval_experiment.py` links reviews that already ran into a dataset run, one run
per model, so the Experiments tab is populated without re-running the reviewer.
One trace per (run, item), the most recent: a PR re-reviewed on every push has
many traces and a run is one output per input.

It posts to the deprecated /api/public/dataset-run-items — the notice exempts
self-hosted v3 from the cutoff date and the pilot is stdlib-only by design.
Revisit at v4.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

gitea_admin changed title from ~~fix(eval): url-safe dataset item ids, filterable metadata, and Experiments runs~~ to feat(eval): LLM-as-judge evaluators, dataset item fixes, and Experiments runs

2026-08-31 17:18:39 +00:00

gitea_admin added 1 commit 2026-08-31 17:18:41 +00:00

feat(eval): LLM-as-judge evaluators for finding actionability and review self-consistency 5d44121b28

Two llm_as_judge evaluators score the review generation directly: a
NUMERIC 0-1 on finding actionability, a BOOLEAN on whether the summary
agrees with the findings. Both run on every observation whose trace
name is pr-review or opencode-review.

The judge is kimi-k2.7-code through the headroom hub. Local Ollama
returns Anthropic-format responses but the thinking blocks lack the
signature field Langfuse Zod schema requires; the evaluator preflight
fails as Invalid JSON response. A small judge-proxy pod on 8802
forwards to the hub and patches every thinking block with a synthetic
signature before returning.

Trace + generation output now includes the findings themselves
(capped at 25) rather than just the count, so a judge has something
to grade. generation input/output mirrors the trace so an
observation-level evaluator can read them.

Idempotent: existing evaluators and rules are skipped on re-run,
not duplicated. The connection is upserted on provider.

gitea_admin added 1 commit 2026-08-31 18:26:08 +00:00

test: retrigger review after eval rule wiring 72ac0f76bc

gitea_admin added 1 commit 2026-08-31 18:31:33 +00:00

test: re-trigger after dedupe window 3543d15677

gitea_admin added 1 commit 2026-08-31 18:33:09 +00:00

chore: enable pragent pilot on this repo (.pr-review.json on PR branch) 1644c7f6b1

gitea_admin merged commit 193a90e63e into main

2026-08-31 18:34:03 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: gitea_admin/pragent#13