gitea_admin/pragent
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

test: eval judges e2e #14

Merged
masi merged 4 commits from test/eval-judges-trigger into main 2026-09-01 00:26:59 +00:00
Conversation 4 Commits 4 Files Changed 4
+25 -10
gitea_admin commented 2026-08-31 18:37:43 +00:00
Owner
Copy Link
Copy Source
No description provided.
gitea_admin added 1 commit 2026-08-31 18:37:43 +00:00
test: trigger review for eval judges d746b1fdc2
pragent-bot bot reviewed 2026-08-31 18:38:29 +00:00
pragent-bot bot left a comment
Copy Link
Copy Source

🤖 AI Review · pragent pilot · headroom/MiniMax-M2.7 · d746b1fd · Merge confidence: 5/5 🟢

PR-Agent pilot on this repo. Comments are LLM-generated; treat as suggestions, not mandates.

Summary of Changes

  • Adds pilot eval judges test 1788201461 marker to README.md end

Key Risks & Concerns

None identified.

🔋 AI Usage & Run Details

  • Model / Engine: headroom/MiniMax-M2.7 · opencode · 3 steps · 45.1s

  • Total Tokens: 3,260 (3.3K) in / 914 out (0 reasoning, cache 23,532 (23.5K) read / 14,906 (14.9K) write, 42,612 (42.6K) total)

  • Actual: $0.00 (headroom/MiniMax-M2.7 — free tier)

  • Scope: Whole-repo checkout at head sha (agent can read any file + run linters, not just the diff) — input tokens include files read beyond the diff. Per-comment output is attributed (one model pass produces all findings; output split by each finding's body weight).

  • Equivalent cost on paid providers (this run's tokens):

Provider Cost
Claude Sonnet 5 $0.01
GPT-5 $0.0054
Gemini 2.5 Pro $0.0058
Grok 4.5 $0.0018
🤖 **AI Review** · pragent pilot · headroom/MiniMax-M2.7 · `d746b1fd` · Merge confidence: 5/5 🟢 > PR-Agent pilot on this repo. Comments are LLM-generated; treat as suggestions, not mandates. ### Summary of Changes - Adds _pilot eval judges test 1788201461_ marker to README.md end ### Key Risks & Concerns _None identified._ <details> <summary>🔋 AI Usage & Run Details</summary> - **Model / Engine**: `headroom/MiniMax-M2.7` · opencode · 3 steps · 45.1s - **Total Tokens**: 3,260 (3.3K) in / 914 out (0 reasoning, cache 23,532 (23.5K) read / 14,906 (14.9K) write, 42,612 (42.6K) total) - **Actual**: $0.00 (headroom/MiniMax-M2.7 — free tier) - **Scope**: Whole-repo checkout at head sha (agent can read any file + run linters, not just the diff) — input tokens include files read beyond the diff. Per-comment output is *attributed* (one model pass produces all findings; output split by each finding's body weight). - **Equivalent cost on paid providers** (this run's tokens): | Provider | Cost | |---|---:| | **Claude Sonnet 5** | $0.01 | | GPT-5 | $0.0054 | | Gemini 2.5 Pro | $0.0058 | | Grok 4.5 | $0.0018 | </details> <!-- pragent:sha=d746b1fdc26cef72f2631f18de51ed032d0e4c26 -->
gitea_admin added 1 commit 2026-08-31 19:19:37 +00:00
pilot(eval): switch rule target from observation to trace 3109bd7c2d 
The standard /api/public/ingestion path feeds only the trace-upsert
queue; evalService.createEvalJobs only dispatches targetObject in
{TRACE, DATASET}. Observation rules fire exclusively from the OTel
pipeline, which this pilot does not use. The trace body already carries
review input/output via langfuse_trace, so a trace rule sees the same
material an observation rule would.
gitea_admin added 1 commit 2026-08-31 19:20:02 +00:00
test: trigger review after DB rule insert 5b8be61e2f
pragent-bot bot reviewed 2026-08-31 19:21:14 +00:00
pragent-bot bot left a comment
Copy Link
Copy Source

🤖 AI Review · pragent pilot · headroom/MiniMax-M2.7 · 5b8be61e · Merge confidence: 5/5 🟢

PR-Agent pilot on this repo. Comments are LLM-generated; treat as suggestions, not mandates.

Summary of Changes

  • pilot/eval_judges.py: new 314-line module bootstrapping two Langfuse LLM-as-judge evaluators (finding_actionability, review_self_consistency) and their trace-filter rules
  • tests/pilot/test_eval_judges.py: 132-line test suite covering rule_body shape, ensure_evaluators/ensure_rules idempotency, and judge config validation
  • .judge-trigger.txt and README.md updated with timestamp/test-marker lines

Key Risks & Concerns

  • Hardcoded default JUDGE_BASE_URL (http://100.74.17.70:8802) points at internal infrastructure IP — not a security issue here (intentional pilot design, env-overrideable), but worth documenting in a comment
🔋 AI Usage & Run Details
  • Model / Engine: headroom/MiniMax-M2.7 · opencode · 7 steps · 70.2s
  • Total Tokens: 19,276 (19.3K) in / 3,815 (3.8K) out (0 reasoning, cache 115,149 (115.1K) read / 35,278 (35.3K) write, 173,518 (173.5K) total)
  • Actual: $0.00 (headroom/MiniMax-M2.7 — free tier)
  • Scope: Whole-repo checkout at head sha (agent can read any file + run linters, not just the diff) — input tokens include files read beyond the diff. Per-comment output is attributed (one model pass produces all findings; output split by each finding's body weight).
🤖 **AI Review** · pragent pilot · headroom/MiniMax-M2.7 · `5b8be61e` · Merge confidence: 5/5 🟢 > PR-Agent pilot on this repo. Comments are LLM-generated; treat as suggestions, not mandates. ### Summary of Changes - pilot/eval_judges.py: new 314-line module bootstrapping two Langfuse LLM-as-judge evaluators (finding_actionability, review_self_consistency) and their trace-filter rules - tests/pilot/test_eval_judges.py: 132-line test suite covering rule_body shape, ensure_evaluators/ensure_rules idempotency, and judge config validation - .judge-trigger.txt and README.md updated with timestamp/test-marker lines ### Key Risks & Concerns - Hardcoded default JUDGE_BASE_URL (http://100.74.17.70:8802) points at internal infrastructure IP — not a security issue here (intentional pilot design, env-overrideable), but worth documenting in a comment <details> <summary>🔋 AI Usage & Run Details</summary> - **Model / Engine**: `headroom/MiniMax-M2.7` · opencode · 7 steps · 70.2s - **Total Tokens**: 19,276 (19.3K) in / 3,815 (3.8K) out (0 reasoning, cache 115,149 (115.1K) read / 35,278 (35.3K) write, 173,518 (173.5K) total) - **Actual**: $0.00 (headroom/MiniMax-M2.7 — free tier) - **Scope**: Whole-repo checkout at head sha (agent can read any file + run linters, not just the diff) — input tokens include files read beyond the diff. Per-comment output is *attributed* (one model pass produces all findings; output split by each finding's body weight). </details> <!-- pragent:sha=5b8be61e2f0e794d25fdd750f923854be1b37e69 -->
pragent-bot bot reviewed 2026-08-31 19:21:30 +00:00
pragent-bot bot left a comment
Copy Link
Copy Source

🤖 AI Review · pragent pilot · headroom/MiniMax-M2.7 · 3109bd7c · Merge confidence: 3/5 🟡

PR-Agent pilot on this repo. Comments are LLM-generated; treat as suggestions, not mandates.

Summary of Changes

  • pilot/eval_judges.py: new file — registers two LLM-as-judge evaluators (finding_actionability, review_self_consistency) with Langfuse and wires evaluation rules to pr-review traces
  • tests/pilot/test_eval_judges.py: new file — tests for rule_body shape, ensure_evaluators idempotency, and ensure_rules idempotency
  • README.md: appends a single timestamp line

Key Risks & Concerns

  • existing_rule_names() filters by target='observation' but rule_body() sets target='trace', so re-runs will never find existing rules and will attempt to create duplicates, getting 409s and adding them to the failed list
  • Linter (ruff) reports 5 issues: unused noqa directives (E402 not enabled), unsorted imports, lambda instead of dict, and a shebang on a non-executable file

Findings Overview

1 inline comment(s); 1 total.

Severity Location Finding
🔴 [HIGH] pilot/eval_judges.py:275 existing_rule_names() filters rules by target='observation' but rule_body() creates rules with target='trace'. On re-run, ensure_rules() will never find the rules it created and will POST them again, getting a 409 Conflict and adding them to the failed list. This violates the stated idempotency: 're-running this script would pile up versions'.
🔋 AI Usage & Run Details
  • Model / Engine: headroom/MiniMax-M2.7 · opencode · 10 steps · 110.3s
  • Total Tokens: 21,353 (21.4K) in / 4,881 (4.9K) out (0 reasoning, cache 219,951 (220.0K) read / 61,983 (62.0K) write, 308,168 (308.2K) total)
  • Actual: $0.00 (headroom/MiniMax-M2.7 — free tier)
  • Scope: Whole-repo checkout at head sha (agent can read any file + run linters, not just the diff) — input tokens include files read beyond the diff. Per-comment output is attributed (one model pass produces all findings; output split by each finding's body weight).
🤖 **AI Review** · pragent pilot · headroom/MiniMax-M2.7 · `3109bd7c` · Merge confidence: 3/5 🟡 > PR-Agent pilot on this repo. Comments are LLM-generated; treat as suggestions, not mandates. ### Summary of Changes - pilot/eval_judges.py: new file — registers two LLM-as-judge evaluators (finding_actionability, review_self_consistency) with Langfuse and wires evaluation rules to pr-review traces - tests/pilot/test_eval_judges.py: new file — tests for rule_body shape, ensure_evaluators idempotency, and ensure_rules idempotency - README.md: appends a single timestamp line ### Key Risks & Concerns - existing_rule_names() filters by target='observation' but rule_body() sets target='trace', so re-runs will never find existing rules and will attempt to create duplicates, getting 409s and adding them to the failed list - Linter (ruff) reports 5 issues: unused noqa directives (E402 not enabled), unsorted imports, lambda instead of dict, and a shebang on a non-executable file ### Findings Overview _1 inline comment(s); 1 total._ | Severity | Location | Finding | |---|---|---| | 🔴 [HIGH] | `pilot/eval_judges.py:275` | existing_rule_names() filters rules by target='observation' but rule_body() creates rules with target='trace'. On re-run, ensure_rules() will never find the rules it created and will POST them again, getting a 409 Conflict and adding them to the failed list. This violates the stated idempotency: 're-running this script would pile up versions'. | <details> <summary>🔋 AI Usage & Run Details</summary> - **Model / Engine**: `headroom/MiniMax-M2.7` · opencode · 10 steps · 110.3s - **Total Tokens**: 21,353 (21.4K) in / 4,881 (4.9K) out (0 reasoning, cache 219,951 (220.0K) read / 61,983 (62.0K) write, 308,168 (308.2K) total) - **Actual**: $0.00 (headroom/MiniMax-M2.7 — free tier) - **Scope**: Whole-repo checkout at head sha (agent can read any file + run linters, not just the diff) — input tokens include files read beyond the diff. Per-comment output is *attributed* (one model pass produces all findings; output split by each finding's body weight). </details> <!-- pragent:sha=3109bd7c2d206e34d32d195749ce3a36902f8a2a -->
pilot/eval_judges.py
@@ -0,0 +272,4 @@
st, body = eb._call("GET", "/api/public/unstable/evaluation-rules?limit=100")
if st == 200 and isinstance(body, dict):
for r in body.get("data") or []:
if r.get("target") == "observation":
pragent-bot bot commented 2026-08-31 19:21:30 +00:00
Copy Link
Copy Source

🔴 [HIGH] existing_rule_names() filters rules by target='observation' but rule_body() creates rules with target='trace'. On re-run, ensure_rules() will never find the rules it created and will POST them again, getting a 409 Conflict and adding them to the failed list. This violates the stated idempotency: 're-running this script would pile up versions'.

Fix: Change r.get('target') == 'observation' to r.get('target') == 'trace' so the function finds the rules it itself creates.

if r.get("target") == "trace":

🪙 ~4,881 (4.9K) tok (100% · attributed output)

🔴 [HIGH] existing_rule_names() filters rules by target='observation' but rule_body() creates rules with target='trace'. On re-run, ensure_rules() will never find the rules it created and will POST them again, getting a 409 Conflict and adding them to the failed list. This violates the stated idempotency: 're-running this script would pile up versions'. **Fix:** Change r.get('target') == 'observation' to r.get('target') == 'trace' so the function finds the rules it itself creates. ```suggestion if r.get("target") == "trace": ``` 🪙 ~4,881 (4.9K) tok (100% · attributed output)
masi added 1 commit 2026-09-01 00:26:35 +00:00
merge: resolve eval judge rule conflict 3f30bdd450
masi merged commit 9a9923d034 into main 2026-09-01 00:26:59 +00:00
pragent-bot bot reviewed 2026-09-01 00:27:39 +00:00
pragent-bot bot left a comment
Copy Link
Copy Source

🤖 AI Review · pragent pilot · headroom/MiniMax-M2.7 · 3f30bdd4 · Merge confidence: 4/5 🟢

PR-Agent pilot on this repo. Comments are LLM-generated; treat as suggestions, not mandates.

Summary of Changes

  • pilot/eval_judges.py: change rule_body target from 'observation' to 'trace', with updated docstring explaining why trace is used
  • pilot/eval_judges.py: existing_rule_names() still filters on target=='observation' — breaks idempotency for trace-target rules on re-run
  • tests/pilot/test_eval_judges.py: rename test and update assertion to match new trace target
  • README.md and .judge-trigger.txt: cosmetic/timestamp markers

Key Risks & Concerns

  • existing_rule_names() filters on target=='observation' but rule_body now produces trace-target rules. A re-run of ensure_rules() will not find the new rules, causing a duplicate-POST 409 instead of a clean skip.
  • No test covers the ensure_rules idempotency path for trace-target rules, so the regression is silent.

Findings Overview

1 finding(s).

Severity Location Finding
🟡 [MEDIUM] pilot/eval_judges.py:275 existing_rule_names() filters rules where target=='observation', but rule_body() now creates rules with target=='trace'. On a re-run of ensure_rules(), the newly created trace rules will not be found by this function, so the idempotency check at line 254 will not suppress the second POST — resulting in a 409 conflict instead of a clean skip. This breaks the stated idempotency guarantee in ensure_rules().

Unanchored Notes

  • 🟡 [MEDIUM] pilot/eval_judges.py:275 — existing_rule_names() filters rules where target=='observation', but rule_body() now creates rules with target=='trace'. On a re-run of ensure_rules(), the newly created trace rules will not be found by this function, so the idempotency check at line 254 will not suppress the second POST — resulting in a 409 conflict instead of a clean skip. This breaks the stated idempotency guarantee in ensure_rules().
    • Fix: Change the filter from target=='observation' to target=='trace', and update the docstring to match
🔋 AI Usage & Run Details
  • Model / Engine: headroom/MiniMax-M2.7 · opencode · 6 steps · 61.5s
  • Total Tokens: 10,092 (10.1K) in / 2,923 (2.9K) out (0 reasoning, cache 76,607 (76.6K) read / 25,242 (25.2K) write, 114,864 (114.9K) total)
  • Actual: $0.00 (headroom/MiniMax-M2.7 — free tier)
  • Scope: Whole-repo checkout at head sha (agent can read any file + run linters, not just the diff) — input tokens include files read beyond the diff. Per-comment output is attributed (one model pass produces all findings; output split by each finding's body weight).
🤖 **AI Review** · pragent pilot · headroom/MiniMax-M2.7 · `3f30bdd4` · Merge confidence: 4/5 🟢 > PR-Agent pilot on this repo. Comments are LLM-generated; treat as suggestions, not mandates. ### Summary of Changes - pilot/eval_judges.py: change rule_body target from 'observation' to 'trace', with updated docstring explaining why trace is used - pilot/eval_judges.py: existing_rule_names() still filters on target=='observation' — breaks idempotency for trace-target rules on re-run - tests/pilot/test_eval_judges.py: rename test and update assertion to match new trace target - README.md and .judge-trigger.txt: cosmetic/timestamp markers ### Key Risks & Concerns - existing_rule_names() filters on target=='observation' but rule_body now produces trace-target rules. A re-run of ensure_rules() will not find the new rules, causing a duplicate-POST 409 instead of a clean skip. - No test covers the ensure_rules idempotency path for trace-target rules, so the regression is silent. ### Findings Overview _1 finding(s)._ | Severity | Location | Finding | |---|---|---| | 🟡 [MEDIUM] | `pilot/eval_judges.py:275` | existing_rule_names() filters rules where target=='observation', but rule_body() now creates rules with target=='trace'. On a re-run of ensure_rules(), the newly created trace rules will not be found by this function, so the idempotency check at line 254 will not suppress the second POST — resulting in a 409 conflict instead of a clean skip. This breaks the stated idempotency guarantee in ensure_rules(). | ### Unanchored Notes - 🟡 [MEDIUM] `pilot/eval_judges.py:275` — existing_rule_names() filters rules where target=='observation', but rule_body() now creates rules with target=='trace'. On a re-run of ensure_rules(), the newly created trace rules will not be found by this function, so the idempotency check at line 254 will not suppress the second POST — resulting in a 409 conflict instead of a clean skip. This breaks the stated idempotency guarantee in ensure_rules(). - **Fix:** Change the filter from target=='observation' to target=='trace', and update the docstring to match <details> <summary>🔋 AI Usage & Run Details</summary> - **Model / Engine**: `headroom/MiniMax-M2.7` · opencode · 6 steps · 61.5s - **Total Tokens**: 10,092 (10.1K) in / 2,923 (2.9K) out (0 reasoning, cache 76,607 (76.6K) read / 25,242 (25.2K) write, 114,864 (114.9K) total) - **Actual**: $0.00 (headroom/MiniMax-M2.7 — free tier) - **Scope**: Whole-repo checkout at head sha (agent can read any file + run linters, not just the diff) — input tokens include files read beyond the diff. Per-comment output is *attributed* (one model pass produces all findings; output split by each finding's body weight). </details> <!-- pragent:sha=3f30bdd4502203f3abcbbcd85d5873d625164dc8 -->
Sign in to join this conversation.
Reviewers
No Reviewers
pragent-bot
Labels
Clear labels
AI-REVIEW
Trigger pragent-bot AI review
 AI-USAGE
Opt-in: append AI token-usage summary + per-comment attribution to the review
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
pragent-bot (pragent AI review bot) gitea_admin
No Assignees
3 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: gitea_admin/pragent#14
Delete Branch "test/eval-judges-trigger"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?