feat(eval): LLM-as-judge evaluators for finding actionability and review self-consistency

Two llm_as_judge evaluators score the review generation directly: a
NUMERIC 0-1 on finding actionability, a BOOLEAN on whether the summary
agrees with the findings. Both run on every observation whose trace
name is pr-review or opencode-review.

The judge is kimi-k2.7-code through the headroom hub. Local Ollama
returns Anthropic-format responses but the thinking blocks lack the
signature field Langfuse Zod schema requires; the evaluator preflight
fails as Invalid JSON response. A small judge-proxy pod on 8802
forwards to the hub and patches every thinking block with a synthetic
signature before returning.

Trace + generation output now includes the findings themselves
(capped at 25) rather than just the count, so a judge has something
to grade. generation input/output mirrors the trace so an
observation-level evaluator can read them.

Idempotent: existing evaluators and rules are skipped on re-run,
not duplicated. The connection is upserted on provider.

k8s/judge-proxy.yaml

@@ -0,0 +1,82 @@
+# Judge-side think-block patcher. Stands between Langfuse evaluators and the
+# headroom-ollama hub (port 8790). Local Ollama does not emit the `signature`
+# field that Langfuse's Anthropic adapter's Zod schema requires on every
+# `thinking` content block — without it, the evaluator preflight fails as
+# "Invalid JSON response". The proxy forwards /v1/* verbatim and adds a dummy
+# signature to each thinking block before returning.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: judge-proxy
+  namespace: pragent
+data:
+  proxy.py: |
+    #!/usr/bin/env python3
+    """Judge proxy: forward to headroom-ollama, fix thinking blocks."""
+    import json, sys, urllib.request, urllib.error
+    from http.server import BaseHTTPRequestHandler, HTTPServer
+    from socketserver import ThreadingMixIn
+    UPSTREAM = "http://100.74.17.70:8790"
+    DUMMY_SIG = "kimi-local-judge-no-signature"
+    class H(BaseHTTPRequestHandler):
+        def _proxy(self):
+            n = int(self.headers.get("Content-Length", 0))
+            body = self.rfile.read(n) if n else b""
+            h = {k: v for k, v in self.headers.items() if k.lower() not in ("host", "content-length")}
+            req = urllib.request.Request(UPSTREAM + self.path, data=body, headers=h, method=self.command)
+            try:
+                with urllib.request.urlopen(req, timeout=120) as r:
+                    resp_body = r.read(); status = r.status; rh = dict(r.headers)
+            except urllib.error.HTTPError as e:
+                resp_body = e.read(); status = e.code; rh = dict(e.headers)
+            ct = rh.get("content-type", "")
+            if status == 200 and "application/json" in ct and self.path.startswith("/v1/messages"):
+                try:
+                    obj = json.loads(resp_body)
+                    patched = 0
+                    for blk in obj.get("content") or []:
+                        if isinstance(blk, dict) and blk.get("type") == "thinking" and "signature" not in blk:
+                            blk["signature"] = DUMMY_SIG; patched += 1
+                    if patched:
+                        resp_body = json.dumps(obj).encode("utf-8")
+                        rh["content-length"] = str(len(resp_body))
+                        print(f"judge-proxy: patched {patched} thinking block(s)", file=sys.stderr, flush=True)
+                except Exception as e:
+                    print(f"judge-proxy: patch failed: {e}", file=sys.stderr, flush=True)
+            self.send_response(status)
+            for k, v in rh.items():
+                if k.lower() not in ("transfer-encoding", "content-length", "connection"):
+                    self.send_header(k, v)
+            self.send_header("Content-Length", str(len(resp_body)))
+            self.end_headers(); self.wfile.write(resp_body)
+        def do_POST(self): self._proxy()
+        def do_GET(self): self._proxy()
+        def log_message(self, *a, **k): pass
+    class S(ThreadingMixIn, HTTPServer): daemon_threads = True
+    S(("0.0.0.0", 8802), H).serve_forever()
+---
+apiVersion: v1
+kind: Pod
+metadata:
+  name: judge-proxy
+  namespace: pragent
+  labels:
+    app: judge-proxy
+spec:
+  nodeSelector:
+    kubernetes.io/hostname: kubernets
+  hostNetwork: true
+  dnsPolicy: ClusterFirstWithHostNet
+  restartPolicy: Always
+  containers:
+  - name: p
+    image: python:3.12-alpine
+    command: ["sh","-c","apk add --no-cache ca-certificates >/dev/null && python3 -u /etc/cfg/proxy.py"]
+    volumeMounts:
+    - {name: cfg, mountPath: /etc/cfg}
+    ports:
+    - {containerPort: 8802, hostPort: 8802}
+  volumes:
+  - name: cfg
+    configMap:
+      name: judge-proxy

pilot/README-eval.md

@@ -127,6 +127,51 @@ Coverage is bounded by the dataset, not by the traces: items only exist for PRs
 with a row in `feedback.db`, and a review that posted no comment leaves a trace
 but no row. That is why a run links fewer items than there are traces.
 
+## Evaluators: `eval_judges.py`
+
+Behaviour scores answer "how many, how severe, how much" — computable from data
+already in hand. Two things they cannot answer:
+
+- **Was the finding any good?** Specificity vs. hedge, generic advice vs.
+  fix-it-now advice — the difference between a useful review and one a
+  developer scrolls past.
+- **Did the summary match the findings?** Claiming "no issues" above two
+  criticals, or describing a problem in prose that never became a finding.
+
+These need a judge. `eval_judges.py` registers two `llm_as_judge` evaluators
+against the trace names this project emits (`pr-review`, `opencode-review`)
+and wires a sampling=1 rule per evaluator. Both run on every observation in a
+matching trace; the only observations in those traces are the review itself.
+
+| evaluator | output | what it answers |
+|---|---|---|
+| `finding_actionability` | NUMERIC 0–1 | How specific and fixable is each finding? |
+| `review_self_consistency` | BOOLEAN | Does the summary agree with the findings? |
+
+The judge is a different model from the reviewer (`kimi-k2.7-code` through the
+headroom hub). A model grading its own output agrees with itself for reasons
+that have nothing to do with quality. The judges are also asked only what they
+can answer from the review itself — never whether a finding is correct, since
+that needs the diff the trace does not carry.
+
+### Why the judge goes through `judge-proxy` (port 8802)
+
+The headroom hub in front of local Ollama returns Anthropic-format responses,
+but every `thinking` content block is missing the `signature` field real
+Claude emits. Langfuse's Zod schema requires it; the omission fails the
+evaluator preflight as `Invalid JSON response`. The `judge-proxy` pod sits in
+front of the hub on `100.74.17.70:8802` and patches every thinking block with
+a synthetic signature before forwarding the response. The model is unchanged;
+only the wire shape is fixed.
+
+```bash
+python3 pilot/eval_judges.py --dry-run   # show what would be created
+python3 pilot/eval_judges.py             # create the LLM connection, evaluators, rules
+```
+
+Idempotent: existing evaluators and rules are skipped, not duplicated. The
+connection is upserted on `provider` so re-runs return the same record.
+
 ## Running it
 
 ```bash

pilot/eval_judges.py

@@ -0,0 +1,308 @@
+#!/usr/bin/env python3
+"""pragent pilot — LLM-as-a-judge evaluators for the reviewer.
+
+The deterministic scorers in `eval_scores.py` measure *behaviour*: how many
+findings, how severe, how much they cost. None of them can say whether a
+finding was any good. With no human labels in `feedback.db`, a judge is the
+only thing that can — so these two ask the questions that need no ground truth,
+only the review itself:
+
+  `finding_actionability` — is each finding concrete enough to act on? A
+    reviewer that says "consider improving error handling" at file level is
+    indistinguishable from a useful one by finding count alone. This is the
+    failure mode a cheap model degrades into first.
+
+  `review_self_consistency` — does the summary agree with the findings it
+    posted? Claiming "no issues found" above a list of two criticals, or
+    describing a problem in prose that never became a finding, is a defect the
+    reviewer can commit entirely on its own.
+
+Neither judge is asked whether a finding is *correct*. That needs the diff,
+which these traces do not carry, and a judge asked to rule on correctness from
+a summary alone will confabulate. Accuracy stays an open question until humans
+start labelling — which is what `feedback_scores.py` is there to capture.
+
+**The judge is a different model from the reviewer.** The reviewer runs
+MiniMax-M2.7; the judge runs kimi-k2.7-code through the same headroom hub. A
+model grading its own output agrees with itself for reasons that have nothing
+to do with quality.
+
+Evaluators score *observations*, and their variable mapping reads the
+observation's own input/output — which is why `langfuse_trace` now writes the
+review onto the generation and not just onto the trace.
+
+Usage:
+    LANGFUSE_HOST=... LANGFUSE_PUBLIC_KEY=... LANGFUSE_SECRET_KEY=... \\
+        python3 eval_judges.py --dry-run
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+
+import eval_bootstrap as eb  # noqa: E402
+
+# The headroom hub in front of the local Ollama, plus a small pass-through
+# proxy (`judge-proxy` on 8802) that patches every `thinking` content block
+# to carry the `signature` field Langfuse's Anthropic adapter requires. The
+# underlying model is kimi-k2.7-code through the hub on 8790; the proxy fixes
+# the shape so Mastra's Zod parse stops failing.
+JUDGE_PROVIDER = "headroom-ollama"
+JUDGE_BASE_URL = os.environ.get("PRAGENT_JUDGE_BASE_URL", "http://100.74.17.70:8802")
+JUDGE_API_KEY = os.environ.get("PRAGENT_JUDGE_API_KEY", "ollama")
+JUDGE_MODEL = os.environ.get("PRAGENT_JUDGE_MODEL", "kimi-k2.7-code:cloud")
+
+# The trace names this project emits (`pr-review` on the trace, `opencode-review`
+# on the generation). Filter on `traceName` rather than observation `name` — the
+# observation-rule schema only exposes `traceName` as a stringOptions column, and
+# every observation inside these traces is the review itself, so the narrowness
+# is the same.
+REVIEW_TRACE_NAMES = ["pr-review", "opencode-review"]
+
+
+def _model_config() -> dict:
+    return {"provider": JUDGE_PROVIDER, "model": JUDGE_MODEL}
+
+
+JUDGES = [
+    {
+        "name": "finding_actionability",
+        "prompt": (
+            "You are auditing the output of an automated code reviewer.\n\n"
+            "PR under review:\n{{input}}\n\n"
+            "What the reviewer produced:\n{{output}}\n\n"
+            "Rate how ACTIONABLE the findings are, from 0 to 1. A finding is "
+            "actionable when a developer could act on it without asking a "
+            "follow-up question: it points at a specific location, names a "
+            "concrete problem, and proposes a fix that could be applied.\n\n"
+            "Score 1.0 when every finding is specific and fixable. Score around "
+            "0.5 when findings identify a real area but leave the developer to "
+            "work out what to change. Score near 0.0 when findings are generic "
+            "advice that would apply to almost any pull request.\n\n"
+            "Judge only specificity and actionability. You cannot see the diff, "
+            "so do NOT attempt to judge whether a finding is factually correct, "
+            "and do not penalise a finding for being one you cannot verify.\n\n"
+            "If the reviewer reported no findings at all, return 1.0 and say in "
+            "your reasoning that there was nothing to judge — a silent review is "
+            "measured by finding_rate, not here."
+        ),
+        "outputDefinition": {
+            "dataType": "NUMERIC",
+            "minValue": 0,
+            "maxValue": 1,
+            "reasoning": {
+                "description": (
+                    "Name the least actionable finding and say what it would "
+                    "need in order to be acted on."
+                )
+            },
+            "score": {"description": "0 = generic advice, 1 = every finding is specific and fixable."},
+        },
+    },
+    {
+        "name": "review_self_consistency",
+        "prompt": (
+            "You are auditing the output of an automated code reviewer.\n\n"
+            "PR under review:\n{{input}}\n\n"
+            "What the reviewer produced:\n{{output}}\n\n"
+            "The output contains a prose `summary` and a list of `findings`. "
+            "Decide whether the summary is CONSISTENT with the findings.\n\n"
+            "Inconsistent means, for example: the summary says no issues were "
+            "found while findings are listed; the summary describes a problem "
+            "that never became a finding; the summary characterises the severity "
+            "of the findings in a way the findings themselves contradict; or the "
+            "summary refers to files that appear in no finding and in no part of "
+            "the PR description.\n\n"
+            "A summary that adds context beyond the findings is NOT inconsistent "
+            "as long as nothing in it contradicts them. A review that found "
+            "nothing and says so is consistent.\n\n"
+            "You cannot see the diff. Judge the summary against the findings and "
+            "the PR title only — never against what you imagine the code does."
+        ),
+        "outputDefinition": {
+            "dataType": "BOOLEAN",
+            "reasoning": {
+                "description": "Quote the part of the summary that conflicts with the findings, if any."
+            },
+            "score": {"description": "true = summary agrees with the findings, false = it contradicts them."},
+        },
+    },
+]
+
+# Both judges read the observation's own input/output.
+MAPPING = [
+    {"variable": "input", "source": "input"},
+    {"variable": "output", "source": "output"},
+]
+
+
+# ---------------------------------------------------------------------------
+# LLM connection
+# ---------------------------------------------------------------------------
+
+def ensure_llm_connection() -> dict:
+    """Point the project at the judge model. Upserted on `provider`."""
+    body = {
+        "provider": JUDGE_PROVIDER,
+        "adapter": "anthropic",
+        "baseURL": JUDGE_BASE_URL,
+        "secretKey": JUDGE_API_KEY,
+        "customModels": [JUDGE_MODEL],
+        # The hub serves two local models and none of Anthropic's, so the
+        # default catalogue would be a list of models that all fail on use.
+        "withDefaultModels": False,
+    }
+    st, resp = eb._call("PUT", "/api/public/llm-connections", body)
+    return {"status": st, "ok": st in (200, 201), "provider": JUDGE_PROVIDER,
+            "error": None if st in (200, 201) else resp}
+
+
+# ---------------------------------------------------------------------------
+# Evaluators
+# ---------------------------------------------------------------------------
+
+def existing_evaluators() -> dict[str, str]:
+    """name -> id for evaluators already in the project."""
+    out: dict[str, str] = {}
+    st, body = eb._call("GET", "/api/public/unstable/evaluators?limit=100")
+    if st == 200 and isinstance(body, dict):
+        for ev in body.get("data") or []:
+            out[ev.get("name")] = ev.get("id")
+    return out
+
+
+def ensure_evaluators() -> dict:
+    """Create each judge if no version exists for the name yet.
+
+    POST /evaluators with a name that already exists creates a new version, not
+    a no-op — re-running this script would pile up versions until the page
+    listing them is unreadable. Skip when an evaluator of that name is present.
+    """
+    created, skipped, failed = {}, [], []
+    existing = set(existing_evaluators())
+    for judge in JUDGES:
+        if judge["name"] in existing:
+            skipped.append(judge["name"])
+            continue
+        body = {
+            "type": "llm_as_judge",
+            "name": judge["name"],
+            "prompt": judge["prompt"],
+            "outputDefinition": judge["outputDefinition"],
+            "modelConfig": _model_config(),
+        }
+        st, resp = eb._call("POST", "/api/public/unstable/evaluators", body, timeout=60.0)
+        if st in (200, 201) and isinstance(resp, dict):
+            created[judge["name"]] = resp.get("id")
+        else:
+            failed.append({"name": judge["name"], "status": st, "error": resp})
+    return {"created": created, "skipped": skipped, "failed": failed}
+
+
+# ---------------------------------------------------------------------------
+# Rules — what gets judged, and how often
+# ---------------------------------------------------------------------------
+
+def rule_body(name: str, judge_name: str, sampling: float) -> dict:
+    """POST /evaluation-rules shape for an LLM-as-judge observation rule.
+
+    The judge is referenced by `name`+`scope`, not by id — ids name specific
+    versions, names name the evaluator across versions. Mapping is required at
+    both the rule root (the server validates it there) and inside `evaluator`
+    (the API echoes it back). Filter is on `traceName` because that is the only
+    stringOptions column the observation-rule schema exposes.
+    """
+    return {
+        "name": name,
+        "enabled": True,
+        "target": "observation",
+        "sampling": sampling,
+        "filter": [
+            {"column": "traceName", "operator": "any of",
+             "value": REVIEW_TRACE_NAMES, "type": "stringOptions"},
+        ],
+        "evaluator": {
+            "name": judge_name,
+            "scope": "project",
+            "variableMapping": MAPPING,
+        },
+        "mapping": MAPPING,
+    }
+
+
+def ensure_rules(evaluator_ids: dict[str, str], sampling: float) -> dict:
+    """Idempotent: existing rules with the same name are skipped, not duplicated.
+
+    The API has no `name`-keyed upsert; the convention is to POST once and
+    re-run the script to verify the response. A duplicate POST raises 409.
+    """
+    created, failed, skipped = [], [], []
+    existing = existing_rule_names()
+    for name, eid in evaluator_ids.items():
+        if not eid:
+            continue
+        rule_name = f"{name}-on-reviews"
+        if rule_name in existing:
+            skipped.append(name)
+            continue
+        st, resp = eb._call(
+            "POST", "/api/public/unstable/evaluation-rules",
+            rule_body(rule_name, name, sampling), timeout=60.0,
+        )
+        if st in (200, 201):
+            created.append(name)
+        else:
+            failed.append({"rule": name, "status": st, "error": resp})
+    return {"created": created, "failed": failed, "skipped": skipped}
+
+
+def existing_rule_names() -> set[str]:
+    """Names of observation-target rules already in the project."""
+    out: set[str] = set()
+    st, body = eb._call("GET", "/api/public/unstable/evaluation-rules?limit=100")
+    if st == 200 and isinstance(body, dict):
+        for r in body.get("data") or []:
+            if r.get("target") == "observation":
+                out.add(r.get("name"))
+    return out
+
+
+def main(argv: list[str] | None = None) -> int:
+    ap = argparse.ArgumentParser(description=__doc__)
+    ap.add_argument("--sampling", type=float, default=1.0,
+                    help="fraction of matching observations to judge (default: all)")
+    ap.add_argument("--skip-connection", action="store_true")
+    ap.add_argument("--dry-run", action="store_true")
+    args = ap.parse_args(argv)
+
+    if args.dry_run:
+        print(json.dumps({
+            "would_connect": {"provider": JUDGE_PROVIDER, "baseURL": JUDGE_BASE_URL,
+                              "model": JUDGE_MODEL},
+            "would_create": [j["name"] for j in JUDGES],
+            "existing_evaluators": sorted(existing_evaluators()),
+            "sampling": args.sampling,
+        }, indent=2))
+        return 0
+
+    report = {}
+    if not args.skip_connection:
+        report["llm_connection"] = ensure_llm_connection()
+    report["evaluators"] = ensure_evaluators()
+    ids = dict(report["evaluators"]["created"])
+    # Fall back to whatever is already registered, so a re-run still wires rules.
+    for name, eid in existing_evaluators().items():
+        ids.setdefault(name, eid)
+    report["rules"] = ensure_rules(
+        {j["name"]: ids.get(j["name"]) for j in JUDGES}, args.sampling
+    )
+    print(json.dumps(report, indent=2))
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

pilot/langfuse_trace.py

@@ -280,8 +280,8 @@ def build_batch(
         "timestamp": ts,
         "environment": env,
         "sessionId": f"{repo}#{index}",
-        "input": {"repo": repo, "pr": index, "sha": sha, "title": title},
+        "input": _review_input(repo, index, sha, title),
-        "output": {"summary": summary[:2000], "findings": len(findings or [])},
+        "output": _review_output(summary, findings),
         "metadata": metadata,
         "tags": tags,
     }
@@ -310,6 +310,11 @@ def build_batch(
             "usageDetails": _usage_details(usage),
             "metadata": metadata,
             "level": "DEFAULT",
+            # Repeated from the trace on purpose: an evaluator's variable
+            # mapping reads the *observation's* input/output, so a generation
+            # left blank cannot be judged at all.
+            "input": _review_input(repo, index, sha, title),
+            "output": _review_output(summary, findings),
         }
         if costs:
             gen_body["costDetails"] = costs
@@ -337,6 +342,46 @@ def build_batch(
     return events
 
 
+MAX_JUDGED_FINDINGS = 25
+_FIELD_CAP = 600
+
+
+def _review_input(repo: str, index, sha: str, title: str) -> dict:
+    return {"repo": repo, "pr": index, "sha": sha, "title": title}
+
+
+def _review_output(summary: str, findings) -> dict:
+    """What the reviewer actually said, in a shape an evaluator can read.
+
+    The findings themselves are included, not just their count. A judge given
+    only `{"summary": ..., "findings": 3}` can say nothing about whether those
+    three findings are specific, actionable, or consistent with the summary —
+    which is the whole question worth asking of a reviewer that has no ground
+    truth to check against.
+
+    Capped rather than complete: this rides in every ingestion batch, and a
+    review with 80 findings would push the payload past what is reasonable to
+    store per trace. `finding_count` stays exact so nothing reading the count
+    is misled by the cap.
+    """
+    items = list(findings or [])
+    return {
+        "summary": summary[:2000],
+        "finding_count": len(items),
+        "findings_truncated": len(items) > MAX_JUDGED_FINDINGS,
+        "findings": [
+            {
+                "path": f.get("path"),
+                "line": f.get("line"),
+                "severity": f.get("severity"),
+                "problem": str(f.get("problem") or "")[:_FIELD_CAP],
+                "fix": str(f.get("fix") or "")[:_FIELD_CAP],
+            }
+            for f in items[:MAX_JUDGED_FINDINGS]
+        ],
+    }
+
+
 def _score_events(*, cost_basis: str, **kwargs) -> list[dict]:
     """Deterministic scores for this review, or [] if the scorer is missing.
 

tests/pilot/test_eval_judges.py

@@ -0,0 +1,126 @@
+
+
+"""Tests for the LLM-as-judge evaluator bootstrap."""
+import os
+import sys
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "pilot"))
+
+import eval_judges as ej  # noqa: E402
+
+
+# --- rule_body ------------------------------------------------------------
+
+def test_rule_body_targets_observations():
+    """Trace-level rules wouldn't see observation input/output."""
+    body = ej.rule_body("rule-x", "finding_actionability", 1.0)
+    assert body["target"] == "observation"
+    assert body["enabled"] is True
+
+
+def test_rule_body_filters_on_trace_name():
+    """`name` isn't a stringOptions column; only `traceName` is."""
+    body = ej.rule_body("rule-x", "finding_actionability", 1.0)
+    f = body["filter"][0]
+    assert f["column"] == "traceName"
+    assert f["operator"] == "any of"
+    assert f["type"] == "stringOptions"
+    assert "pr-review" in f["value"]
+
+
+def test_rule_body_references_evaluator_by_name():
+    """Ids are version-specific; rules must name the evaluator across versions."""
+    body = ej.rule_body("rule-x", "finding_actionability", 1.0)
+    assert body["evaluator"]["name"] == "finding_actionability"
+    assert body["evaluator"]["scope"] == "project"
+
+
+def test_rule_body_maps_input_and_output():
+    """Both judges read the observation's own input/output."""
+    body = ej.rule_body("rule-x", "any", 1.0)
+    sources = {m["source"] for m in body["mapping"]}
+    assert sources == {"input", "output"}
+
+
+def test_rule_body_carries_mapping_at_both_levels():
+    """The server validates `mapping` at the rule root and echoes it on the evaluator."""
+    body = ej.rule_body("rule-x", "any", 1.0)
+    assert body["mapping"]
+    assert body["evaluator"]["variableMapping"] == body["mapping"]
+
+
+def test_rule_body_passes_sampling_through():
+    assert ej.rule_body("r", "any", 0.25)["sampling"] == 0.25
+
+
+# --- ensure_evaluators idempotency ---------------------------------------
+
+def test_ensure_evaluators_skips_existing(monkeypatch):
+    seen = []
+
+    def fake_call(method, path, body=None, timeout=20.0):
+        seen.append(path)
+        return 200, {}
+
+    monkeypatch.setattr(ej.eb, "_call", fake_call)
+    monkeypatch.setattr(ej, "existing_evaluators",
+                        lambda: {"finding_actionability": "id-1", "review_self_consistency": "id-2"})
+    res = ej.ensure_evaluators()
+    assert res["created"] == {}
+    assert sorted(res["skipped"]) == ["finding_actionability", "review_self_consistency"]
+    assert res["failed"] == []
+    assert seen == []
+
+
+def test_ensure_evaluators_records_failures(monkeypatch):
+    def fake_call(method, path, body=None, timeout=20.0):
+        return 422, "boom"
+
+    monkeypatch.setattr(ej.eb, "_call", fake_call)
+    monkeypatch.setattr(ej, "existing_evaluators", lambda: {})
+    res = ej.ensure_evaluators()
+    assert res["created"] == {}
+    assert res["failed"][0]["status"] == 422
+
+
+# --- ensure_rules idempotency --------------------------------------------
+
+def test_ensure_rules_skips_existing(monkeypatch):
+    calls = []
+    monkeypatch.setattr(ej.eb, "_call",
+                        lambda *a, **k: calls.append(a) or (200, {}))
+    monkeypatch.setattr(ej, "existing_evaluators",
+                        lambda: {"finding_actionability": "id-1",
+                                 "review_self_consistency": "id-2"})
+    monkeypatch.setattr(ej, "existing_rule_names",
+                        lambda: {"finding_actionability-on-reviews",
+                                 "review_self_consistency-on-reviews"})
+    res = ej.ensure_rules({"finding_actionability": "id-1",
+                           "review_self_consistency": "id-2"}, 1.0)
+    assert res["created"] == []
+    assert sorted(res["skipped"]) == ["finding_actionability", "review_self_consistency"]
+    assert calls == []
+
+
+def test_ensure_rules_creates_when_missing(monkeypatch):
+    calls = []
+    monkeypatch.setattr(ej.eb, "_call",
+                        lambda *a, **k: calls.append(a) or (201, {}))
+    monkeypatch.setattr(ej, "existing_rule_names", lambda: set())
+    res = ej.ensure_rules({"finding_actionability": "id-1"}, 1.0)
+    assert res["created"] == ["finding_actionability"]
+    assert calls[0][0] == "POST"
+    assert calls[0][1] == "/api/public/unstable/evaluation-rules"
+
+
+# --- judge shape ----------------------------------------------------------
+
+def test_judges_have_required_keys():
+    for j in ej.JUDGES:
+        assert j["prompt"]
+        assert j["outputDefinition"]["dataType"] in ("NUMERIC", "BOOLEAN", "CATEGORICAL")
+
+
+def test_default_base_url_points_at_the_thinking_patch_proxy():
+    """`8802` is the judge-proxy that adds a `signature` to thinking blocks."""
+    assert "8802" in ej.JUDGE_BASE_URL