Claude
5d44121b28
Two llm_as_judge evaluators score the review generation directly: a NUMERIC 0-1 on finding actionability, a BOOLEAN on whether the summary agrees with the findings. Both run on every observation whose trace name is pr-review or opencode-review. The judge is kimi-k2.7-code through the headroom hub. Local Ollama returns Anthropic-format responses but the thinking blocks lack the signature field Langfuse Zod schema requires; the evaluator preflight fails as Invalid JSON response. A small judge-proxy pod on 8802 forwards to the hub and patches every thinking block with a synthetic signature before returning. Trace + generation output now includes the findings themselves (capped at 25) rather than just the count, so a judge has something to grade. generation input/output mirrors the trace so an observation-level evaluator can read them. Idempotent: existing evaluators and rules are skipped on re-run, not duplicated. The connection is upserted on provider.
83 lines
3.5 KiB
YAML
83 lines
3.5 KiB
YAML
# Judge-side think-block patcher. Stands between Langfuse evaluators and the
|
|
# headroom-ollama hub (port 8790). Local Ollama does not emit the `signature`
|
|
# field that Langfuse's Anthropic adapter's Zod schema requires on every
|
|
# `thinking` content block — without it, the evaluator preflight fails as
|
|
# "Invalid JSON response". The proxy forwards /v1/* verbatim and adds a dummy
|
|
# signature to each thinking block before returning.
|
|
apiVersion: v1
|
|
kind: ConfigMap
|
|
metadata:
|
|
name: judge-proxy
|
|
namespace: pragent
|
|
data:
|
|
proxy.py: |
|
|
#!/usr/bin/env python3
|
|
"""Judge proxy: forward to headroom-ollama, fix thinking blocks."""
|
|
import json, sys, urllib.request, urllib.error
|
|
from http.server import BaseHTTPRequestHandler, HTTPServer
|
|
from socketserver import ThreadingMixIn
|
|
UPSTREAM = "http://100.74.17.70:8790"
|
|
DUMMY_SIG = "kimi-local-judge-no-signature"
|
|
class H(BaseHTTPRequestHandler):
|
|
def _proxy(self):
|
|
n = int(self.headers.get("Content-Length", 0))
|
|
body = self.rfile.read(n) if n else b""
|
|
h = {k: v for k, v in self.headers.items() if k.lower() not in ("host", "content-length")}
|
|
req = urllib.request.Request(UPSTREAM + self.path, data=body, headers=h, method=self.command)
|
|
try:
|
|
with urllib.request.urlopen(req, timeout=120) as r:
|
|
resp_body = r.read(); status = r.status; rh = dict(r.headers)
|
|
except urllib.error.HTTPError as e:
|
|
resp_body = e.read(); status = e.code; rh = dict(e.headers)
|
|
ct = rh.get("content-type", "")
|
|
if status == 200 and "application/json" in ct and self.path.startswith("/v1/messages"):
|
|
try:
|
|
obj = json.loads(resp_body)
|
|
patched = 0
|
|
for blk in obj.get("content") or []:
|
|
if isinstance(blk, dict) and blk.get("type") == "thinking" and "signature" not in blk:
|
|
blk["signature"] = DUMMY_SIG; patched += 1
|
|
if patched:
|
|
resp_body = json.dumps(obj).encode("utf-8")
|
|
rh["content-length"] = str(len(resp_body))
|
|
print(f"judge-proxy: patched {patched} thinking block(s)", file=sys.stderr, flush=True)
|
|
except Exception as e:
|
|
print(f"judge-proxy: patch failed: {e}", file=sys.stderr, flush=True)
|
|
self.send_response(status)
|
|
for k, v in rh.items():
|
|
if k.lower() not in ("transfer-encoding", "content-length", "connection"):
|
|
self.send_header(k, v)
|
|
self.send_header("Content-Length", str(len(resp_body)))
|
|
self.end_headers(); self.wfile.write(resp_body)
|
|
def do_POST(self): self._proxy()
|
|
def do_GET(self): self._proxy()
|
|
def log_message(self, *a, **k): pass
|
|
class S(ThreadingMixIn, HTTPServer): daemon_threads = True
|
|
S(("0.0.0.0", 8802), H).serve_forever()
|
|
---
|
|
apiVersion: v1
|
|
kind: Pod
|
|
metadata:
|
|
name: judge-proxy
|
|
namespace: pragent
|
|
labels:
|
|
app: judge-proxy
|
|
spec:
|
|
nodeSelector:
|
|
kubernetes.io/hostname: kubernets
|
|
hostNetwork: true
|
|
dnsPolicy: ClusterFirstWithHostNet
|
|
restartPolicy: Always
|
|
containers:
|
|
- name: p
|
|
image: python:3.12-alpine
|
|
command: ["sh","-c","apk add --no-cache ca-certificates >/dev/null && python3 -u /etc/cfg/proxy.py"]
|
|
volumeMounts:
|
|
- {name: cfg, mountPath: /etc/cfg}
|
|
ports:
|
|
- {containerPort: 8802, hostPort: 8802}
|
|
volumes:
|
|
- name: cfg
|
|
configMap:
|
|
name: judge-proxy
|