refactor: split pilot architecture

Remove the obsolete dashboard now that Langfuse is the analytics surface.\nIntroduce focused transport, model, and configuration modules while preserving the ai_review facade, and document the current runtime architecture.

.judge-trigger.txt

@@ -0,0 +1 @@
+# judge trigger 1788203999

README.md

@@ -35,9 +35,9 @@ code never has to leave your network.
 
 ## Status
 
-A **pilot** is live and reviewing real PRs. The full framework (`pragent init`,
-tiering as code, analyzer fan-out, `explain` / `replay`) is designed but not
-built — see [`docs/plans/`](docs/plans/).
+A **pilot** is live and reviewing real PRs. The current runtime architecture is
+documented in [`docs/architecture.md`](docs/architecture.md); older framework
+plans remain in [`docs/plans/`](docs/plans/) as historical design material.
 
 What works today:
 
@@ -51,10 +51,12 @@ What works today:
 - `.pr-review.json` for per-repo focus and house rules (also the opt-in flag)
 - token-usage reporting on every review, measured from opencode `step_finish`
   events
+- Langfuse traces, equivalent-cost reporting, evaluation scores, and feedback
+  harvesting
 - containment against hostile PR content (see [Security](#security))
 
 Not yet: status checks, fail-close, attention tiering enforced in code (it is
-currently a skill the agent follows), multi-model routing.
+currently a skill the agent follows), multi-model routing, and a CLI framework.
 
 ## How a review runs
 
@@ -94,6 +96,10 @@ path is in [`pilot/README.md`](pilot/README.md).
 The model endpoint is supplied at runtime via `PRAGENT_MODEL_BASE_URL`; the
 committed `opencode.json` carries a placeholder.
 
+Per-review token spend, latency, equivalent cost, and evaluation scores are
+shipped to a self-hosted Langfuse: [`pilot/README-langfuse.md`](pilot/README-langfuse.md).
+Emission is a silent no-op unless `LANGFUSE_HOST` and the key pair are set.
+
 ## Extending it
 
 The review "factory" is [`.opencode/`](.opencode/README.md) — agent definitions
@@ -170,7 +176,7 @@ python3 pilot/cost_model.py --help      # other mixes, volumes, models
 ## Development
 
 ```bash
-python3 -m pytest tests -q    # 137 tests, stdlib only, no network
+python3 -m pytest tests -q    # stdlib-only tests, no network
 ```
 
 The pilot is stdlib-only Python by design — it runs from a bare `python:slim`
@@ -180,3 +186,5 @@ review time.
 ## License
 
 Not yet chosen. Until one is added, no reuse rights are granted.
+
+_pilot eval judges test 1788201461_

docs/architecture.md

@@ -0,0 +1,68 @@
+# pragent current architecture
+
+Status: pilot implementation, September 2026.
+
+## System shape
+
+```text
+Gitea pull_request webhook
+          │ signed HTTP
+          ▼
+webhook_server ── trusted base config ──► review_config
+          │ bounded worker
+          ▼
+review_pr facade/orchestrator
+   ├── gitea_client       fetch diff, reviews, config; publish review
+   ├── diff_compress      reduce prompt context
+   ├── opencode_review     isolated checkout + agent execution
+   │       └── model / repo factory (.opencode)
+   ├── review parsing      normalize findings + validate anchors
+   ├── feedback            persist reactions and derive scores
+   └── langfuse_trace      usage, cost, evaluation telemetry
+```
+
+## Seams and responsibilities
+
+The external seam is `ai_review.review_pr(...)`: one call represents one review
+attempt and returns success/skip status. The module is retained as a facade for
+the CI and webhook callers that already import it.
+
+The internal seams are deliberately narrower:
+
+- `review_config.repo_enabled(get, ...)` owns the security-sensitive opt-in
+  decision. It receives a transport function, so malformed configuration and
+  failure behavior are deterministic in tests.
+- `gitea_client.request()` and `GiteaClient` own HTTP authentication, JSON
+  request encoding, timeout, and Gitea URL construction.
+- `model_client.complete()` owns the legacy Anthropic-compatible request shape.
+  `opencode_review` is the preferred agent adapter and keeps Gitea I/O out of
+  the autonomous process.
+- `diff_compress`, finding parsing, config filtering, and rendering remain
+  pure transformations. Their callers do not need to know how model or Gitea
+  transport works.
+- `langfuse_trace` is an optional sink. It is fail-open and cannot change the
+  review result.
+
+## Trust model
+
+The review config is read from the PR base branch, never the PR head. The agent
+checkout is treated as hostile: instruction files are removed, credentials are
+not inherited, and the agent only returns text to the Python publisher. Python
+validates finding paths and post-change line anchors before sending comments.
+
+## Observability
+
+Langfuse is the operational analytics surface. A trace groups runs by
+`owner/repo#PR`; generations carry usage and cost basis; evaluation scores and
+human-feedback scores are attached later. The former SQLite-backed dashboard
+was removed. SQLite remains only as the feedback/evaluation ingestion store.
+
+## Removed surface
+
+The dashboard server, dashboard data module, dashboard tests, dashboard README,
+and dashboard Kubernetes manifest are intentionally gone. Operators use the
+Langfuse UI for review trends and cost analysis, and Gitea for review details
+and configuration changes.
+
+Historical design/implementation plans under `docs/plans/` describe the
+earlier TypeScript framework proposal and are not the runtime architecture.

k8s/pragent-dashboard.yaml

@@ -1,118 +0,0 @@
-# pragent pilot — central dashboard service.
-#
-# Read-only overview + per-repo / per-PR drilldown over the same SQLite
-# feedback DB the webhook writes. Also mutates `.pr-review.json` on covered
-# repos via the Gitea contents API (Tasks C+D in pilot/dashboard.py). Same
-# image as the webhook (`pragent-webhook:optin`) — all pilot modules are
-# baked in at /app/pilot/.
-#
-# Routes: GET /  (overview), GET /r/<o>/<n>  (repo), GET /r/<o>/<n>/<i>  (PR),
-# GET /r/<o>/<n>/<i>/raw  (PR markdown raw), GET /login, GET /static/style.css,
-# POST /login, POST /r/<o>/<n>/edit.
-#
-# Auth: PRAGENT_DASHBOARD_TOKEN in the pragent-webhook Secret, cookie
-# `pragent_dash=<token>`, single-user. Empty / unset = no auth (tailnet-only).
-#
-# NodePort 30082 — only reachable on the Tailscale / LAN side of kubernets
-# (100.74.17.70 / 192.168.1.80) until/unconfigured. Mirrors pragent-webhook.yaml
-# in every other respect (uid 10001, nodeSelector, /data PVC).
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: pragent-dashboard
-  namespace: pragent
-  labels:
-    app: pragent-dashboard
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      app: pragent-dashboard
-  template:
-    metadata:
-      labels:
-        app: pragent-dashboard
-    spec:
-      # Same node as the webhook — holds the headroom proxy + the /data PVC.
-      nodeSelector:
-        kubernetes.io/hostname: kubernets
-      # Dashboard is read-only over /data and only mutates Gitea (not local
-      # files), so unprivileged is fine. fsGroup matches the image's USER
-      # directive (10001) so the RO mount is readable.
-      securityContext:
-        runAsNonRoot: true
-        runAsUser: 10001
-        runAsGroup: 10001
-        fsGroup: 10001
-      containers:
-        - name: dashboard
-          image: pragent-webhook:optin
-          imagePullPolicy: Never
-          workingDir: /app
-          command: ["python3", "-m", "pilot.dashboard"]
-          ports:
-            - name: http
-              containerPort: 8081
-          env:
-            - name: PRAGENT_FEEDBACK_DB
-              value: /data/feedback.db
-            - name: PRAGENT_GITEA_API
-              value: http://gitea-http.gitea.svc.cluster.local:3000
-            # Dashboard reads DASHBOARD_PORT (not PORT) — verified in
-            # pilot/dashboard.py:51. Default 8081 if unset.
-            - name: DASHBOARD_PORT
-              value: "8081"
-            # Used by /r/<o>/<n>/edit to PUT updated JSON to the repo's
-            # contents API. Reuses the same bot token the webhook uses.
-            - name: PRAGENT_BOT_TOKEN
-              valueFrom:
-                secretKeyRef:
-                  name: pragent-webhook
-                  key: PRAGENT_BOT_TOKEN
-            # Auth cookie value. Add to the pragent-webhook Secret with:
-            #   kubectl patch secret pragent-webhook -n pragent --type=json \
-            #     -p='[{"op":"add","path":"/data/PRAGENT_DASHBOARD_TOKEN","value":"<base64>"}]'
-            - name: PRAGENT_DASHBOARD_TOKEN
-              valueFrom:
-                secretKeyRef:
-                  name: pragent-webhook
-                  key: PRAGENT_DASHBOARD_TOKEN
-          # /data is read-only — the dashboard doesn't write the SQLite file;
-          # .pr-review.json mutations go through the Gitea contents API, not
-          # local fs. RO avoids any chance of two pods racing the same RWO PVC.
-          volumeMounts:
-            - name: feedback-data
-              mountPath: /data
-              readOnly: true
-          # No /health route in dashboard.py (returns 404 on unknown paths).
-          # Probes omitted intentionally — see pilot/dashboard.py:687-721.
-          # Resources: dashboard is read-heavy + tiny writes. /data RO + no
-          # subprocess fan-out (no opencode) keeps footprint small.
-          resources:
-            requests:
-              cpu: 100m
-              memory: 256Mi
-            limits:
-              cpu: 500m
-              memory: 512Mi
-      volumes:
-        - name: feedback-data
-          persistentVolumeClaim:
-            claimName: pragent-feedback-data
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: pragent-dashboard
-  namespace: pragent
-spec:
-  selector:
-    app: pragent-dashboard
-  ports:
-    - name: http
-      port: 80
-      targetPort: http
-      nodePort: 31540
-  type: NodePort
-  # 31540 — auto-allocated at first apply (30082 was already taken by
-  # habitsnow/habitsnow-proxy). Tailscale / LAN only until a Caddy route is set.

pilot/README-dashboard.md

@@ -1,269 +0,0 @@
-# pragent pilot — central dashboard service
-
-A read-only overview + per-repo / per-PR drilldown over the same SQLite
-feedback DB the webhook writes, plus a small form to mutate `.pr-review.json`
-on a covered repo via the Gitea contents API. Companion to the
-[webhook service](README-webhook.md); reuses the webhook image
-(`pragent-webhook:dashboard`) — the pilot modules are baked into `/app/pilot/`,
-and the dashboard is just `python3 -m pilot.dashboard`.
-
-## Architecture
-
-```
-Browser
-   │
-   ▼
-Caddy (TLS, wildcard cert via Cloudflare DNS-01)
-   │   https://pragent-dashboard.marcospaulo.dev.br → 100.74.17.70:31541
-   ▼
-Service oauth2-proxy-dashboard.pragent.svc.cluster.local  (NodePort 31541, ns pragent)
-   │
-   │   oauth2-proxy fronts the dashboard, enforces Logto SSO + email allowlist
-   │   sets X-Forwarded-User / X-Forwarded-Email on accepted requests
-   ▼
-Service pragent-dashboard.pragent.svc.cluster.local  (ClusterIP, ns pragent)
-   │
-   ▼
-pragent-dashboard pod  (uid 10001, /data RO, no subprocess fan-out)
-   │
-   ├── read /data/feedback.db           (PVC pragent-feedback-data, RO)
-   ├── GET  .../repos/{o}/{r}/...        (Gitea contents API, bot token)
-   └── PUT  .../repos/{o}/{r}/contents/.pr-review.json
-            (edit form submit; Gitea commits a new sha)
-```
-
-Fail-soft. Nothing is ever written to local disk by the dashboard — the
-SQLite file is read-only and `.pr-review.json` mutations go through Gitea's
-contents API so the commit history records who changed what.
-
-The dashboard `Service` is **ClusterIP** — only oauth2-proxy can reach it.
-Public access is gated by Caddy (TLS termination) → oauth2-proxy (Logto SSO
-+ allowlist) → dashboard.
-
-## What it does
-
-- **Overview** (`GET /`): summary stats across all onboarded repos — total
-  reviews, distinct PRs, finding counts by severity, false-positive /
-  accepted-pattern scores (see "Feedback loop" in README-webhook.md), and a
-  sparkline of review activity.
-- **Repo drilldown** (`GET /r/<owner>/<name>`): per-repo PRs with their
-  last-review status, finding counts, and links to PR-level drilldowns.
-- **PR drilldown** (`GET /r/<owner>/<name>/<index>`): the bot's review(s)
-  on that PR, inline findings, and reaction / resolved status harvested
-  by `feedback_harvest.py`.
-- **Raw review** (`GET /r/<owner>/<name>/<index>/raw`): the markdown body
-  of the most recent review, for copy-paste / diff-with-prose workflows.
-- **Edit form** (`POST /r/<owner>/<name>/edit`): a small HTML page that
-  loads the current `.pr-review.json` from the repo's default branch and
-  lets the operator edit the JSON (validated, then PUT to Gitea contents
-  API). This is how repo-local `focus` / `instructions` /
-  `reviewers` / `severity_threshold` get tuned per-repo after seeing
-  the feedback roll-up.
-
-All routes return HTML (or plain text for `/raw`) with the same stylesheet
-(`/static/style.css`).
-
-## Routes
-
-| method | path                              | auth | description                                  |
-|--------|-----------------------------------|------|----------------------------------------------|
-| GET    | `/`                               | yes  | Overview                                     |
-| GET    | `/static/style.css`               | no   | Stylesheet                                   |
-| GET    | `/r/<owner>/<name>`               | yes  | Repo drilldown                               |
-| GET    | `/r/<owner>/<name>/<index>`       | yes  | PR drilldown                                 |
-| GET    | `/r/<owner>/<name>/<index>/raw`   | yes  | Most recent review body as markdown          |
-| POST   | `/r/<owner>/<name>/edit`          | yes  | Edit `.pr-review.json` on the default branch |
-
-Auth is enforced by oauth2-proxy upstream; the dashboard itself only
-checks the `X-Forwarded-User` header that oauth2-proxy sets after a
-successful Logto login + email allowlist match.
-
-There is no `/health` route — don't add one to the k8s probes without
-updating `pilot/dashboard.py` (the handler returns 404 on unknown paths,
-so a probe would loop forever).
-
-## Mutations flow through Gitea, not local fs
-
-The edit endpoint reads the current `.pr-review.json` from
-`GET /repos/{o}/{r}/contents/.pr-review.json?ref=<default-branch>`, lets
-the operator edit it in a form (validated as JSON, length-capped per
-field, no schema migration), and PUTs the new content back via the
-contents API with a commit message like
-`pragent dashboard: update .pr-review.json`. Every edit is a real Gitea
-commit on the default branch, attributable to `pragent-bot`, and the
-next webhook fire picks up the new config — no Pod restart, no image
-rebuild, no pod-level state.
-
-The `/data` mount is **read-only** (see the `readOnly: true` on the
-volumeMount in `~/k8s/pragent-dashboard.yaml`): the dashboard never
-writes the SQLite file, only the webhook + the daily cronjob do, and
-keeping it RO means a buggy deploy can't corrupt the harvested feedback.
-
-## Auth (Logto SSO via oauth2-proxy)
-
-Authentication is delegated to oauth2-proxy, which fronts the dashboard
-in-cluster. The dashboard never sees a cookie or a token — it only
-inspects `X-Forwarded-User` (set by oauth2-proxy after a successful
-Logto login + email allowlist match). Missing header → 401 with
-`WWW-Authenticate: Basic realm="pragent-dashboard"`, which lets
-oauth2-proxy intercept and bounce the browser to Logto.
-
-Email allowlist lives in the ConfigMap `oauth2-proxy-dashboard-emails`
-in namespace `pragent`:
-
-```yaml
-data:
-  authenticated-emails: |
-    marcos.paulodasilva.mp@gmail.com
-    thiago@marcospaulo.dev.br
-```
-
-Edit the ConfigMap to add/remove users; oauth2-proxy hot-reloads the
-file (it logs `watching ... for updates`), no restart needed. This is
-the same isolation pattern as the minecraft-sso / code-server
-allowlists — see `~/.claude/memory/minecraft-sso.md`.
-
-The Logto app is `pragent-dashboard` (tenant `default`, type
-`Traditional`), created by direct INSERT into Logto Postgres mirroring
-the proven `minecraft-sso` pattern. Credentials live in
-`~/k8s/oauth2-proxy-dashboard-secret.yaml` (mode 600, NOT in git).
-
-Public URL: **https://pragent-dashboard.marcospaulo.dev.br** (Caddy
-TLS termination via wildcard cert → Tailscale → NodePort 31541 →
-oauth2-proxy → dashboard ClusterIP).
-
-### Emergency bypass (cookie)
-
-If Logto goes down and you need to access the dashboard before the
-oauth2-proxy restart dance (see `~/.claude/memory/logto-fix.md`),
-`pilot/dashboard.py` can be patched to accept a fallback cookie by
-re-adding the `PRAGENT_DASHBOARD_TOKEN` env path — the route gate is
-isolated in `_is_authed` and the logic is straightforward. The current
-commit intentionally has no bypass because Logto SSO is the single
-source of truth for "who can touch `.pr-review.json`".
-
-## Deploy
-
-The dashboard shares the webhook image, so there's nothing to rebuild
-beyond what the webhook already does. After editing `pilot/dashboard.py`
-or `pilot/dashboard_data.py`, redo the webhook image rebuild + containerd
-import (see `README-webhook.md` § "K8s deployment") and roll both
-deployments.
-
-```bash
-K="microk8s kubectl"
-
-# 1. (one-time) create the Logto app + cookie secret + oauth2-proxy
-#    See ~/.claude/memory/minecraft-sso.md for the SQL INSERT recipe
-#    and ~/k8s/oauth2-proxy-dashboard*.yaml for the manifests.
-
-# 2. apply all pragent-dashboard manifests (dashboard + oauth2-proxy)
-$K apply -f ~/k8s/oauth2-proxy-dashboard.yaml
-$K apply -f ~/k8s/pragent-dashboard.yaml
-
-# 3. roll on image / code changes
-$K -n pragent rollout restart deploy/pragent-dashboard
-$K -n pragent rollout status  deploy/pragent-dashboard --timeout=120s
-$K -n pragent logs -f deploy/pragent-dashboard
-```
-
-K8s manifests:
-
-- `~/k8s/pragent-dashboard.yaml` — Deployment + ClusterIP Service.
-  - `image: pragent-webhook:dashboard` + `imagePullPolicy: Never` —
-    local containerd only, same image as the webhook.
-  - `nodeSelector: kubernetes.io/hostname: kubernets` — pinned to the
-    node holding the `/data` PVC.
-  - `securityContext: runAsNonRoot: true, runAsUser: 10001, runAsGroup:
-    10001, fsGroup: 10001` — matches the image's USER directive;
-    fsGroup makes the RO hostpath volume readable.
-  - `volumeMounts.feedback-data.readOnly: true` — dashboard is
-    read-only over `/data`; mutations go through Gitea, not local fs.
-  - No `readinessProbe` / `livenessProbe` — the dashboard has no
-    `/health` route. If you add one to `pilot/dashboard.py`, add a
-    probe here too.
-  - `resources.requests: {cpu: 100m, memory: 256Mi}` /
-    `limits: {cpu: 500m, memory: 512Mi}` — read-heavy + tiny writes,
-    no opencode subprocess fan-out, much smaller than the webhook.
-  - `Service.type: ClusterIP` — only oauth2-proxy can reach it.
-
-- `~/k8s/oauth2-proxy-dashboard.yaml` — Deployment + ConfigMap +
-  NodePort Service (`oauth2-proxy-dashboard`, NodePort 31541,
-  namespace `pragent`). Same shape as the code-server /
-  minecraft-sso oauth2-proxy. NodePort 31541 was chosen because
-  31540 was the old dashboard NodePort and the 30096..30969 media
-  range + 30350-30351 (other oauth2-proxy NodePorts) were taken.
-
-- `~/k8s/oauth2-proxy-dashboard-secret.yaml` — client-id /
-  client-secret / cookie-secret (mode 600, NOT in git).
-
-## Smoke test
-
-```bash
-# 1. anonymous request → 302 redirect to Logto
-curl -I https://pragent-dashboard.marcospaulo.dev.br/
-
-# 2. pod logs
-microk8s kubectl logs -n pragent -l app=oauth2-proxy-dashboard --tail=50
-microk8s kubectl logs -n pragent -l app=pragent-dashboard --tail=50
-
-# 3. in-cluster direct probe (should 401 without X-Forwarded-User)
-microk8s kubectl port-forward -n pragent svc/pragent-dashboard 8181:80 &
-sleep 2
-curl -I http://localhost:8181/                       # expect 401 + WWW-Authenticate: Basic
-curl -I -H "X-Forwarded-User: marcos@example.com" http://localhost:8181/  # expect 200
-kill %1
-```
-
-The HTML returned with a valid `X-Forwarded-User` should contain a
-`<title>` (whatever the dashboard renders) and **never** `Traceback` or
-any Python exception output. A 401 on the unauthenticated GET is the
-expected behaviour — oauth2-proxy catches it and redirects to Logto.
-
-## Threat model / security notes
-
-- **Behind Logto SSO.** Anonymous traffic gets 302 → Logto. Allowed
-  emails (marcos, thiago) reach the dashboard after Logto login; all
-  others see oauth2-proxy's "not authorized" page. Adding a user is a
-  one-line ConfigMap edit; oauth2-proxy hot-reloads the allowlist.
-- **`PRAGENT_BOT_TOKEN` is Gitea Write scoped** to onboarded repos, so
-  a successful auth bypass on the dashboard is Gitea repo write access,
-  not just read. oauth2-proxy's email allowlist is the only
-  authentication factor — there is no second factor. If this becomes a
-  concern, swap oauth2-proxy for an IdP that supports TOTP/WebAuthn
-  and the dashboard needs no further changes (it just reads the
-  forwarded headers).
-- **CSRF on the edit form.** Per-process random secret embedded as a
-  hidden input + double-submit via the `X-Forwarded-User` context. An
-  attacker would need to (a) steal the user's Logto session cookie
-  from oauth2-proxy and (b) read the rendered HTML to harvest the
-  CSRF token. Both have to happen in the same browser.
-- **Read-only `/data` mount.** The dashboard can't corrupt the
-  harvested SQLite file even if it's compromised. The webhook and the
-  daily cronjob are the only writers.
-- **ClusterIP dashboard Service.** Even if a malicious actor discovered
-  the dashboard's container port, they cannot reach it from outside the
-  cluster — only oauth2-proxy can. NetworkPolicy is the cluster
-  default deny.
-- **`uid 10001` + `runAsNonRoot: true`.** No host-level escalation if
-  the dashboard is popped — it has no caps, no `/proc` mounts.
-- **No author-controlled input is `eval`-ed.** The edit form parses the
-  JSON, validates types / lengths, and re-serialises before the Gitea
-  PUT. The review-side hostile-input concerns from `README-webhook.md`
-  § "Threat model" do **not** apply to the dashboard — the dashboard
-  is a read-mostly viewer over already-harvested, already-posted data.
-
-## Known limitations (pilot)
-
-- Logto SSO is the only auth factor — no per-user sessions, no CSRF
-  token tied to a per-user identity (the per-process CSRF secret is
-  global). Adequate for a single-operator dashboard; not adequate for
-  multi-tenant.
-- No `/health` route — if the dashboard process wedges on a Gitea hang,
-  k8s won't restart it. Add a `/health` route to `pilot/dashboard.py`
-  + a probe here before relying on this in production.
-- The overview is a single-process render over a SQLite file that the
-  daily cronjob also writes. A long Gitea hang during a page render can
-  stall the dashboard until the client request times out (30 s). The
-  underlying SQLite reader is read-only and concurrent-safe, so no
-  data corruption — just a slow page.

pilot/README-eval.md

@@ -0,0 +1,123 @@
+# Evaluation — scorers, ground truth, and the dataset
+
+Langfuse already receives one trace per review (`README-langfuse.md`). This is
+the layer on top: numbers attached to those traces that say how the reviewer
+*behaved*, and the beginnings of a ground-truth signal that says whether it was
+*right*.
+
+Those two things are deliberately kept apart, because only one of them exists
+yet.
+
+## What could and could not be built
+
+`feedback.db` has recorded 113 reviews across 4 repos. It has recorded **zero**
+reactions, zero thread resolutions and zero replies. The harvester, the schema
+and the daily analyzer are all working; nobody has ever reacted to a bot
+comment.
+
+That rules out an accuracy metric today. Correctness needs labels, and a
+judge scored against no labels is theatre. So the scorers here measure
+behaviour, which is computable from data already in hand, and a separate
+bridge exists to turn human reactions into scores the moment any arrive.
+
+## The five behavioural scores
+
+Emitted with every review by `eval_scores.py`, folded into the same ingestion
+batch as the trace so they cost no extra request.
+
+| score | type | what a change in it means |
+|---|---|---|
+| `finding_rate` | NUMERIC | Findings posted. 0 is the restraint case — good on clean code, a failure when the run degraded. Only the rate over time separates those. |
+| `severity_info_ratio` | NUMERIC 0–1 | Share of findings the model rated `info`/`trivial`. Rising = the model is hedging rather than committing. `None` when the review was silent: a ratio over an empty set is undefined, and charting it as 0 would read as perfect calibration. |
+| `severity_max` | CATEGORICAL | Highest severity surfaced, `none` when silent. Categorical because "did this ever surface something serious" is the real question, and a mean of severity ranks answers nothing. |
+| `dropped_findings` | NUMERIC | Findings the model emitted that the parser rejected for an unusable `path`/`line`. This is the only score here that measures the model's raw output. |
+| `cost_per_finding` | NUMERIC | Equivalent USD per finding. A cheaper model that finds nothing is not cheaper. |
+
+### Why `dropped_findings` needed a change to the parser
+
+`parse_findings` and `parse_review_output` discard any finding with a missing or
+unusable location. That happens silently, so a model emitting ten findings at
+invalid locations was indistinguishable from a model that found nothing — both
+produce an empty list. `ai_review.last_parse_dropped()` exposes the delta,
+recorded at parse time.
+
+It must be read at parse time specifically: by the time findings reach
+`_emit_langfuse`, `apply_repo_config` has already filtered them by
+`severity_threshold` and `max_findings`, and those drops are the config working
+as intended, not the model misbehaving.
+
+## Ground truth: `feedback_scores.py`
+
+Turns `feedback.db` into two session-level scores, keyed on `"{repo}#{pr}"`
+(which is what `langfuse_trace` already sets as `sessionId`).
+
+| score | meaning |
+|---|---|
+| `review_engagement` | Share of a PR's findings that drew any human reaction, resolution or reply. **Watch this first** — every quality number is vapour until it moves off 0. |
+| `review_acceptance` | Net verdict over engaged findings, −1 to +1. Absent, not 0, when nothing was engaged: zero would claim humans judged the review neutral, when the truth is nobody looked. |
+
+Session-level rather than trace-level because feedback arrives days later
+against a PR, and nothing in `feedback.db` records which re-run of the reviewer
+produced which comment. The session is both the available join and the honest
+granularity.
+
+Score ids are `uuid5(namespace, repo#pr#name)`, so the daily backfill updates
+rather than duplicates.
+
+## The dataset
+
+`pragent-reviews`, one item per PR the reviewer has run on, seeded by
+`eval_bootstrap.py` from `feedback.db`.
+
+`expectedOutput` is **the reviewer's own prior output**, not human-verified
+truth — every item carries `metadata.labelled_by_human: false`. Read it as a
+regression baseline: re-run a candidate model over these PRs and the diff
+against this column is the behaviour change. Promoting an item to real ground
+truth means a human editing it in the dataset view after re-reading the PR.
+
+## Running it
+
+```bash
+# once per project: score configs + dataset (+ score historical traces)
+python3 pilot/eval_bootstrap.py --db /data/feedback.db --backfill-traces
+
+# ship feedback verdicts (runs daily from the feedback CronJob)
+python3 pilot/feedback_scores.py --db /data/feedback.db
+```
+
+Both need `LANGFUSE_HOST`, `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`. In
+cluster they come from the `pragent-langfuse` Secret and point at the ClusterIP
+— never the NodePort, whose oauth2-proxy 302s ingestion to Logto and drops it.
+
+## Gotcha: HTTP 207 is not success
+
+The ingestion endpoint answers `207 Multi-Status` when *some* events failed, so
+a batch where **every** event was rejected still returns 207. An early version
+of these scorers omitted the required per-event `timestamp` and silently
+ingested nothing while reporting success. `langfuse_trace._warn_on_rejected_events`
+now logs the per-event errors under `LANGFUSE_DEBUG=1`. If scores are missing,
+check that before anything else.
+
+## What the first run showed
+
+Backfilled over 42 existing traces and 13 PRs:
+
+```
+cost_per_finding     n=42  mean=0.3133  min=0.0880  max=0.9042
+finding_rate         n=42  mean=0.4762  min=0.0000  max=4.0000
+severity_info_ratio  n=14  mean=0.0000
+review_engagement    n=14  mean=0.0000
+severity_max         {none: 28, medium: 11, high: 1, critical: 2}
+```
+
+Two things worth keeping:
+
+- **The reviewer is not info-heavy.** `feedback.db` shows 61 of 62 findings at
+  `INFO`, which looked like a badly calibrated model. It is not: `severity_max`
+  reads `medium`/`high`/`critical` on every trace that found anything, and
+  `severity_info_ratio` is flat 0. The `INFO` in the DB comes from
+  `feedback_harvest._parse_severity`, which defaults to `INFO` when its regex
+  misses the severity badge in the rendered comment. The DB severity is a
+  re-parse artifact; the score reads the model's structured output directly.
+- **28 of 42 reviews found nothing** (67%), and **engagement is flat zero**. The
+  first is not yet interpretable without the second.

pilot/README-langfuse.md

@@ -0,0 +1,122 @@
+# pragent → Langfuse
+
+Every review the pilot runs ships one **trace** to a self-hosted Langfuse. The
+review body already prints a usage table, but that table lives and dies inside
+one Gitea PR. Langfuse is where the same numbers become a trend: tokens per
+review, latency per model, equivalent cost per repo, and how those move when
+the model or the tiering changes.
+
+## The ollama / claude split
+
+Both paths route through the same headroom proxy, so the provider prefix does
+not distinguish them — `headroom/claude-sonnet-5` is Claude spend,
+`headroom/glm-5.2:cloud` is not. The split is keyed off the **bare model name**
+and lands on the trace's `environment`:
+
+| resolved model              | environment |
+| --------------------------- | ----------- |
+| `headroom/claude-sonnet-5`  | `claude`    |
+| `claude-opus-5`             | `claude`    |
+| `headroom/glm-5.2:cloud`    | `ollama`    |
+| `headroom/MiniMax-M2.7`     | `ollama`    |
+| `vllm-qwen38/qwen3.8-27b`   | `ollama`    |
+
+Langfuse takes an environment selector on every view, filter and cost
+breakdown, so the two spend stories stay separate inside one project — one key
+pair to rotate instead of two. Tags carry the finer cut:
+`provider:headroom`, `model:<bare>`, `engine:opencode`, `repo:<owner/name>`,
+`lens:<id>` per fan-out lens.
+
+To split into two *projects* later, point `LANGFUSE_PUBLIC_KEY` /
+`LANGFUSE_SECRET_KEY` at the second project on whichever deployment runs the
+Claude path. Nothing in the code needs to change.
+
+## What a trace carries
+
+- **trace** `pr-review` — `sessionId` = `owner/repo#index`, so every push to one
+  PR groups together. Input is the PR identity; output is the summary + finding
+  count; metadata carries steps, duration, severity counts and the provider's
+  own reported cost.
+- **generation** `opencode-review` — `model`, `usageDetails`, `costDetails`.
+
+`usageDetails.input` is the **uncached** input. opencode reports `cache_read`
+*inside* `input`, and Langfuse sums the keys it is given, so passing both
+verbatim would bill the resent prefix twice.
+
+### How cost is priced
+
+Langfuse has no price table of its own here — we compute the number and ship it
+as `costDetails.total`, so what Langfuse charts is exactly what
+`cost_model.PRICES` says.
+
+A model that genuinely bills (`claude-*`, `gpt-*`, `gemini-*`, `grok-*`) is
+priced **as itself**: basis `actual`.
+
+A model that costs nothing through the headroom proxy is priced against a
+**comparison target** instead: basis `equivalent:<target>`. That covers the
+models absent from `PRICES` (`MiniMax-M2.7` — which is what the webhook
+actually runs — and `glm-5.2:cloud`) as well as entries priced at all zeros
+(the self-hosted vLLM `qwen3.8-27b`). Without this Langfuse would show a
+flat $0.00 line, since the pilot's own path is free.
+
+The target follows the same precedence as the review body, so the PR and
+Langfuse never disagree:
+
+    .pr-review.json:cost_target  >  PRAGENT_PRICE_TARGET  >  claude-sonnet-5
+
+An equivalent cost is a hypothetical, not money spent, so every trace is tagged
+`cost:actual` or `cost:equivalent:<target>` and the generation metadata carries
+`cost_basis`. Filter on it before reading any cost chart as spend.
+
+If the comparison target itself is unknown, the trace ships usage with **no**
+cost block — better no number than a wrong one.
+
+Anthropic prices in `cost_model.PRICES` were fetched 2026-08-18; re-check them
+before quoting anything externally.
+
+## Configuration
+
+| env                   | meaning                                                   |
+| --------------------- | --------------------------------------------------------- |
+| `LANGFUSE_HOST`       | `http://langfuse-web.langfuse.svc.cluster.local:3000`      |
+| `LANGFUSE_PUBLIC_KEY` | `pk-lf-…`                                                  |
+| `LANGFUSE_SECRET_KEY` | `sk-lf-…`                                                  |
+| `LANGFUSE_TIMEOUT`    | seconds, default `5`                                       |
+| `LANGFUSE_DEBUG`      | `1` to log ingestion failures to stderr                    |
+
+Unset host or either key ⇒ emission is a silent no-op. That is the default, so
+a checkout without Langfuse behaves exactly as before.
+
+## Fail-open
+
+`langfuse_trace` is stdlib-only (`urllib`) and every entry point swallows its
+own exceptions; `_emit_langfuse` in `ai_review.py` wraps even the import. A
+Langfuse outage cannot fail, delay past `LANGFUSE_TIMEOUT`, or alter a review.
+
+Both token-spending exit paths emit — the normal post **and** the salvage path
+where the agent produced unparseable output. That run cost the same as a clean
+one, and is precisely the failure worth trending.
+
+## Deployment
+
+Cluster side lives outside this repo: `~/k8s/langfuse.yaml` (ClickHouse +
+web + worker, reusing the gitea postgres, gitea valkey and minio),
+`~/k8s/oauth2-proxy-langfuse.yaml` (the Logto gate), and
+`~/k8s/langfuse-setup.sh`, which provisions the database, the bucket, the
+secrets, and wires `pragent-webhook` with the three env vars above.
+
+The UI is at **https://langfuse.marcospaulo.dev.br**:
+
+    browser -> Caddy (VPS, TLS, DNS-01) -> tailscale
+            -> 100.74.17.70:30361 -> oauth2-proxy (Logto, email allowlist)
+            -> langfuse-web (ClusterIP)
+
+Logto sits at *both* layers off one app (`langfuse`, two redirect URIs): the
+proxy gates the domain, and Langfuse's own NextAuth uses the same Logto as a
+custom OIDC provider, so the inner login is a silent redirect rather than a
+second password.
+
+pragent does **not** go through any of that. It posts to
+`langfuse-web.langfuse.svc.cluster.local:3000` from inside the cluster, on
+API-key auth — putting ingestion behind an interactive SSO gate would break it
+on the first review.

pilot/README.md

@@ -1,100 +1,76 @@
-# pragent pilot — AI Review bot
+# pragent pilot
 
-A minimal AI code-review bot for Gitea, running as a CI step on the existing
-`act-runner`. This is the **pilot** — a small, self-contained reviewer that
-predates the full `pragent` framework (whose design lives in
-`docs/plans/2026-08-04-pragent-design.md`). The framework will later absorb
-this; until then, this is what runs.
+The pilot is a central, stdlib-only Gitea webhook service. It reviews opted-in
+pull requests with an on-network model, posts inline findings, and emits review
+telemetry to Langfuse. The service is fail-open: a review failure is reported
+as a PR comment and does not block CI.
 
-## How it works
+## Runtime flow
 
-1. You add `pragent-bot` to a repo and commit `.gitea/workflows/ai-review.yml`.
-2. On a PR, you add the **`AI-REVIEW`** label.
-3. Gitea Actions runs the workflow on the `act-runner`; it fetches the PR diff,
-   asks `glm-5.2:cloud` (on-network via the headroom proxy) to review it, and
-   posts the findings back as a PR review authored by `pragent-bot`.
-4. Remove the label to stop re-reviews on further pushes.
+1. Gitea sends a signed `pull_request` webhook.
+2. `webhook_server.py` validates the request, checks the base branch's
+   `.pr-review.json` for `"enabled": true`, and claims `(repo, PR, SHA)`.
+3. `ai_review.review_pr()` fetches the diff, trusted config, and prior reviews.
+4. `opencode_review.py` checks out the PR head in a sanitized temporary
+   directory and runs the review agent. The legacy Ollama-compatible path is
+   still available through `PRAGENT_ENGINE`.
+5. The review output is parsed and normalized, valid post-change line anchors
+   are separated from summary-only findings, and Gitea receives the result.
+6. `langfuse_trace.py` records usage, cost basis, findings, and evaluation
+   scores when Langfuse credentials are configured.
 
-Fail-open: the job always exits 0 and never blocks CI. Errors become a short
-"review failed" comment.
+## Module map
 
-## Onboard a repo (3 steps)
-
-### 1. Add `pragent-bot` as collaborator
-
-Repo → Settings → Collaborators → Add → `pragent-bot` → permission **Write**.
-(Write is required to post reviews/comments.)
-
-Or via API (with an admin/owner token):
-
-```bash
-curl -X PUT -H "Authorization: token $OWNER_TOKEN" \
-  -H "Content-Type: application/json" \
-  -d '{"permission":"write"}' \
-  "http://<gitea-host>:3000/api/v1/repos/OWNER/REPO/collaborators/pragent-bot"
-```
-
-### 2. Add the `PRAGENT_BOT_TOKEN` secret
-
-Repo → Settings → Actions → Secrets → New secret → name `PRAGENT_BOT_TOKEN`,
-value = the bot's access token (ask the platform admin; stored mode-600 at
-`~/.claude/.pragent-bot-token` on the admin host).
-
-### 3. Commit the workflow
-
-Copy `pilot/workflow-template.yml` into the target repo as
-`.gitea/workflows/ai-review.yml` and commit it. That's it.
-
-## Use it
-
-Open a PR (or push to an open one), add the **`AI-REVIEW`** label. The review
-appears within ~30–90s depending on diff size and model latency.
-
-## What's intentionally NOT in the pilot
-
-Deferred to the full framework (by design, see the design doc):
-
-- Attention tiering (trivial/lite/full/oversized) and per-tier cost control.
-- Multiple analyzer fan-out over a shared cached prompt prefix.
-- Prior-comment synthesis (so each push re-posts; the latest review is tagged
-  with the head SHA so it's easy to spot).
-- Inline line comments and status checks.
-- `pragent explain` / `replay` / analytics JSONL.
-- A second forge (GitLab) and the provider matrix.
-
-## Pieces
-
-| File | Role |
+| Module | Responsibility |
 |---|---|
-| `pilot/ai_review.py` | The reviewer script (stdlib only). Single source of truth — fetched at runtime by each repo's workflow. |
-| `pilot/workflow-template.yml` | The Gitea Action consumers copy into `.gitea/workflows/ai-review.yml`. |
-| `tests/pilot/test_ai_review.py` | Unit tests for the pure helpers (no network). |
+| `webhook_server.py` | HTTP ingress, signature verification, opt-in gate, concurrency |
+| `review_config.py` | Trusted base-branch opt-in policy; transport injected for tests |
+| `gitea_client.py` | HTTP transport adapter and repository-scoped client |
+| `ai_review.py` | Compatibility facade and review orchestration |
+| `model_client.py` | Anthropic-compatible model adapter and response text extraction |
+| `opencode_review.py` | Hostile-checkout containment and agent execution |
+| `diff_compress.py` | Diff compression and prior-review extraction |
+| `feedback*.py` | Feedback persistence, harvesting, analysis, and Langfuse scores |
+| `langfuse_trace.py` | Fail-open Langfuse ingestion and cost metadata |
+| `cost_model.py` | Provider price catalog and equivalent-cost calculations |
+| `eval_*.py` | Dataset bootstrap, evaluators, and behavioral scoring |
 
-## Run the tests
+`ai_review.py` remains the stable import surface for existing workflow and
+webhook deployments. New code should put policy, adapters, and pure transforms
+in the focused modules above rather than adding unrelated functions there.
 
-```bash
-cd ~/Projects/pragent
-PYTHONPATH=pilot python3 -m pytest tests/pilot/    # if pytest available
-# or, without pytest:
-python3 - <<'PY'
-import os, sys, importlib.util
-sys.path.insert(0, os.path.abspath("pilot"))
-import ai_review  # noqa: F401
-spec = importlib.util.spec_from_file_location("t", "tests/pilot/test_ai_review.py")
-m = importlib.util.module_from_spec(spec); spec.loader.exec_module(m)
-fails = 0
-for n in sorted(x for x in dir(m) if x.startswith("test_")):
-    try: getattr(m, n)(); print("PASS", n)
-    except Exception as e: fails += 1; print("FAIL", n, e)
-print("failed:", fails)
-PY
+## Onboard a repository
+
+1. Add `pragent-bot` as a Write collaborator.
+2. Commit this file to the default branch:
+
+   ```json
+   {"enabled": true}
    ```
 
-## Configuration knobs (env in the workflow)
+3. Open or update a pull request.
 
-| Env | Default | Purpose |
-|---|---|---|
-| `OLLAMA_MODEL` | `glm-5.2:cloud` | Model id passed to the headroom proxy. |
-| `OLLAMA_MAX_TOKENS` | `6000` | Output token cap. |
-| `DIFF_MAX_CHARS` | `150000` | Diff truncation cap (with a noted truncation marker). |
-| `OLLAMA_URL` | `http://<model-proxy-host>:8789` | headroom proxy (tailnet). If the act-runner can't reach the tailnet IP, expose 8789 as an in-cluster Service+Endpoints and set this to the cluster DNS name. |
+No per-repository workflow, secret, or label is required for the central
+webhook path. See [`README-webhook.md`](README-webhook.md) for deployment,
+security, and webhook registration details.
+
+## Configuration
+
+| Variable | Default | Purpose |
+|---|---:|---|
+| `GITEA_API` | in-cluster URL | Gitea API base URL |
+| `PRAGENT_BOT_TOKEN` | — | Bot credential |
+| `OLLAMA_URL` / `OLLAMA_MODEL` | headroom / `glm-5.2:cloud` | Legacy model path |
+| `PRAGENT_ENGINE` | `opencode` | `opencode` or legacy model path |
+| `DIFF_MAX_CHARS` | `150000` | Diff input cap |
+| `PRAGENT_MAX_CONCURRENT_REVIEWS` | `2` | Process concurrency bound |
+| `LANGFUSE_HOST` + keys | unset | Enables telemetry; unset is a no-op |
+
+## Tests
+
+```bash
+python3 -m pytest tests -q
+```
+
+Tests use mocked transports and local fixtures. They do not require Gitea,
+Langfuse, a model endpoint, or network access.

pilot/ai_review.py

@@ -185,15 +185,8 @@ def parse_text_blocks(content: list) -> str:
     Drops `thinking` blocks (glm-5.2:cloud is a reasoning model and emits them).
     Tolerates missing/malformed blocks by skipping them.
     """
-    if not isinstance(content, list):
-        return ""
-    out = []
-    for block in content:
-        if not isinstance(block, dict):
-            continue
-        if block.get("type") == "text" and isinstance(block.get("text"), str):
-            out.append(block["text"])
-    return "\n".join(out).strip()
+    from model_client import parse_text_blocks as _parse_text_blocks
+    return _parse_text_blocks(content)
 
 
 def _int_env(name: str, default: int) -> int:
@@ -678,6 +671,19 @@ def _strip_path_prefix(p: str) -> str:
 # ---------------------------------------------------------------------------
 
 
+# How many raw findings the last `parse_review_output` / `parse_findings` call
+# rejected for an unusable path/line. A side channel rather than a return value
+# because both parsers already return fixed-width tuples that several callers
+# and their tests unpack positionally; widening them to carry a telemetry
+# number would be a breaking change for a fail-open signal.
+_LAST_PARSE_DROPPED: dict[str, int] = {"n": 0}
+
+
+def last_parse_dropped() -> int:
+    """Findings the last parse discarded. Read it immediately after parsing."""
+    return int(_LAST_PARSE_DROPPED.get("n") or 0)
+
+
 def _normalize_finding(f: dict) -> dict | None:
     """Validate + normalize one raw finding dict. Returns None if it's unusable
     (missing path/line). Normalises severity, keeps `reference` (default "")."""
@@ -754,6 +760,7 @@ def parse_findings(text: str) -> list[dict]:
     Also accepts a bare JSON array as the outer value: ``[{...}, {...}]`` —
     some agents skip the ``{"summary":..., "findings":[...]}`` wrapper.
     """
+    _LAST_PARSE_DROPPED["n"] = 0
     data = _parse_json_tolerant(text)
     if isinstance(data, dict):
         findings = data.get("findings")
@@ -768,6 +775,7 @@ def parse_findings(text: str) -> list[dict]:
         n = _normalize_finding(f)
         if n is not None:
             out.append(n)
+    _LAST_PARSE_DROPPED["n"] = len(findings) - len(out)
     return out
 
 
@@ -823,6 +831,7 @@ def parse_review_output(
     block), with a tolerant fallback that scans for the last balanced
     object/array in the prose tail. Never raises.
     """
+    _LAST_PARSE_DROPPED["n"] = 0
     blob = _last_json_block(text)
     if blob is None:
         return "", [], [], [], [], "", ""
@@ -856,6 +865,12 @@ def parse_review_output(
             n = _normalize_finding(f)
             if n is not None:
                 out.append(n)
+        # A model that emits findings at unusable locations is indistinguishable
+        # from one that found nothing, because both end up with an empty `out`.
+        # Stash the delta so the caller can score it (see `eval_scores`).
+        _LAST_PARSE_DROPPED["n"] = len(findings_raw) - len(out)
+    else:
+        _LAST_PARSE_DROPPED["n"] = 0
     return summary, out, summary_changes, risks, walkthrough, risk_verdict, test_coverage
 
 
@@ -1781,19 +1796,8 @@ def compact_prior_reviews(prior_bodies: list[str]) -> list[str]:
 
 
 def _http(method: str, url: str, token: str, body: dict | None = None, accept: str = "application/json") -> tuple[int, bytes]:
-    headers = {"Authorization": f"token {token}", "Accept": accept}
-    data = None
-    if body is not None:
-        data = json.dumps(body).encode()
-        headers["Content-Type"] = "application/json"
-    req = urllib.request.Request(url, data=data, headers=headers, method=method)
-    try:
-        with urllib.request.urlopen(req, timeout=180) as r:
-            return r.status, r.read()
-    except urllib.error.HTTPError as e:
-        return e.code, e.read()
-    except urllib.error.URLError as e:
-        raise RuntimeError(f"network error: {e.reason}") from e
+    from gitea_client import request
+    return request(method, url, token, body, accept)
 
 
 def gitea_get(api: str, repo: str, path: str, token: str, accept: str = "application/json") -> tuple[int, bytes]:
@@ -1986,22 +1990,8 @@ def fetch_repo_config(api: str, repo: str, token: str, ref: str = "") -> dict:
 
 
 def call_model(ollama_url: str, model: str, system: str, user: str, max_tokens: int) -> str:
-    payload = {
-        "model": model,
-        "max_tokens": max_tokens,
-        "system": system,
-        "messages": [{"role": "user", "content": user}],
-    }
-    status, raw = _http(
-        "POST",
-        f"{ollama_url.rstrip('/')}/v1/messages",
-        "ollama",  # headroom ollama hub uses x-api-key: ollama
-        payload,
-    )
-    if status != 200:
-        raise RuntimeError(f"model call failed: HTTP {status}: {raw[:500].decode('utf-8', errors='replace')}")
-    data = json.loads(raw)
-    return parse_text_blocks(data.get("content", []))
+    from model_client import complete
+    return complete(ollama_url, model, system, user, max_tokens)
 
 
 def post_review(api: str, repo: str, index: str, token: str, body: str) -> None:
@@ -2068,6 +2058,50 @@ def _need(name: str) -> str:
     return v
 
 
+def _emit_langfuse(
+    *,
+    repo: str,
+    index: str,
+    sha: str,
+    title: str,
+    model: str,
+    usage: dict | None,
+    findings: list[dict],
+    summary: str,
+    engine: str,
+    config: dict | None = None,
+    dropped_count: float | None = None,
+) -> None:
+    """Ship this review's usage to Langfuse, if one is configured.
+
+    Called on both exit paths that spent tokens — the normal post and the
+    salvage path — because an unparseable run costs the same as a clean one and
+    is exactly the kind of thing worth trending.
+
+    Local import + blanket except: `langfuse_trace` is stdlib-only but optional,
+    and telemetry is never allowed to fail a review (see the fail-open contract
+    in `review_pr`). The trace's `environment` is `claude` or `ollama`, so the
+    two spend stories stay separated in every Langfuse view.
+    """
+    try:
+        import langfuse_trace
+
+        # Same comparison model the review body prices against, so the number
+        # in Langfuse and the number in the PR agree. Free/unknown models
+        # (MiniMax, glm, self-hosted qwen) are priced against it; a paid model
+        # is priced as itself.
+        price_target, _err = _resolve_price_target(config)
+
+        langfuse_trace.emit_review_trace(
+            repo=repo, index=index, sha=sha, title=title, model=model,
+            usage=usage, findings=findings, summary=summary or "",
+            engine=engine, lenses=(usage or {}).get("lenses"),
+            price_target=price_target, dropped_count=dropped_count,
+        )
+    except Exception as e:
+        print(f"pragent: langfuse emit skipped: {e}", file=sys.stderr)
+
+
 def review_pr(
     api: str,
     repo: str,
@@ -2192,6 +2226,7 @@ def review_pr(
                     additional_context=additional_context,
                 )
             review_summary, findings, summary_changes, risks, _walkthrough, _risk_verdict, _test_coverage = parse_review_output(stdout)
+            parse_dropped = last_parse_dropped()
             if not findings and not review_summary:
                 # The findings JSON was missing or malformed. Don't discard the
                 # run: salvage the prose, keep the usage report (the tokens were
@@ -2208,11 +2243,18 @@ def review_pr(
                     salvaged or "AI review produced no parseable output.",
                     display_model, sha, usage_section=usage_section,
                     static_message=(config or {}).get("static_message", "")))
+                _emit_langfuse(
+                    repo=repo, index=index, sha=sha, title=title,
+                    model=display_model, usage=usage, findings=[],
+                    summary=salvaged, engine=engine, config=config,
+                    dropped_count=parse_dropped,
+                )
                 return True
         else:
             user_prompt = build_user_prompt(title, body + compression_note, diff, config, prior, additional_context)
             raw_findings = call_model(ollama_url, model, SYSTEM_PROMPT, user_prompt, max_tokens)
             findings = parse_findings(raw_findings)
+            parse_dropped = last_parse_dropped()
             usage = None
 
         # Filter / cap findings per `.pr-review.json` (style, threshold, max,
@@ -2291,6 +2333,12 @@ def review_pr(
         )
 
         post_inline_review(api, repo, index, token, summary_body, anchored)
+        _emit_langfuse(
+            repo=repo, index=index, sha=sha, title=title,
+            model=display_model, usage=usage, findings=findings,
+            summary=review_summary, engine=engine, config=config,
+            dropped_count=parse_dropped,
+        )
         print(
             f"pragent: reviewed {repo}#{index} sha={sha[:8]} "
             f"engine={engine} findings={len(findings)} inline={len(anchored)}",

pilot/dashboard.py

@@ -1,754 +0,0 @@
-#!/usr/bin/env python3
-"""pragent pilot — read-mostly dashboard.
-
-Stdlib HTTP server (mirrors `webhook_server.py`'s BaseHTTPRequestHandler +
-ThreadingHTTPServer shape) that renders three views off the feedback SQLite:
-
-  GET /                                    overview
-  GET /r/<owner>/<name>                    repo summary + edit form
-  GET /r/<owner>/<name>/<index>            one PR's findings
-  GET /r/<owner>/<name>/<index>/raw        raw Markdown body (via Gitea)
-  GET /static/style.css                    CSS
-  POST /r/<owner>/<name>/edit              mutate .pr-review.json (Tasks C+D)
-
-Auth: oauth2-proxy fronts this service in-cluster. Every route except
-`/static/*` requires the `X-Forwarded-User` header (set by oauth2-proxy
-once the user has logged in via Logto). Missing header → 401 +
-`WWW-Authenticate: Basic realm="pragent-dashboard"` so oauth2-proxy
-intercepts the response.
-
-DB: `PRAGENT_FEEDBACK_DB` points at the SQLite file the webhook server
-also writes. Per-request open (SQLite is cheap, no concurrency hazard,
-no stale-conn surprise after the file rotates).
-
-All HTML is rendered via `string.Template` and every dynamic value is
-escaped with `html.escape(..., quote=True)`. No `.format`, no f-string
-templates — see `_render_*` for the discipline.
-"""
-
-from __future__ import annotations
-
-import base64
-import datetime
-import html
-import json
-import os
-import secrets
-import string
-import urllib.error
-import urllib.parse
-import urllib.request
-from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
-
-from pilot import dashboard_data
-
-
-# ---------------------------------------------------------------------------
-# Config
-# ---------------------------------------------------------------------------
-
-FEEDBACK_DB = ""  # legacy; readers should call _feedback_db()
-PORT = int(os.environ.get("DASHBOARD_PORT", "8081"))
-
-GITEA_API = ""  # legacy; readers should call _gitea_api()
-BOT_TOKEN = ""  # legacy; readers should call _bot_token()
-
-# CSRF secret for the edit form. Regenerated per process (each Python
-# interpreter launch). Behind oauth2-proxy this is enough — only an
-# already-authenticated same-tab request can read this and echo it back.
-_CSRF_SECRET: str = secrets.token_urlsafe(24)
-
-
-# ---------------------------------------------------------------------------
-# Lazy config readers — tests set env after import, so each request re-reads.
-# Production: env is fixed for the process lifetime; the per-request lookup is
-# a dict access, not a syscall.
-# ---------------------------------------------------------------------------
-
-
-def _feedback_db() -> str:
-    return os.environ.get("PRAGENT_FEEDBACK_DB", "")
-
-
-def _bot_token() -> str:
-    return os.environ.get("PRAGENT_BOT_TOKEN", "")
-
-
-def _gitea_api() -> str:
-    return os.environ.get("GITEA_API", "http://gitea-http.gitea.svc.cluster.local:3000")
-
-
-# ---------------------------------------------------------------------------
-# Stylesheet — small, dark-mode-friendly, deliberately under 100 lines
-# ---------------------------------------------------------------------------
-
-STYLE_CSS = """
-:root { color-scheme: light dark; }
-* { box-sizing: border-box; }
-body {
-  font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", system-ui, sans-serif;
-  margin: 0; padding: 0;
-  background: #0f1115; color: #e6e6e6;
-  line-height: 1.5;
-}
-header {
-  background: #1a1d23; padding: 12px 20px;
-  border-bottom: 1px solid #2a2f38;
-  display: flex; align-items: center; gap: 18px;
-}
-header h1 { font-size: 18px; margin: 0; }
-header nav a {
-  color: #8ab4f8; text-decoration: none; margin-right: 12px;
-}
-header nav a:hover { text-decoration: underline; }
-main { padding: 20px; max-width: 1100px; margin: 0 auto; }
-h2 { margin-top: 24px; font-size: 16px; color: #c9d1d9; }
-.metric-row { display: flex; gap: 16px; flex-wrap: wrap; margin-bottom: 16px; }
-.metric {
-  background: #1a1d23; padding: 14px 18px; border-radius: 8px;
-  min-width: 140px; border: 1px solid #2a2f38;
-}
-.metric .v { font-size: 28px; font-weight: 600; }
-.metric .l { font-size: 12px; color: #8b949e; text-transform: uppercase; letter-spacing: 0.04em; }
-table { width: 100%; border-collapse: collapse; margin: 8px 0 16px; font-size: 14px; }
-th, td { text-align: left; padding: 6px 10px; border-bottom: 1px solid #2a2f38; }
-th { color: #8b949e; font-weight: 500; text-transform: uppercase; font-size: 11px; letter-spacing: 0.04em; }
-tr:hover td { background: #161922; }
-.sev-critical { color: #ff7b72; font-weight: 600; }
-.sev-high { color: #f0883e; }
-.sev-medium { color: #d29922; }
-.sev-low { color: #8b949e; }
-.muted { color: #8b949e; font-size: 12px; }
-.sparkline { font-family: ui-monospace, "SF Mono", monospace; letter-spacing: 1px; }
-form { background: #1a1d23; padding: 14px 18px; border-radius: 8px; border: 1px solid #2a2f38; margin: 12px 0; }
-form label { display: block; margin: 8px 0 4px; color: #c9d1d9; font-size: 13px; }
-form input[type=text], form textarea, form select {
-  background: #0f1115; color: #e6e6e6; border: 1px solid #2a2f38;
-  border-radius: 4px; padding: 6px 8px; font-family: inherit; font-size: 14px;
-  width: 100%;
-}
-form textarea { min-height: 80px; }
-form .row { display: flex; gap: 8px; align-items: center; margin-top: 12px; }
-form button {
-  background: #2ea043; color: white; border: none; border-radius: 4px;
-  padding: 6px 14px; font-size: 14px; cursor: pointer;
-}
-form button:hover { background: #3fb950; }
-.flash { background: #3d1e1e; color: #ff7b72; padding: 8px 12px; border-radius: 4px; margin-bottom: 12px; }
-code { background: #161922; padding: 1px 4px; border-radius: 3px; font-size: 13px; }
-pre { background: #161922; padding: 12px; border-radius: 6px; overflow-x: auto; }
-"""
-
-
-# ---------------------------------------------------------------------------
-# Templates — string.Template so dynamic values are always escaped explicitly
-# ---------------------------------------------------------------------------
-
-_BASE = string.Template("""<!doctype html>
-<html lang="en">
-<head>
-<meta charset="utf-8">
-<title>${title}</title>
-<link rel="stylesheet" href="/static/style.css">
-</head>
-<body>
-<header>
-  <h1>pragent dashboard</h1>
-  <nav>
-    <a href="/">Home</a>
-    <a href="/r/${repos_first}">repos</a>
-  </nav>
-  <span class="muted" style="margin-left:auto">${db_status}</span>
-</header>
-<main>
-${body}
-</main>
-</body>
-</html>""")
-
-
-_OVERVIEW = string.Template("""<h2>Overview</h2>
-<div class="metric-row">
-  <div class="metric"><div class="v">${total_reviews}</div><div class="l">reviews</div></div>
-  <div class="metric"><div class="v">${total_findings}</div><div class="l">findings</div></div>
-  <div class="metric"><div class="v">${total_repos}</div><div class="l">repos</div></div>
-  <div class="metric"><div class="v">${last_30d_reviews}</div><div class="l">last 30d</div></div>
-</div>
-
-<h2>Last 7 days</h2>
-<div class="sparkline">${sparkline}</div>
-<div class="muted">total cost: $${total_cost_usd} — no per-review cost logged</div>
-
-<h2>Top repos</h2>
-${top_repos_table}
-""")
-
-
-_REPO = string.Template("""<h2>Repo: <code>${repo}</code></h2>
-<div class="metric-row">
-  <div class="metric"><div class="v">${total_runs}</div><div class="l">runs</div></div>
-  <div class="metric"><div class="v">${sev_critical}</div><div class="l sev-critical">critical</div></div>
-  <div class="metric"><div class="v">${sev_high}</div><div class="l sev-high">high</div></div>
-  <div class="metric"><div class="v">${sev_medium}</div><div class="l sev-medium">medium</div></div>
-  <div class="metric"><div class="v">${sev_low}</div><div class="l sev-low">low</div></div>
-</div>
-
-<h2>Edit .pr-review.json</h2>
-${flash}
-<form method="post" action="/r/${repo_url}/edit">
-  <input type="hidden" name="_csrf" value="${csrf}">
-  <label for="static_message">Static banner message (max 400 chars)</label>
-  <textarea id="static_message" name="static_message" maxlength="400">${current_static_message}</textarea>
-  <label for="model">Model (PRICES keys)</label>
-  <select id="model" name="model">${model_options}</select>
-  <div class="row">
-    <button type="submit">Save</button>
-    <span class="muted">posted via the bot identity; one commit on the base branch</span>
-  </div>
-</form>
-
-<h2>Top findings (by occurrence)</h2>
-${top_findings_table}
-
-<h2>Runs by day (last 30d)</h2>
-${runs_by_day_table}
-
-<h2>Reviews</h2>
-${reviews_table}
-""")
-
-
-_PR = string.Template("""<h2>PR <code>${repo}</code> #${pr}</h2>
-<div class="muted">head sha: <code>${head_sha}</code></div>
-<div class="muted">posted_at: ${posted_at_iso}</div>
-<div class="muted">review_id_gitea: ${review_id_gitea} · body_comment_id: ${body_comment_id}</div>
-
-<h2>Findings</h2>
-${findings_table}
-
-<p><a href="/r/${repo_url}/${pr}/raw">raw review body (Markdown)</a></p>
-""")
-
-
-# ---------------------------------------------------------------------------
-# Small helpers
-# ---------------------------------------------------------------------------
-
-
-def _esc(s) -> str:
-    """HTML-escape any value to a string."""
-    return html.escape(str(s), quote=True)
-
-
-def _ts_iso(ts: int) -> str:
-    if not ts:
-        return "—"
-    return datetime.datetime.fromtimestamp(int(ts), tz=datetime.timezone.utc).isoformat()
-
-
-def _sparkline(buckets: list[dict]) -> str:
-    """7-bucket sparkline as unicode bars."""
-    bars = "▁▂▃▄▅▆▇█"
-    if not buckets:
-        return ""
-    mx = max((b.get("count", 0) for b in buckets), default=0) or 1
-    out = []
-    for b in buckets:
-        n = b.get("count", 0)
-        idx = min(len(bars) - 1, int(round(n / mx * (len(bars) - 1))))
-        out.append(bars[idx])
-    return "".join(out)
-
-
-# ---------------------------------------------------------------------------
-# Renderers — one per page
-# ---------------------------------------------------------------------------
-
-
-def _overview_body(data: dict) -> str:
-    top_rows = "".join(
-        f"<tr><td><a href=\"/r/{_esc(r['repo'])}\">{_esc(r['repo'])}</a></td>"
-        f"<td>{int(r['run_count'])}</td>"
-        f"<td class=\"muted\">{_ts_iso(int(r['last_seen']))}</td></tr>"
-        for r in data.get("top_repos", [])
-    ) or "<tr><td class=\"muted\">no reviews yet</td></tr>"
-    top_table = f"<table><thead><tr><th>repo</th><th>runs</th><th>last seen</th></tr></thead><tbody>{top_rows}</tbody></table>"
-    return _OVERVIEW.substitute(
-        total_reviews=_esc(data.get("total_reviews", 0)),
-        total_findings=_esc(data.get("total_findings", 0)),
-        total_repos=_esc(data.get("total_repos", 0)),
-        last_30d_reviews=_esc(data.get("last_30d_reviews", 0)),
-        sparkline=_esc(_sparkline(data.get("daily", []))),
-        total_cost_usd=f"{float(data.get('total_cost_usd', 0.0)):.2f}",
-        top_repos_table=top_table,
-    )
-
-
-def _repo_body(data: dict, *, repo_url: str, csrf: str, current_model: str,
-               current_static_message: str, flash: str = "") -> str:
-    fbs = data.get("findings_by_severity", {})
-    tf = data.get("top_findings", [])
-
-    # Top findings table.
-    if tf:
-        rows = "".join(
-            f"<tr><td><code>{_esc(f['path'])}:{_esc(f['line'])}</code></td>"
-            f"<td class=\"sev-{_esc(f.get('severity', 'low').lower())}\">{_esc(f.get('severity', ''))}</td>"
-            f"<td>{_esc(f.get('problem', ''))}</td>"
-            f"<td>{int(f.get('occurrences', 0))}</td>"
-            f"<td>+{int(f.get('upvotes', 0))} / -{int(f.get('downvotes', 0))}</td>"
-            f"<td>{'resolved' if int(f.get('resolved', 0)) else 'open'}</td>"
-            f"<td>{int(f.get('reply_count', 0))}</td></tr>"
-            for f in tf
-        )
-        top_findings_table = (
-            "<table><thead><tr><th>location</th><th>severity</th>"
-            "<th>problem</th><th>occurrences</th><th>votes</th>"
-            "<th>state</th><th>replies</th></tr></thead><tbody>"
-            f"{rows}</tbody></table>"
-        )
-    else:
-        top_findings_table = "<p class=\"muted\">no findings yet</p>"
-
-    # Runs by day.
-    runs = data.get("runs_by_day", [])
-    if runs:
-        rows = "".join(
-            f"<tr><td>{_esc(r['date'])}</td><td>{int(r.get('count', 0))}</td></tr>"
-            for r in runs
-        )
-        runs_by_day_table = (
-            "<table><thead><tr><th>date</th><th>runs</th></tr></thead>"
-            f"<tbody>{rows}</tbody></table>"
-        )
-    else:
-        runs_by_day_table = "<p class=\"muted\">no runs in the last 30 days</p>"
-
-    # Reviews list — derived from finding timestamps; cheap because we
-    # just enumerate the repo's review rows.
-    reviews_table = _repo_reviews_table(repo_url, data.get("recent_reviews", []))
-
-    # Model select (Task D) — sorted PRICES keys + "keep current".
-    from cost_model import PRICES  # local: pilot-only dep
-    model_options = (
-        f"<option value=\"\">— keep current ({_esc(current_model or 'unset')}) —</option>"
-        + "".join(
-            f"<option value=\"{_esc(k)}\" {'selected' if k == current_model else ''}>{_esc(k)}</option>"
-            for k in sorted(PRICES)
-        )
-    )
-
-    return _REPO.substitute(
-        repo=_esc(data.get("repo", "")),
-        repo_url=_esc(repo_url),
-        total_runs=_esc(data.get("total_runs", 0)),
-        sev_critical=_esc(fbs.get("critical", 0)),
-        sev_high=_esc(fbs.get("high", 0)),
-        sev_medium=_esc(fbs.get("medium", 0)),
-        sev_low=_esc(fbs.get("low", 0)),
-        csrf=_esc(csrf),
-        current_static_message=_esc(current_static_message),
-        model_options=model_options,
-        flash=_esc(flash),
-        top_findings_table=top_findings_table,
-        runs_by_day_table=runs_by_day_table,
-        reviews_table=reviews_table,
-    )
-
-
-def _repo_reviews_table(repo_url: str, rows: list[dict]) -> str:
-    if not rows:
-        return "<p class=\"muted\">no reviews yet</p>"
-    out = "<table><thead><tr><th>PR</th><th>head sha</th><th>posted</th></tr></thead><tbody>"
-    for r in rows:
-        out += (
-            f"<tr><td><a href=\"/r/{_esc(repo_url)}/{int(r['pr'])}\">#{int(r['pr'])}</a></td>"
-            f"<td><code>{_esc(r['head_sha'][:10])}</code></td>"
-            f"<td class=\"muted\">{_ts_iso(int(r.get('posted_at', 0)))}</td></tr>"
-        )
-    out += "</tbody></table>"
-    return out
-
-
-def _pr_body(data: dict, *, repo_url: str) -> str:
-    findings = data.get("findings", [])
-    if findings:
-        rows = "".join(
-            f"<tr><td><code>{_esc(f['path'])}:{_esc(f['line'])}</code></td>"
-            f"<td class=\"sev-{_esc(f.get('severity', 'low').lower())}\">{_esc(f.get('severity', ''))}</td>"
-            f"<td>{_esc(f.get('problem', ''))}</td>"
-            f"<td>{_esc(f.get('fix', ''))}</td>"
-            f"<td>{_esc(f.get('suggestion', ''))}</td>"
-            f"<td>+{int(f.get('upvotes', 0))} / -{int(f.get('downvotes', 0))}</td>"
-            f"<td>{'resolved' if int(f.get('resolved', 0)) else 'open'}</td>"
-            f"<td>{int(f.get('reply_count', 0))}</td></tr>"
-            for f in findings
-        )
-        findings_table = (
-            "<table><thead><tr><th>location</th><th>severity</th>"
-            "<th>problem</th><th>fix</th><th>suggestion</th>"
-            "<th>votes</th><th>state</th><th>replies</th></tr></thead>"
-            f"<tbody>{rows}</tbody></table>"
-        )
-    else:
-        findings_table = "<p class=\"muted\">no findings</p>"
-
-    return _PR.substitute(
-        repo=_esc(data.get("repo", "")),
-        repo_url=_esc(repo_url),
-        pr=_esc(data.get("pr", 0)),
-        head_sha=_esc(data.get("head_sha", "")),
-        posted_at_iso=_ts_iso(int(data.get("posted_at", 0))),
-        review_id_gitea=_esc(data.get("review_id_gitea", "") or "—"),
-        body_comment_id=_esc(data.get("body_comment_id", "") or "—"),
-        findings_table=findings_table,
-    )
-
-
-def _page(title: str, body: str, *, repos_first: str = "") -> str:
-    db_status = _feedback_db() or "(no DB configured)"
-    return _BASE.substitute(
-        title=_esc(title),
-        body=body,
-        repos_first=_esc(repos_first),
-        db_status=_esc(db_status),
-    )
-
-
-# ---------------------------------------------------------------------------
-# Gitea HTTP helper — minimal, used by the raw body fetch and the edit endpoint
-# ---------------------------------------------------------------------------
-
-
-def _http(method: str, url: str, *, token: str = "", body: dict | None = None,
-          raw_body: bytes | None = None) -> tuple[int, bytes]:
-    """Like ai_review._http but local: this module is stdlib-only and doesn't
-    depend on the ai_review import (which pulls in a 1700-line reviewer)."""
-    headers = {"Accept": "application/json"}
-    data: bytes | None = None
-    if raw_body is not None:
-        data = raw_body
-        headers["Content-Type"] = "application/json"
-    elif body is not None:
-        data = json.dumps(body).encode()
-        headers["Content-Type"] = "application/json"
-    if token:
-        headers["Authorization"] = f"token {token}"
-    req = urllib.request.Request(url, data=data, headers=headers, method=method)
-    try:
-        with urllib.request.urlopen(req, timeout=30) as r:
-            return r.status, r.read()
-    except urllib.error.HTTPError as e:
-        return e.code, e.read()
-    except urllib.error.URLError as e:
-        raise RuntimeError(f"network error: {e.reason}") from e
-
-
-# ---------------------------------------------------------------------------
-# Auth
-# ---------------------------------------------------------------------------
-
-
-def _is_authed(headers) -> bool:
-    """True when oauth2-proxy forwarded a verified user.
-
-    oauth2-proxy sets `X-Forwarded-User` (and friends) only after a
-    successful Logto login + email allowlist check. Unauthenticated
-    requests never see the header, so the dashboard never has to know
-    about cookies, secrets, or Logto's token shape.
-    """
-    return bool((headers.get("X-Forwarded-User") or "").strip())
-
-
-# ---------------------------------------------------------------------------
-# Routes
-# ---------------------------------------------------------------------------
-
-
-def _route_overview() -> bytes:
-    data = dashboard_data.overview(_feedback_db())
-    body = _overview_body(data)
-    # nav: first repo if any
-    repos_first = ""
-    if data.get("top_repos"):
-        repos_first = data["top_repos"][0]["repo"]
-    return _page("Overview", body, repos_first=repos_first).encode()
-
-
-def _route_repo(owner: str, name: str) -> bytes:
-    repo_url = f"{owner}/{name}"
-    data = dashboard_data.repo_summary(_feedback_db(), repo_url)
-    # Pull current .pr-review.json (best-effort) so the form fields prefill.
-    current_static_message, current_model, flash = "", "", ""
-    cfg, err = _fetch_pr_review_json(repo_url)
-    if cfg:
-        current_static_message = cfg.get("static_message", "")
-        current_model = cfg.get("model", "")
-    elif err and err != "404":
-        flash = f"could not read .pr-review.json: {err}"
-    body = _repo_body(
-        data,
-        repo_url=repo_url,
-        csrf=_CSRF_SECRET,
-        current_model=current_model,
-        current_static_message=current_static_message,
-        flash=flash,
-    )
-    return _page(f"repo {repo_url}", body, repos_first=repo_url).encode()
-
-
-def _route_pr(owner: str, name: str, index: int) -> bytes:
-    repo_url = f"{owner}/{name}"
-    data = dashboard_data.pr_summary(_feedback_db(), repo_url, int(index))
-    body = _pr_body(data, repo_url=repo_url)
-    return _page(f"PR {repo_url}#{index}", body, repos_first=repo_url).encode()
-
-
-def _route_pr_raw(owner: str, name: str, index: int) -> tuple[int, bytes]:
-    repo_url = f"{owner}/{name}"
-    data = dashboard_data.pr_summary(_feedback_db(), repo_url, int(index))
-    body_comment_id = data.get("body_comment_id")
-    if not body_comment_id:
-        return 404, b"no body_comment_id"
-    status, raw = _http(
-        "GET",
-        f"{_gitea_api()}/api/v1/repos/{repo_url}/issues/{index}/comments/{body_comment_id}",
-        token=_bot_token(),
-    )
-    if status != 200:
-        return 404, f"Gitea returned {status}".encode()
-    try:
-        parsed = json.loads(raw)
-        md = parsed.get("body", "")
-    except (json.JSONDecodeError, ValueError):
-        return 404, b"could not parse Gitea response"
-    return 200, md.encode()
-
-
-def _route_static_css() -> bytes:
-    return STYLE_CSS.encode()
-
-
-def _route_edit(owner: str, name: str, form: dict) -> tuple[int, dict, bytes]:
-    """Mutate .pr-review.json via the Gitea contents API (Tasks C+D)."""
-    repo_url = f"{owner}/{name}"
-    csrf = form.get("_csrf", "")
-    if csrf != _CSRF_SECRET:
-        return 302, {"Location": f"/r/{repo_url}"}, b""
-    static_message = (form.get("static_message") or "").strip()[:400]
-    model = (form.get("model") or "").strip()
-
-    # Validate model against PRICES.
-    from cost_model import PRICES
-    if model and model not in PRICES:
-        flash = urllib.parse.quote(f"unknown model {model!r}; not saved")
-        return 302, {"Location": f"/r/{repo_url}?flash={flash}"}, b""
-
-    cfg, err = _fetch_pr_review_json(repo_url)
-    if err and err != "404":
-        flash = urllib.parse.quote(f"could not read .pr-review.json: {err}")
-        return 302, {"Location": f"/r/{repo_url}?flash={flash}"}, b""
-    if cfg is None:
-        cfg = {}
-
-    if static_message:
-        cfg["static_message"] = static_message
-    elif "static_message" in cfg and not static_message:
-        # Empty submission clears the banner.
-        del cfg["static_message"]
-    if model:
-        cfg["model"] = model
-    elif "model" in cfg and not model:
-        del cfg["model"]
-
-    payload = json.dumps(cfg, indent=2, sort_keys=True).encode()
-    b64 = base64.b64encode(payload).decode()
-    body = {"content": b64, "message": "pragent dashboard: update .pr-review.json"}
-    if err == "404":
-        # File didn't exist — Gitea contents PUT still creates the file when
-        # `sha` is omitted, but only on certain versions; passing sha=None is
-        # safer.
-        pass
-    else:
-        # GET returned a sha — include it so Gitea enforces optimistic lock.
-        # The sha lives in cfg's wrapper: re-fetch once to capture it.
-        _, raw = _http(
-            "GET",
-            f"{_gitea_api()}/api/v1/repos/{repo_url}/contents/.pr-review.json",
-            token=_bot_token(),
-        )
-        try:
-            existing = json.loads(raw)
-            sha = existing.get("sha")
-            if sha:
-                body["sha"] = sha
-        except (json.JSONDecodeError, ValueError):
-            pass
-
-    status, _ = _http(
-        "PUT",
-        f"{_gitea_api()}/api/v1/repos/{repo_url}/contents/.pr-review.json",
-        token=_bot_token(),
-        body=body,
-    )
-    if status not in (200, 201):
-        flash = urllib.parse.quote(f"Gitea PUT failed: status {status}")
-        return 302, {"Location": f"/r/{repo_url}?flash={flash}"}, b""
-    return 302, {"Location": f"/r/{repo_url}"}, b""
-
-
-def _fetch_pr_review_json(repo_url: str) -> tuple[dict | None, str | None]:
-    """Return (cfg, None) on success, (None, None) when the file doesn't exist,
-    (None, 'reason') on error."""
-    if not _bot_token():
-        return None, "PRAGENT_BOT_TOKEN not set"
-    status, raw = _http(
-        "GET",
-        f"{_gitea_api()}/api/v1/repos/{repo_url}/contents/.pr-review.json",
-        token=_bot_token(),
-    )
-    if status == 404:
-        return None, "404"
-    if status != 200:
-        return None, f"status {status}"
-    try:
-        wrapper = json.loads(raw)
-        content_b64 = wrapper.get("content", "").replace("\n", "")
-        decoded = base64.b64decode(content_b64).decode("utf-8", errors="replace")
-        cfg = json.loads(decoded)
-    except (json.JSONDecodeError, ValueError) as e:
-        return None, f"parse error: {e}"
-    if not isinstance(cfg, dict):
-        return None, "not a JSON object"
-    return cfg, None
-
-
-# ---------------------------------------------------------------------------
-# Handler
-# ---------------------------------------------------------------------------
-
-
-class Handler(BaseHTTPRequestHandler):
-    def _send(self, status: int, body: bytes, *, content_type: str = "text/html; charset=utf-8",
-              extra_headers: dict | None = None) -> None:
-        self.send_response(status)
-        self.send_header("Content-Type", content_type)
-        self.send_header("Content-Length", str(len(body)))
-        if extra_headers:
-            for k, v in extra_headers.items():
-                self.send_header(k, v)
-        self.end_headers()
-        self.wfile.write(body)
-
-    def _redirect(self, location: str) -> None:
-        body = b""
-        self.send_response(302)
-        self.send_header("Location", location)
-        self.send_header("Content-Length", "0")
-        self.end_headers()
-        self.wfile.write(body)
-
-    def _unauthorized(self) -> None:
-        """401 + Basic challenge so oauth2-proxy intercepts and redirects to Logto."""
-        body = b"unauthorized\n"
-        self.send_response(401)
-        self.send_header("Content-Type", "text/plain; charset=utf-8")
-        self.send_header("Content-Length", str(len(body)))
-        self.send_header("WWW-Authenticate", 'Basic realm="pragent-dashboard"')
-        self.end_headers()
-        self.wfile.write(body)
-
-    # --- GET -----------------------------------------------------------------
-
-    def do_GET(self):
-        path = self.path
-        # Static is exempt from auth (also unauthenticated browser fingerprinting
-        # noise, but it's the same CSS regardless of viewer).
-        if path == "/static/style.css":
-            self._send(200, _route_static_css(), content_type="text/css; charset=utf-8")
-            return
-        if not _is_authed(self.headers):
-            self._unauthorized()
-            return
-
-        if path == "/" or path == "":
-            self._send(200, _route_overview())
-            return
-
-        # /r/<owner>/<name>            → repo
-        # /r/<owner>/<name>/<index>    → PR
-        # /r/<owner>/<name>/<index>/raw → raw Markdown
-        m = _REPO_PR_RAW_RE.match(path)
-        if m:
-            owner, name, idx, raw = m.group(1), m.group(2), m.group(3), m.group(4)
-            if raw:
-                status, body = _route_pr_raw(owner, name, int(idx))
-                self._send(status, body,
-                           content_type="text/plain; charset=utf-8" if status == 200 else "text/plain")
-                return
-            if idx:
-                self._send(200, _route_pr(owner, name, int(idx)))
-                return
-            self._send(200, _route_repo(owner, name))
-            return
-
-        self._send(404, b"not found", content_type="text/plain")
-
-    # --- POST ----------------------------------------------------------------
-
-    def do_POST(self):
-        path = self.path
-        if not _is_authed(self.headers):
-            self._unauthorized()
-            return
-        # /r/<owner>/<name>/edit
-        m = _EDIT_RE.match(path)
-        if m:
-            owner, name = m.group(1), m.group(2)
-            length = int(self.headers.get("Content-Length", "0") or "0")
-            raw = self.rfile.read(length) if length else b""
-            form = urllib.parse.parse_qs(raw.decode("utf-8", errors="replace"))
-            # Collapse lists to single values.
-            form_single = {k: v[0] for k, v in form.items()}
-            status, extra, body = _route_edit(owner, name, form_single)
-            self._send(status, body, content_type="text/plain", extra_headers=extra)
-            return
-        self._send(404, b"not found", content_type="text/plain")
-
-    def log_message(self, fmt, *args):
-        print(f"pragent-dashboard: {self.address_string()} {fmt % args}", flush=True)
-
-
-# ---------------------------------------------------------------------------
-# Routing regexes (compiled at import time)
-# ---------------------------------------------------------------------------
-
-import re  # noqa: E402
-
-_REPO_PR_RAW_RE = re.compile(
-    r"^/r/([^/]+)/([^/]+)(?:/(\d+)(?:/(raw))?)?/?$"
-)
-_EDIT_RE = re.compile(r"^/r/([^/]+)/([^/]+)/edit/?$")
-
-
-# ---------------------------------------------------------------------------
-# Main
-# ---------------------------------------------------------------------------
-
-
-def main() -> int:
-    if not _feedback_db():
-        print("pragent-dashboard: WARNING: PRAGENT_FEEDBACK_DB not set; dashboard will be empty",
-              flush=True)
-    print("pragent-dashboard: auth via oauth2-proxy (X-Forwarded-User required)", flush=True)
-    server = ThreadingHTTPServer(("0.0.0.0", PORT), Handler)
-    print(f"pragent-dashboard: listening on :{PORT}", flush=True)
-    try:
-        server.serve_forever()
-    except KeyboardInterrupt:
-        pass
-    return 0
-
-
-if __name__ == "__main__":
-    raise SystemExit(main())

pilot/dashboard_data.py

@@ -1,302 +0,0 @@
-"""pragent pilot — dashboard read-only query layer.
-
-Three functions: overview / repo_summary / pr_summary. Each opens the SQLite
-feedback DB via `feedback.init`, runs the queries it needs, and returns plain
-dicts/lists. NEVER writes — that's the dashboard_server's job (via the Gitea
-contents API). This module is what the dashboard_server's templates render.
-
-All three functions are tolerant of a missing or empty DB: they return the
-shaped dict with zeros/empty lists rather than crashing. The dashboard is a
-read-only view; the pilot can boot with no feedback DB and the dashboard
-should still load.
-
-Cost note: `total_cost_usd` is hardcoded to 0.0. Per-review `usage:cost` is
-not in the feedback SQLite — only the raw `review` / `inline_finding` rows
-are stored there. The equivalent-cost calc lives in `ai_review._render_collapsible_usage`
-and only knows about the latest review's tokens. Surfacing a rolled-up dollar
-figure without per-row telemetry would be guessing, so we don't.
-"""
-from __future__ import annotations
-
-import datetime
-import os
-import sqlite3
-
-from pilot import feedback
-
-
-# ---------------------------------------------------------------------------
-# Helpers
-# ---------------------------------------------------------------------------
-
-
-def _empty_overview() -> dict:
-    return {
-        "total_reviews": 0,
-        "total_findings": 0,
-        "total_repos": 0,
-        "last_30d_reviews": 0,
-        "daily": [{"date": _iso_date(i), "count": 0} for i in range(7)],
-        "top_repos": [],
-        "total_cost_usd": 0.0,
-    }
-
-
-def _empty_repo_summary(repo: str) -> dict:
-    return {
-        "repo": repo,
-        "total_runs": 0,
-        "last_run_ts": 0,
-        "runs_by_day": [],
-        "findings_by_severity": {"critical": 0, "high": 0, "medium": 0, "low": 0},
-        "top_findings": [],
-        # NOTE: review rows don't carry a `model` column in the schema today,
-        # so we have nothing to aggregate. When that lands, replace this
-        # empty list with a `SELECT model, COUNT(*) …` over `review`.
-        "models_used": [],
-    }
-
-
-def _empty_pr_summary(repo: str, pr: int) -> dict:
-    return {
-        "repo": repo,
-        "pr": pr,
-        "head_sha": "",
-        "posted_at": 0,
-        "review_id_gitea": None,
-        "body_comment_id": None,
-        "findings": [],
-        # usage isn't on the review row today; ai_review.py renders it
-        # in-memory at review time. Leave empty.
-        "usage": {},
-    }
-
-
-def _iso_date(days_ago: int) -> str:
-    """Return YYYY-MM-DD for `days_ago` days before today (UTC)."""
-    d = datetime.datetime.now(datetime.timezone.utc).date() - datetime.timedelta(days=days_ago)
-    return d.isoformat()
-
-
-def _open_or_none(db_path: str) -> sqlite3.Connection | None:
-    """Open the DB if it exists and looks like a feedback DB. Else None.
-
-    Tolerates missing files (fresh container) and a schema-less file (the
-    operator dropped a stray DB at the path). Returns a connection with
-    Row factory set so callers can use `row["col"]`.
-    """
-    if not db_path or not os.path.exists(db_path):
-        return None
-    try:
-        conn = feedback.init(db_path)
-    except sqlite3.DatabaseError:
-        return None
-    return conn
-
-
-# ---------------------------------------------------------------------------
-# Public API
-# ---------------------------------------------------------------------------
-
-
-def overview(db_path: str) -> dict:
-    """Top-of-page summary: totals + 7-bucket daily sparkline + top 5 repos."""
-    conn = _open_or_none(db_path)
-    if conn is None:
-        return _empty_overview()
-    try:
-        cur = conn.execute("SELECT COUNT(*) FROM review")
-        total_reviews = cur.fetchone()[0]
-        cur = conn.execute("SELECT COUNT(*) FROM inline_finding")
-        total_findings = cur.fetchone()[0]
-        cur = conn.execute("SELECT COUNT(DISTINCT repo) FROM review")
-        total_repos = cur.fetchone()[0]
-
-        # Last 30d window — reviews AND findings posted within the window.
-        ts_30d_ago = int(datetime.datetime.now(datetime.timezone.utc).timestamp()) - 30 * 86400
-        cur = conn.execute("SELECT COUNT(*) FROM review WHERE posted_at >= ?", (ts_30d_ago,))
-        last_30d_reviews = cur.fetchone()[0]
-
-        # 7-bucket daily sparkline, oldest first. Bucket key is UTC date.
-        cur = conn.execute(
-            "SELECT posted_at FROM review WHERE posted_at >= ?",
-            (int(datetime.datetime.now(datetime.timezone.utc).timestamp()) - 7 * 86400,),
-        )
-        buckets: dict[str, int] = {_iso_date(i): 0 for i in range(7)}
-        for (ts,) in cur.fetchall():
-            d = datetime.datetime.fromtimestamp(ts, tz=datetime.timezone.utc).date().isoformat()
-            if d in buckets:
-                buckets[d] += 1
-        daily = [{"date": _iso_date(i), "count": buckets[_iso_date(i)]} for i in range(7)]
-
-        # Top 5 repos by run count, descending. last_seen is the most recent
-        # review timestamp on that repo.
-        cur = conn.execute(
-            "SELECT repo, COUNT(*) AS runs, MAX(posted_at) AS last_seen "
-            "FROM review GROUP BY repo ORDER BY runs DESC, last_seen DESC LIMIT 5"
-        )
-        top_repos = [
-            {"repo": row[0], "run_count": row[1], "last_seen": int(row[2])}
-            for row in cur.fetchall()
-        ]
-
-        return {
-            "total_reviews": total_reviews,
-            "total_findings": total_findings,
-            "total_repos": total_repos,
-            "last_30d_reviews": last_30d_reviews,
-            "daily": daily,
-            "top_repos": top_repos,
-            "total_cost_usd": 0.0,
-        }
-    finally:
-        conn.close()
-
-
-def repo_summary(db_path: str, repo: str) -> dict:
-    """Per-repo drill-down: runs by day, severity histogram, top findings."""
-    conn = _open_or_none(db_path)
-    if conn is None:
-        return _empty_repo_summary(repo)
-    try:
-        cur = conn.execute(
-            "SELECT COUNT(*), MAX(posted_at) FROM review WHERE repo = ?", (repo,)
-        )
-        row = cur.fetchone()
-        total_runs = row[0] or 0
-        last_run_ts = int(row[1]) if row[1] else 0
-
-        # runs_by_day for the last 30 days, oldest first; zero-buckets included.
-        cur = conn.execute(
-            "SELECT posted_at FROM review WHERE repo = ? AND posted_at >= ?",
-            (repo, int(datetime.datetime.now(datetime.timezone.utc).timestamp()) - 30 * 86400),
-        )
-        buckets: dict[str, int] = {}
-        for d in range(30):
-            buckets[_iso_date(d)] = 0  # newest-day mapped to 0; we'll iterate
-        # Re-key: build oldest-first, days_ago goes 29..0
-        oldest_first = {}
-        for d in range(30):
-            oldest_first[_iso_date(29 - d)] = 0
-        for (ts,) in cur.fetchall():
-            d = datetime.datetime.fromtimestamp(ts, tz=datetime.timezone.utc).date().isoformat()
-            if d in oldest_first:
-                oldest_first[d] += 1
-        runs_by_day = [{"date": k, "count": v} for k, v in oldest_first.items()]
-
-        # findings_by_severity — case-insensitive match; bucket unknown as 'low'.
-        cur = conn.execute(
-            "SELECT severity, COUNT(*) FROM inline_finding WHERE repo = ? GROUP BY severity",
-            (repo,),
-        )
-        fbs = {"critical": 0, "high": 0, "medium": 0, "low": 0}
-        for sev, n in cur.fetchall():
-            k = (sev or "").strip().lower()
-            if k not in fbs:
-                k = "low"
-            fbs[k] += n
-
-        # top_findings — top 5 posthashes by occurrence count, joined with
-        # vote rollups via feedback.findings_with_votes.
-        cur = conn.execute(
-            "SELECT f.path, f.line, MAX(f.severity) AS severity, MAX(f.problem) AS problem, "
-            "COUNT(*) AS occurrences, "
-            "COALESCE(SUM(CASE WHEN rct.content = '+1' THEN 1 ELSE 0 END), 0) AS upvotes, "
-            "COALESCE(SUM(CASE WHEN rct.content = '-1' THEN 1 ELSE 0 END), 0) AS downvotes, "
-            "MAX(ts.resolved) AS resolved, "
-            "COALESCE((SELECT COUNT(*) FROM reply WHERE finding_id IN "
-            "  (SELECT id FROM inline_finding WHERE posthash = f.posthash AND repo = f.repo AND path = f.path AND line = f.line)), 0) AS reply_count "
-            "FROM inline_finding f "
-            "LEFT JOIN reaction rct ON rct.comment_id = f.comment_id "
-            "LEFT JOIN thread_state ts ON ts.finding_id = f.id "
-            "WHERE f.repo = ? "
-            "GROUP BY f.posthash, f.repo, f.path, f.line "
-            "ORDER BY occurrences DESC, upvotes DESC LIMIT 5",
-            (repo,),
-        )
-        top_findings = [
-            {
-                "path": r[0],
-                "line": r[1],
-                "severity": r[2],
-                "problem": r[3],
-                "occurrences": r[4],
-                "upvotes": int(r[5] or 0),
-                "downvotes": int(r[6] or 0),
-                "resolved": int(r[7] or 0),
-                "reply_count": int(r[8] or 0),
-            }
-            for r in cur.fetchall()
-        ]
-
-        return {
-            "repo": repo,
-            "total_runs": total_runs,
-            "last_run_ts": last_run_ts,
-            "runs_by_day": runs_by_day,
-            "findings_by_severity": fbs,
-            "top_findings": top_findings,
-            "models_used": [],  # see _empty_repo_summary NOTE
-        }
-    finally:
-        conn.close()
-
-
-def pr_summary(db_path: str, repo: str, pr: int) -> dict:
-    """Per-PR view: meta + every finding the bot ever posted on that PR."""
-    conn = _open_or_none(db_path)
-    if conn is None:
-        return _empty_pr_summary(repo, pr)
-    try:
-        cur = conn.execute(
-            "SELECT head_sha, posted_at, review_id_gitea, body_comment_id "
-            "FROM review WHERE repo = ? AND pr = ? ORDER BY posted_at DESC LIMIT 1",
-            (repo, pr),
-        )
-        row = cur.fetchone()
-        if row is None:
-            return _empty_pr_summary(repo, pr)
-        head_sha, posted_at, review_id_gitea, body_comment_id = row
-
-        cur = conn.execute(
-            "SELECT f.path, f.line, f.severity, f.problem, f.fix, f.suggestion, "
-            "COALESCE(SUM(CASE WHEN rct.content = '+1' THEN 1 ELSE 0 END), 0) AS upvotes, "
-            "COALESCE(SUM(CASE WHEN rct.content = '-1' THEN 1 ELSE 0 END), 0) AS downvotes, "
-            "MAX(ts.resolved) AS resolved, "
-            "COALESCE((SELECT COUNT(*) FROM reply WHERE finding_id = f.id), 0) AS reply_count "
-            "FROM inline_finding f "
-            "LEFT JOIN reaction rct ON rct.comment_id = f.comment_id "
-            "LEFT JOIN thread_state ts ON ts.finding_id = f.id "
-            "WHERE f.repo = ? AND f.pr = ? "
-            "GROUP BY f.id "
-            "ORDER BY f.path, f.line",
-            (repo, pr),
-        )
-        findings = [
-            {
-                "path": r[0],
-                "line": r[1],
-                "severity": r[2],
-                "problem": r[3],
-                "fix": r[4],
-                "suggestion": r[5],
-                "upvotes": int(r[6] or 0),
-                "downvotes": int(r[7] or 0),
-                "resolved": int(r[8] or 0),
-                "reply_count": int(r[9] or 0),
-            }
-            for r in cur.fetchall()
-        ]
-
-        return {
-            "repo": repo,
-            "pr": pr,
-            "head_sha": head_sha,
-            "posted_at": int(posted_at),
-            "review_id_gitea": review_id_gitea,
-            "body_comment_id": body_comment_id,
-            "findings": findings,
-            "usage": {},
-        }
-    finally:
-        conn.close()

pilot/eval_bootstrap.py

@@ -0,0 +1,299 @@
+#!/usr/bin/env python3
+"""pragent pilot — one-time Langfuse project setup for evaluation.
+
+Three jobs, each idempotent so it can be re-run after any change:
+
+  1. **Score configs.** Registers the schema for every score pragent emits
+     (`eval_scores.SCORE_CONFIGS` + `feedback_scores.SCORE_CONFIGS`). Without
+     these the scores still ingest, but nothing stops a later scorer writing
+     `severity_max="HIGH"` beside today's `"high"` and quietly splitting one
+     series into two. Configs are immutable in Langfuse — a name that already
+     exists is left alone rather than updated.
+
+  2. **Dataset.** Seeds `pragent-reviews` from `feedback.db`: one item per PR
+     the reviewer has actually run on, carrying the repo/PR/sha as input and
+     the findings it posted as `expectedOutput`.
+
+     Read `expectedOutput` here as "what the reviewer said last time", not "what
+     is correct" — no human has labelled any of it. It is a regression baseline:
+     re-run a candidate model over these PRs and the diff against this column is
+     the behaviour change. Promoting an item to real ground truth means a human
+     editing it after reviewing the PR, which is what the dataset view is for.
+
+  3. **Trace backfill** (`--backfill-traces`). Scores only ride along with new
+     reviews, so without this the charts stay empty until the next PR lands.
+     Every trace `langfuse_trace` has ever written already carries the finding
+     count, the severity histogram and the cost in its metadata, which is
+     everything four of the five scorers need. `dropped_findings` is absent from
+     historical traces and is left unscored rather than backfilled as zero.
+
+  4. **Reports** what it found, so the gap between "reviews recorded" and
+     "reviews with human feedback" is visible rather than assumed.
+
+Usage:
+    LANGFUSE_HOST=... LANGFUSE_PUBLIC_KEY=... LANGFUSE_SECRET_KEY=... \\
+        python3 eval_bootstrap.py --db /data/feedback.db
+"""
+from __future__ import annotations
+
+import argparse
+import base64
+import json
+import os
+import sqlite3
+import sys
+import urllib.error
+import urllib.request
+
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+
+import eval_scores  # noqa: E402
+import feedback_scores  # noqa: E402
+
+DATASET_NAME = "pragent-reviews"
+
+
+def _conf() -> tuple[str, str, str]:
+    host = (os.environ.get("LANGFUSE_HOST") or "").strip().rstrip("/")
+    pk = (os.environ.get("LANGFUSE_PUBLIC_KEY") or "").strip()
+    sk = (os.environ.get("LANGFUSE_SECRET_KEY") or "").strip()
+    if not host or not pk or not sk:
+        raise SystemExit("LANGFUSE_HOST / LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY must be set")
+    return host, pk, sk
+
+
+def _call(method: str, path: str, body: dict | None = None, timeout: float = 20.0):
+    host, pk, sk = _conf()
+    auth = base64.b64encode(f"{pk}:{sk}".encode()).decode("ascii")
+    data = json.dumps(body).encode() if body is not None else None
+    req = urllib.request.Request(
+        host + path,
+        data=data,
+        headers={
+            "Content-Type": "application/json",
+            "Authorization": f"Basic {auth}",
+            "User-Agent": "pragent-pilot/1.0",
+        },
+        method=method,
+    )
+    try:
+        with urllib.request.urlopen(req, timeout=timeout) as resp:
+            raw = resp.read()
+            return resp.status, (json.loads(raw) if raw else None)
+    except urllib.error.HTTPError as e:
+        return e.code, e.read()[:400].decode("utf-8", "replace")
+
+
+# ---------------------------------------------------------------------------
+# 1. Score configs
+# ---------------------------------------------------------------------------
+
+def ensure_score_configs() -> dict:
+    status, existing = _call("GET", "/api/public/score-configs?limit=100")
+    have = set()
+    if status == 200 and isinstance(existing, dict):
+        have = {c.get("name") for c in existing.get("data", [])}
+
+    created, skipped, failed = [], [], []
+    for cfg in list(eval_scores.SCORE_CONFIGS) + list(feedback_scores.SCORE_CONFIGS):
+        if cfg["name"] in have:
+            skipped.append(cfg["name"])
+            continue
+        st, resp = _call("POST", "/api/public/score-configs", cfg)
+        if st in (200, 201):
+            created.append(cfg["name"])
+        else:
+            failed.append({"name": cfg["name"], "status": st, "error": resp})
+    return {"created": created, "already_present": skipped, "failed": failed}
+
+
+# ---------------------------------------------------------------------------
+# 2. Dataset from recorded reviews
+# ---------------------------------------------------------------------------
+
+def read_review_items(db_path: str) -> list[dict]:
+    """One dataset item per (repo, pr) the reviewer has run on.
+
+    Keyed on the PR rather than on each individual review row: the same PR is
+    re-reviewed on every push, and 113 rows over 26 PRs would make a benchmark
+    that is 4x redundant and weighted towards whichever PR churned most.
+    """
+    conn = sqlite3.connect(db_path)
+    conn.row_factory = sqlite3.Row
+    try:
+        prs = conn.execute(
+            """
+            SELECT repo, pr, MAX(posted_at) AS last_seen, COUNT(*) AS reviews,
+                   MAX(head_sha) AS head_sha
+            FROM review GROUP BY repo, pr ORDER BY repo, pr
+            """
+        ).fetchall()
+        items = []
+        for row in prs:
+            findings = conn.execute(
+                """
+                SELECT path, line, severity, problem, fix
+                FROM inline_finding WHERE repo = ? AND pr = ?
+                ORDER BY path, line
+                """,
+                (row["repo"], row["pr"]),
+            ).fetchall()
+            items.append(
+                {
+                    "id": f'{row["repo"]}#{row["pr"]}',
+                    "input": {
+                        "repo": row["repo"],
+                        "pr": int(row["pr"]),
+                        "head_sha": row["head_sha"],
+                    },
+                    "expectedOutput": {
+                        "findings": [dict(f) for f in findings],
+                        "finding_count": len(findings),
+                    },
+                    "metadata": {
+                        "reviews_run": int(row["reviews"]),
+                        "last_reviewed_at": int(row["last_seen"]),
+                        # Flags that this row is the reviewer's own past output,
+                        # not a human judgement. Filter on it before anyone
+                        # treats the dataset as ground truth.
+                        "labelled_by_human": False,
+                    },
+                }
+            )
+        return items
+    finally:
+        conn.close()
+
+
+def ensure_dataset(items: list[dict], name: str = DATASET_NAME) -> dict:
+    st, _ = _call(
+        "POST",
+        "/api/public/datasets",
+        {
+            "name": name,
+            "description": (
+                "PRs the pragent pilot has reviewed, seeded from feedback.db. "
+                "expectedOutput is the reviewer's own prior output — a regression "
+                "baseline, not human-verified ground truth."
+            ),
+            "metadata": {"source": "feedback.db", "seeded_by": "eval_bootstrap.py"},
+        },
+    )
+    # A duplicate name is fine: the dataset already exists from an earlier run.
+    dataset_ok = st in (200, 201, 409)
+
+    created, failed = 0, []
+    for item in items:
+        body = {
+            "datasetName": name,
+            "id": item["id"],  # idempotent: same PR updates rather than duplicates
+            "input": item["input"],
+            "expectedOutput": item["expectedOutput"],
+            "metadata": item["metadata"],
+        }
+        ist, resp = _call("POST", "/api/public/dataset-items", body)
+        if ist in (200, 201):
+            created += 1
+        else:
+            failed.append({"item": item["id"], "status": ist, "error": resp})
+    return {"dataset": name, "dataset_created": dataset_ok, "items_upserted": created, "failed": failed}
+
+
+# ---------------------------------------------------------------------------
+# 3. Backfill scores onto traces that predate the scorers
+# ---------------------------------------------------------------------------
+
+def _synth_findings(severities: dict) -> list[dict]:
+    """Rebuild a findings list from a trace's severity histogram.
+
+    Only severity matters to the scorers, and that is all the histogram kept.
+    Reconstructing placeholders is honest here because every scorer being
+    backfilled reads nothing else off a finding.
+    """
+    out = []
+    for sev, count in (severities or {}).items():
+        out.extend({"severity": sev} for _ in range(int(count)))
+    return out
+
+
+def backfill_traces(limit_pages: int = 20) -> dict:
+    import eval_scores as es
+
+    scored, skipped, events = 0, 0, []
+    page = 1
+    while page <= limit_pages:
+        st, resp = _call("GET", f"/api/public/traces?limit=50&page={page}&name=pr-review")
+        if st != 200 or not isinstance(resp, dict):
+            break
+        rows = resp.get("data") or []
+        if not rows:
+            break
+        for tr in rows:
+            meta = tr.get("metadata") or {}
+            severities = meta.get("severities") or {}
+            count = meta.get("findings")
+            if count is None:
+                skipped += 1
+                continue
+            findings = _synth_findings(severities)
+            # The histogram is authoritative when present; a trace that recorded
+            # a count but no histogram still scores its rate.
+            if not findings and count:
+                findings = [{"severity": "medium"} for _ in range(int(count))]
+            batch = es.build_scores(
+                trace_id=tr["id"],
+                findings=findings,
+                environment=tr.get("environment") or "default",
+                cost_usd=(tr.get("totalCost") or meta.get("provider_cost_usd")),
+                timestamp=tr.get("timestamp"),
+                comment="backfilled from trace metadata",
+            )
+            events.extend(batch)
+            scored += 1
+        page += 1
+
+    posted = False
+    status = None
+    if events:
+        import langfuse_trace
+
+        host, pk, sk = _conf()
+        # Chunked: one 2000-event POST is refused, and a partial backfill that
+        # reports success is worse than a slow one.
+        for i in range(0, len(events), 200):
+            status = langfuse_trace._post(host, pk, sk, events[i:i + 200], 30.0)
+            posted = status in (200, 201, 207)
+            if not posted:
+                break
+    return {"traces_scored": scored, "traces_skipped": skipped, "scores": len(events),
+            "posted": posted, "http_status": status}
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description="Bootstrap Langfuse evaluation for the pragent pilot")
+    ap.add_argument("--db", default=os.environ.get("PRAGENT_FEEDBACK_DB", "/data/feedback.db"))
+    ap.add_argument("--skip-dataset", action="store_true")
+    ap.add_argument("--skip-configs", action="store_true")
+    ap.add_argument("--backfill-traces", action="store_true",
+                    help="score traces written before the scorers existed")
+    args = ap.parse_args()
+
+    out: dict = {}
+    if not args.skip_configs:
+        out["score_configs"] = ensure_score_configs()
+    if not args.skip_dataset:
+        items = read_review_items(args.db)
+        out["dataset"] = ensure_dataset(items)
+        out["dataset"]["items_read"] = len(items)
+    if args.backfill_traces:
+        out["trace_backfill"] = backfill_traces()
+    print(json.dumps(out, indent=2))
+
+    failed = (out.get("score_configs", {}).get("failed") or []) + (
+        out.get("dataset", {}).get("failed") or []
+    )
+    return 1 if failed else 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

pilot/eval_judges.py

@@ -0,0 +1,314 @@
+#!/usr/bin/env python3
+"""pragent pilot — LLM-as-a-judge evaluators for the reviewer.
+
+The deterministic scorers in `eval_scores.py` measure *behaviour*: how many
+findings, how severe, how much they cost. None of them can say whether a
+finding was any good. With no human labels in `feedback.db`, a judge is the
+only thing that can — so these two ask the questions that need no ground truth,
+only the review itself:
+
+  `finding_actionability` — is each finding concrete enough to act on? A
+    reviewer that says "consider improving error handling" at file level is
+    indistinguishable from a useful one by finding count alone. This is the
+    failure mode a cheap model degrades into first.
+
+  `review_self_consistency` — does the summary agree with the findings it
+    posted? Claiming "no issues found" above a list of two criticals, or
+    describing a problem in prose that never became a finding, is a defect the
+    reviewer can commit entirely on its own.
+
+Neither judge is asked whether a finding is *correct*. That needs the diff,
+which these traces do not carry, and a judge asked to rule on correctness from
+a summary alone will confabulate. Accuracy stays an open question until humans
+start labelling — which is what `feedback_scores.py` is there to capture.
+
+**The judge is a different model from the reviewer.** The reviewer runs
+MiniMax-M2.7; the judge runs kimi-k2.7-code through the same headroom hub. A
+model grading its own output agrees with itself for reasons that have nothing
+to do with quality.
+
+Evaluators score *observations*, and their variable mapping reads the
+observation's own input/output — which is why `langfuse_trace` now writes the
+review onto the generation and not just onto the trace.
+
+Usage:
+    LANGFUSE_HOST=... LANGFUSE_PUBLIC_KEY=... LANGFUSE_SECRET_KEY=... \\
+        python3 eval_judges.py --dry-run
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+
+import eval_bootstrap as eb  # noqa: E402
+
+# The headroom hub in front of the local Ollama, plus a small pass-through
+# proxy (`judge-proxy` on 8802) that patches every `thinking` content block
+# to carry the `signature` field Langfuse's Anthropic adapter requires. The
+# underlying model is kimi-k2.7-code through the hub on 8790; the proxy fixes
+# the shape so Mastra's Zod parse stops failing.
+JUDGE_PROVIDER = "headroom-ollama"
+JUDGE_BASE_URL = os.environ.get("PRAGENT_JUDGE_BASE_URL", "http://100.74.17.70:8802")
+JUDGE_API_KEY = os.environ.get("PRAGENT_JUDGE_API_KEY", "ollama")
+JUDGE_MODEL = os.environ.get("PRAGENT_JUDGE_MODEL", "kimi-k2.7-code:cloud")
+
+# The trace names this project emits (`pr-review` on the trace, `opencode-review`
+# on the generation). Filter on `traceName` rather than observation `name` — the
+# observation-rule schema only exposes `traceName` as a stringOptions column, and
+# every observation inside these traces is the review itself, so the narrowness
+# is the same.
+REVIEW_TRACE_NAMES = ["pr-review", "opencode-review"]
+
+
+def _model_config() -> dict:
+    return {"provider": JUDGE_PROVIDER, "model": JUDGE_MODEL}
+
+
+JUDGES = [
+    {
+        "name": "finding_actionability",
+        "prompt": (
+            "You are auditing the output of an automated code reviewer.\n\n"
+            "PR under review:\n{{input}}\n\n"
+            "What the reviewer produced:\n{{output}}\n\n"
+            "Rate how ACTIONABLE the findings are, from 0 to 1. A finding is "
+            "actionable when a developer could act on it without asking a "
+            "follow-up question: it points at a specific location, names a "
+            "concrete problem, and proposes a fix that could be applied.\n\n"
+            "Score 1.0 when every finding is specific and fixable. Score around "
+            "0.5 when findings identify a real area but leave the developer to "
+            "work out what to change. Score near 0.0 when findings are generic "
+            "advice that would apply to almost any pull request.\n\n"
+            "Judge only specificity and actionability. You cannot see the diff, "
+            "so do NOT attempt to judge whether a finding is factually correct, "
+            "and do not penalise a finding for being one you cannot verify.\n\n"
+            "If the reviewer reported no findings at all, return 1.0 and say in "
+            "your reasoning that there was nothing to judge — a silent review is "
+            "measured by finding_rate, not here."
+        ),
+        "outputDefinition": {
+            "dataType": "NUMERIC",
+            "minValue": 0,
+            "maxValue": 1,
+            "reasoning": {
+                "description": (
+                    "Name the least actionable finding and say what it would "
+                    "need in order to be acted on."
+                )
+            },
+            "score": {"description": "0 = generic advice, 1 = every finding is specific and fixable."},
+        },
+    },
+    {
+        "name": "review_self_consistency",
+        "prompt": (
+            "You are auditing the output of an automated code reviewer.\n\n"
+            "PR under review:\n{{input}}\n\n"
+            "What the reviewer produced:\n{{output}}\n\n"
+            "The output contains a prose `summary` and a list of `findings`. "
+            "Decide whether the summary is CONSISTENT with the findings.\n\n"
+            "Inconsistent means, for example: the summary says no issues were "
+            "found while findings are listed; the summary describes a problem "
+            "that never became a finding; the summary characterises the severity "
+            "of the findings in a way the findings themselves contradict; or the "
+            "summary refers to files that appear in no finding and in no part of "
+            "the PR description.\n\n"
+            "A summary that adds context beyond the findings is NOT inconsistent "
+            "as long as nothing in it contradicts them. A review that found "
+            "nothing and says so is consistent.\n\n"
+            "You cannot see the diff. Judge the summary against the findings and "
+            "the PR title only — never against what you imagine the code does."
+        ),
+        "outputDefinition": {
+            "dataType": "BOOLEAN",
+            "reasoning": {
+                "description": "Quote the part of the summary that conflicts with the findings, if any."
+            },
+            "score": {"description": "true = summary agrees with the findings, false = it contradicts them."},
+        },
+    },
+]
+
+# Both judges read the observation's own input/output.
+MAPPING = [
+    {"variable": "input", "source": "input"},
+    {"variable": "output", "source": "output"},
+]
+
+
+# ---------------------------------------------------------------------------
+# LLM connection
+# ---------------------------------------------------------------------------
+
+def ensure_llm_connection() -> dict:
+    """Point the project at the judge model. Upserted on `provider`."""
+    body = {
+        "provider": JUDGE_PROVIDER,
+        "adapter": "anthropic",
+        "baseURL": JUDGE_BASE_URL,
+        "secretKey": JUDGE_API_KEY,
+        "customModels": [JUDGE_MODEL],
+        # The hub serves two local models and none of Anthropic's, so the
+        # default catalogue would be a list of models that all fail on use.
+        "withDefaultModels": False,
+    }
+    st, resp = eb._call("PUT", "/api/public/llm-connections", body)
+    return {"status": st, "ok": st in (200, 201), "provider": JUDGE_PROVIDER,
+            "error": None if st in (200, 201) else resp}
+
+
+# ---------------------------------------------------------------------------
+# Evaluators
+# ---------------------------------------------------------------------------
+
+def existing_evaluators() -> dict[str, str]:
+    """name -> id for evaluators already in the project."""
+    out: dict[str, str] = {}
+    st, body = eb._call("GET", "/api/public/unstable/evaluators?limit=100")
+    if st == 200 and isinstance(body, dict):
+        for ev in body.get("data") or []:
+            out[ev.get("name")] = ev.get("id")
+    return out
+
+
+def ensure_evaluators() -> dict:
+    """Create each judge if no version exists for the name yet.
+
+    POST /evaluators with a name that already exists creates a new version, not
+    a no-op — re-running this script would pile up versions until the page
+    listing them is unreadable. Skip when an evaluator of that name is present.
+    """
+    created, skipped, failed = {}, [], []
+    existing = set(existing_evaluators())
+    for judge in JUDGES:
+        if judge["name"] in existing:
+            skipped.append(judge["name"])
+            continue
+        body = {
+            "type": "llm_as_judge",
+            "name": judge["name"],
+            "prompt": judge["prompt"],
+            "outputDefinition": judge["outputDefinition"],
+            "modelConfig": _model_config(),
+        }
+        st, resp = eb._call("POST", "/api/public/unstable/evaluators", body, timeout=60.0)
+        if st in (200, 201) and isinstance(resp, dict):
+            created[judge["name"]] = resp.get("id")
+        else:
+            failed.append({"name": judge["name"], "status": st, "error": resp})
+    return {"created": created, "skipped": skipped, "failed": failed}
+
+
+# ---------------------------------------------------------------------------
+# Rules — what gets judged, and how often
+# ---------------------------------------------------------------------------
+
+def rule_body(name: str, judge_name: str, sampling: float) -> dict:
+    """POST /evaluation-rules shape for an LLM-as-judge trace rule.
+
+    Target is `trace` rather than `observation` on purpose: the standard
+    `/api/public/ingestion` path that ships review traces here feeds only
+    the trace-upsert queue, and `evalService.createEvalJobs` only creates
+    jobs for `targetObject ∈ {TRACE, DATASET}`. Observation rules are
+    triggered exclusively from the OTel ingestion pipeline, which this
+    pilot does not use. A trace rule reads the trace's own input/output —
+    `langfuse_trace` already writes `_review_input`/`_review_output` onto
+    the trace body for exactly this reason.
+
+    Mapping is required at both the rule root (server validates it there)
+    and inside `evaluator` (the API echoes it back).
+    """
+    return {
+        "name": name,
+        "enabled": True,
+        "target": "trace",
+        "sampling": sampling,
+        "filter": [
+            {"column": "traceName", "operator": "any of",
+             "value": REVIEW_TRACE_NAMES, "type": "stringOptions"},
+        ],
+        "evaluator": {
+            "name": judge_name,
+            "scope": "project",
+            "variableMapping": MAPPING,
+        },
+        "mapping": MAPPING,
+    }
+
+
+def ensure_rules(evaluator_ids: dict[str, str], sampling: float) -> dict:
+    """Idempotent: existing rules with the same name are skipped, not duplicated.
+
+    The API has no `name`-keyed upsert; the convention is to POST once and
+    re-run the script to verify the response. A duplicate POST raises 409.
+    """
+    created, failed, skipped = [], [], []
+    existing = existing_rule_names()
+    for name, eid in evaluator_ids.items():
+        if not eid:
+            continue
+        rule_name = f"{name}-on-reviews"
+        if rule_name in existing:
+            skipped.append(name)
+            continue
+        st, resp = eb._call(
+            "POST", "/api/public/unstable/evaluation-rules",
+            rule_body(rule_name, name, sampling), timeout=60.0,
+        )
+        if st in (200, 201):
+            created.append(name)
+        else:
+            failed.append({"rule": name, "status": st, "error": resp})
+    return {"created": created, "failed": failed, "skipped": skipped}
+
+
+def existing_rule_names() -> set[str]:
+    """Names of observation-target rules already in the project."""
+    out: set[str] = set()
+    st, body = eb._call("GET", "/api/public/unstable/evaluation-rules?limit=100")
+    if st == 200 and isinstance(body, dict):
+        for r in body.get("data") or []:
+            if r.get("target") == "observation":
+                out.add(r.get("name"))
+    return out
+
+
+def main(argv: list[str] | None = None) -> int:
+    ap = argparse.ArgumentParser(description=__doc__)
+    ap.add_argument("--sampling", type=float, default=1.0,
+                    help="fraction of matching observations to judge (default: all)")
+    ap.add_argument("--skip-connection", action="store_true")
+    ap.add_argument("--dry-run", action="store_true")
+    args = ap.parse_args(argv)
+
+    if args.dry_run:
+        print(json.dumps({
+            "would_connect": {"provider": JUDGE_PROVIDER, "baseURL": JUDGE_BASE_URL,
+                              "model": JUDGE_MODEL},
+            "would_create": [j["name"] for j in JUDGES],
+            "existing_evaluators": sorted(existing_evaluators()),
+            "sampling": args.sampling,
+        }, indent=2))
+        return 0
+
+    report = {}
+    if not args.skip_connection:
+        report["llm_connection"] = ensure_llm_connection()
+    report["evaluators"] = ensure_evaluators()
+    ids = dict(report["evaluators"]["created"])
+    # Fall back to whatever is already registered, so a re-run still wires rules.
+    for name, eid in existing_evaluators().items():
+        ids.setdefault(name, eid)
+    report["rules"] = ensure_rules(
+        {j["name"]: ids.get(j["name"]) for j in JUDGES}, args.sampling
+    )
+    print(json.dumps(report, indent=2))
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

pilot/eval_scores.py

@@ -0,0 +1,233 @@
+#!/usr/bin/env python3
+"""pragent pilot — deterministic review scorers.
+
+Four numbers computed from a review that already happened, shipped to Langfuse
+as scores on the review's trace. All are derived from data the reviewer already
+has in hand: no LLM judge, no ground truth, no extra token spend.
+
+Why these four and not `helpfulness`/`quality`
+----------------------------------------------
+They come from what the recorded reviews actually did, not from a generic eval
+checklist:
+
+  * `severity_info_ratio` — of the findings ever posted to a PR, effectively all
+    landed at `info`. Either the model will not commit to a severity or the
+    per-repo `severity_threshold` is filtering the rest out. Trending the ratio
+    per model says which.
+  * `finding_rate` — most reviews post nothing at all. Silence on clean code is
+    the goal; silence because the run degraded is a failure. Same output, two
+    causes, and only the rate over time separates them.
+  * `dropped_findings` — `ai_review.parse_findings` discards any finding whose
+    `path`/`line` is unusable. That happens silently, so a model that emits ten
+    findings at invalid locations is indistinguishable from one that found
+    nothing. This is the only signal here that measures the *model's* output
+    rather than the review's.
+  * `cost_per_finding` — the equivalent-cost number is already trended per
+    review; per finding is what actually compares two models, since a cheaper
+    model that finds nothing is not cheaper.
+
+None of these say whether a finding was *correct*. That needs labels, and the
+labels come from `feedback_scores.py` once maintainers start reacting to review
+comments. Read these as behavioural drift detectors, not as accuracy.
+
+Fail-open, like every other telemetry path here: a scorer that raises returns no
+score rather than failing the review.
+"""
+from __future__ import annotations
+
+import uuid
+from datetime import datetime, timezone
+
+# Mirrors ai_review.SEVERITY_RANK. Duplicated rather than imported because this
+# module is also run standalone (backfill) where ai_review's import side effects
+# are unwanted.
+SEVERITY_RANK = {"info": -1, "trivial": 0, "low": 1, "medium": 2, "high": 3, "critical": 4}
+
+# Findings at or below this rank are "the model declined to commit". `trivial`
+# and `info` are advisory by the reviewer's own prompt contract.
+_ADVISORY_MAX_RANK = 0
+
+# Score names. Named for what is measured, not for the mechanism producing it —
+# these land on every trace and become the axis of every chart.
+FINDING_RATE = "finding_rate"
+SEVERITY_INFO_RATIO = "severity_info_ratio"
+SEVERITY_MAX = "severity_max"
+DROPPED_FINDINGS = "dropped_findings"
+COST_PER_FINDING = "cost_per_finding"
+
+
+def _sev(f: dict) -> str:
+    return str(f.get("severity") or "medium").strip().lower()
+
+
+def finding_rate(findings: list[dict] | None) -> float:
+    """How many findings this review posted. 0.0 is the restraint case."""
+    return float(len(findings or []))
+
+
+def severity_info_ratio(findings: list[dict] | None) -> float | None:
+    """Share of findings the model rated advisory (`info`/`trivial`).
+
+    `None` for a review with no findings — a ratio over an empty set is not 0,
+    it is undefined, and charting it as 0 would read as "perfectly calibrated".
+    """
+    fs = findings or []
+    if not fs:
+        return None
+    advisory = sum(1 for f in fs if SEVERITY_RANK.get(_sev(f), 2) <= _ADVISORY_MAX_RANK)
+    return round(advisory / len(fs), 4)
+
+
+def severity_max(findings: list[dict] | None) -> str:
+    """Highest severity present, or `none` when the review was silent.
+
+    Categorical on purpose: the useful question is "did this review ever surface
+    something serious", and an average of severity ranks answers nothing.
+    """
+    fs = findings or []
+    if not fs:
+        return "none"
+    top = max(fs, key=lambda f: SEVERITY_RANK.get(_sev(f), 2))
+    sev = _sev(top)
+    return sev if sev in SEVERITY_RANK else "medium"
+
+
+def dropped_findings(raw_count: int | None, kept_count: int | None) -> float | None:
+    """Findings the model emitted that the parser could not use.
+
+    `raw_count` is what came back in the JSON; `kept_count` is what survived
+    `_normalize_finding`. `None` when the caller could not determine the raw
+    count — better no score than a fabricated zero.
+    """
+    if raw_count is None or kept_count is None:
+        return None
+    return float(max(0, int(raw_count) - int(kept_count)))
+
+
+def cost_per_finding(cost_usd: float | None, findings: list[dict] | None) -> float | None:
+    """Equivalent USD spent per finding posted.
+
+    `None` when nothing could be priced. A silent review divides by one, not by
+    zero: the run still cost money, and attributing that whole cost to "found
+    nothing" is the honest reading.
+    """
+    if cost_usd is None:
+        return None
+    try:
+        c = float(cost_usd)
+    except (TypeError, ValueError):
+        return None
+    return round(c / max(1, len(findings or [])), 6)
+
+
+def build_scores(
+    *,
+    trace_id: str,
+    findings: list[dict] | None,
+    environment: str,
+    cost_usd: float | None = None,
+    dropped_count: float | None = None,
+    timestamp: str | None = None,
+    comment: str = "",
+) -> list[dict]:
+    """The `score-create` ingestion events for one review.
+
+    `dropped_count` must be measured at parse time, not here: by the time
+    `findings` reaches this function the per-repo config has already filtered it
+    by severity threshold and `max_findings`, and those drops are the config
+    working as intended, not the model emitting garbage.
+
+    Returns [] rather than raising if something is unscoreable — scores are
+    telemetry and must never cost a review.
+    """
+    # The ingestion envelope requires a timestamp on every event; omitting it
+    # gets the whole batch rejected with an HTTP 207 whose per-event 400s are
+    # easy to mistake for success.
+    ts = timestamp or datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
+    out: list[dict] = []
+
+    def add(name: str, value, data_type: str) -> None:
+        if value is None:
+            return
+        body = {
+            "id": str(uuid.uuid4()),
+            "traceId": trace_id,
+            "name": name,
+            "dataType": data_type,
+            "environment": environment,
+        }
+        if data_type == "CATEGORICAL":
+            body["value"] = str(value)
+        else:
+            body["value"] = float(value)
+        if comment:
+            body["comment"] = comment
+        out.append(
+            {
+                "id": str(uuid.uuid4()),
+                "type": "score-create",
+                "timestamp": ts,
+                "body": body,
+            }
+        )
+
+    try:
+        add(FINDING_RATE, finding_rate(findings), "NUMERIC")
+        add(SEVERITY_INFO_RATIO, severity_info_ratio(findings), "NUMERIC")
+        add(SEVERITY_MAX, severity_max(findings), "CATEGORICAL")
+        add(DROPPED_FINDINGS, dropped_count, "NUMERIC")
+        add(COST_PER_FINDING, cost_per_finding(cost_usd, findings), "NUMERIC")
+    except Exception:  # pragma: no cover - defensive
+        return out
+    return out
+
+
+# ---------------------------------------------------------------------------
+# Score configs — the schema these scores must comply with
+# ---------------------------------------------------------------------------
+
+# Registered once per project via `eval_bootstrap.py`. Without configs the
+# scores still ingest, but nothing constrains a future scorer from writing
+# `severity_max="HIGH"` next to today's `"high"` and silently splitting the
+# series in two.
+SCORE_CONFIGS = [
+    {
+        "name": FINDING_RATE,
+        "dataType": "NUMERIC",
+        "minValue": 0,
+        "description": "Findings posted by one review. 0 = the reviewer stayed silent.",
+    },
+    {
+        "name": SEVERITY_INFO_RATIO,
+        "dataType": "NUMERIC",
+        "minValue": 0,
+        "maxValue": 1,
+        "description": "Share of a review's findings rated info/trivial. High = the model is not committing to a severity.",
+    },
+    {
+        "name": SEVERITY_MAX,
+        "dataType": "CATEGORICAL",
+        "categories": [
+            {"label": "none", "value": 0},
+            {"label": "info", "value": 1},
+            {"label": "trivial", "value": 2},
+            {"label": "low", "value": 3},
+            {"label": "medium", "value": 4},
+            {"label": "high", "value": 5},
+            {"label": "critical", "value": 6},
+        ],
+        "description": "Highest severity surfaced by one review; 'none' when it posted nothing.",
+    },
+    {
+        "name": DROPPED_FINDINGS,
+        "dataType": "NUMERIC",
+        "minValue": 0,
+        "description": "Findings the model emitted that the parser rejected for an unusable path/line.",
+    },
+    {
+        "name": COST_PER_FINDING,
+        "dataType": "NUMERIC",
+        "minValue": 0,
+        "description": "Equivalent USD per finding posted. Silent reviews divide by 1, not 0.",
+    },
+]

pilot/feedback_analyze.py

@@ -163,7 +163,7 @@ def _md_escape(s: str) -> str:
 def analyze(db_path: str, *, since_ts: Optional[int] = None,
             as_json: bool = False) -> str:
     """Build the daily report. Returns a markdown string by default;
-    `as_json=True` returns a structured dict (for tests + dashboards)."""
+    `as_json=True` returns a structured dict (for tests + automation)."""
     conn = feedback.init(db_path)
     try:
         findings = list(feedback.findings_with_votes(conn, since_ts=since_ts))

pilot/feedback_scores.py

@@ -0,0 +1,247 @@
+#!/usr/bin/env python3
+"""pragent pilot — feedback DB to Langfuse scores.
+
+`feedback.db` already records every reaction, thread resolution and reply a
+maintainer leaves on a bot comment. That is the only ground truth pragent has
+about whether a finding was any good, and until now it went to a markdown report
+nobody reads and nowhere else. This ships it to Langfuse as session-level
+scores, so "was the reviewer right" sits on the same axis as "what did it cost".
+
+Session, not trace
+------------------
+`langfuse_trace` sets `sessionId` to `"{repo}#{pr}"` and lets the trace id be a
+fresh uuid per review. Feedback arrives days later against a PR, not against one
+particular re-run of the reviewer, and nothing in `feedback.db` records which
+trace produced which comment. Scoring the session is therefore both the
+available join and the honest granularity: this is feedback on the review of
+this PR, not on one invocation.
+
+Two scores, deliberately separated
+----------------------------------
+  * `review_engagement` — the share of a PR's findings that got any human
+    response at all. This is a signal about the *feedback loop*, not the
+    reviewer: at the time of writing it is 0.0 across all 113 recorded reviews,
+    which is exactly the fact that makes an accuracy metric impossible today.
+    It must be watched first, because every other quality number is vapour
+    until it moves.
+  * `review_acceptance` — net verdict over the findings that *did* get a
+    response: (upvotes + resolved) - (downvotes + negation replies), normalised
+    to -1..1. Computed only over engaged findings, so an ignored review scores
+    `None` rather than 0. Zero would read as "humans judged this exactly
+    neutral"; the truth is nobody looked.
+
+Fail-open and idempotent. Score ids are derived from (repo, pr, name) so a
+re-run overwrites rather than duplicates.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sqlite3
+import sys
+import uuid
+from datetime import datetime, timezone
+
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+
+from feedback_harvest import classify_reaction, _is_negation_reply  # noqa: E402
+
+REVIEW_ENGAGEMENT = "review_engagement"
+REVIEW_ACCEPTANCE = "review_acceptance"
+
+# Stable namespace so the same (repo, pr, score) always produces the same score
+# id — Langfuse treats a repeated id as an update, which is what a backfill of a
+# still-accumulating PR should do.
+_NS = uuid.UUID("6f1d9c2e-4a77-4f2a-9c1a-0d3b5e8a7c41")
+
+
+def _score_id(repo: str, pr: int, name: str) -> str:
+    return str(uuid.uuid5(_NS, f"{repo}#{pr}#{name}"))
+
+
+def collect_pr_feedback(conn: sqlite3.Connection, repo: str, pr: int) -> dict:
+    """Tally one PR's findings and the human responses attached to them.
+
+    Returns counts only — the scoring maths lives in `score_pr` so it can be
+    tested without a database.
+    """
+    rows = conn.execute(
+        "SELECT id, comment_id FROM inline_finding WHERE repo = ? AND pr = ?",
+        (repo, pr),
+    ).fetchall()
+    total = len(rows)
+    engaged = 0
+    positive = 0
+    negative = 0
+
+    for row in rows:
+        fid = row["id"] if isinstance(row, sqlite3.Row) else row[0]
+        cid = row["comment_id"] if isinstance(row, sqlite3.Row) else row[1]
+        pos = neg = 0
+
+        if cid is not None:
+            for r in conn.execute(
+                "SELECT content FROM reaction WHERE comment_id = ?", (cid,)
+            ):
+                kind = classify_reaction(r[0])
+                if kind == "positive":
+                    pos += 1
+                elif kind == "negative":
+                    neg += 1
+
+        for r in conn.execute(
+            "SELECT resolved FROM thread_state WHERE finding_id = ?", (fid,)
+        ):
+            # A resolved thread means the maintainer acted on the finding.
+            if r[0]:
+                pos += 1
+
+        # A reply counts as engagement either way; only a negation phrase makes
+        # it a vote against. A neutral reply ("done", "good catch, but…") is
+        # deliberately not a positive vote — it says someone looked, not that
+        # they agreed.
+        replied = 0
+        for r in conn.execute(
+            "SELECT body FROM reply WHERE finding_id = ?", (fid,)
+        ):
+            replied += 1
+            if _is_negation_reply(r[0]):
+                neg += 1
+
+        if pos or neg or replied:
+            engaged += 1
+        positive += pos
+        negative += neg
+
+    return {"total": total, "engaged": engaged, "positive": positive, "negative": negative}
+
+
+def score_pr(tally: dict) -> dict:
+    """Turn one PR's tally into score values.
+
+    `review_acceptance` is `None` when nothing was engaged — see the module
+    docstring on why that is not 0.
+    """
+    total = int(tally.get("total") or 0)
+    engaged = int(tally.get("engaged") or 0)
+    pos = int(tally.get("positive") or 0)
+    neg = int(tally.get("negative") or 0)
+
+    engagement = round(engaged / total, 4) if total else None
+    acceptance = None
+    if pos or neg:
+        acceptance = round((pos - neg) / (pos + neg), 4)
+    return {REVIEW_ENGAGEMENT: engagement, REVIEW_ACCEPTANCE: acceptance}
+
+
+def build_score_events(
+    repo: str, pr: int, values: dict, environment: str = "default",
+    timestamp: str | None = None,
+) -> list[dict]:
+    """`score-create` events for one PR's feedback.
+
+    Every event carries a timestamp: the ingestion endpoint rejects those that
+    do not, and it reports the rejection as a per-event 400 inside an HTTP 207,
+    which reads as success to a caller that only checks the status code.
+    """
+    ts = timestamp or datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
+    events = []
+    for name, value in values.items():
+        if value is None:
+            continue
+        events.append(
+            {
+                "id": str(uuid.uuid4()),
+                "type": "score-create",
+                "timestamp": ts,
+                "body": {
+                    "id": _score_id(repo, pr, name),
+                    "sessionId": f"{repo}#{pr}",
+                    "name": name,
+                    "value": float(value),
+                    "dataType": "NUMERIC",
+                    "environment": environment,
+                    "comment": f"from feedback.db · {repo}#{pr}",
+                },
+            }
+        )
+    return events
+
+
+SCORE_CONFIGS = [
+    {
+        "name": REVIEW_ENGAGEMENT,
+        "dataType": "NUMERIC",
+        "minValue": 0,
+        "maxValue": 1,
+        "description": "Share of a PR's findings that drew any human reaction, resolution or reply. 0 = nobody engaged with the review.",
+    },
+    {
+        "name": REVIEW_ACCEPTANCE,
+        "dataType": "NUMERIC",
+        "minValue": -1,
+        "maxValue": 1,
+        "description": "Net human verdict over engaged findings: +1 all accepted, -1 all rejected. Absent when nothing was engaged.",
+    },
+]
+
+
+def iter_prs(conn: sqlite3.Connection):
+    for row in conn.execute(
+        "SELECT DISTINCT repo, pr FROM inline_finding ORDER BY repo, pr"
+    ):
+        yield row[0], int(row[1])
+
+
+def backfill(db_path: str, *, environment: str = "default", dry_run: bool = False) -> dict:
+    """Score every PR in the feedback DB. Returns a summary dict."""
+    conn = sqlite3.connect(db_path)
+    conn.row_factory = sqlite3.Row
+    events: list[dict] = []
+    scanned = 0
+    engaged_prs = 0
+    try:
+        for repo, pr in iter_prs(conn):
+            scanned += 1
+            tally = collect_pr_feedback(conn, repo, pr)
+            values = score_pr(tally)
+            if (values.get(REVIEW_ENGAGEMENT) or 0) > 0:
+                engaged_prs += 1
+            events.extend(build_score_events(repo, pr, values, environment))
+    finally:
+        conn.close()
+
+    summary = {"prs_scanned": scanned, "prs_with_engagement": engaged_prs, "scores": len(events)}
+    if dry_run or not events:
+        summary["posted"] = False
+        return summary
+
+    import langfuse_trace
+
+    conf = langfuse_trace._enabled()
+    if conf is None:
+        summary["posted"] = False
+        summary["error"] = "Langfuse not configured (LANGFUSE_HOST / keys unset)"
+        return summary
+    host, pk, sk = conf
+    status = langfuse_trace._post(host, pk, sk, events, 15.0)
+    summary["posted"] = status in (200, 201, 207)
+    summary["http_status"] = status
+    return summary
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description="Ship feedback.db verdicts to Langfuse as scores")
+    ap.add_argument("--db", default=os.environ.get("PRAGENT_FEEDBACK_DB", "/data/feedback.db"))
+    ap.add_argument("--environment", default="default")
+    ap.add_argument("--dry-run", action="store_true")
+    args = ap.parse_args()
+
+    summary = backfill(args.db, environment=args.environment, dry_run=args.dry_run)
+    print(json.dumps(summary, indent=2))
+    return 0 if summary.get("posted") or args.dry_run else 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

pilot/gitea_client.py

@@ -0,0 +1,46 @@
+"""Gitea transport adapter.
+
+This module owns HTTP mechanics only. Review policy, parsing, and publishing
+decisions stay in the review layer so they can be tested without a network.
+"""
+from __future__ import annotations
+
+import json
+import urllib.error
+import urllib.request
+
+
+def request(
+    method: str,
+    url: str,
+    token: str,
+    body: dict | None = None,
+    accept: str = "application/json",
+) -> tuple[int, bytes]:
+    headers = {"Authorization": f"token {token}", "Accept": accept}
+    data = None
+    if body is not None:
+        data = json.dumps(body).encode()
+        headers["Content-Type"] = "application/json"
+    req = urllib.request.Request(url, data=data, headers=headers, method=method)
+    try:
+        with urllib.request.urlopen(req, timeout=180) as response:
+            return response.status, response.read()
+    except urllib.error.HTTPError as exc:
+        return exc.code, exc.read()
+    except urllib.error.URLError as exc:
+        raise RuntimeError(f"network error: {exc.reason}") from exc
+
+
+class GiteaClient:
+    """Small adapter for repository-scoped Gitea calls."""
+
+    def __init__(self, api: str, token: str):
+        self.api = api.rstrip("/")
+        self.token = token
+
+    def get(self, path: str, accept: str = "application/json") -> tuple[int, bytes]:
+        return request("GET", f"{self.api}/api/v1/repos/{path}", self.token, accept=accept)
+
+    def post(self, path: str, body: dict) -> tuple[int, bytes]:
+        return request("POST", f"{self.api}/api/v1/repos/{path}", self.token, body)

pilot/langfuse_trace.py

@@ -0,0 +1,423 @@
+#!/usr/bin/env python3
+"""pragent pilot — Langfuse trace emission.
+
+Ships one trace per PR review to a self-hosted Langfuse (v3) so the reviewer's
+token spend, latency and per-model behaviour are queryable outside the review
+body. The review body already renders a usage table; that table is per-PR and
+disappears into Gitea. This is the same numbers, aggregated.
+
+Why hand-rolled instead of the `langfuse` SDK: the pilot image is stdlib-only
+(see pilot/Dockerfile — no requirements.txt anywhere in the repo), and the
+ingestion API is a single authenticated POST of a JSON batch. Pulling an SDK
+plus its otel dependency tree into a fail-open telemetry side-path is a bad
+trade.
+
+Provider split
+--------------
+`environment` on every trace is either `ollama` or `claude`, derived from the
+resolved display model (`resolve_environment`). That is what keeps the two
+spend stories separate in Langfuse: every view, filter and cost breakdown
+takes an environment selector, so "what did the local/self-hosted path cost"
+and "what did the Claude path cost" are two views of one project rather than
+two projects with two key pairs to rotate. Tags carry the finer split
+(`provider:headroom`, `model:...`, `engine:opencode`).
+
+Cost
+----
+The pilot's own path bills $0 (headroom proxy, no per-token charge), so the
+`cost` reported to Langfuse is the *equivalent* cost from `cost_model` — what
+the same tokens would bill on the comparison model. That is the number worth
+trending; a chart of $0.00 is not.
+
+A model is "free" when `cost_model.PRICES` has no entry for it (MiniMax-M2.7,
+glm-5.2:cloud) or when its entry is all zeros (the self-hosted vLLM qwen). In
+both cases the reported cost is priced against the comparison target instead —
+same precedence the review body uses: `.pr-review.json:cost_target` >
+`PRAGENT_PRICE_TARGET` > `claude-sonnet-5`. A paid model is priced as itself.
+
+Because a hypothetical and a real charge must never be read as the same
+number, every trace is tagged `cost:actual` or `cost:equivalent:<target>`, and
+the generation's metadata carries `cost_basis`.
+
+Fail-open: every entry point swallows its own exceptions. Telemetry must never
+cost a review.
+
+Env:
+  LANGFUSE_HOST         e.g. http://langfuse-web.langfuse.svc.cluster.local:3000
+  LANGFUSE_PUBLIC_KEY   pk-lf-...
+  LANGFUSE_SECRET_KEY   sk-lf-...
+  LANGFUSE_TIMEOUT      seconds, default 5
+  LANGFUSE_DEBUG        1 to log ingestion failures to stderr
+Disabled (silently) when host or either key is unset.
+"""
+
+from __future__ import annotations
+
+import base64
+import json
+import os
+import sys
+import time
+import urllib.error
+import urllib.request
+import uuid
+from datetime import datetime, timezone
+
+INGESTION_PATH = "/api/public/ingestion"
+
+# Model-key prefixes that mean "this review ran against Anthropic-shaped
+# billing". Everything else (glm, MiniMax, qwen, local vLLM) is the ollama /
+# self-hosted side of the split.
+_CLAUDE_PREFIXES = ("claude-", "anthropic/")
+
+
+def _now_iso() -> str:
+    return datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
+
+
+def _enabled() -> tuple[str, str, str] | None:
+    host = (os.environ.get("LANGFUSE_HOST") or "").strip().rstrip("/")
+    pk = (os.environ.get("LANGFUSE_PUBLIC_KEY") or "").strip()
+    sk = (os.environ.get("LANGFUSE_SECRET_KEY") or "").strip()
+    if not host or not pk or not sk:
+        return None
+    return host, pk, sk
+
+
+def _debug(msg: str) -> None:
+    if os.environ.get("LANGFUSE_DEBUG"):
+        print(f"pragent/langfuse: {msg}", file=sys.stderr, flush=True)
+
+
+def strip_provider(model: str) -> str:
+    """`headroom/claude-sonnet-5` -> `claude-sonnet-5`. Bare names pass through."""
+    return model.split("/", 1)[1] if "/" in model else model
+
+
+def provider_of(model: str) -> str:
+    """The opencode provider block a display model routes through."""
+    return model.split("/", 1)[0] if "/" in model else "headroom"
+
+
+def resolve_environment(model: str) -> str:
+    """Which spend story this review belongs to: `claude` or `ollama`.
+
+    Keyed off the bare model name, not the provider, because both paths route
+    through the same `headroom` proxy — `headroom/claude-sonnet-5` is Claude
+    spend, `headroom/glm-5.2:cloud` is not.
+    """
+    bare = strip_provider(model).lower()
+    return "claude" if bare.startswith(_CLAUDE_PREFIXES) else "ollama"
+
+
+def _usage_details(usage: dict) -> dict:
+    """opencode's usage dict -> Langfuse `usageDetails`.
+
+    Langfuse sums every key except the ones it knows are derived, so `input`
+    here is the *uncached* portion: reporting both `input` (which opencode
+    reports as the full input, cache included) and `cache_read_input_tokens`
+    would double-count.
+    """
+    inp = int(usage.get("input") or 0)
+    cache_read = int(usage.get("cache_read") or 0)
+    cache_write = int(usage.get("cache_write") or 0)
+    details = {
+        "input": max(0, inp - cache_read),
+        "output": int(usage.get("output") or 0),
+    }
+    if cache_read:
+        details["cache_read_input_tokens"] = cache_read
+    if cache_write:
+        details["cache_write_input_tokens"] = cache_write
+    reasoning = int(usage.get("reasoning") or 0)
+    if reasoning:
+        details["reasoning"] = reasoning
+    return details
+
+
+DEFAULT_PRICE_TARGET = "claude-sonnet-5"
+
+
+def resolve_price_target(price_target: str | None = None) -> str:
+    """The model to price free/unknown runs against.
+
+    Mirrors `ai_review._resolve_price_target`: an explicit target (which the
+    caller reads from `.pr-review.json:cost_target`) wins, then
+    `PRAGENT_PRICE_TARGET`, then Sonnet.
+    """
+    if price_target and price_target.strip():
+        return price_target.strip()
+    env = os.environ.get("PRAGENT_PRICE_TARGET", "").strip()
+    return env or DEFAULT_PRICE_TARGET
+
+
+def _is_free(price) -> bool:
+    """A price entry that charges nothing — self-hosted or proxied at no cost."""
+    return price.input == 0 and price.output == 0
+
+
+def _cost_details(usage: dict, model: str, price_target: str | None = None) -> tuple[dict, str]:
+    """USD for this usage plus the basis it was computed on.
+
+    Returns `({"total": …}, basis)` where basis is `actual` for a model that
+    genuinely bills, or `equivalent:<target>` for one that does not. `({}, "")`
+    when nothing can be priced at all — better no number than a wrong one.
+
+    Local import + broad except: `cost_model` is only present on the opencode
+    path, and an unknown model key must not break telemetry.
+    """
+    try:
+        from cost_model import PRICES, Usage, cost
+
+        bare = strip_provider(model)
+        price = PRICES.get(bare)
+        basis = "actual"
+        if price is None or _is_free(price):
+            # MiniMax / glm / self-hosted qwen: $0 through the proxy, so the
+            # useful number is what these tokens would have billed elsewhere.
+            target = resolve_price_target(price_target)
+            price = PRICES.get(target)
+            if price is None:
+                _debug(f"comparison target {target!r} not in PRICES")
+                return {}, ""
+            basis = f"equivalent:{target}"
+
+        u = Usage(
+            uncached_input=max(0, int(usage.get("input") or 0) - int(usage.get("cache_read") or 0)),
+            cached_input=int(usage.get("cache_read") or 0),
+            cache_writes=int(usage.get("cache_write") or 0),
+            output=int(usage.get("output") or 0),
+        )
+        return {"total": round(cost(u, price), 6)}, basis
+    except Exception as e:  # pragma: no cover - defensive
+        _debug(f"cost lookup failed for {model!r}: {e}")
+        return {}, ""
+
+
+def _severity_counts(findings: list[dict] | None) -> dict:
+    counts: dict[str, int] = {}
+    for f in findings or []:
+        sev = str(f.get("severity") or "unknown").lower()
+        counts[sev] = counts.get(sev, 0) + 1
+    return counts
+
+
+def build_batch(
+    *,
+    repo: str,
+    index: str,
+    sha: str,
+    title: str,
+    model: str,
+    usage: dict | None,
+    findings: list[dict] | None = None,
+    summary: str = "",
+    engine: str = "opencode",
+    tier: str = "",
+    lenses: list[str] | None = None,
+    trace_id: str | None = None,
+    release: str = "",
+    price_target: str | None = None,
+    dropped_count: float | None = None,
+) -> list[dict]:
+    """The ingestion batch for one review: a trace, a generation, and scores.
+
+    Split out from `emit_review_trace` so the shape is testable without a
+    Langfuse to POST to.
+
+    `dropped_count` is how many findings the parser rejected for an unusable
+    `path`/`line`, measured where the model output was parsed. Passing it turns
+    on the `dropped_findings` score; leaving it `None` omits that score rather
+    than reporting a zero the caller never measured.
+    """
+    usage = usage or {}
+    tid = trace_id or str(uuid.uuid4())
+    ts = _now_iso()
+    env = resolve_environment(model)
+    duration = float(usage.get("duration_s") or 0.0)
+    started = datetime.fromtimestamp(
+        time.time() - duration, tz=timezone.utc
+    ).isoformat().replace("+00:00", "Z")
+
+    tags = [
+        f"provider:{provider_of(model)}",
+        f"model:{strip_provider(model)}",
+        f"engine:{engine}",
+        f"repo:{repo}",
+    ]
+    if tier:
+        tags.append(f"tier:{tier}")
+    for lens in lenses or []:
+        tags.append(f"lens:{lens}")
+
+    costs, cost_basis = _cost_details(usage, model, price_target) if usage else ({}, "")
+    if cost_basis:
+        # Filterable in Langfuse, so an equivalent-cost chart can never be
+        # mistaken for money actually spent.
+        tags.append(f"cost:{cost_basis}")
+
+    metadata = {
+        "repo": repo,
+        "pr": index,
+        "sha": sha,
+        "engine": engine,
+        "steps": usage.get("steps"),
+        "duration_s": duration or None,
+        "findings": len(findings or []),
+        "severities": _severity_counts(findings),
+        "provider_cost_usd": usage.get("cost"),
+        "cost_basis": cost_basis or None,
+    }
+    if lenses:
+        metadata["lenses"] = lenses
+    if tier:
+        metadata["tier"] = tier
+    metadata = {k: v for k, v in metadata.items() if v not in (None, {}, [])}
+
+    trace_body = {
+        "id": tid,
+        "name": "pr-review",
+        "timestamp": ts,
+        "environment": env,
+        "sessionId": f"{repo}#{index}",
+        "input": {"repo": repo, "pr": index, "sha": sha, "title": title},
+        "output": {"summary": summary[:2000], "findings": len(findings or [])},
+        "metadata": metadata,
+        "tags": tags,
+    }
+    if release:
+        trace_body["release"] = release
+
+    events = [
+        {
+            "id": str(uuid.uuid4()),
+            "type": "trace-create",
+            "timestamp": ts,
+            "body": trace_body,
+        }
+    ]
+
+    if usage:
+        gen_body = {
+            "id": str(uuid.uuid4()),
+            "traceId": tid,
+            "type": "GENERATION",
+            "name": f"{engine}-review",
+            "environment": env,
+            "startTime": started,
+            "endTime": ts,
+            "model": strip_provider(model),
+            "usageDetails": _usage_details(usage),
+            "metadata": metadata,
+            "level": "DEFAULT",
+        }
+        if costs:
+            gen_body["costDetails"] = costs
+        events.append(
+            {
+                "id": str(uuid.uuid4()),
+                "type": "generation-create",
+                "timestamp": ts,
+                "body": gen_body,
+            }
+        )
+
+    events.extend(
+        _score_events(
+            trace_id=tid,
+            findings=findings,
+            environment=env,
+            cost_usd=costs.get("total"),
+            dropped_count=dropped_count,
+            timestamp=ts,
+            cost_basis=cost_basis,
+        )
+    )
+
+    return events
+
+
+def _score_events(*, cost_basis: str, **kwargs) -> list[dict]:
+    """Deterministic scores for this review, or [] if the scorer is missing.
+
+    Local import + blanket except for the same reason the rest of this module
+    swallows: `eval_scores` is optional, and a scoring bug must not cost the
+    trace it was supposed to annotate.
+    """
+    try:
+        import eval_scores
+
+        # The cost score is only meaningful next to its basis — a $/finding
+        # figure computed from an equivalent price is not money that was spent.
+        comment = f"cost basis: {cost_basis}" if cost_basis else ""
+        return eval_scores.build_scores(comment=comment, **kwargs)
+    except Exception as e:  # pragma: no cover - defensive
+        _debug(f"scoring failed: {e}")
+        return []
+
+
+def _post(host: str, pk: str, sk: str, batch: list[dict], timeout: float) -> int:
+    payload = json.dumps({"batch": batch}).encode("utf-8")
+    auth = base64.b64encode(f"{pk}:{sk}".encode("utf-8")).decode("ascii")
+    req = urllib.request.Request(
+        host + INGESTION_PATH,
+        data=payload,
+        headers={
+            "Content-Type": "application/json",
+            "Authorization": f"Basic {auth}",
+            "User-Agent": "pragent-pilot/1.0",
+        },
+        method="POST",
+    )
+    with urllib.request.urlopen(req, timeout=timeout) as resp:
+        _warn_on_rejected_events(resp.read())
+        return resp.status
+
+
+def _warn_on_rejected_events(raw: bytes) -> None:
+    """Surface per-event rejections hiding inside a 207.
+
+    The ingestion endpoint answers 207 Multi-Status when *some* events failed,
+    so a caller that only checks the status code reads a batch where every
+    single event was rejected as a success. That failure mode is invisible
+    exactly when it matters — the traces simply never appear.
+    """
+    try:
+        body = json.loads(raw or b"{}")
+        errors = body.get("errors") or []
+        if errors:
+            first = errors[0]
+            _debug(
+                f"{len(errors)} event(s) rejected by ingestion; "
+                f"first: status={first.get('status')} {first.get('error')}"
+            )
+    except Exception:  # pragma: no cover - never let logging break emission
+        pass
+
+
+def emit_review_trace(**kwargs) -> bool:
+    """Ship one review's trace. Returns True if Langfuse accepted it.
+
+    No-op (False) when Langfuse is unconfigured. Never raises — a telemetry
+    outage must not turn into a failed review.
+    """
+    conf = _enabled()
+    if conf is None:
+        return False
+    host, pk, sk = conf
+    try:
+        timeout = float(os.environ.get("LANGFUSE_TIMEOUT", "5"))
+    except ValueError:
+        timeout = 5.0
+    try:
+        batch = build_batch(**kwargs)
+        status = _post(host, pk, sk, batch, timeout)
+        if status not in (200, 201, 207):
+            _debug(f"ingestion returned HTTP {status}")
+            return False
+        return True
+    except urllib.error.HTTPError as e:
+        _debug(f"ingestion HTTP {e.code}: {e.read()[:300]!r}")
+    except Exception as e:
+        _debug(f"ingestion failed: {e}")
+    return False

pilot/model_client.py

@@ -0,0 +1,36 @@
+"""Model-provider adapter for the legacy Anthropic-compatible endpoint."""
+from __future__ import annotations
+
+import json
+
+try:  # Works both as `python pilot/ai_review.py` and `import pilot.model_client`.
+    from .gitea_client import request
+except ImportError:  # pragma: no cover - script-style runtime
+    from gitea_client import request
+
+
+def parse_text_blocks(content: object) -> str:
+    """Return only text blocks from an Anthropic-style response."""
+    if not isinstance(content, list):
+        return ""
+    return "\n".join(
+        block["text"]
+        for block in content
+        if isinstance(block, dict)
+        and block.get("type") == "text"
+        and isinstance(block.get("text"), str)
+    ).strip()
+
+
+def complete(base_url: str, model: str, system: str, user: str, max_tokens: int) -> str:
+    payload = {
+        "model": model,
+        "max_tokens": max_tokens,
+        "system": system,
+        "messages": [{"role": "user", "content": user}],
+    }
+    status, raw = request("POST", f"{base_url.rstrip('/')}/v1/messages", "ollama", payload)
+    if status != 200:
+        detail = raw[:500].decode("utf-8", errors="replace")
+        raise RuntimeError(f"model call failed: HTTP {status}: {detail}")
+    return parse_text_blocks(json.loads(raw).get("content", []))

pilot/review_config.py

@@ -0,0 +1,32 @@
+"""Trusted repository configuration and opt-in policy."""
+from __future__ import annotations
+
+import base64
+import json
+import urllib.parse
+from collections.abc import Callable
+
+
+def repo_enabled(
+    get: Callable[..., tuple[int, bytes]],
+    api: str,
+    repo: str,
+    ref: str,
+    token: str,
+) -> bool:
+    """Read the opt-in flag from the trusted base branch.
+
+    The transport is injected so the policy is testable without a live Gitea.
+    Any missing, malformed, or non-boolean value disables review.
+    """
+    path = "contents/.pr-review.json?ref=" + urllib.parse.quote(ref, safe="")
+    status, raw = get(api, repo, path, token)
+    if status != 200:
+        return False
+    try:
+        envelope = json.loads(raw)
+        encoded = envelope.get("content", "").replace("\n", "")
+        config = json.loads(base64.b64decode(encoded).decode("utf-8", errors="replace"))
+    except (AttributeError, TypeError, ValueError, json.JSONDecodeError):
+        return False
+    return isinstance(config, dict) and config.get("enabled") is True

pilot/review_ports.py

@@ -0,0 +1,17 @@
+"""Stable interfaces shared by the review pipeline and its adapters."""
+from __future__ import annotations
+
+from typing import Protocol
+
+
+class Forge(Protocol):
+    def get(self, path: str, accept: str = "application/json") -> tuple[int, bytes]: ...
+    def post(self, path: str, body: dict) -> tuple[int, bytes]: ...
+
+
+class Reviewer(Protocol):
+    def review(self, system: str, user: str, max_tokens: int) -> str: ...
+
+
+class Telemetry(Protocol):
+    def emit(self, **event: object) -> None: ...

pilot/webhook_server.py

@@ -46,6 +46,7 @@ import urllib.parse
 from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
 
 from ai_review import gitea_get, review_pr
+from review_config import repo_enabled
 
 try:
     import feedback_harvest  # optional — absent in CI-step pod, present in
@@ -102,21 +103,7 @@ def is_repo_enabled(api: str, repo: str, ref: str, token: str) -> bool:
     The bool-coerce of `.get("enabled") is True` rejects the common
     gotchas (`null`, `1`, `"yes"`, missing field all yield False).
     """
-    code, raw = gitea_get(
-        api, repo,
-        "contents/.pr-review.json?ref=" + urllib.parse.quote(ref, safe=""),
-        token,
-    )
-    if code != 200:
-        return False
-    try:
-        data = json.loads(raw)
-        content_b64 = data.get("content", "").replace("\n", "")
-        decoded = base64.b64decode(content_b64).decode("utf-8", errors="replace")
-        cfg = json.loads(decoded)
-    except (json.JSONDecodeError, ValueError):
-        return False
-    return isinstance(cfg, dict) and cfg.get("enabled") is True
+    return repo_enabled(gitea_get, api, repo, ref, token)
 
 
 def _verify_signature(raw_body: bytes, headers) -> bool:

tests/pilot/test_dashboard.py

@@ -1,203 +0,0 @@
-"""Tests for pilot/dashboard.py — stdlib HTTP server rendering dashboard HTML.
-
-We spin up the server on an ephemeral port in setUp, drive it with
-http.client, and tear it down in tearDown. Auth is now performed by
-oauth2-proxy: the dashboard trusts `X-Forwarded-User` set by the proxy
-and returns 401 (with a Basic challenge) when the header is missing.
-
-The dashboard reads `PRAGENT_FEEDBACK_DB` and renders views via
-`dashboard_data`. We seed an in-memory SQLite at `tmp_path` for each
-scenario that needs rows.
-"""
-import http.client
-import os
-import socket
-import sys
-import tempfile
-import threading
-import time
-import unittest
-
-HERE = os.path.dirname(os.path.abspath(__file__))
-ROOT = os.path.abspath(os.path.join(HERE, "..", ".."))
-sys.path.insert(0, os.path.join(ROOT, "pilot"))
-
-import dashboard as dash  # noqa: E402
-from pilot import feedback  # noqa: E402
-
-
-def _free_port() -> int:
-    s = socket.socket()
-    s.bind(("127.0.0.1", 0))
-    port = s.getsockname()[1]
-    s.close()
-    return port
-
-
-class _ServerThread:
-    def __init__(self, port: int, handler):
-        self.server = handler((host := "127.0.0.1", port), None)
-        self.port = port
-        self.thread = threading.Thread(target=self.server.serve_forever, daemon=True)
-        self.thread.start()
-
-    def stop(self):
-        self.server.shutdown()
-        self.server.server_close()
-        self.thread.join(timeout=2)
-
-
-def _get(port: int, path: str, headers: dict | None = None) -> tuple[int, dict, bytes]:
-    conn = http.client.HTTPConnection("127.0.0.1", port, timeout=5)
-    conn.request("GET", path, headers=headers or {})
-    r = conn.getresponse()
-    body = r.read()
-    h = dict(r.getheaders())
-    conn.close()
-    return r.status, h, body
-
-
-def _post(port: int, path: str, body: bytes, headers: dict | None = None) -> tuple[int, dict, bytes]:
-    conn = http.client.HTTPConnection("127.0.0.1", port, timeout=5)
-    hdrs = {"Content-Type": "application/x-www-form-urlencoded"}
-    if headers:
-        hdrs.update(headers)
-    conn.request("POST", path, body=body, headers=hdrs)
-    r = conn.getresponse()
-    body_b = r.read()
-    h = dict(r.getheaders())
-    conn.close()
-    return r.status, h, body_b
-
-
-class TestDashboardAuth(unittest.TestCase):
-    """Auth gate: require X-Forwarded-User (set by oauth2-proxy).
-
-    When the header is missing every non-static route returns 401 with a
-    Basic challenge, which lets oauth2-proxy redirect the browser to
-    Logto. Static is exempt so the unauthenticated probe traffic doesn't
-    loop the proxy through the auth flow.
-    """
-
-    def setUp(self):
-        self.tmp = tempfile.TemporaryDirectory()
-        self.db = f"{self.tmp.name}/f.db"
-        os.environ["PRAGENT_FEEDBACK_DB"] = self.db
-        os.environ["DASHBOARD_PORT"] = str(0)  # we override below
-
-        self.port = _free_port()
-        from http.server import ThreadingHTTPServer
-        self.srv = ThreadingHTTPServer(("127.0.0.1", self.port), dash.Handler)
-        self.thread = threading.Thread(target=self.srv.serve_forever, daemon=True)
-        self.thread.start()
-
-    def tearDown(self):
-        self.srv.shutdown()
-        self.srv.server_close()
-        self.thread.join(timeout=2)
-        for k in ("PRAGENT_FEEDBACK_DB", "DASHBOARD_PORT"):
-            os.environ.pop(k, None)
-        self.tmp.cleanup()
-
-    def test_anonymous_overview_returns_401_with_basic_challenge(self):
-        status, h, body = _get(self.port, "/")
-        self.assertEqual(status, 401)
-        self.assertEqual(h.get("WWW-Authenticate"), 'Basic realm="pragent-dashboard"')
-        self.assertEqual(body, b"unauthorized\n")
-
-    def test_anonymous_repo_returns_401(self):
-        status, _h, _body = _get(self.port, "/r/alpha/one")
-        self.assertEqual(status, 401)
-
-    def test_anonymous_post_returns_401(self):
-        status, _h, _body = _post(self.port, "/r/alpha/one/edit", b"x=1")
-        self.assertEqual(status, 401)
-
-    def test_authenticated_overview_succeeds(self):
-        status, h, body = _get(self.port, "/", headers={"X-Forwarded-User": "marcos@example.com"})
-        self.assertEqual(status, 200)
-        self.assertIn(b"Overview", body)
-
-    def test_static_does_not_require_auth(self):
-        status, h, body = _get(self.port, "/static/style.css")
-        self.assertEqual(status, 200)
-        self.assertIn("text/css", h.get("Content-Type", ""))
-        self.assertGreater(len(body), 50)
-
-    def test_empty_x_forwarded_user_treated_as_anonymous(self):
-        status, _h, _body = _get(self.port, "/", headers={"X-Forwarded-User": "   "})
-        self.assertEqual(status, 401)
-
-
-class TestDashboardRender(unittest.TestCase):
-    """Render-only tests — X-Forwarded-User set, real seeded data."""
-
-    def setUp(self):
-        self.tmp = tempfile.TemporaryDirectory()
-        self.db = f"{self.tmp.name}/f.db"
-        os.environ["PRAGENT_FEEDBACK_DB"] = self.db
-        # Seed: 2 repos, a couple of reviews + findings each.
-        conn = feedback.init(self.db)
-        for repo, n_prs in (("alpha/one", 2), ("beta/two", 1)):
-            for n in range(n_prs):
-                rid = feedback.record_review(
-                    conn, repo=repo, pr=n + 1, head_sha=f"sha{repo}-{n}",
-                    review_id_gitea=1000 + n, body_comment_id=2000 + n,
-                    posted_at=int(time.time()) - n * 60,
-                )
-                for k in range(3):
-                    feedback.record_inline_finding(
-                        conn, review_id=rid, repo=repo, pr=n + 1,
-                        path=f"src/file_{k}.py", line=k + 1,
-                        severity=["critical", "high", "medium"][k],
-                        problem=f"problem {k}",
-                        fix=f"fix {k}", suggestion=f"suggestion {k}",
-                        comment_id=3000 + n * 10 + k,
-                    )
-        conn.close()
-
-        self.port = _free_port()
-        from http.server import ThreadingHTTPServer
-        self.srv = ThreadingHTTPServer(("127.0.0.1", self.port), dash.Handler)
-        self.thread = threading.Thread(target=self.srv.serve_forever, daemon=True)
-        self.thread.start()
-        self.auth_hdr = {"X-Forwarded-User": "marcos@example.com"}
-
-    def tearDown(self):
-        self.srv.shutdown()
-        self.srv.server_close()
-        self.thread.join(timeout=2)
-        os.environ.pop("PRAGENT_FEEDBACK_DB", None)
-        self.tmp.cleanup()
-
-    def test_overview_200_contains_repo_names(self):
-        status, _h, body = _get(self.port, "/", headers=self.auth_hdr)
-        self.assertEqual(status, 200)
-        text = body.decode()
-        self.assertIn("Overview", text)
-        self.assertIn("alpha/one", text)
-        self.assertIn("beta/two", text)
-
-    def test_repo_page_200(self):
-        status, _h, body = _get(self.port, "/r/alpha/one", headers=self.auth_hdr)
-        self.assertEqual(status, 200)
-        text = body.decode()
-        self.assertIn("alpha/one", text)
-        # The findings table should appear.
-        self.assertIn("src/file_0.py", text)
-
-    def test_pr_page_200(self):
-        status, _h, body = _get(self.port, "/r/alpha/one/1", headers=self.auth_hdr)
-        self.assertEqual(status, 200)
-        text = body.decode()
-        self.assertIn("alpha/one", text)
-        self.assertIn("#1", text)
-        self.assertIn("src/file_0.py", text)
-
-    def test_unknown_route_404(self):
-        status, _h, _body = _get(self.port, "/no/such/route", headers=self.auth_hdr)
-        self.assertEqual(status, 404)
-
-
-if __name__ == "__main__":
-    unittest.main()

tests/pilot/test_dashboard_data.py

@@ -1,249 +0,0 @@
-"""Tests for pilot/dashboard_data.py — read-only query layer over the feedback SQLite.
-
-Covers: empty-DB fallbacks (no crash on missing/empty DB), overview rollups,
-per-repo drill-down (findings by severity, top findings, runs by day), and
-the per-PR view. The dashboard never writes — only reads.
-"""
-import os
-import sys
-import tempfile
-import time
-import unittest
-
-HERE = os.path.dirname(os.path.abspath(__file__))
-ROOT = os.path.abspath(os.path.join(HERE, "..", ".."))
-sys.path.insert(0, os.path.join(HERE, "..", ".."))  # so `from pilot import …` works
-
-from pilot import dashboard_data, feedback
-
-
-def _seed_repo(conn, *, repo: str, prs: int, findings_per_pr: int, day_offset: int = 0):
-    """Seed one repo with `prs` PRs each with `findings_per_pr` findings.
-
-    All timestamps cluster on (now - day_offset days). Returns list of review ids.
-    """
-    base = int(time.time()) - day_offset * 86400
-    rids = []
-    for n in range(prs):
-        rid = feedback.record_review(
-            conn, repo=repo, pr=n + 1, head_sha=f"sha{n}",
-            review_id_gitea=1000 + n, body_comment_id=2000 + n,
-            posted_at=base + n * 60,
-        )
-        rids.append(rid)
-        for k in range(findings_per_pr):
-            feedback.record_inline_finding(
-                conn, review_id=rid, repo=repo, pr=n + 1,
-                path=f"src/file_{k}.py", line=k + 1,
-                severity=["critical", "high", "medium", "low"][k % 4],
-                problem=f"problem {k}",
-                fix=f"fix {k}", suggestion=f"suggestion {k}",
-                comment_id=3000 + n * 10 + k,
-                posted_at=base + n * 60,
-            )
-    return rids
-
-
-class TestEmptyDB(unittest.TestCase):
-    def test_missing_file_returns_zero_dict(self):
-        with tempfile.TemporaryDirectory() as d:
-            missing = f"{d}/nope.db"
-            data = dashboard_data.overview(missing)
-            self.assertEqual(data["total_reviews"], 0)
-            self.assertEqual(data["total_findings"], 0)
-            self.assertEqual(data["total_repos"], 0)
-            self.assertEqual(data["last_30d_reviews"], 0)
-            self.assertEqual(len(data["daily"]), 7)
-            self.assertEqual(data["top_repos"], [])
-            self.assertEqual(data["total_cost_usd"], 0.0)
-
-    def test_missing_file_repo_summary_safe(self):
-        with tempfile.TemporaryDirectory() as d:
-            data = dashboard_data.repo_summary(f"{d}/nope.db", "o/r")
-            self.assertEqual(data["repo"], "o/r")
-            self.assertEqual(data["total_runs"], 0)
-            self.assertEqual(data["runs_by_day"], [])
-            for sev in ("critical", "high", "medium", "low"):
-                self.assertEqual(data["findings_by_severity"][sev], 0)
-            self.assertEqual(data["top_findings"], [])
-            self.assertEqual(data["models_used"], [])
-
-    def test_missing_file_pr_summary_safe(self):
-        with tempfile.TemporaryDirectory() as d:
-            data = dashboard_data.pr_summary(f"{d}/nope.db", "o/r", 1)
-            self.assertEqual(data["repo"], "o/r")
-            self.assertEqual(data["pr"], 1)
-            self.assertEqual(data["findings"], [])
-            self.assertEqual(data["usage"], {})
-
-
-class TestEmptyButExistingDB(unittest.TestCase):
-    """`init` creates the schema — DB exists but has no rows."""
-
-    def setUp(self):
-        self.tmp = tempfile.TemporaryDirectory()
-        self.db = f"{self.tmp.name}/f.db"
-        feedback.init(self.db)
-
-    def tearDown(self):
-        self.tmp.cleanup()
-
-    def test_overview_is_zero(self):
-        data = dashboard_data.overview(self.db)
-        self.assertEqual(data["total_reviews"], 0)
-        self.assertEqual(data["total_findings"], 0)
-        self.assertEqual(data["total_repos"], 0)
-
-    def test_repo_summary_is_zero(self):
-        data = dashboard_data.repo_summary(self.db, "o/r")
-        self.assertEqual(data["total_runs"], 0)
-        self.assertEqual(data["findings_by_severity"], {"critical": 0, "high": 0, "medium": 0, "low": 0})
-
-    def test_pr_summary_is_zero(self):
-        data = dashboard_data.pr_summary(self.db, "o/r", 1)
-        self.assertEqual(data["findings"], [])
-
-
-class TestOverview(unittest.TestCase):
-    def setUp(self):
-        self.tmp = tempfile.TemporaryDirectory()
-        self.db = f"{self.tmp.name}/f.db"
-        self.conn = feedback.init(self.db)
-        _seed_repo(self.conn, repo="alpha/one", prs=3, findings_per_pr=2)
-        _seed_repo(self.conn, repo="beta/two", prs=1, findings_per_pr=4)
-        self.conn.close()
-
-    def tearDown(self):
-        self.tmp.cleanup()
-
-    def test_totals(self):
-        data = dashboard_data.overview(self.db)
-        self.assertEqual(data["total_reviews"], 4)
-        self.assertEqual(data["total_findings"], 6 + 4)  # 3*2 + 1*4 = 10
-        self.assertEqual(data["total_repos"], 2)
-        self.assertEqual(data["total_cost_usd"], 0.0)
-
-    def test_top_repos_sorted_by_run_count(self):
-        data = dashboard_data.overview(self.db)
-        repos = [r["repo"] for r in data["top_repos"]]
-        # alpha/one has 3 runs, beta/two has 1.
-        self.assertEqual(repos[0], "alpha/one")
-        self.assertEqual(data["top_repos"][0]["run_count"], 3)
-        self.assertEqual(data["top_repos"][1]["run_count"], 1)
-        # last_seen is a unix timestamp int.
-        for r in data["top_repos"]:
-            self.assertIsInstance(r["last_seen"], int)
-
-    def test_daily_buckets_are_7(self):
-        data = dashboard_data.overview(self.db)
-        self.assertEqual(len(data["daily"]), 7)
-        for b in data["daily"]:
-            self.assertIn("date", b)
-            self.assertIn("count", b)
-
-    def test_last_30d_reviews(self):
-        data = dashboard_data.overview(self.db)
-        self.assertEqual(data["last_30d_reviews"], 4)
-
-
-class TestRepoSummary(unittest.TestCase):
-    def setUp(self):
-        self.tmp = tempfile.TemporaryDirectory()
-        self.db = f"{self.tmp.name}/f.db"
-        self.conn = feedback.init(self.db)
-        # 4 PRs with 2 findings each → 8 findings, severity cycle [c,h,m,l,c,h,m,l]
-        _seed_repo(self.conn, repo="o/r", prs=4, findings_per_pr=2)
-        # Add some reactions so top_findings has signal.
-        rows = self.conn.execute(
-            "SELECT id, comment_id FROM inline_finding WHERE repo=? ORDER BY id LIMIT 3",
-            ("o/r",),
-        ).fetchall()
-        for r in rows:
-            feedback.record_reaction(self.conn, comment_id=r["comment_id"], user="u", content="+1")
-        self.conn.close()
-
-    def tearDown(self):
-        self.tmp.cleanup()
-
-    def test_basic_shape(self):
-        data = dashboard_data.repo_summary(self.db, "o/r")
-        self.assertEqual(data["repo"], "o/r")
-        self.assertEqual(data["total_runs"], 4)
-        self.assertIsInstance(data["last_run_ts"], int)
-
-    def test_findings_by_severity(self):
-        data = dashboard_data.repo_summary(self.db, "o/r")
-        fbs = data["findings_by_severity"]
-        # 4 PRs × 2 findings; per-PR severities are [critical, high].
-        # (k in range(2) → k=0 critical, k=1 high for every PR.)
-        self.assertEqual(fbs["critical"], 4)
-        self.assertEqual(fbs["high"], 4)
-        self.assertEqual(fbs["medium"], 0)
-        self.assertEqual(fbs["low"], 0)
-
-    def test_runs_by_day_is_list(self):
-        data = dashboard_data.repo_summary(self.db, "o/r")
-        self.assertIsInstance(data["runs_by_day"], list)
-        for r in data["runs_by_day"]:
-            self.assertIn("date", r)
-            self.assertIn("count", r)
-
-    def test_top_findings_structure(self):
-        data = dashboard_data.repo_summary(self.db, "o/r")
-        self.assertGreater(len(data["top_findings"]), 0)
-        first = data["top_findings"][0]
-        for k in ("path", "line", "severity", "problem", "occurrences", "upvotes", "downvotes", "resolved", "reply_count"):
-            self.assertIn(k, first)
-
-    def test_models_used_is_empty_list_with_note(self):
-        # The schema has no `model` column on review — the dashboard can't show
-        # model usage from this DB today. We document that via an empty list.
-        data = dashboard_data.repo_summary(self.db, "o/r")
-        self.assertEqual(data["models_used"], [])
-
-
-class TestPRSummary(unittest.TestCase):
-    def setUp(self):
-        self.tmp = tempfile.TemporaryDirectory()
-        self.db = f"{self.tmp.name}/f.db"
-        self.conn = feedback.init(self.db)
-        rid = feedback.record_review(
-            self.conn, repo="o/r", pr=42, head_sha="abc",
-            review_id_gitea=9001, body_comment_id=8001,
-            posted_at=1700000000,
-        )
-        for k in range(3):
-            feedback.record_inline_finding(
-                self.conn, review_id=rid, repo="o/r", pr=42,
-                path=f"src/x_{k}.py", line=k + 10,
-                severity=["critical", "high", "low"][k],
-                problem=f"p{k}", fix=f"f{k}", suggestion=f"s{k}",
-                comment_id=7000 + k,
-            )
-        self.conn.close()
-
-    def tearDown(self):
-        self.tmp.cleanup()
-
-    def test_meta(self):
-        data = dashboard_data.pr_summary(self.db, "o/r", 42)
-        self.assertEqual(data["repo"], "o/r")
-        self.assertEqual(data["pr"], 42)
-        self.assertEqual(data["head_sha"], "abc")
-        self.assertEqual(data["review_id_gitea"], 9001)
-        self.assertEqual(data["body_comment_id"], 8001)
-        self.assertEqual(data["posted_at"], 1700000000)
-        # usage is empty because the schema has no usage column.
-        self.assertEqual(data["usage"], {})
-
-    def test_findings(self):
-        data = dashboard_data.pr_summary(self.db, "o/r", 42)
-        self.assertEqual(len(data["findings"]), 3)
-        for f in data["findings"]:
-            for k in ("path", "line", "severity", "problem", "fix", "suggestion",
-                      "upvotes", "downvotes", "resolved", "reply_count"):
-                self.assertIn(k, f)
-
-
-if __name__ == "__main__":
-    unittest.main()

tests/pilot/test_dashboard_edit.py

@@ -1,209 +0,0 @@
-"""Tests for the edit endpoint — POST /r/<owner>/<name>/edit (Task C).
-
-We mock the Gitea HTTP layer (urllib.request.urlopen) so the test never
-touches the network. The dashboard handler is responsible for:
-  * auth (X-Forwarded-User set by oauth2-proxy) + CSRF
-  * read .pr-review.json via GET (404 → start from {})
-  * validate model against cost_model.PRICES
-  * PUT the updated file back, with sha + base64 content
-  * redirect to /r/<owner>/<name> on success
-"""
-import base64
-import http.client
-import json
-import os
-import socket
-import sys
-import tempfile
-import threading
-import unittest
-from unittest.mock import patch
-
-HERE = os.path.dirname(os.path.abspath(__file__))
-ROOT = os.path.abspath(os.path.join(HERE, "..", ".."))
-sys.path.insert(0, os.path.join(ROOT, "pilot"))
-
-import dashboard as dash  # noqa: E402
-from pilot import feedback  # noqa: E402
-
-
-def _free_port() -> int:
-    s = socket.socket()
-    s.bind(("127.0.0.1", 0))
-    port = s.getsockname()[1]
-    s.close()
-    return port
-
-
-def _post(port: int, path: str, body: bytes, *, headers: dict | None = None) -> tuple[int, dict, bytes]:
-    conn = http.client.HTTPConnection("127.0.0.1", port, timeout=5)
-    hdrs = {"Content-Type": "application/x-www-form-urlencoded"}
-    if headers:
-        hdrs.update(headers)
-    conn.request("POST", path, body=body, headers=hdrs)
-    r = conn.getresponse()
-    body_b = r.read()
-    h = dict(r.getheaders())
-    conn.close()
-    return r.status, h, body_b
-
-
-class _FakeResp:
-    def __init__(self, status: int, body: bytes):
-        self.status = status
-        self._body = body
-
-    def read(self):
-        return self._body
-
-    def __enter__(self):
-        return self
-
-    def __exit__(self, *a):
-        return False
-
-
-class TestDashboardEdit(unittest.TestCase):
-    def setUp(self):
-        self.tmp = tempfile.TemporaryDirectory()
-        self.db = f"{self.tmp.name}/f.db"
-        os.environ["PRAGENT_FEEDBACK_DB"] = self.db
-        os.environ["PRAGENT_BOT_TOKEN"] = "bot-token"
-        # Seed a row so the repo page is meaningful.
-        conn = feedback.init(self.db)
-        feedback.record_review(
-            conn, repo="o/r", pr=1, head_sha="x",
-        )
-        conn.close()
-
-        self.port = _free_port()
-        from http.server import ThreadingHTTPServer
-        self.srv = ThreadingHTTPServer(("127.0.0.1", self.port), dash.Handler)
-        self.thread = threading.Thread(target=self.srv.serve_forever, daemon=True)
-        self.thread.start()
-        # Pull the per-process CSRF secret from the rendered repo page — the
-        # edit form embeds the same token as a hidden input.
-        self.csrf = dash._CSRF_SECRET
-        self.auth_hdr = {"X-Forwarded-User": "marcos@example.com"}
-
-        # Records of HTTP calls made by the handler.
-        self.calls: list[tuple[str, str, dict | None, bytes | None]] = []
-
-    def tearDown(self):
-        self.srv.shutdown()
-        self.srv.server_close()
-        self.thread.join(timeout=2)
-        for k in ("PRAGENT_FEEDBACK_DB", "PRAGENT_BOT_TOKEN"):
-            os.environ.pop(k, None)
-        self.tmp.cleanup()
-
-    def _urlopen(self, req, timeout=30):
-        """Replacement for urllib.request.urlopen that the handler uses."""
-        url = req.full_url if hasattr(req, "full_url") else req
-        method = getattr(req, "method", None) or "GET"
-        body = getattr(req, "data", None)
-        headers = dict(getattr(req, "headers", {}) or {})
-        self.calls.append((method, url, headers, body))
-        # Route based on URL: GET contents/.../raw vs PUT contents/.pr-review.json
-        if method == "GET" and ".pr-review.json" in url:
-            return _FakeResp(200, json.dumps({
-                "content": base64.b64encode(b'{"focus":["x"],"model":"claude-haiku-4-5"}').decode(),
-                "sha": "deadbeef",
-            }).encode())
-        if method == "PUT" and ".pr-review.json" in url:
-            return _FakeResp(200, b'{}')
-        return _FakeResp(404, b'{"message":"not found"}')
-
-    def test_edit_updates_static_message_and_model(self):
-        form = (
-            f"_csrf={self.csrf}"
-            f"&static_message=Hello%20world"
-            f"&model=claude-sonnet-5"
-        ).encode()
-        with patch.object(dash.urllib.request, "urlopen", side_effect=self._urlopen):
-            status, h, _b = _post(self.port, "/r/o/r/edit", form, headers=self.auth_hdr)
-        self.assertEqual(status, 302)
-        self.assertEqual(h.get("Location"), "/r/o/r")
-
-        # Find the PUT call.
-        put_calls = [c for c in self.calls if c[0] == "PUT"]
-        self.assertEqual(len(put_calls), 1, self.calls)
-        method, url, _hdrs, body = put_calls[0]
-        self.assertIn(".pr-review.json", url)
-        payload = json.loads(body)
-        self.assertIn("content", payload)
-        self.assertEqual(payload["sha"], "deadbeef")
-        decoded = base64.b64decode(payload["content"]).decode()
-        cfg = json.loads(decoded)
-        self.assertEqual(cfg.get("static_message"), "Hello world")
-        self.assertEqual(cfg.get("model"), "claude-sonnet-5")
-
-    def test_edit_strips_static_message_to_400(self):
-        long_msg = "x" * 600
-        form = (
-            f"_csrf={self.csrf}"
-            f"&static_message={long_msg}"
-            f"&model=claude-haiku-4-5"
-        ).encode()
-        with patch.object(dash.urllib.request, "urlopen", side_effect=self._urlopen):
-            _post(self.port, "/r/o/r/edit", form, headers=self.auth_hdr)
-        put = next(c for c in self.calls if c[0] == "PUT")
-        cfg = json.loads(base64.b64decode(json.loads(put[3])["content"]))
-        self.assertEqual(len(cfg["static_message"]), 400)
-
-    def test_edit_rejects_unknown_model_with_flash(self):
-        form = (
-            f"_csrf={self.csrf}"
-            f"&static_message=hi"
-            f"&model=does-not-exist"
-        ).encode()
-        with patch.object(dash.urllib.request, "urlopen", side_effect=self._urlopen):
-            status, h, _b = _post(self.port, "/r/o/r/edit", form, headers=self.auth_hdr)
-        self.assertEqual(status, 302)
-        self.assertIn("flash=", h.get("Location", ""))
-        # No PUT should have been issued.
-        put_calls = [c for c in self.calls if c[0] == "PUT"]
-        self.assertEqual(put_calls, [])
-
-    def test_edit_requires_auth(self):
-        form = f"_csrf={self.csrf}&static_message=x&model=claude-haiku-4-5".encode()
-        with patch.object(dash.urllib.request, "urlopen", side_effect=self._urlopen):
-            status, h, _b = _post(self.port, "/r/o/r/edit", form)
-        self.assertEqual(status, 401)
-        self.assertEqual(h.get("WWW-Authenticate"), 'Basic realm="pragent-dashboard"')
-        # No Gitea calls at all — auth gate fires first.
-        self.assertEqual(self.calls, [])
-
-    def test_edit_csrf_mismatch_redirects_without_save(self):
-        form = f"_csrf=wrong&static_message=x&model=claude-haiku-4-5".encode()
-        with patch.object(dash.urllib.request, "urlopen", side_effect=self._urlopen):
-            status, h, _b = _post(self.port, "/r/o/r/edit", form, headers=self.auth_hdr)
-        self.assertEqual(status, 302)
-        self.assertEqual(h.get("Location"), "/r/o/r")
-        put_calls = [c for c in self.calls if c[0] == "PUT"]
-        self.assertEqual(put_calls, [])
-
-    def test_edit_creates_file_when_missing(self):
-        """When GET returns 404, the PUT must still happen (no sha)."""
-        def _route(req, timeout=30):
-            url = req.full_url
-            method = getattr(req, "method", None) or "GET"
-            body = getattr(req, "data", None)
-            self.calls.append((method, url, {}, body))
-            if method == "GET" and ".pr-review.json" in url:
-                return _FakeResp(404, b'{"message":"not found"}')
-            if method == "PUT" and ".pr-review.json" in url:
-                return _FakeResp(201, b"{}")
-            return _FakeResp(404, b"")
-
-        form = f"_csrf={self.csrf}&static_message=hi&model=claude-haiku-4-5".encode()
-        with patch.object(dash.urllib.request, "urlopen", side_effect=_route):
-            status, h, _b = _post(self.port, "/r/o/r/edit", form, headers=self.auth_hdr)
-        self.assertEqual(status, 302)
-        put = next(c for c in self.calls if c[0] == "PUT")
-        payload = json.loads(put[3])
-        self.assertNotIn("sha", payload, "missing-file PUT should omit sha")
-
-
-if __name__ == "__main__":
-    unittest.main()

tests/pilot/test_dashboard_select.py

@@ -1,74 +0,0 @@
-"""Tests for the model <select> in the repo edit form (Task D)."""
-import http.client
-import os
-import socket
-import sys
-import tempfile
-import threading
-import unittest
-
-HERE = os.path.dirname(os.path.abspath(__file__))
-ROOT = os.path.abspath(os.path.join(HERE, "..", ".."))
-sys.path.insert(0, os.path.join(ROOT, "pilot"))
-
-import dashboard as dash  # noqa: E402
-from pilot import cost_model, feedback  # noqa: E402
-
-
-def _free_port() -> int:
-    s = socket.socket()
-    s.bind(("127.0.0.1", 0))
-    port = s.getsockname()[1]
-    s.close()
-    return port
-
-
-def _get(port: int, path: str, headers: dict | None = None) -> tuple[int, dict, bytes]:
-    conn = http.client.HTTPConnection("127.0.0.1", port, timeout=5)
-    conn.request("GET", path, headers=headers or {})
-    r = conn.getresponse()
-    body = r.read()
-    h = dict(r.getheaders())
-    conn.close()
-    return r.status, h, body
-
-
-class TestRepoEditSelect(unittest.TestCase):
-    def setUp(self):
-        self.tmp = tempfile.TemporaryDirectory()
-        self.db = f"{self.tmp.name}/f.db"
-        os.environ["PRAGENT_FEEDBACK_DB"] = self.db
-        os.environ["PRAGENT_BOT_TOKEN"] = ""
-        conn = feedback.init(self.db)
-        feedback.record_review(conn, repo="o/r", pr=1, head_sha="x")
-        conn.close()
-
-        self.port = _free_port()
-        from http.server import ThreadingHTTPServer
-        self.srv = ThreadingHTTPServer(("127.0.0.1", self.port), dash.Handler)
-        self.thread = threading.Thread(target=self.srv.serve_forever, daemon=True)
-        self.thread.start()
-        self.auth_hdr = {"X-Forwarded-User": "marcos@example.com"}
-
-    def tearDown(self):
-        self.srv.shutdown()
-        self.srv.server_close()
-        self.thread.join(timeout=2)
-        for k in ("PRAGENT_FEEDBACK_DB", "PRAGENT_BOT_TOKEN"):
-            os.environ.pop(k, None)
-        self.tmp.cleanup()
-
-    def test_repo_page_renders_select_with_one_option_per_price(self):
-        status, _h, body = _get(self.port, "/r/o/r", headers=self.auth_hdr)
-        self.assertEqual(status, 200)
-        text = body.decode()
-        self.assertIn('<select id="model" name="model">', text)
-        # Every PRICES key should appear as an <option value="…">.
-        for k in sorted(cost_model.PRICES):
-            self.assertIn(f'<option value="{k}"', text, f"missing {k} in select")
-        # Plus the "keep current" placeholder.
-        self.assertIn("— keep current", text)
-
-
-if __name__ == "__main__":
-    unittest.main()

tests/pilot/test_eval_judges.py

@@ -0,0 +1,132 @@
+
+
+"""Tests for the LLM-as-judge evaluator bootstrap."""
+import os
+import sys
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "pilot"))
+
+import eval_judges as ej  # noqa: E402
+
+
+# --- rule_body ------------------------------------------------------------
+
+def test_rule_body_targets_traces():
+    """Trace target matches the path `/api/public/ingestion` triggers.
+
+    Observation rules only fire from the OTel ingestion pipeline; this
+    pilot uses standard ingestion, so its jobs only come from
+    `evalService.createEvalJobs` and that dispatcher handles
+    `targetObject ∈ {TRACE, DATASET}`.
+    """
+    body = ej.rule_body("rule-x", "finding_actionability", 1.0)
+    assert body["target"] == "trace"
+    assert body["enabled"] is True
+
+
+def test_rule_body_filters_on_trace_name():
+    """`name` isn't a stringOptions column; only `traceName` is."""
+    body = ej.rule_body("rule-x", "finding_actionability", 1.0)
+    f = body["filter"][0]
+    assert f["column"] == "traceName"
+    assert f["operator"] == "any of"
+    assert f["type"] == "stringOptions"
+    assert "pr-review" in f["value"]
+
+
+def test_rule_body_references_evaluator_by_name():
+    """Ids are version-specific; rules must name the evaluator across versions."""
+    body = ej.rule_body("rule-x", "finding_actionability", 1.0)
+    assert body["evaluator"]["name"] == "finding_actionability"
+    assert body["evaluator"]["scope"] == "project"
+
+
+def test_rule_body_maps_input_and_output():
+    """Both judges read the observation's own input/output."""
+    body = ej.rule_body("rule-x", "any", 1.0)
+    sources = {m["source"] for m in body["mapping"]}
+    assert sources == {"input", "output"}
+
+
+def test_rule_body_carries_mapping_at_both_levels():
+    """The server validates `mapping` at the rule root and echoes it on the evaluator."""
+    body = ej.rule_body("rule-x", "any", 1.0)
+    assert body["mapping"]
+    assert body["evaluator"]["variableMapping"] == body["mapping"]
+
+
+def test_rule_body_passes_sampling_through():
+    assert ej.rule_body("r", "any", 0.25)["sampling"] == 0.25
+
+
+# --- ensure_evaluators idempotency ---------------------------------------
+
+def test_ensure_evaluators_skips_existing(monkeypatch):
+    seen = []
+
+    def fake_call(method, path, body=None, timeout=20.0):
+        seen.append(path)
+        return 200, {}
+
+    monkeypatch.setattr(ej.eb, "_call", fake_call)
+    monkeypatch.setattr(ej, "existing_evaluators",
+                        lambda: {"finding_actionability": "id-1", "review_self_consistency": "id-2"})
+    res = ej.ensure_evaluators()
+    assert res["created"] == {}
+    assert sorted(res["skipped"]) == ["finding_actionability", "review_self_consistency"]
+    assert res["failed"] == []
+    assert seen == []
+
+
+def test_ensure_evaluators_records_failures(monkeypatch):
+    def fake_call(method, path, body=None, timeout=20.0):
+        return 422, "boom"
+
+    monkeypatch.setattr(ej.eb, "_call", fake_call)
+    monkeypatch.setattr(ej, "existing_evaluators", lambda: {})
+    res = ej.ensure_evaluators()
+    assert res["created"] == {}
+    assert res["failed"][0]["status"] == 422
+
+
+# --- ensure_rules idempotency --------------------------------------------
+
+def test_ensure_rules_skips_existing(monkeypatch):
+    calls = []
+    monkeypatch.setattr(ej.eb, "_call",
+                        lambda *a, **k: calls.append(a) or (200, {}))
+    monkeypatch.setattr(ej, "existing_evaluators",
+                        lambda: {"finding_actionability": "id-1",
+                                 "review_self_consistency": "id-2"})
+    monkeypatch.setattr(ej, "existing_rule_names",
+                        lambda: {"finding_actionability-on-reviews",
+                                 "review_self_consistency-on-reviews"})
+    res = ej.ensure_rules({"finding_actionability": "id-1",
+                           "review_self_consistency": "id-2"}, 1.0)
+    assert res["created"] == []
+    assert sorted(res["skipped"]) == ["finding_actionability", "review_self_consistency"]
+    assert calls == []
+
+
+def test_ensure_rules_creates_when_missing(monkeypatch):
+    calls = []
+    monkeypatch.setattr(ej.eb, "_call",
+                        lambda *a, **k: calls.append(a) or (201, {}))
+    monkeypatch.setattr(ej, "existing_rule_names", lambda: set())
+    res = ej.ensure_rules({"finding_actionability": "id-1"}, 1.0)
+    assert res["created"] == ["finding_actionability"]
+    assert calls[0][0] == "POST"
+    assert calls[0][1] == "/api/public/unstable/evaluation-rules"
+
+
+# --- judge shape ----------------------------------------------------------
+
+def test_judges_have_required_keys():
+    for j in ej.JUDGES:
+        assert j["prompt"]
+        assert j["outputDefinition"]["dataType"] in ("NUMERIC", "BOOLEAN", "CATEGORICAL")
+
+
+def test_default_base_url_points_at_the_thinking_patch_proxy():
+    """`8802` is the judge-proxy that adds a `signature` to thinking blocks."""
+    assert "8802" in ej.JUDGE_BASE_URL

tests/pilot/test_eval_scores.py

@@ -0,0 +1,200 @@
+"""Tests for the deterministic review scorers."""
+import os
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "pilot"))
+
+import eval_scores as es  # noqa: E402
+
+
+def f(sev, path="a.py", line=1):
+    return {"severity": sev, "path": path, "line": line, "problem": "p", "fix": ""}
+
+
+# --- finding_rate ---------------------------------------------------------
+
+def test_finding_rate_counts_findings():
+    assert es.finding_rate([f("high"), f("low")]) == 2.0
+
+
+def test_finding_rate_zero_for_silent_review():
+    assert es.finding_rate([]) == 0.0
+    assert es.finding_rate(None) == 0.0
+
+
+# --- severity_info_ratio --------------------------------------------------
+
+def test_info_ratio_all_advisory():
+    assert es.severity_info_ratio([f("info"), f("trivial")]) == 1.0
+
+
+def test_info_ratio_mixed():
+    assert es.severity_info_ratio([f("info"), f("high")]) == 0.5
+
+
+def test_info_ratio_none_when_no_findings():
+    # Undefined, not zero — zero would read as perfectly calibrated.
+    assert es.severity_info_ratio([]) is None
+
+
+def test_info_ratio_unknown_severity_treated_as_medium():
+    # Matches _normalize_finding's fallback, so an odd severity is not
+    # silently counted as advisory.
+    assert es.severity_info_ratio([f("bogus")]) == 0.0
+
+
+# --- severity_max ---------------------------------------------------------
+
+def test_severity_max_picks_highest():
+    assert es.severity_max([f("info"), f("critical"), f("low")]) == "critical"
+
+
+def test_severity_max_none_when_silent():
+    assert es.severity_max([]) == "none"
+
+
+def test_severity_max_case_insensitive():
+    assert es.severity_max([f("HIGH")]) == "high"
+
+
+# --- dropped_findings -----------------------------------------------------
+
+def test_dropped_findings_delta():
+    assert es.dropped_findings(5, 2) == 3.0
+
+
+def test_dropped_findings_never_negative():
+    assert es.dropped_findings(1, 3) == 0.0
+
+
+def test_dropped_findings_none_when_unknown():
+    assert es.dropped_findings(None, 2) is None
+
+
+# --- cost_per_finding -----------------------------------------------------
+
+def test_cost_per_finding_divides():
+    assert es.cost_per_finding(1.0, [f("high"), f("low")]) == 0.5
+
+
+def test_cost_per_finding_silent_review_divides_by_one():
+    # The run still cost money; attributing all of it to "found nothing" is
+    # the honest reading, and it avoids a division by zero.
+    assert es.cost_per_finding(0.25, []) == 0.25
+
+
+def test_cost_per_finding_none_when_unpriced():
+    assert es.cost_per_finding(None, [f("high")]) is None
+
+
+def test_cost_per_finding_none_on_garbage():
+    assert es.cost_per_finding("abc", [f("high")]) is None
+
+
+# --- build_scores ---------------------------------------------------------
+
+def _by_name(events):
+    return {e["body"]["name"]: e["body"] for e in events}
+
+
+def test_build_scores_emits_expected_set():
+    events = es.build_scores(
+        trace_id="t1", findings=[f("high"), f("info")], environment="claude",
+        cost_usd=0.5, dropped_count=2, timestamp="2026-01-01T00:00:00Z",
+    )
+    names = _by_name(events)
+    assert set(names) == {
+        es.FINDING_RATE, es.SEVERITY_INFO_RATIO, es.SEVERITY_MAX,
+        es.DROPPED_FINDINGS, es.COST_PER_FINDING,
+    }
+    assert names[es.FINDING_RATE]["value"] == 2.0
+    assert names[es.SEVERITY_MAX]["value"] == "high"
+    assert names[es.DROPPED_FINDINGS]["value"] == 2.0
+    assert names[es.COST_PER_FINDING]["value"] == 0.25
+
+
+def test_build_scores_all_events_are_score_create_on_the_trace():
+    events = es.build_scores(
+        trace_id="t9", findings=[f("low")], environment="ollama", cost_usd=1.0,
+    )
+    assert all(e["type"] == "score-create" for e in events)
+    assert all(e["body"]["traceId"] == "t9" for e in events)
+    assert all(e["body"]["environment"] == "ollama" for e in events)
+
+
+def test_build_scores_omits_undefined_scores():
+    # No cost and no drop count measured -> those scores are absent, not zero.
+    events = es.build_scores(trace_id="t2", findings=[], environment="ollama")
+    names = set(_by_name(events))
+    assert es.COST_PER_FINDING not in names
+    assert es.DROPPED_FINDINGS not in names
+    assert es.SEVERITY_INFO_RATIO not in names
+    assert names == {es.FINDING_RATE, es.SEVERITY_MAX}
+
+
+def test_build_scores_categorical_value_is_string():
+    events = es.build_scores(trace_id="t3", findings=[f("high")], environment="claude")
+    sev = _by_name(events)[es.SEVERITY_MAX]
+    assert sev["dataType"] == "CATEGORICAL"
+    assert isinstance(sev["value"], str)
+
+
+def test_build_scores_numeric_values_are_floats():
+    events = es.build_scores(
+        trace_id="t4", findings=[f("high")], environment="claude", cost_usd=1,
+    )
+    for name, body in _by_name(events).items():
+        if body["dataType"] == "NUMERIC":
+            assert isinstance(body["value"], float), name
+
+
+def test_build_scores_comment_propagates():
+    events = es.build_scores(
+        trace_id="t5", findings=[f("high")], environment="claude",
+        cost_usd=1.0, comment="cost basis: equivalent:claude-sonnet-5",
+    )
+    assert all("equivalent" in e["body"]["comment"] for e in events)
+
+
+# --- score configs --------------------------------------------------------
+
+def test_every_emitted_score_has_a_config():
+    configured = {c["name"] for c in es.SCORE_CONFIGS}
+    events = es.build_scores(
+        trace_id="t6", findings=[f("high")], environment="claude",
+        cost_usd=1.0, dropped_count=0,
+    )
+    assert set(_by_name(events)) <= configured
+
+
+def test_severity_max_config_covers_every_severity_it_can_emit():
+    labels = {c["label"] for c in
+              next(c for c in es.SCORE_CONFIGS if c["name"] == es.SEVERITY_MAX)["categories"]}
+    assert set(es.SEVERITY_RANK) | {"none"} == labels
+
+
+# --- ingestion envelope ---------------------------------------------------
+
+def test_every_event_carries_a_timestamp():
+    # Ingestion rejects events without one, and reports the rejection as a
+    # per-event 400 inside an HTTP 207 that reads as success.
+    events = es.build_scores(
+        trace_id="t7", findings=[f("high")], environment="claude", cost_usd=1.0,
+    )
+    assert events
+    assert all(e.get("timestamp") for e in events)
+
+
+def test_timestamp_defaults_when_caller_omits_it():
+    events = es.build_scores(trace_id="t8", findings=[f("low")], environment="claude")
+    assert all(isinstance(e["timestamp"], str) and e["timestamp"].endswith("Z") for e in events)
+
+
+def test_explicit_timestamp_is_used():
+    events = es.build_scores(
+        trace_id="t9", findings=[f("low")], environment="claude",
+        timestamp="2026-01-02T03:04:05Z",
+    )
+    assert all(e["timestamp"] == "2026-01-02T03:04:05Z" for e in events)

tests/pilot/test_feedback_scores.py

@@ -0,0 +1,207 @@
+"""Tests for the feedback.db -> Langfuse score bridge."""
+import os
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "pilot"))
+
+import feedback  # noqa: E402
+import feedback_scores as fs  # noqa: E402
+
+
+@pytest.fixture
+def db(tmp_path):
+    conn = feedback.init(str(tmp_path / "fb.db"))
+    yield conn
+    conn.close()
+
+
+def _seed_finding(conn, repo="o/r", pr=1, comment_id=100, path="a.py", line=1):
+    cur = conn.execute(
+        "INSERT INTO review (repo, pr, head_sha, posted_at) VALUES (?,?,?,?)",
+        (repo, pr, "deadbeef", 1000),
+    )
+    review_id = cur.lastrowid
+    cur = conn.execute(
+        """INSERT INTO inline_finding
+           (review_id, repo, pr, path, line, severity, problem, comment_id, posthash, posted_at)
+           VALUES (?,?,?,?,?,?,?,?,?,?)""",
+        (review_id, repo, pr, path, line, "HIGH", "problem", comment_id, f"h{comment_id}", 1000),
+    )
+    conn.commit()
+    return cur.lastrowid
+
+
+# --- score_pr maths -------------------------------------------------------
+
+def test_engagement_zero_when_nobody_responded():
+    v = fs.score_pr({"total": 4, "engaged": 0, "positive": 0, "negative": 0})
+    assert v[fs.REVIEW_ENGAGEMENT] == 0.0
+
+
+def test_acceptance_absent_when_nobody_engaged():
+    # Not 0.0 — zero would claim humans judged it neutral.
+    v = fs.score_pr({"total": 4, "engaged": 0, "positive": 0, "negative": 0})
+    assert v[fs.REVIEW_ACCEPTANCE] is None
+
+
+def test_engagement_is_a_share_of_findings():
+    v = fs.score_pr({"total": 4, "engaged": 1, "positive": 1, "negative": 0})
+    assert v[fs.REVIEW_ENGAGEMENT] == 0.25
+
+
+def test_acceptance_all_positive():
+    v = fs.score_pr({"total": 2, "engaged": 2, "positive": 3, "negative": 0})
+    assert v[fs.REVIEW_ACCEPTANCE] == 1.0
+
+
+def test_acceptance_all_negative():
+    v = fs.score_pr({"total": 2, "engaged": 2, "positive": 0, "negative": 2})
+    assert v[fs.REVIEW_ACCEPTANCE] == -1.0
+
+
+def test_acceptance_mixed_is_normalised():
+    v = fs.score_pr({"total": 4, "engaged": 4, "positive": 3, "negative": 1})
+    assert v[fs.REVIEW_ACCEPTANCE] == 0.5
+
+
+def test_engagement_absent_when_no_findings_at_all():
+    v = fs.score_pr({"total": 0, "engaged": 0, "positive": 0, "negative": 0})
+    assert v[fs.REVIEW_ENGAGEMENT] is None
+
+
+# --- collect_pr_feedback over a real sqlite ------------------------------
+
+def test_collect_counts_nothing_on_untouched_findings(db):
+    _seed_finding(db)
+    tally = fs.collect_pr_feedback(db, "o/r", 1)
+    assert tally == {"total": 1, "engaged": 0, "positive": 0, "negative": 0}
+
+
+def test_collect_counts_positive_reaction(db):
+    _seed_finding(db, comment_id=101)
+    db.execute(
+        "INSERT INTO reaction (comment_id, user, content, created_at) VALUES (?,?,?,?)",
+        (101, "alice", "+1", 1),
+    )
+    db.commit()
+    tally = fs.collect_pr_feedback(db, "o/r", 1)
+    assert tally["positive"] == 1 and tally["engaged"] == 1
+
+
+def test_collect_counts_negative_reaction(db):
+    _seed_finding(db, comment_id=102)
+    db.execute(
+        "INSERT INTO reaction (comment_id, user, content, created_at) VALUES (?,?,?,?)",
+        (102, "bob", "-1", 1),
+    )
+    db.commit()
+    tally = fs.collect_pr_feedback(db, "o/r", 1)
+    assert tally["negative"] == 1 and tally["engaged"] == 1
+
+
+def test_resolved_thread_counts_positive(db):
+    fid = _seed_finding(db, comment_id=103)
+    db.execute(
+        "INSERT INTO thread_state (finding_id, resolved, checked_at) VALUES (?,?,?)",
+        (fid, 1, 1),
+    )
+    db.commit()
+    tally = fs.collect_pr_feedback(db, "o/r", 1)
+    assert tally["positive"] == 1 and tally["engaged"] == 1
+
+
+def test_unresolved_thread_is_not_a_vote(db):
+    fid = _seed_finding(db, comment_id=104)
+    db.execute(
+        "INSERT INTO thread_state (finding_id, resolved, checked_at) VALUES (?,?,?)",
+        (fid, 0, 1),
+    )
+    db.commit()
+    tally = fs.collect_pr_feedback(db, "o/r", 1)
+    assert tally == {"total": 1, "engaged": 0, "positive": 0, "negative": 0}
+
+
+def test_negation_reply_counts_negative(db):
+    fid = _seed_finding(db, comment_id=105)
+    db.execute(
+        "INSERT INTO reply (finding_id, author, body, created_at) VALUES (?,?,?,?)",
+        (fid, "carol", "this is a false positive", 1),
+    )
+    db.commit()
+    tally = fs.collect_pr_feedback(db, "o/r", 1)
+    assert tally["negative"] == 1 and tally["engaged"] == 1
+
+
+def test_neutral_reply_is_engagement_but_not_a_vote(db):
+    fid = _seed_finding(db, comment_id=106)
+    db.execute(
+        "INSERT INTO reply (finding_id, author, body, created_at) VALUES (?,?,?,?)",
+        (fid, "dave", "done", 1),
+    )
+    db.commit()
+    tally = fs.collect_pr_feedback(db, "o/r", 1)
+    assert tally["engaged"] == 1
+    assert tally["positive"] == 0 and tally["negative"] == 0
+
+
+# --- event shape ----------------------------------------------------------
+
+def test_build_score_events_shape():
+    events = fs.build_score_events("o/r", 7, {fs.REVIEW_ENGAGEMENT: 0.5}, "claude")
+    assert len(events) == 1
+    body = events[0]["body"]
+    assert events[0]["type"] == "score-create"
+    assert body["sessionId"] == "o/r#7"
+    assert body["value"] == 0.5
+    assert body["environment"] == "claude"
+
+
+def test_build_score_events_skips_none():
+    events = fs.build_score_events("o/r", 7, {fs.REVIEW_ACCEPTANCE: None})
+    assert events == []
+
+
+def test_score_ids_are_stable_across_runs():
+    # A backfill re-run must update, not duplicate.
+    a = fs.build_score_events("o/r", 7, {fs.REVIEW_ENGAGEMENT: 0.5})[0]["body"]["id"]
+    b = fs.build_score_events("o/r", 7, {fs.REVIEW_ENGAGEMENT: 0.9})[0]["body"]["id"]
+    assert a == b
+
+
+def test_score_ids_differ_per_pr_and_name():
+    e1 = fs.build_score_events("o/r", 7, {fs.REVIEW_ENGAGEMENT: 1})[0]["body"]["id"]
+    e2 = fs.build_score_events("o/r", 8, {fs.REVIEW_ENGAGEMENT: 1})[0]["body"]["id"]
+    e3 = fs.build_score_events("o/r", 7, {fs.REVIEW_ACCEPTANCE: 1})[0]["body"]["id"]
+    assert len({e1, e2, e3}) == 3
+
+
+def test_backfill_dry_run_reports_without_posting(db, tmp_path):
+    _seed_finding(db, comment_id=107)
+    db.commit()
+    path = db.execute("PRAGMA database_list").fetchone()[2]
+    summary = fs.backfill(path, dry_run=True)
+    assert summary["prs_scanned"] == 1
+    assert summary["prs_with_engagement"] == 0
+    assert summary["posted"] is False
+
+
+def test_every_emitted_score_has_a_config():
+    configured = {c["name"] for c in fs.SCORE_CONFIGS}
+    assert {fs.REVIEW_ENGAGEMENT, fs.REVIEW_ACCEPTANCE} == configured
+
+
+def test_every_event_carries_a_timestamp():
+    # Without one the ingestion endpoint 400s the event inside a 207 that the
+    # caller reads as success.
+    events = fs.build_score_events("o/r", 1, {fs.REVIEW_ENGAGEMENT: 0.0})
+    assert events
+    assert all(e.get("timestamp") for e in events)
+
+
+def test_explicit_timestamp_is_used():
+    events = fs.build_score_events(
+        "o/r", 1, {fs.REVIEW_ENGAGEMENT: 0.0}, timestamp="2026-01-02T03:04:05Z"
+    )
+    assert events[0]["timestamp"] == "2026-01-02T03:04:05Z"

tests/pilot/test_langfuse_trace.py

@@ -0,0 +1,326 @@
+"""Unit tests for Langfuse trace emission. No network.
+
+`_post` is monkeypatched everywhere a POST would happen; a test that reaches
+the real network is a bug in the test, not a slow test.
+"""
+import json
+import os
+import sys
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+ROOT = os.path.abspath(os.path.join(HERE, "..", ".."))
+sys.path.insert(0, os.path.join(ROOT, "pilot"))
+
+import langfuse_trace as lt  # noqa: E402
+
+
+USAGE = {
+    "input": 2_000_000,
+    "output": 17_000,
+    "reasoning": 500,
+    "cache_read": 400_000,
+    "cache_write": 50_000,
+    "total": 2_017_000,
+    "cost": 0.0,
+    "steps": 28,
+    "duration_s": 348.3,
+}
+
+BASE = dict(
+    repo="techspark/pragent",
+    index="42",
+    sha="2613b3e1122334455",
+    title="Harden the review path",
+    usage=USAGE,
+    findings=[
+        {"severity": "critical", "path": "a.py"},
+        {"severity": "minor", "path": "b.py"},
+        {"severity": "minor", "path": "c.py"},
+    ],
+    summary="Three findings.",
+)
+
+
+# ---------------------------------------------------------------------------
+# model -> environment split (the whole point of the integration)
+# ---------------------------------------------------------------------------
+
+
+def test_claude_models_land_in_the_claude_environment():
+    assert lt.resolve_environment("headroom/claude-sonnet-5") == "claude"
+    assert lt.resolve_environment("claude-opus-5") == "claude"
+
+
+def test_everything_else_lands_in_the_ollama_environment():
+    for m in (
+        "headroom/glm-5.2:cloud",
+        "headroom/MiniMax-M2.7",
+        "vllm-qwen38/qwen3.8-27b",
+        "gpt-5",
+    ):
+        assert lt.resolve_environment(m) == "ollama", m
+
+
+def test_provider_and_bare_model_are_split_on_the_first_slash_only():
+    assert lt.provider_of("vllm-qwen38/qwen3.8-27b") == "vllm-qwen38"
+    assert lt.strip_provider("headroom/glm-5.2:cloud") == "glm-5.2:cloud"
+    # A bare name has no provider prefix; default to the pilot's proxy.
+    assert lt.provider_of("glm-5.2:cloud") == "headroom"
+    assert lt.strip_provider("glm-5.2:cloud") == "glm-5.2:cloud"
+
+
+# ---------------------------------------------------------------------------
+# usage accounting
+# ---------------------------------------------------------------------------
+
+
+def test_cache_reads_are_subtracted_from_input_not_added():
+    # Langfuse sums usageDetails keys; opencode reports cache_read *inside*
+    # input, so reporting both raw would bill the prefix twice.
+    d = lt._usage_details(USAGE)
+    assert d["input"] == 2_000_000 - 400_000
+    assert d["cache_read_input_tokens"] == 400_000
+    assert d["cache_write_input_tokens"] == 50_000
+    assert d["output"] == 17_000
+    assert d["reasoning"] == 500
+
+
+def test_zero_cache_fields_are_omitted_rather_than_sent_as_zero():
+    d = lt._usage_details({"input": 100, "output": 10})
+    assert d == {"input": 100, "output": 10}
+
+
+def test_a_paid_model_is_priced_as_itself():
+    costs, basis = lt._cost_details(USAGE, "headroom/claude-sonnet-5")
+    assert costs["total"] > 0
+    assert basis == "actual"
+
+
+def test_minimax_is_priced_against_the_comparison_target_not_zero():
+    # MiniMax-M2.7 is the model the webhook actually runs and it is absent from
+    # PRICES; charting it at $0 would make the whole dashboard a flat line.
+    costs, basis = lt._cost_details(USAGE, "headroom/MiniMax-M2.7")
+    assert costs["total"] > 0
+    assert basis == "equivalent:claude-sonnet-5"
+
+
+def test_glm_is_priced_against_the_comparison_target():
+    costs, basis = lt._cost_details(USAGE, "headroom/glm-5.2:cloud")
+    assert costs["total"] > 0
+    assert basis.startswith("equivalent:")
+
+
+def test_an_all_zero_price_entry_counts_as_free_not_as_priced():
+    # The self-hosted vLLM qwen IS in PRICES, at 0.00 across the board.
+    costs, basis = lt._cost_details(USAGE, "vllm-qwen38/qwen3.8-27b")
+    assert costs["total"] > 0
+    assert basis.startswith("equivalent:")
+
+
+def test_explicit_price_target_wins_over_the_default():
+    costs, basis = lt._cost_details(USAGE, "headroom/MiniMax-M2.7", "claude-opus-5")
+    assert basis == "equivalent:claude-opus-5"
+    sonnet, _ = lt._cost_details(USAGE, "headroom/MiniMax-M2.7", "claude-sonnet-5")
+    assert costs["total"] > sonnet["total"]
+
+
+def test_env_overrides_the_default_target(monkeypatch):
+    monkeypatch.setenv("PRAGENT_PRICE_TARGET", "claude-haiku-4-5")
+    assert lt.resolve_price_target() == "claude-haiku-4-5"
+    # An explicit argument still beats the env.
+    assert lt.resolve_price_target("gpt-5") == "gpt-5"
+
+
+def test_unknown_comparison_target_yields_no_cost_block_rather_than_a_wrong_one():
+    costs, basis = lt._cost_details(USAGE, "headroom/MiniMax-M2.7", "not-a-real-model")
+    assert costs == {}
+    assert basis == ""
+
+
+# ---------------------------------------------------------------------------
+# batch shape
+# ---------------------------------------------------------------------------
+
+
+def test_batch_has_a_trace_and_a_generation_linked_by_trace_id():
+    batch = lt.build_batch(model="headroom/claude-sonnet-5", **BASE)
+    types = [e["type"] for e in batch]
+    # Scores ride in the same batch; the trace and generation lead it.
+    assert types[:2] == ["trace-create", "generation-create"]
+    trace, gen = batch[0], batch[1]
+    assert gen["body"]["traceId"] == trace["body"]["id"]
+    assert trace["body"]["environment"] == gen["body"]["environment"] == "claude"
+
+
+def test_batch_without_usage_has_no_generation():
+    batch = lt.build_batch(model="headroom/glm-5.2:cloud", **{**BASE, "usage": None})
+    types = [e["type"] for e in batch]
+    assert "generation-create" not in types
+    assert types[0] == "trace-create"
+
+
+def test_trace_carries_repo_pr_session_and_severity_counts():
+    batch = lt.build_batch(model="headroom/glm-5.2:cloud", **BASE)
+    body = batch[0]["body"]
+    assert body["sessionId"] == "techspark/pragent#42"
+    assert body["metadata"]["severities"] == {"critical": 1, "minor": 2}
+    assert body["metadata"]["findings"] == 3
+    assert "provider:headroom" in body["tags"]
+    assert "model:glm-5.2:cloud" in body["tags"]
+
+
+def test_lens_names_become_tags():
+    batch = lt.build_batch(
+        model="headroom/glm-5.2:cloud", lenses=["security", "tests"], **BASE
+    )
+    assert "lens:security" in batch[0]["body"]["tags"]
+    assert "lens:tests" in batch[0]["body"]["tags"]
+
+
+def test_cost_basis_is_tagged_so_equivalent_is_never_read_as_spend():
+    batch = lt.build_batch(model="headroom/MiniMax-M2.7", **BASE)
+    trace = batch[0]["body"]
+    assert "cost:equivalent:claude-sonnet-5" in trace["tags"]
+    assert trace["metadata"]["cost_basis"] == "equivalent:claude-sonnet-5"
+
+    paid = lt.build_batch(model="headroom/claude-sonnet-5", **BASE)
+    assert "cost:actual" in paid[0]["body"]["tags"]
+
+
+def test_minimax_generation_carries_a_nonzero_cost():
+    batch = lt.build_batch(model="headroom/MiniMax-M2.7", **BASE)
+    assert batch[1]["body"]["costDetails"]["total"] > 0
+
+
+def test_batch_is_json_serializable():
+    batch = lt.build_batch(model="headroom/claude-sonnet-5", **BASE)
+    json.dumps({"batch": batch})
+
+
+# ---------------------------------------------------------------------------
+# emit_review_trace — config gate and fail-open
+# ---------------------------------------------------------------------------
+
+
+def _configure(monkeypatch):
+    monkeypatch.setenv("LANGFUSE_HOST", "http://langfuse.test:3000/")
+    monkeypatch.setenv("LANGFUSE_PUBLIC_KEY", "pk-lf-test")
+    monkeypatch.setenv("LANGFUSE_SECRET_KEY", "sk-lf-test")
+
+
+def test_no_config_means_no_post_and_no_error(monkeypatch):
+    for k in ("LANGFUSE_HOST", "LANGFUSE_PUBLIC_KEY", "LANGFUSE_SECRET_KEY"):
+        monkeypatch.delenv(k, raising=False)
+    calls = []
+    monkeypatch.setattr(lt, "_post", lambda *a, **k: calls.append(a) or 200)
+    assert lt.emit_review_trace(model="headroom/glm-5.2:cloud", **BASE) is False
+    assert calls == []
+
+
+def test_configured_emit_posts_to_the_ingestion_endpoint(monkeypatch):
+    _configure(monkeypatch)
+    seen = {}
+
+    def fake_post(host, pk, sk, batch, timeout):
+        seen.update(host=host, pk=pk, sk=sk, batch=batch, timeout=timeout)
+        return 207
+
+    monkeypatch.setattr(lt, "_post", fake_post)
+    assert lt.emit_review_trace(model="headroom/claude-sonnet-5", **BASE) is True
+    # Trailing slash stripped so the path is not doubled.
+    assert seen["host"] == "http://langfuse.test:3000"
+    kinds = [e["type"] for e in seen["batch"]]
+    assert kinds[:2] == ["trace-create", "generation-create"]
+    assert "score-create" in kinds
+
+
+def test_transport_failure_is_swallowed(monkeypatch):
+    _configure(monkeypatch)
+
+    def boom(*a, **k):
+        raise OSError("connection refused")
+
+    monkeypatch.setattr(lt, "_post", boom)
+    assert lt.emit_review_trace(model="headroom/glm-5.2:cloud", **BASE) is False
+
+
+def test_non_success_status_reports_failure_without_raising(monkeypatch):
+    _configure(monkeypatch)
+    monkeypatch.setattr(lt, "_post", lambda *a, **k: 401)
+    assert lt.emit_review_trace(model="headroom/glm-5.2:cloud", **BASE) is False
+
+
+# ---------------------------------------------------------------------------
+# Scores folded into the review batch (added with eval_scores)
+# ---------------------------------------------------------------------------
+
+def _scores(events):
+    return {e["body"]["name"]: e["body"] for e in events if e["type"] == "score-create"}
+
+
+def test_build_batch_appends_scores():
+    events = lt.build_batch(
+        repo="o/r", index="1", sha="abc", title="t",
+        model="headroom/claude-sonnet-5",
+        usage={"input": 100, "output": 10},
+        findings=[{"severity": "high", "path": "a.py", "line": 1}],
+    )
+    names = set(_scores(events))
+    assert "finding_rate" in names
+    assert "severity_max" in names
+
+
+def test_scores_attach_to_the_same_trace():
+    events = lt.build_batch(
+        repo="o/r", index="1", sha="abc", title="t", model="m",
+        usage={"input": 1, "output": 1}, findings=[], trace_id="fixed-id",
+    )
+    for body in _scores(events).values():
+        assert body["traceId"] == "fixed-id"
+
+
+def test_scores_inherit_the_trace_environment():
+    events = lt.build_batch(
+        repo="o/r", index="1", sha="abc", title="t",
+        model="headroom/glm-5.2:cloud",
+        usage={"input": 1, "output": 1}, findings=[],
+    )
+    for body in _scores(events).values():
+        assert body["environment"] == "ollama"
+
+
+def test_dropped_findings_scored_when_provided():
+    events = lt.build_batch(
+        repo="o/r", index="1", sha="abc", title="t", model="m",
+        usage={"input": 1, "output": 1}, findings=[], dropped_count=3,
+    )
+    assert _scores(events)["dropped_findings"]["value"] == 3.0
+
+
+def test_dropped_findings_absent_when_not_measured():
+    events = lt.build_batch(
+        repo="o/r", index="1", sha="abc", title="t", model="m",
+        usage={"input": 1, "output": 1}, findings=[],
+    )
+    assert "dropped_findings" not in _scores(events)
+
+
+def test_cost_score_carries_its_basis_in_the_comment():
+    # An equivalent-cost $/finding must never be read as money spent.
+    events = lt.build_batch(
+        repo="o/r", index="1", sha="abc", title="t",
+        model="headroom/glm-5.2:cloud",
+        usage={"input": 1000, "output": 100}, findings=[{"severity": "low", "path": "a", "line": 1}],
+    )
+    cpf = _scores(events).get("cost_per_finding")
+    if cpf is not None:  # only when cost_model could price the comparison target
+        assert "equivalent" in cpf["comment"]
+
+
+def test_batch_without_usage_still_scores_findings():
+    # A run with no usage report still produced findings worth scoring.
+    events = lt.build_batch(
+        repo="o/r", index="1", sha="abc", title="t", model="m",
+        usage=None, findings=[{"severity": "critical", "path": "a", "line": 2}],
+    )
+    assert _scores(events)["severity_max"]["value"] == "critical"

tests/pilot/test_parse_dropped.py

@@ -0,0 +1,90 @@
+"""The parse-time drop counter feeding the `dropped_findings` score.
+
+A model that emits findings at unusable locations produces an empty findings
+list, exactly like a model that found nothing. These tests pin the signal that
+tells the two apart.
+"""
+import json
+import os
+import sys
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, os.path.abspath(os.path.join(HERE, "..", "..", "pilot")))
+
+import ai_review  # noqa: E402
+
+
+def _payload(findings):
+    return "```json\n" + json.dumps({"summary": "s", "findings": findings}) + "\n```"
+
+
+GOOD = {"severity": "high", "path": "a.py", "line": 3, "problem": "p", "fix": "f"}
+NO_PATH = {"severity": "high", "line": 3, "problem": "p"}
+NO_LINE = {"severity": "high", "path": "a.py", "problem": "p"}
+BAD_LINE = {"severity": "high", "path": "a.py", "line": 0, "problem": "p"}
+
+
+def test_no_drops_on_clean_output():
+    _, findings, *_ = ai_review.parse_review_output(_payload([GOOD, GOOD]))
+    assert len(findings) == 2
+    assert ai_review.last_parse_dropped() == 0
+
+
+def test_counts_findings_missing_path():
+    _, findings, *_ = ai_review.parse_review_output(_payload([GOOD, NO_PATH]))
+    assert len(findings) == 1
+    assert ai_review.last_parse_dropped() == 1
+
+
+def test_counts_findings_missing_line():
+    _, findings, *_ = ai_review.parse_review_output(_payload([NO_LINE, NO_LINE]))
+    assert findings == []
+    assert ai_review.last_parse_dropped() == 2
+
+
+def test_counts_findings_with_unusable_line():
+    _, findings, *_ = ai_review.parse_review_output(_payload([BAD_LINE]))
+    assert findings == []
+    assert ai_review.last_parse_dropped() == 1
+
+
+def test_all_dropped_is_distinguishable_from_found_nothing():
+    ai_review.parse_review_output(_payload([NO_PATH, NO_PATH, NO_PATH]))
+    all_dropped = ai_review.last_parse_dropped()
+    ai_review.parse_review_output(_payload([]))
+    found_nothing = ai_review.last_parse_dropped()
+    assert all_dropped == 3 and found_nothing == 0
+
+
+def test_counter_resets_on_unparseable_output():
+    # Otherwise a salvage-path review inherits the previous review's count.
+    ai_review.parse_review_output(_payload([NO_PATH, NO_PATH]))
+    assert ai_review.last_parse_dropped() == 2
+    ai_review.parse_review_output("no json here at all")
+    assert ai_review.last_parse_dropped() == 0
+
+
+def test_counter_resets_on_malformed_json():
+    ai_review.parse_review_output(_payload([NO_PATH]))
+    ai_review.parse_review_output("```json\n{not valid json,,,}\n```")
+    assert ai_review.last_parse_dropped() == 0
+
+
+def test_parse_findings_tracks_drops_too():
+    # The non-opencode path must be scored on the same basis.
+    findings = ai_review.parse_findings(json.dumps({"findings": [GOOD, NO_PATH]}))
+    assert len(findings) == 1
+    assert ai_review.last_parse_dropped() == 1
+
+
+def test_parse_findings_resets_on_garbage():
+    ai_review.parse_findings(json.dumps({"findings": [NO_PATH]}))
+    assert ai_review.last_parse_dropped() == 1
+    ai_review.parse_findings("not json")
+    assert ai_review.last_parse_dropped() == 0
+
+
+def test_bare_array_output_is_counted():
+    _, findings, *_ = ai_review.parse_review_output("```json\n" + json.dumps([GOOD, NO_PATH]) + "\n```")
+    assert len(findings) == 1
+    assert ai_review.last_parse_dropped() == 1