Group review, feedback, evaluation, observability, and entrypoint code into packages. Keep thin top-level compatibility shims for existing scripts and imports, and mirror the structure in the tests.
Remove the obsolete dashboard now that Langfuse is the analytics surface.\nIntroduce focused transport, model, and configuration modules while preserving the ai_review facade, and document the current runtime architecture.
The standard /api/public/ingestion path feeds only the trace-upsert
queue; evalService.createEvalJobs only dispatches targetObject in
{TRACE, DATASET}. Observation rules fire exclusively from the OTel
pipeline, which this pilot does not use. The trace body already carries
review input/output via langfuse_trace, so a trace rule sees the same
material an observation rule would.
Two llm_as_judge evaluators score the review generation directly: a
NUMERIC 0-1 on finding actionability, a BOOLEAN on whether the summary
agrees with the findings. Both run on every observation whose trace
name is pr-review or opencode-review.
The judge is kimi-k2.7-code through the headroom hub. Local Ollama
returns Anthropic-format responses but the thinking blocks lack the
signature field Langfuse Zod schema requires; the evaluator preflight
fails as Invalid JSON response. A small judge-proxy pod on 8802
forwards to the hub and patches every thinking block with a synthetic
signature before returning.
Trace + generation output now includes the findings themselves
(capped at 25) rather than just the count, so a judge has something
to grade. generation input/output mirrors the trace so an
observation-level evaluator can read them.
Idempotent: existing evaluators and rules are skipped on re-run,
not duplicated. The connection is upserted on provider.
The filter bar matches on metadata only — not on input and not on the item id —
so a dataset seeded with repo/pr in `input` alone could not be sliced by repo
at all. Every facet worth filtering on is now a flat primitive in `metadata`:
repo, owner, repo_name, pr, head_sha, finding_count, has_findings,
max_severity, reviews_run and the review timestamp both ways. `owner` is split
out because a filter on the joined repo matches one repo, never a whole org,
and `max_severity` is "none" rather than absent because an absent key matches
no filter.
`eval_experiment.py` links reviews that already ran into a dataset run, one run
per model, so the Experiments tab is populated without re-running the reviewer.
One trace per (run, item), the most recent: a PR re-reviewed on every push has
many traces and a run is one output per input.
It posts to the deprecated /api/public/dataset-run-items — the notice exempts
self-hosted v3 from the cutoff date and the pilot is stdlib-only by design.
Revisit at v4.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Items were keyed `{repo}#{pr}`, e.g. `netcracker/interview#29`. Both
characters break the UI's item route: the `/` in `owner/repo` splits into
extra path segments, and everything after the `#` is a fragment the browser
never sends. Items were created successfully and then 404'd when opened.
Ids are now `{owner}__{repo}__pr{n}`, which needs no percent-encoding. The
real repo and pr stay intact in `input`, so nothing downstream reads the id
back apart. Session ids elsewhere keep the `{repo}#{pr}` form — those are
never path segments and feedback_scores depends on that shape.
The 28 existing items were unusable and are regenerable from feedback.db;
they were deleted and recreated under the new ids.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds the evaluation layer on top of the review traces: five deterministic
scores describing how the reviewer behaved, a bridge that turns human reactions
into ground truth, and a dataset seeded from the reviews already run.
The two are kept apart on purpose. feedback.db has recorded 113 reviews and
zero reactions, resolutions or replies — nobody has ever responded to a bot
comment — so an accuracy metric cannot be built yet. The scorers therefore
measure behaviour, which is computable from data in hand, and feedback_scores
turns verdicts into scores the moment any arrive.
eval_scores.py emits finding_rate, severity_info_ratio, severity_max,
dropped_findings and cost_per_finding into the same ingestion batch as the
trace. Undefined values are omitted rather than reported as zero: an info ratio
over a silent review is undefined, and charting it as 0 would read as perfect
calibration.
dropped_findings needed a parser change. Both parsers silently discard findings
with an unusable path/line, which made a model emitting garbage locations
indistinguishable from one that found nothing. last_parse_dropped() exposes the
delta, read at parse time — after apply_repo_config the drops are the config
working as intended, not the model misbehaving.
feedback_scores.py scores the session ("{repo}#{pr}"), because feedback arrives
days later against a PR and nothing records which re-run produced which
comment. review_acceptance is absent rather than 0 when nothing was engaged.
eval_bootstrap.py registers the score configs, seeds the pragent-reviews
dataset, and can backfill scores onto traces that predate the scorers.
expectedOutput is the reviewer's own prior output, flagged
labelled_by_human: false — a regression baseline, not verified truth.
Also fixes a silent telemetry failure: the ingestion endpoint answers 207 when
only some events succeed, so a batch with every event rejected still looked
like success. Score events were missing the required per-event timestamp and
ingested nothing while reporting 207. _warn_on_rejected_events now logs the
per-event errors under LANGFUSE_DEBUG.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ship token spend, latency and equivalent cost for every review to the
self-hosted Langfuse so per-model behaviour is queryable as a trend rather
than one PR comment at a time.
langfuse_trace.py is stdlib-only and emits via the public ingestion API.
Traces split into `ollama` and `claude` environments keyed off the bare model
name, not the provider: both paths go through the same headroom proxy, so the
provider prefix says nothing about which spend story a review belongs to. The
pilot's own path bills $0, so the reported cost is the equivalent price from
cost_model.PRICES.
ai_review.py calls _emit_langfuse on both token-spending exit paths (the
normal post and the salvage path). Import and emission are wrapped in a
blanket except: with no LANGFUSE_HOST or key pair the whole thing is a silent
no-op, and a telemetry failure must never fail a review.
These files were previously deployed only by way of the image build's
`COPY . /app`, so a clean checkout would have silently dropped tracing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- opencode_review.install_config: opencode 1.3.10 rejects factory
opencode.json$'\''s top-level "$schema" key with 'Unrecognized key'
at parse time, causing every fresh review to fail as the misleading
'opencode empty text (rc=0)'. Add a small drop-list of known-bad
top-level keys + a sanitizer applied to both the fast (no-env) and
slow (env-substituted) write paths.
- feedback_harvest: Gitea versions occasionally serialize reaction
content / inline-comment resolver as a dict instead of a string,
crashing harvester with 'dict object has no attribute strip'. Coerce
both via str() with a brief comment documenting the WHY.
Affected reviews: techspark/suaspark-dashboard #11 (fixed),
PRAgent reviews on PR #9 era (recovered).
Verified: opencode 1.3.10 emits real '### Summary of Changes' body.
The repo edit form already renders a <select id="model"> with one
<option value="…"> per cost_model.PRICES key (sorted) plus a
"— keep current —" placeholder. This commit adds the test that pins
that behaviour: every key must appear as an <option>, no others.
1 test under tests/pilot/test_dashboard_select.py.
POST /r/<owner>/<name>/edit reads the current .pr-review.json via the
Gitea contents API (404 → start from {}), updates static_message and
model, then PUTs the file back as the bot identity. Static message is
stripped and capped at 400 chars; model is validated against
cost_model.PRICES (unknown → flash + redirect, no save). CSRF token is
the dashboard auth token (same value, simplest binding).
Helpers (_http) moved to urllib.request at module top so tests can patch
the entry point; the helper returns the Gitea status code so 404 (file
missing) cleanly omits sha on the create-PUT path.
6 tests under tests/pilot/test_dashboard_edit.py covering the happy
path, the 400-char cap, the unknown-model rejection, auth + CSRF gates,
and the file-missing creation path.
Mirrors webhook_server.py's BaseHTTPRequestHandler + ThreadingHTTPServer
shape. Pure stdlib, no pip deps. Routes:
GET / overview (totals + 7-day sparkline + top repos)
GET /r/<owner>/<name> repo summary (severity histogram + top findings)
GET /r/<owner>/<name>/<index> one PR's findings
GET /r/<owner>/<name>/<index>/raw raw Markdown body (via Gitea contents API)
GET /static/style.css dark-mode stylesheet
GET /login login form
POST /login compare token, set HttpOnly+SameSite cookie
POST /r/<owner>/<name>/edit (Tasks C+D)
Auth: when PRAGENT_DASHBOARD_TOKEN is set, every route except /login and
/static/* requires Cookie: pragent_dash=<token>. Unset → tailnet-only.
All HTML rendered via string.Template; every dynamic value is escaped
with html.escape(..., quote=True). No .format, no f-string templates.
12 tests under tests/pilot/test_dashboard.py.
Three pure functions — overview / repo_summary / pr_summary — that open
the feedback SQLite via feedback.init, run their queries, and return
plain dicts/lists. All three tolerate a missing or empty DB by returning
a zero-shaped dict.
Cost is hardcoded 0.0: per-review usage:cost isn't stored, only the raw
review rows are. Surfacing a rolled-up dollar figure without telemetry
would be guessing, so we don't.
17 new tests under tests/pilot/test_dashboard_data.py.
Repos that need to pin the review engine (e.g. 'this project requires
claude-sonnet-5 for the budget line' or 'route everything through
gpt-5.6-luna for now') can declare a top-level 'model' string in
.pr-review.json. The parser validates the value against cost_model.PRICES
(lazy import — ollama path stays dep-free) and silently drops unknown
values with a stderr pointer to the valid key set so a typo in the
config file surfaces in the logs instead of silently falling back.
The orchestrator gains a small _resolve_display_model(base, config)
helper that implements a 3-way precedence:
1. OPENCODE_MODEL env (operator override, used verbatim)
2. config['model'] (per-repo override)
3. f'headroom/{base}' (default)
review_pr resolves once (lazy fallback before config is loaded) and
re-resolves after .pr-review.json is fetched, then threads the result
into the opencode subprocess, REVIEW_HEADER, and the cost-line
parenthetical. Same single value everywhere — no more mix of bare
OLLAMA_MODEL id in the header and a stale free-tier literal in the cost
line.
Tests cover parsing acceptance, parsing rejection (capsys stderr),
type validation, precedence in all 4 (env×config) combinations, and an
end-to-end sanity check that format_review_body shows the override and
not the base id.
masi
Claude
gitea_admin