Remove the obsolete dashboard now that Langfuse is the analytics surface.\nIntroduce focused transport, model, and configuration modules while preserving the ai_review facade, and document the current runtime architecture.
The standard /api/public/ingestion path feeds only the trace-upsert
queue; evalService.createEvalJobs only dispatches targetObject in
{TRACE, DATASET}. Observation rules fire exclusively from the OTel
pipeline, which this pilot does not use. The trace body already carries
review input/output via langfuse_trace, so a trace rule sees the same
material an observation rule would.
Two llm_as_judge evaluators score the review generation directly: a
NUMERIC 0-1 on finding actionability, a BOOLEAN on whether the summary
agrees with the findings. Both run on every observation whose trace
name is pr-review or opencode-review.
The judge is kimi-k2.7-code through the headroom hub. Local Ollama
returns Anthropic-format responses but the thinking blocks lack the
signature field Langfuse Zod schema requires; the evaluator preflight
fails as Invalid JSON response. A small judge-proxy pod on 8802
forwards to the hub and patches every thinking block with a synthetic
signature before returning.
Trace + generation output now includes the findings themselves
(capped at 25) rather than just the count, so a judge has something
to grade. generation input/output mirrors the trace so an
observation-level evaluator can read them.
Idempotent: existing evaluators and rules are skipped on re-run,
not duplicated. The connection is upserted on provider.
The filter bar matches on metadata only — not on input and not on the item id —
so a dataset seeded with repo/pr in `input` alone could not be sliced by repo
at all. Every facet worth filtering on is now a flat primitive in `metadata`:
repo, owner, repo_name, pr, head_sha, finding_count, has_findings,
max_severity, reviews_run and the review timestamp both ways. `owner` is split
out because a filter on the joined repo matches one repo, never a whole org,
and `max_severity` is "none" rather than absent because an absent key matches
no filter.
`eval_experiment.py` links reviews that already ran into a dataset run, one run
per model, so the Experiments tab is populated without re-running the reviewer.
One trace per (run, item), the most recent: a PR re-reviewed on every push has
many traces and a run is one output per input.
It posts to the deprecated /api/public/dataset-run-items — the notice exempts
self-hosted v3 from the cutoff date and the pilot is stdlib-only by design.
Revisit at v4.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Items were keyed `{repo}#{pr}`, e.g. `netcracker/interview#29`. Both
characters break the UI's item route: the `/` in `owner/repo` splits into
extra path segments, and everything after the `#` is a fragment the browser
never sends. Items were created successfully and then 404'd when opened.
Ids are now `{owner}__{repo}__pr{n}`, which needs no percent-encoding. The
real repo and pr stay intact in `input`, so nothing downstream reads the id
back apart. Session ids elsewhere keep the `{repo}#{pr}` form — those are
never path segments and feedback_scores depends on that shape.
The 28 existing items were unusable and are regenerable from feedback.db;
they were deleted and recreated under the new ids.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds the evaluation layer on top of the review traces: five deterministic
scores describing how the reviewer behaved, a bridge that turns human reactions
into ground truth, and a dataset seeded from the reviews already run.
The two are kept apart on purpose. feedback.db has recorded 113 reviews and
zero reactions, resolutions or replies — nobody has ever responded to a bot
comment — so an accuracy metric cannot be built yet. The scorers therefore
measure behaviour, which is computable from data in hand, and feedback_scores
turns verdicts into scores the moment any arrive.
eval_scores.py emits finding_rate, severity_info_ratio, severity_max,
dropped_findings and cost_per_finding into the same ingestion batch as the
trace. Undefined values are omitted rather than reported as zero: an info ratio
over a silent review is undefined, and charting it as 0 would read as perfect
calibration.
dropped_findings needed a parser change. Both parsers silently discard findings
with an unusable path/line, which made a model emitting garbage locations
indistinguishable from one that found nothing. last_parse_dropped() exposes the
delta, read at parse time — after apply_repo_config the drops are the config
working as intended, not the model misbehaving.
feedback_scores.py scores the session ("{repo}#{pr}"), because feedback arrives
days later against a PR and nothing records which re-run produced which
comment. review_acceptance is absent rather than 0 when nothing was engaged.
eval_bootstrap.py registers the score configs, seeds the pragent-reviews
dataset, and can backfill scores onto traces that predate the scorers.
expectedOutput is the reviewer's own prior output, flagged
labelled_by_human: false — a regression baseline, not verified truth.
Also fixes a silent telemetry failure: the ingestion endpoint answers 207 when
only some events succeed, so a batch with every event rejected still looked
like success. Score events were missing the required per-event timestamp and
ingested nothing while reporting 207. _warn_on_rejected_events now logs the
per-event errors under LANGFUSE_DEBUG.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ship token spend, latency and equivalent cost for every review to the
self-hosted Langfuse so per-model behaviour is queryable as a trend rather
than one PR comment at a time.
langfuse_trace.py is stdlib-only and emits via the public ingestion API.
Traces split into `ollama` and `claude` environments keyed off the bare model
name, not the provider: both paths go through the same headroom proxy, so the
provider prefix says nothing about which spend story a review belongs to. The
pilot's own path bills $0, so the reported cost is the equivalent price from
cost_model.PRICES.
ai_review.py calls _emit_langfuse on both token-spending exit paths (the
normal post and the salvage path). Import and emission are wrapped in a
blanket except: with no LANGFUSE_HOST or key pair the whole thing is a silent
no-op, and a telemetry failure must never fail a review.
These files were previously deployed only by way of the image build's
`COPY . /app`, so a clean checkout would have silently dropped tracing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- opencode_review.install_config: opencode 1.3.10 rejects factory
opencode.json$'\''s top-level "$schema" key with 'Unrecognized key'
at parse time, causing every fresh review to fail as the misleading
'opencode empty text (rc=0)'. Add a small drop-list of known-bad
top-level keys + a sanitizer applied to both the fast (no-env) and
slow (env-substituted) write paths.
- feedback_harvest: Gitea versions occasionally serialize reaction
content / inline-comment resolver as a dict instead of a string,
crashing harvester with 'dict object has no attribute strip'. Coerce
both via str() with a brief comment documenting the WHY.
Affected reviews: techspark/suaspark-dashboard #11 (fixed),
PRAgent reviews on PR #9 era (recovered).
Verified: opencode 1.3.10 emits real '### Summary of Changes' body.
The repo edit form already renders a <select id="model"> with one
<option value="…"> per cost_model.PRICES key (sorted) plus a
"— keep current —" placeholder. This commit adds the test that pins
that behaviour: every key must appear as an <option>, no others.
1 test under tests/pilot/test_dashboard_select.py.
POST /r/<owner>/<name>/edit reads the current .pr-review.json via the
Gitea contents API (404 → start from {}), updates static_message and
model, then PUTs the file back as the bot identity. Static message is
stripped and capped at 400 chars; model is validated against
cost_model.PRICES (unknown → flash + redirect, no save). CSRF token is
the dashboard auth token (same value, simplest binding).
Helpers (_http) moved to urllib.request at module top so tests can patch
the entry point; the helper returns the Gitea status code so 404 (file
missing) cleanly omits sha on the create-PUT path.
6 tests under tests/pilot/test_dashboard_edit.py covering the happy
path, the 400-char cap, the unknown-model rejection, auth + CSRF gates,
and the file-missing creation path.
Mirrors webhook_server.py's BaseHTTPRequestHandler + ThreadingHTTPServer
shape. Pure stdlib, no pip deps. Routes:
GET / overview (totals + 7-day sparkline + top repos)
GET /r/<owner>/<name> repo summary (severity histogram + top findings)
GET /r/<owner>/<name>/<index> one PR's findings
GET /r/<owner>/<name>/<index>/raw raw Markdown body (via Gitea contents API)
GET /static/style.css dark-mode stylesheet
GET /login login form
POST /login compare token, set HttpOnly+SameSite cookie
POST /r/<owner>/<name>/edit (Tasks C+D)
Auth: when PRAGENT_DASHBOARD_TOKEN is set, every route except /login and
/static/* requires Cookie: pragent_dash=<token>. Unset → tailnet-only.
All HTML rendered via string.Template; every dynamic value is escaped
with html.escape(..., quote=True). No .format, no f-string templates.
12 tests under tests/pilot/test_dashboard.py.
Three pure functions — overview / repo_summary / pr_summary — that open
the feedback SQLite via feedback.init, run their queries, and return
plain dicts/lists. All three tolerate a missing or empty DB by returning
a zero-shaped dict.
Cost is hardcoded 0.0: per-review usage:cost isn't stored, only the raw
review rows are. Surfacing a rolled-up dollar figure without telemetry
would be guessing, so we don't.
17 new tests under tests/pilot/test_dashboard_data.py.
Repos that need to pin the review engine (e.g. 'this project requires
claude-sonnet-5 for the budget line' or 'route everything through
gpt-5.6-luna for now') can declare a top-level 'model' string in
.pr-review.json. The parser validates the value against cost_model.PRICES
(lazy import — ollama path stays dep-free) and silently drops unknown
values with a stderr pointer to the valid key set so a typo in the
config file surfaces in the logs instead of silently falling back.
The orchestrator gains a small _resolve_display_model(base, config)
helper that implements a 3-way precedence:
1. OPENCODE_MODEL env (operator override, used verbatim)
2. config['model'] (per-repo override)
3. f'headroom/{base}' (default)
review_pr resolves once (lazy fallback before config is loaded) and
re-resolves after .pr-review.json is fetched, then threads the result
into the opencode subprocess, REVIEW_HEADER, and the cost-line
parenthetical. Same single value everywhere — no more mix of bare
OLLAMA_MODEL id in the header and a stale free-tier literal in the cost
line.
Tests cover parsing acceptance, parsing rejection (capsys stderr),
type validation, precedence in all 4 (env×config) combinations, and an
end-to-end sanity check that format_review_body shows the override and
not the base id.
Repos can pin a free-text notice (e.g. 'this repo is in maintenance mode',
or 'reviewers: focus on the public API only for this quarter') in
.pr-review.json:static_message. The string is stripped and capped at 400
chars (mirror of the existing instructions cap), then rendered as a
Markdown blockquote (> {msg}) directly under the REVIEW_HEADER so it
surfaces on every review without scrolling.
Plumbing: parse_repo_config exposes 'static_message'; format_review_body
accepts a static_message kwarg and inserts the blockquote before the
'### Summary of Changes' section; review_pr threads
config.get('static_message') to both call sites (salvage path + happy
path). Empty / non-string values are silently dropped, mirroring the
parser's 'ignore blank' handling for every other text field.
The webhook pod now routes through headroom's MiniMax-M2.7 endpoint, but the
cost line in the AI-usage collapsible still read 'headroom glm-5.2:cloud'.
Resolve a single display_model at the top of review_pr (OPENCODE_MODEL env
wins, else headroom/{OLLAMA_MODEL}) and pass it to:
* the opencode subprocess (was already doing this on the same line, now
sharing the value)
* format_review_body so REVIEW_HEADER also reflects the actual run
* _render_collapsible_usage so the parenthetical reads
'({display_model} — free tier)' or '({display_model} — billed)'.
Test additions in tests/pilot/test_ai_review.py cover:
* the parenthetical picks up the passed-in model verbatim
* the full provider prefix survives (headroom/<id>) for the opencode path
* nonzero cost flips the inner clause from 'free tier' to 'billed'
* the existing nonzero-cost assertion flips to assert 'billed' instead
of dropping the parenthetical entirely
opencode's @ai-sdk/anthropic provider sends the config's apiKey as the
x-api-key header. The committed opencode.json carries apiKey="ollama"
(placeholder) so the repo can be public. When the headroom upstream
switches to an auth-gated provider (e.g. MiniMax), the placeholder
returns 'No credentials found'. install_config now also patches
options.apiKey from PRAGENT_MODEL_API_KEY when set, mirroring the
existing baseURL patching.
Co-Authored-By: Claude <noreply@anthropic.com>
headroom-hub 8789 was repointed to api.minimax.io/anthropic.
glm-5.2:cloud is no longer a registered model there.
Co-Authored-By: Claude <noreply@anthropic.com>
masi
Claude
gitea_admin