Claude
92283c44e8
Ship token spend, latency and equivalent cost for every review to the self-hosted Langfuse so per-model behaviour is queryable as a trend rather than one PR comment at a time. langfuse_trace.py is stdlib-only and emits via the public ingestion API. Traces split into `ollama` and `claude` environments keyed off the bare model name, not the provider: both paths go through the same headroom proxy, so the provider prefix says nothing about which spend story a review belongs to. The pilot's own path bills $0, so the reported cost is the equivalent price from cost_model.PRICES. ai_review.py calls _emit_langfuse on both token-spending exit paths (the normal post and the salvage path). Import and emission are wrapped in a blanket except: with no LANGFUSE_HOST or key pair the whole thing is a silent no-op, and a telemetry failure must never fail a review. These files were previously deployed only by way of the image build's `COPY . /app`, so a clean checkout would have silently dropped tracing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
5.6 KiB
pragent → Langfuse
Every review the pilot runs ships one trace to a self-hosted Langfuse. The review body already prints a usage table, but that table lives and dies inside one Gitea PR. Langfuse is where the same numbers become a trend: tokens per review, latency per model, equivalent cost per repo, and how those move when the model or the tiering changes.
The ollama / claude split
Both paths route through the same headroom proxy, so the provider prefix does
not distinguish them — headroom/claude-sonnet-5 is Claude spend,
headroom/glm-5.2:cloud is not. The split is keyed off the bare model name
and lands on the trace's environment:
| resolved model | environment |
|---|---|
headroom/claude-sonnet-5 |
claude |
claude-opus-5 |
claude |
headroom/glm-5.2:cloud |
ollama |
headroom/MiniMax-M2.7 |
ollama |
vllm-qwen38/qwen3.8-27b |
ollama |
Langfuse takes an environment selector on every dashboard, filter and cost
breakdown, so the two spend stories stay separate inside one project — one key
pair to rotate instead of two. Tags carry the finer cut:
provider:headroom, model:<bare>, engine:opencode, repo:<owner/name>,
lens:<id> per fan-out lens.
To split into two projects later, point LANGFUSE_PUBLIC_KEY /
LANGFUSE_SECRET_KEY at the second project on whichever deployment runs the
Claude path. Nothing in the code needs to change.
What a trace carries
- trace
pr-review—sessionId=owner/repo#index, so every push to one PR groups together. Input is the PR identity; output is the summary + finding count; metadata carries steps, duration, severity counts and the provider's own reported cost. - generation
opencode-review—model,usageDetails,costDetails.
usageDetails.input is the uncached input. opencode reports cache_read
inside input, and Langfuse sums the keys it is given, so passing both
verbatim would bill the resent prefix twice.
How cost is priced
Langfuse has no price table of its own here — we compute the number and ship it
as costDetails.total, so what Langfuse charts is exactly what
cost_model.PRICES says.
A model that genuinely bills (claude-*, gpt-*, gemini-*, grok-*) is
priced as itself: basis actual.
A model that costs nothing through the headroom proxy is priced against a
comparison target instead: basis equivalent:<target>. That covers the
models absent from PRICES (MiniMax-M2.7 — which is what the webhook
actually runs — and glm-5.2:cloud) as well as entries priced at all zeros
(the self-hosted vLLM qwen3.8-27b). Without this the dashboard would be a
flat $0.00 line, since the pilot's own path is free.
The target follows the same precedence as the review body, so the PR and the dashboard never disagree:
.pr-review.json:cost_target > PRAGENT_PRICE_TARGET > claude-sonnet-5
An equivalent cost is a hypothetical, not money spent, so every trace is tagged
cost:actual or cost:equivalent:<target> and the generation metadata carries
cost_basis. Filter on it before reading any cost chart as spend.
If the comparison target itself is unknown, the trace ships usage with no cost block — better no number than a wrong one.
Anthropic prices in cost_model.PRICES were fetched 2026-08-18; re-check them
before quoting anything externally.
Configuration
| env | meaning |
|---|---|
LANGFUSE_HOST |
http://langfuse-web.langfuse.svc.cluster.local:3000 |
LANGFUSE_PUBLIC_KEY |
pk-lf-… |
LANGFUSE_SECRET_KEY |
sk-lf-… |
LANGFUSE_TIMEOUT |
seconds, default 5 |
LANGFUSE_DEBUG |
1 to log ingestion failures to stderr |
Unset host or either key ⇒ emission is a silent no-op. That is the default, so a checkout without Langfuse behaves exactly as before.
Fail-open
langfuse_trace is stdlib-only (urllib) and every entry point swallows its
own exceptions; _emit_langfuse in ai_review.py wraps even the import. A
Langfuse outage cannot fail, delay past LANGFUSE_TIMEOUT, or alter a review.
Both token-spending exit paths emit — the normal post and the salvage path where the agent produced unparseable output. That run cost the same as a clean one, and is precisely the failure worth trending.
Deployment
Cluster side lives outside this repo: ~/k8s/langfuse.yaml (ClickHouse +
web + worker, reusing the gitea postgres, gitea valkey and minio),
~/k8s/oauth2-proxy-langfuse.yaml (the Logto gate), and
~/k8s/langfuse-setup.sh, which provisions the database, the bucket, the
secrets, and wires pragent-webhook with the three env vars above.
The UI is at https://langfuse.marcospaulo.dev.br:
browser -> Caddy (VPS, TLS, DNS-01) -> tailscale
-> 100.74.17.70:30361 -> oauth2-proxy (Logto, email allowlist)
-> langfuse-web (ClusterIP)
Logto sits at both layers off one app (langfuse, two redirect URIs): the
proxy gates the domain, and Langfuse's own NextAuth uses the same Logto as a
custom OIDC provider, so the inner login is a silent redirect rather than a
second password.
pragent does not go through any of that. It posts to
langfuse-web.langfuse.svc.cluster.local:3000 from inside the cluster, on
API-key auth — putting ingestion behind an interactive SSO gate would break it
on the first review.