Claude
b4041f6892
Remove the obsolete dashboard now that Langfuse is the analytics surface.\nIntroduce focused transport, model, and configuration modules while preserving the ai_review facade, and document the current runtime architecture.
123 lines
5.6 KiB
Markdown
123 lines
5.6 KiB
Markdown
# pragent → Langfuse
|
|
|
|
Every review the pilot runs ships one **trace** to a self-hosted Langfuse. The
|
|
review body already prints a usage table, but that table lives and dies inside
|
|
one Gitea PR. Langfuse is where the same numbers become a trend: tokens per
|
|
review, latency per model, equivalent cost per repo, and how those move when
|
|
the model or the tiering changes.
|
|
|
|
## The ollama / claude split
|
|
|
|
Both paths route through the same headroom proxy, so the provider prefix does
|
|
not distinguish them — `headroom/claude-sonnet-5` is Claude spend,
|
|
`headroom/glm-5.2:cloud` is not. The split is keyed off the **bare model name**
|
|
and lands on the trace's `environment`:
|
|
|
|
| resolved model | environment |
|
|
| --------------------------- | ----------- |
|
|
| `headroom/claude-sonnet-5` | `claude` |
|
|
| `claude-opus-5` | `claude` |
|
|
| `headroom/glm-5.2:cloud` | `ollama` |
|
|
| `headroom/MiniMax-M2.7` | `ollama` |
|
|
| `vllm-qwen38/qwen3.8-27b` | `ollama` |
|
|
|
|
Langfuse takes an environment selector on every view, filter and cost
|
|
breakdown, so the two spend stories stay separate inside one project — one key
|
|
pair to rotate instead of two. Tags carry the finer cut:
|
|
`provider:headroom`, `model:<bare>`, `engine:opencode`, `repo:<owner/name>`,
|
|
`lens:<id>` per fan-out lens.
|
|
|
|
To split into two *projects* later, point `LANGFUSE_PUBLIC_KEY` /
|
|
`LANGFUSE_SECRET_KEY` at the second project on whichever deployment runs the
|
|
Claude path. Nothing in the code needs to change.
|
|
|
|
## What a trace carries
|
|
|
|
- **trace** `pr-review` — `sessionId` = `owner/repo#index`, so every push to one
|
|
PR groups together. Input is the PR identity; output is the summary + finding
|
|
count; metadata carries steps, duration, severity counts and the provider's
|
|
own reported cost.
|
|
- **generation** `opencode-review` — `model`, `usageDetails`, `costDetails`.
|
|
|
|
`usageDetails.input` is the **uncached** input. opencode reports `cache_read`
|
|
*inside* `input`, and Langfuse sums the keys it is given, so passing both
|
|
verbatim would bill the resent prefix twice.
|
|
|
|
### How cost is priced
|
|
|
|
Langfuse has no price table of its own here — we compute the number and ship it
|
|
as `costDetails.total`, so what Langfuse charts is exactly what
|
|
`cost_model.PRICES` says.
|
|
|
|
A model that genuinely bills (`claude-*`, `gpt-*`, `gemini-*`, `grok-*`) is
|
|
priced **as itself**: basis `actual`.
|
|
|
|
A model that costs nothing through the headroom proxy is priced against a
|
|
**comparison target** instead: basis `equivalent:<target>`. That covers the
|
|
models absent from `PRICES` (`MiniMax-M2.7` — which is what the webhook
|
|
actually runs — and `glm-5.2:cloud`) as well as entries priced at all zeros
|
|
(the self-hosted vLLM `qwen3.8-27b`). Without this Langfuse would show a
|
|
flat $0.00 line, since the pilot's own path is free.
|
|
|
|
The target follows the same precedence as the review body, so the PR and
|
|
Langfuse never disagree:
|
|
|
|
.pr-review.json:cost_target > PRAGENT_PRICE_TARGET > claude-sonnet-5
|
|
|
|
An equivalent cost is a hypothetical, not money spent, so every trace is tagged
|
|
`cost:actual` or `cost:equivalent:<target>` and the generation metadata carries
|
|
`cost_basis`. Filter on it before reading any cost chart as spend.
|
|
|
|
If the comparison target itself is unknown, the trace ships usage with **no**
|
|
cost block — better no number than a wrong one.
|
|
|
|
Anthropic prices in `cost_model.PRICES` were fetched 2026-08-18; re-check them
|
|
before quoting anything externally.
|
|
|
|
## Configuration
|
|
|
|
| env | meaning |
|
|
| --------------------- | --------------------------------------------------------- |
|
|
| `LANGFUSE_HOST` | `http://langfuse-web.langfuse.svc.cluster.local:3000` |
|
|
| `LANGFUSE_PUBLIC_KEY` | `pk-lf-…` |
|
|
| `LANGFUSE_SECRET_KEY` | `sk-lf-…` |
|
|
| `LANGFUSE_TIMEOUT` | seconds, default `5` |
|
|
| `LANGFUSE_DEBUG` | `1` to log ingestion failures to stderr |
|
|
|
|
Unset host or either key ⇒ emission is a silent no-op. That is the default, so
|
|
a checkout without Langfuse behaves exactly as before.
|
|
|
|
## Fail-open
|
|
|
|
`langfuse_trace` is stdlib-only (`urllib`) and every entry point swallows its
|
|
own exceptions; `_emit_langfuse` in `ai_review.py` wraps even the import. A
|
|
Langfuse outage cannot fail, delay past `LANGFUSE_TIMEOUT`, or alter a review.
|
|
|
|
Both token-spending exit paths emit — the normal post **and** the salvage path
|
|
where the agent produced unparseable output. That run cost the same as a clean
|
|
one, and is precisely the failure worth trending.
|
|
|
|
## Deployment
|
|
|
|
Cluster side lives outside this repo: `~/k8s/langfuse.yaml` (ClickHouse +
|
|
web + worker, reusing the gitea postgres, gitea valkey and minio),
|
|
`~/k8s/oauth2-proxy-langfuse.yaml` (the Logto gate), and
|
|
`~/k8s/langfuse-setup.sh`, which provisions the database, the bucket, the
|
|
secrets, and wires `pragent-webhook` with the three env vars above.
|
|
|
|
The UI is at **https://langfuse.marcospaulo.dev.br**:
|
|
|
|
browser -> Caddy (VPS, TLS, DNS-01) -> tailscale
|
|
-> 100.74.17.70:30361 -> oauth2-proxy (Logto, email allowlist)
|
|
-> langfuse-web (ClusterIP)
|
|
|
|
Logto sits at *both* layers off one app (`langfuse`, two redirect URIs): the
|
|
proxy gates the domain, and Langfuse's own NextAuth uses the same Logto as a
|
|
custom OIDC provider, so the inner login is a silent redirect rather than a
|
|
second password.
|
|
|
|
pragent does **not** go through any of that. It posts to
|
|
`langfuse-web.langfuse.svc.cluster.local:3000` from inside the cluster, on
|
|
API-key auth — putting ingestion behind an interactive SSO gate would break it
|
|
on the first review.
|