Files
Claude 2f96e66aab feat(pilot): behavioural scorers, feedback ground truth, and an eval dataset
Adds the evaluation layer on top of the review traces: five deterministic
scores describing how the reviewer behaved, a bridge that turns human reactions
into ground truth, and a dataset seeded from the reviews already run.

The two are kept apart on purpose. feedback.db has recorded 113 reviews and
zero reactions, resolutions or replies — nobody has ever responded to a bot
comment — so an accuracy metric cannot be built yet. The scorers therefore
measure behaviour, which is computable from data in hand, and feedback_scores
turns verdicts into scores the moment any arrive.

eval_scores.py emits finding_rate, severity_info_ratio, severity_max,
dropped_findings and cost_per_finding into the same ingestion batch as the
trace. Undefined values are omitted rather than reported as zero: an info ratio
over a silent review is undefined, and charting it as 0 would read as perfect
calibration.

dropped_findings needed a parser change. Both parsers silently discard findings
with an unusable path/line, which made a model emitting garbage locations
indistinguishable from one that found nothing. last_parse_dropped() exposes the
delta, read at parse time — after apply_repo_config the drops are the config
working as intended, not the model misbehaving.

feedback_scores.py scores the session ("{repo}#{pr}"), because feedback arrives
days later against a PR and nothing records which re-run produced which
comment. review_acceptance is absent rather than 0 when nothing was engaged.

eval_bootstrap.py registers the score configs, seeds the pragent-reviews
dataset, and can backfill scores onto traces that predate the scorers.
expectedOutput is the reviewer's own prior output, flagged
labelled_by_human: false — a regression baseline, not verified truth.

Also fixes a silent telemetry failure: the ingestion endpoint answers 207 when
only some events succeed, so a batch with every event rejected still looked
like success. Score events were missing the required per-event timestamp and
ingested nothing while reporting 207. _warn_on_rejected_events now logs the
per-event errors under LANGFUSE_DEBUG.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-31 14:22:55 +00:00
..

pragent pilot — AI Review bot

A minimal AI code-review bot for Gitea, running as a CI step on the existing act-runner. This is the pilot — a small, self-contained reviewer that predates the full pragent framework (whose design lives in docs/plans/2026-08-04-pragent-design.md). The framework will later absorb this; until then, this is what runs.

How it works

  1. You add pragent-bot to a repo and commit .gitea/workflows/ai-review.yml.
  2. On a PR, you add the AI-REVIEW label.
  3. Gitea Actions runs the workflow on the act-runner; it fetches the PR diff, asks glm-5.2:cloud (on-network via the headroom proxy) to review it, and posts the findings back as a PR review authored by pragent-bot.
  4. Remove the label to stop re-reviews on further pushes.

Fail-open: the job always exits 0 and never blocks CI. Errors become a short "review failed" comment.

Onboard a repo (3 steps)

1. Add pragent-bot as collaborator

Repo → Settings → Collaborators → Add → pragent-bot → permission Write. (Write is required to post reviews/comments.)

Or via API (with an admin/owner token):

curl -X PUT -H "Authorization: token $OWNER_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"permission":"write"}' \
  "http://<gitea-host>:3000/api/v1/repos/OWNER/REPO/collaborators/pragent-bot"

2. Add the PRAGENT_BOT_TOKEN secret

Repo → Settings → Actions → Secrets → New secret → name PRAGENT_BOT_TOKEN, value = the bot's access token (ask the platform admin; stored mode-600 at ~/.claude/.pragent-bot-token on the admin host).

3. Commit the workflow

Copy pilot/workflow-template.yml into the target repo as .gitea/workflows/ai-review.yml and commit it. That's it.

Use it

Open a PR (or push to an open one), add the AI-REVIEW label. The review appears within ~3090s depending on diff size and model latency.

What's intentionally NOT in the pilot

Deferred to the full framework (by design, see the design doc):

  • Attention tiering (trivial/lite/full/oversized) and per-tier cost control.
  • Multiple analyzer fan-out over a shared cached prompt prefix.
  • Prior-comment synthesis (so each push re-posts; the latest review is tagged with the head SHA so it's easy to spot).
  • Inline line comments and status checks.
  • pragent explain / replay / analytics JSONL.
  • A second forge (GitLab) and the provider matrix.

Pieces

File Role
pilot/ai_review.py The reviewer script (stdlib only). Single source of truth — fetched at runtime by each repo's workflow.
pilot/workflow-template.yml The Gitea Action consumers copy into .gitea/workflows/ai-review.yml.
tests/pilot/test_ai_review.py Unit tests for the pure helpers (no network).

Run the tests

cd ~/Projects/pragent
PYTHONPATH=pilot python3 -m pytest tests/pilot/    # if pytest available
# or, without pytest:
python3 - <<'PY'
import os, sys, importlib.util
sys.path.insert(0, os.path.abspath("pilot"))
import ai_review  # noqa: F401
spec = importlib.util.spec_from_file_location("t", "tests/pilot/test_ai_review.py")
m = importlib.util.module_from_spec(spec); spec.loader.exec_module(m)
fails = 0
for n in sorted(x for x in dir(m) if x.startswith("test_")):
    try: getattr(m, n)(); print("PASS", n)
    except Exception as e: fails += 1; print("FAIL", n, e)
print("failed:", fails)
PY

Configuration knobs (env in the workflow)

Env Default Purpose
OLLAMA_MODEL glm-5.2:cloud Model id passed to the headroom proxy.
OLLAMA_MAX_TOKENS 6000 Output token cap.
DIFF_MAX_CHARS 150000 Diff truncation cap (with a noted truncation marker).
OLLAMA_URL http://<model-proxy-host>:8789 headroom proxy (tailnet). If the act-runner can't reach the tailnet IP, expose 8789 as an in-cluster Service+Endpoints and set this to the cluster DNS name.