gitea_admin/pragent
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(cost-model): calibrate against the first measured review

Browse Source 
PR #7 ran under the AI-USAGE label and reported real numbers: 28 agent steps,
348s, 2,071,025 input / 17,303 output tokens, and zero cache reads or writes.
The model predicted ~$0.73 on Opus 5 for that tier. The measurement prices it
at $10.79 — the model was ~15x low.

Two wrong assumptions:

- Step count and per-step growth. `full` assumed 12 steps and 1,200 tokens per
  tool result; the run did 28 steps averaging ~3,300. Cost is roughly quadratic
  in steps, so this compounds. Tier defaults are re-derived from the measured
  per-step growth rather than from guesses.
- Caching. The model defaulted to prompt caching on. The headroom/glm-5.2 path
  reports 0 read / 0 write, so the stable prefix is paid at full input price on
  every step. Budget with caching off until that column is nonzero.

Adds OBSERVED_RUNS as an append-only calibration anchor, an observed-runs
section in the report, and a regression test asserting the model stays within
2.5x of the measurement — so the next drift is caught by the suite rather than
by a surprising invoice.

Corrected blended figures at 350 PRs/month: ~$1,740 Opus 5, ~$1,755 GPT-5.6
Sol, ~$696 Sonnet 5, ~$348 Haiku 4.5, ~$70 GPT-5.6 Luna.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B11e8TZZxJyzHW7jj7KWUN
This commit is contained in:
Marcos
2026-08-18 05:08:22 +00:00
parent 30d2a3d7da
commit 2613b3e3af
3 changed files with 145 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+32 -13
View File
@@ -60,21 +60,40 @@ mysterious.
The pilot runs on `glm-5.2:cloud` through the on-network headroom proxy, so today it
bills nothing per token — but the token *work* is real, and `pilot/cost_model.py`
prices it against published API rates. The factory's prompt sizes are measured from
the files in this repo; the per-tier workloads come from the `attention-tiering`
budgets. Blended over a 5/35/55/5 tier mix, prompt caching on:
prices it against published API rates. Factory prompt sizes are measured from the
files in this repo; the per-tier workloads are calibrated against runs actually
measured through the `AI-USAGE` label (`OBSERVED_RUNS` in that file).
| Model | per PR | 350 PRs/month |
|---|---:|---:|
| Claude Opus 5 / GPT-5.6 Sol | ~$0.61 | ~$212 |
| Claude Sonnet 5 / GPT-5.6 Terra | ~$0.24 | ~$85 |
| Claude Haiku 4.5 | ~$0.12 | ~$43 |
| GPT-5.6 Luna | ~$0.02 | ~$8.5 |
**The measured anchor.** The hardening PR (`#7`, 16 files / ~1100 changed lines,
tier `full`) took 28 agent steps and 348s, and consumed **2,071,025 input** and
**17,303 output** tokens — with **zero cache reads or writes**, because the current
headroom/glm path does no prompt caching. Priced elsewhere, that single review is:
Run `python3 pilot/cost_model.py --help` for other mixes and PR volumes. The
dominant cost is the agent loop resending its own context each step, not the diff —
turning prompt caching off multiplies the bill by ~2.3x, which is why the tiering
skill caps steps, file reads, and subagent fan-out per tier.
| Model | that review | blended per PR | 350 PRs/month |
|---|---:|---:|---:|
| Claude Opus 5 | $10.79 | ~$4.97 | ~$1,740 |
| GPT-5.6 Sol | $10.87 | ~$5.02 | ~$1,755 |
| Claude Sonnet 5 | $4.32 | ~$1.99 | ~$696 |
| GPT-5.6 Terra | $4.35 | ~$2.01 | ~$702 |
| Claude Haiku 4.5 | $2.16 | ~$0.99 | ~$348 |
| GPT-5.6 Luna | $0.43 | ~$0.20 | ~$70 |
Blended figures use a 5/35/55/5 tier mix with caching off, matching what is
actually observed. Run `python3 pilot/cost_model.py --help` for other mixes and
volumes.
Two things dominate, and neither is the diff:
1. **The loop resends its context every step.** 28 steps over a ~17k-token diff
produced 2M input tokens. Cost is roughly quadratic in step count, which is why
`attention-tiering` caps steps, file reads and subagent fan-out per tier.
2. **Prompt caching is worth about a third of the bill** and is currently not
happening. Any move to a paid provider should confirm the `cache_read` column
goes nonzero before budgeting.
An earlier version of this model assumed 12 steps and caching on, and was ~15x
low. The lesson is in the file: budget from `OBSERVED_RUNS`, not from the tier
table, and append a row every time a real review reports usage.
## Extension points