The measurement, in detail
This page accompanies the post "What you won’t code, you pay for in tokens". It describes the protocol, gives the raw numbers, and offers the code for download — for whoever wants to rerun the calculation rather than take my word for it.
Protocol
- Task: count the functions of a module that return a certain type. Exact answer established separately, cross-checked by hand and by two other routes: eighteen (four of them in union types a plain keyword search misses — it sees only fourteen).
- Model: Claude Haiku, deliberately a small model, where the effect is most visible.
- Corpus: a module frozen at a public version. The dedicated tool is ast_search (axm-forge); the generic shell, grep and the like.
- Four regimes, eight runs each, thirty-two runs in all. Cost converted at Haiku’s public rate ($1 / $5 / $0.10 per million input / output / cache tokens).
Results (mean over eight runs)
| Regime | Cost/run | Calls | Accuracy | |
|---|---|---|---|---|
| A — reads everything | 0,081 $ | 17,9 | 18/18 sur 8/8 | right, but costly |
| C — shell expert | 0,068 $ | 15,6 | 13,1/18 (min 5) | collapses 3 of 8 times |
| B — tool + free | 0,059 $ | 10,8 | 3/8, said 117 once | scope drift |
| B' — bounded tool | 0,011 $ | 1,0 | 18/18 sur 8/8 | exact, deterministic |
The code and the numbers
Everything is open and inspectable. The raw numbers (metrics.json) read without installing anything; the two scripts document the exact protocol:
run.py— arms A and Brun_bc.py— arms B' and Cmetrics.json— the 32 raw runsground_truth.json— the 18 functions (AST ground truth)RESULTS.md— the write-up
An honest caveat: the scripts rely on axm-harness, my test bench that runs an agent and counts its tokens. So they don’t run "out of the box" outside my ecosystem (you’d need axm-harness and model access). They’re here to show exactly what I did, not as a ready-to-run kit. The file that actually matters for checking is readable by anyone: metrics.json, the thirty-two raw runs.
One measurement, a single small model, a single task, eight runs per regime: an illustration, not a proof. A more powerful model would probably go off the rails less.