Gabriel Jarry

← All posts

What you won't code, you pay for in tokens

· This article is also available in French · Accessibility General audience

A question with a single answer

Eighteen. That’s the right answer, the only one. Same model, same file, a question that admits just one. Depending on the slack I gave the agent, I got eighteen, fourteen, or, once, a hundred and seventeen. Nothing moved but its room to maneuver. And the bill, counted in tokens: those units of text you pay the machine for every time it thinks.

From this small experiment I take a larger conviction. Agentic AI is replaying the cloud’s growing pains, and we settle the difference in tokens.

Back to the question. It was about a piece of software: how many of its functions return a certain kind of result? An exact answer, eighteen, and one you count, the way you’d count the fixed-term contracts in a stack of them. The snag is a detail: four of those functions are written in an unusual way (a condensed form, a lambda, the kind of phrasing that says the same thing differently). Hunt for them by the letter and you miss exactly those four, capping out at fourteen. Reading a page without grasping its grammar means missing whatever is worded otherwise. So I put the same question four times to the same small model, changing each round what I let it do. Eight trials per variant, on a file frozen at a public version, so the measurement can be redone. Here is what each approach costs, and what it’s worth.

  • The agent that reads everything. It works through the code itself, page after page. Right every time, but slow and costly: it takes about fifteen passes on average to reconstruct what an index (a book’s table of contents, prepared ahead of time) would hand over at a glance.
  • The tinkerer agent. It gets the generic toolkit (keyword search, the freedom to combine it all like a virtuoso) but not the dedicated tool that understands the grammar of code. It’s the objection you hear everywhere: surely good tinkering is enough? The data say no, and not “a little less well.” Across eight trials, five land around fourteen, three collapse to five. Almost never the right count, and unpredictable from one run to the next.
  • The tooled agent, turned loose. It calls the right tool, the one that gives the exact answer, then runs off the leash: it widens the search area on its own, rummages through other folders, reruns, recounts. Half the time it doesn’t even return a usable count. One trial claimed a hundred and seventeen functions: it had extended its search, on its own initiative, to a whole folder instead of the single file it was asked about.
  • The tooled agent, kept on a leash. It calls the right tool, once, on the right area, and stops there. Eighteen out of eighteen, on each of the eight trials, the four hard functions included. No variation, and seven times cheaper than the agent that reads everything.

You might think this last one wins by default, since the tool it calls is also the one that serves as the reference. That would be misreading it. The right answer, eighteen, I established separately, cross- checked by hand and by two other routes. The tool doesn’t decree the truth; it finds it reliably, where the tinkerer agent, with access to the same code, caps out at fourteen. The dividing line isn’t “the tool is right because it’s the tool.” It’s simpler: a question with an exact answer is handled by what computes it, not by what guesses it.

I’d been expecting a factor of a hundred: the engineer’s fantasy, the perfect tool against the struggling agent. The real gap is seven. For a more reliable result, seven times cheaper is already a deal you sign without hesitating. My expectation was outsized; the gain itself holds up.

That left the other half of the story, the half that surprised me. The fourth agent had exactly the same tool as the third. Everything turns on the frame: a fixed scope, a capped number of steps, a ban on re-checking its own output. Which of the three restraints carries most of the effect? I haven’t teased it apart, and I’ll refrain from making it a law. Together, they turn an erratic agent into an exact one. The third one was free. It redefined its task, and lost its way.

A category error

Beneath this result lies a simple mistake. Counting these functions is not a matter of judgment: one answer, verifiable, always the same. A mechanical task, in the sense a calculator is: same inputs, same output, every time. Handing it to a machine that guesses, however gifted, is like flipping a coin to decide whether 17 is prime.

The trap is insidious. The probabilistic machine knows how to imitate the deterministic one; it often gives the right answer. But “often” is not “always.” And once turned loose, it doesn’t just answer: it reinterprets the question. That’s exactly what the third agent did. Nobody asked it to comb through the whole repository. It decided on its own that this was the right reading of the task.

The same move as the cloud

This temptation has an exact economic precedent: the cloud. Before it, you bought servers, capital paid once. The cloud turned that capital into an expense: you rent compute by the hour, you own nothing, you pay as you go. A decisive advance, most of the time. Its trap has a name: lift-and-shift. You drop your old application onto rented machines as-is, rethinking nothing, and then discover the bill: paying by the hour, endlessly, for what a little work would have frozen once and for all.

Agentic AI replays that film, a notch higher. It’s no longer compute you rent, it’s reasoning, billed by the word the machine produces, the famous “token.” Writing a program used to cost once; rerunning an agent costs every time. Having it reread an entire module for a question an index would settle in one shot is the same lift-and-shift. The token meter, for its part, runs on every execution.

The tooling debt, paid in tokens

Why do we do it anyway? The reason is real, and it isn’t laziness. A reliable tool has an entry cost. The quick fix costs nothing; the robust tool, the one that trips on no awkward case and that you can rerun a thousand times without surprise, costs something to write. The agent, for its part, abolishes that entry cost: it does the work right away, without your building anything. There’s the hidden bargain. What you won’t, or can’t, pay for in development, you pay for in tokens.

The objection will come that agents already write little programs for these tasks: to count, to search, to rename. That’s precisely the right answer: writing a program means freezing the result instead of re-guessing it. The agent that codes was never the problem. The problem is the one that redoes the same task by eye, without ever freezing or testing its tool. To code is to land on the right side of the line. The cost of writing the tool is paid once and amortizes from the first uses, like a recipe you write once to make a hundred times. And more often than not the tool already exists: the agent just has to use it.

Sometimes the free hand is rational. A one-off, throwaway task doesn’t warrant its own tool. The trap is when the throwaway becomes recurring. You then re-pay the token on every run of a task that one good pass would have closed. And the tab comes due twice, since an agent left free never quite returns the same result: you lose the money, and your confidence in the number you got.

The dividing line

From there, a simple decision test. Faced with a task, ask yourself: does it have a verifiable, stable answer? Listing, counting, renaming, checking a format, tracing dependencies: all of that is deterministic. The agent shouldn’t redo it by eye: it should build the tool, then call it. When the task instead calls for judging an open situation, for writing, for exploring the unknown, then fallible judgment takes back its full place.

My measurement adds a condition I hadn’t seen coming. The right tool isn’t enough; you still have to bound the agent around it. The regime that derailed had the right analyzer, after all; it had simply been left free to reinterpret its task.

Bounding the tool doesn’t mean trusting it blindly. Trust is earned, and it’s earned through tests: that’s the whole point of having written it once and for good. My count of eighteen, I cross-checked three ways before making it a reference. Picture the worst case: a deterministic tool that’s wrong returns an answer that’s wrong, and reproducible, and no one questions it anymore. Hence the rule, in one sentence: verify a deterministic result with something deterministic. A tested tool, a second pass, a schema, an invariant. Never a model left to recount on intuition: it would reopen the door you’d just closed.

The discipline that follows is concrete, and it’s mine day to day. Calling an analyzer rather than reading the code from memory. Entrusting the rewrite of a symbol to a tool that doesn’t get it wrong, never to my own hand. Running the agent’s work through a mechanical quality gate, instead of letting it judge alone. The move, every time, is the same: take from the model the decision that has a right answer, and give it back only where there isn’t one. None of this is new. It’s the ordinary hygiene of an engineer, applied to a collaborator that happens to be a machine.

An illustration, not a law

Let me be honest about scope. One measurement, a single small model, a single task, eight trials per regime: it’s an illustration, not a proof. A more powerful model would surely derail less, and the cost gap depends on the task. I draw no science from it. I draw a direction, and it squares with what an engineer’s common sense already knows.

For agentic AI is not a trap. It’s a real economic shift, and often progress. It simply replays the cloud’s growing pains: mistaking “the machine can do it” for “the machine should do it.” The cloud eventually learned to sort what gains from elasticity from what is better kept frozen. Agentic AI isn’t there yet. The whole discipline lies in that sorting: recognizing what must stay deterministic, bounding it firmly, and paying once in code for what you’d refuse to re-pay endlessly in tokens. An AI you let guess what it could compute is a taxi you let drive the same block over and over: the meter climbs, you end up back where you started. At some point, you have to cut the engine.

Going further

If the gist is enough for you, you can stop here. What follows is for whoever wants the sources and the detail of the measurement.

The idea that a model gains by delegating to tools what it does badly is not new. Toolformer puts it in one line: language models “struggle with basic functionality, such as arithmetic or factual look-up, where much simpler and smaller models excel.”

A few reproducibility notes, for whoever wants to redo the calculation rather than take my word for it. The model was Claude Haiku, deliberately a small model, where the effect shows most; a more powerful model would surely derail less. The dedicated tool of the fourth agent is ast_search, the code analyzer from axm-forge, the toolkit I build; the generic shell of the other agents is grep and the like. The measured module is frozen at a public version. The detailed protocol, the thirty-two raw runs, and the code are on the measurement page.