Cost per token is the wrong metric for evaluating AI coding agents

A coding agent doesn't just respond once and move on. It plans, generates code, runs tests, fails, tries again. A single task can consume 10 to 50 times more tokens than a simple chat prompt. The more useful the agent becomes, the more calls it makes, and with per-token pricing, that means the workflow that generates the most value is also the one that weighs the most on the bill.

This isn't a hypothetical problem. Uber reportedly exhausted its entire 2026 AI budget as early as April, after expanding its use of Claude Code. GitHub Copilot dropped fixed per-seat pricing and moved to usage-metered credits. And Gartner projects that AI coding token costs will exceed the average developer's salary by 2028.

These numbers come from content sponsored by HPE in MIT Technology Review, published on November 2, 2026, and it's worth saying upfront: this is material paid for by a vendor that sells on-premises AI infrastructure. That doesn't invalidate the core argument, but it changes the weight I should give to the case it makes about itself.

The metric everyone is using is wrong

The most common reaction to this cost pressure is the simplest one: usage quotas, per-seat limits, prior approval before running the agent. The problem is that this optimizes the wrong number. Cost per token measures consumption, not outcome. Part of the spend is waste (retries, loops that don't converge), part is the actual work of shipping software, and a consumption meter doesn't distinguish between the two. A spending cap doesn't distinguish either.

I agree with the diagnosis. I've seen this same mistake in other technology cycles: when the control metric is raw cost, the team learns to save on what's easy to measure (API calls) and not necessarily on what actually drains the budget (rework, PRs that fail review, an agent that iterates too much on a poorly specified task).

The proposal: cost per merged pull request

The alternative metric the piece advocates for is total AI-assisted development cost divided by the number of changes that pass review and get merged, closely tracked alongside rework and defect rate, so that a cheaper PR doesn't mask worse quality. If that cost falls as AI usage rises, without compromising quality, the team is actually converting consumption into useful software.

As an engineering management metric, this is a real improvement over counting tokens. It shifts the conversation from "how much are we spending" to "how much are we delivering per dollar spent," which is exactly the question a product board should have been asking a year ago.

The blind spot the report doesn't confront

But no engineering productivity metric survives intact its first perverse incentive, and this is where I need to disagree with the piece's optimistic reading. Cost per merged PR can be gamed in obvious ways: breaking a large change into several small PRs inflates the denominator without increasing delivered value; a reviewer under metric pressure approves faster and more shallowly; and rework that only shows up two sprints later, as technical debt, doesn't get counted in the month the PR was merged.

The report itself tries to guard against this by recommending that rework and defects be tracked alongside the metric, which is correct, but that's the part that requires engineering discipline, not a financial dashboard. If the product team approving budget only looks at the headline number (cost per PR falling), the pressure to game the metric disappears into a single line in a report and off the radar of whoever signs the check.

Model routing: the decision that falls to the CTO

The most concrete and most interesting part of the piece is the division of labor between models. The recommendation is to reserve frontier models (or on-premises flagship models, when policy requires it) for difficult planning and novel reasoning, and route repetitive work (code generation, test creation, recurring refactoring) to open models operated by the company itself.

A gateway decides the route based on task complexity, data sensitivity, cost, and latency: tax the exception, not the entire flow. This is an architecture decision criterion, not an off-the-shelf package, and it's up to each team to validate the candidate open model against its own code, its own tests, and its own acceptance criteria before putting it into production, because a third-party benchmark doesn't guarantee equivalence on a specific repository.

Rented vs. owned infrastructure: the table missing from the report

The report is honest on one point: owned capacity isn't free, and the number that matters remains the fully loaded cost per merged PR, not the price of the card.

HPE's case deserves skepticism, not dismissal

HPE says its hybrid platform routes routine workloads from more than 150 internal applications to open models running on HPE Private Cloud AI, with token cost falling by 30% to 50%. It also claims that the top 5% most intensive users generate most of the consumption, and that the heaviest user consumes about 450 times more than the lightest one on the same license: if the heavy-usage profile goes from a small minority to half the team, monthly spend can rise roughly 4.4 times with the same number of seats.

These are the vendor's own numbers, with no independent audit visible in the source, and that weighs on my reading: this is data that serves the sales argument of the company that published it. That said, the logic behind it (heavy users, not seat count, drive spend) is consistent with what any team running generative AI FinOps should already be tracking, regardless of which infrastructure vendor they choose.

What changes for builders

For anyone leading an engineering team, the practical question isn't whether to adopt cost per merged PR tomorrow, it's whether the CI pipeline can already attribute inference cost to each individual PR, today. Without that granular data, the metric is a promise, not a tool.

And before any decision to purchase owned capacity, the criterion should be simple: run a full development cycle on owned capacity and compare cost per active developer and cost per merged PR against the current model. If neither improves with increased usage, the current model is still the right choice, and switching infrastructure vendors won't fix a poorly designed metric problem.