40 trillion tokens of data show why teams stall when adopting AI agents

In a presentation at QCon AI, Brendan O'Leary, Founding Developer Relations Engineer at Kilo Code, brought a rare angle: real-world usage data from AI coding agents, not lab benchmarks. The talk's original title promised 25 trillion tokens analyzed; by the day of the presentation, the volume had already passed 40 trillion, which O'Leary himself cites as a sign of how fast AI adoption in coding is accelerating.

What Kilo Code is and where the data comes from

Kilo Code is an open-source agentic coding tool, comparable to Claude Code, Codex, or Cursor, but with its code publicly available. It runs as a CLI extension, as a VS Code and JetBrains extension, and also offers cloud agents. Because it's open source, users can use any inference provider they want, but the tool also offers its own gateway with access to more than 500 different models, including the leading models from OpenAI and Anthropic and open models via OpenRouter. According to O'Leary, Kilo Code led OpenRouter's published token-consumption rankings for months.

The data comes from that gateway, with a specific methodology: anonymous users and the company's internal users were excluded, as was the autocomplete system (which has a different usage pattern from the rest) and any traffic identified as automated. The model-choice analysis considers only what the developer manually selected in the interface, not the infrastructure's internal routing.

O'Leary himself makes an important caveat about sample bias: Kilo Code's user base is mostly made up of early adopters who actively sought out an open-source, configurable tool, which already says something about their relationship with experimentation. He gives a hypothetical example to illustrate the point: if at some point in the talk a figure like "68% of developers do X" comes up, that number needs to come with a mental asterisk, because it reflects this specific slice, not the market average.

The agent trust ladder

The core of the presentation is what O'Leary calls the AI adoption ladder, with four rungs. Each rung requires a bigger leap of trust than the one before it, because the feedback loop gets longer and the cost of a mistake gets higher.

The first rung is classic autocomplete, accepted with Tab: the cost of a mistake is nearly zero, because the developer sees it and decides on the spot. The second is the chat model from Copilot's early era (2022-2023), where you ask for a block of code or an explanation and review it manually before pasting it in.

In the third, the agent already reads and writes files, runs commands, and acts authenticated inside the developer's environment, which O'Leary describes as a big leap of trust. At the top, teams orchestrate several specialized agents that split a task into subtasks and come back with something coherent, carrying the biggest risk because an error may not surface for hours.

According to O'Leary, most teams today are stuck between rung 1 and 2, or between 2 and 3, while the gap between levels grows as you climb the ladder. He cites a Gartner study on the high failure rate of AI projects before they reach production, attributing it to governance gaps: teams skipped steps on the ladder without building the trust needed at the lower rungs.

I don't think that gap is random. I think it really is a lot about trust.

Brendan O'Leary, Founding Developer Relations Engineer at Kilo Code

The first point of failure: context building

The presentation identifies three technical points where adoption usually stalls: context building, model routing, and feedback loops. The available material details the first one in depth.

Context building is everything that happens before the request reaches the model: which files go in, whether the codebase has been indexed, whether there's semantic reasoning about the relationships between parts of the project, and whether the problem was well described to the model. When autocomplete suggests something that doesn't make sense for the project's structure, according to O'Leary, it's rarely the model that's bad, it's the context that arrived wrong.

The example he gives is direct: if the developer asks "where is this project's authentication logic?" and the agent starts down the wrong path, that retrieval failure contaminates the rest of the conversation. The model starts working with incomplete context and begins taking risky guesses. When that same broken context feeds an autonomous agent or a multi-agent orchestration, the error only multiplies, because it becomes harder to backtrack and extract code that actually works.

There's an indirect signal of this in the usage data itself: developers who are still stuck at the heavy-chat rung, without having migrated to agents, consume a noticeably higher number of tokens per request. For O'Leary, this indicates that when automatic context building doesn't work, the developer tries to compensate by manually pasting in code snippets, a solution that works up to a point, but doesn't scale as the task grows.

What's still left open

Model routing (which model to use for which type of task) and feedback loops (how and when the system warns that something went wrong, especially at the higher rungs of the ladder, where the signal arrives later and more sparsely) are cited as the second and third points of failure, but the detail on each one is left for the rest of the presentation, available in full on InfoQ.

What this changes for those building AI agents

For those evaluating or already operating agentic coding tools day to day, O'Leary's framing suggests a practical path before promising leadership swarms of autonomous agents: first validate whether autocomplete and chat already generate consistent trust within the team, because jumping straight to multi-agent orchestration without that foundation tends to reproduce the same governance gaps that Gartner associates with AI projects that never leave the drawing board.

The data point on token consumption per request is also a simple, cheap proxy to monitor: if a team is spending much more tokens per chat than expected, it's a sign that codebase indexing or automatic context-file selection isn't working, and that people are compensating manually. This holds both for those using closed commercial tools and for those who, like Kilo Code itself, choose to bring their own inference via an open gateway, an option that tends to appeal to teams in Brazil that are more sensitive to API costs in dollars.