JioHotstar benchmark compares six retrieval approaches for coding agents and finds no single winner

JioHotstar, an Indian streaming platform, published a benchmark on November 11, 2026 comparing six ways of giving context to AI agents that navigate large codebases. The work, led by Abhishek Bansal, Senior Staff Engineer at the company, tested whether adding retrieval layers (search and information retrieval) actually improves the performance of agents like Claude Sonnet, or whether it only adds cost and complexity without a corresponding gain.

The question is practical for any team that has already connected a coding agent to a large repository: is it worth indexing everything semantically, or does the agent do just as well with grep, file reading, and symbol search? JioHotstar tried to answer with data instead of guesswork, and the result dismantles the idea that more context is always better context.

The six methods at play

The study compared the following code navigation approaches for agents:

• Filesystem baseline: local repository, with standard search and file-reading operations, no extra layer.
• Sourcebot: indexed text search, symbol definition and reference, file reading.
• Graphify: graph representation of the code.
• CocoIndex Code: semantic code indexing.
• Repowise: another indexing and search layer over repositories.
• JioHotstar's internal RAG: a proprietary retrieval-augmented generation pipeline built by the company.

All approaches were evaluated using subsets of SWE-bench and CodeScaleBench, plus a question-and-answer harness built on JioHotstar's own codebase. The model used to solve the tasks was Claude Sonnet, and the metrics included task accuracy, token consumption, number of agent turns, and tool usability from the agent's own perspective.

Where structural search had the edge

On SWE-bench Lite, which gathers simpler, more concrete tasks, the filesystem baseline (with no retrieval layer at all) remained competitive. Adding indexed or semantic search generally brought overhead without consistently improving results on these more straightforward tasks.

On SWE-bench Verified, though, which concentrates harder tasks, the picture changed: Sourcebot (indexed text search, symbol definition and reference, file reading) outperformed the baseline in both accuracy and cost, consuming fewer tokens to reach the result. This suggests that the gain from structural search appears precisely when the problem requires navigating many cross-references in the code, not in one-off tasks.

When semantic search pays off

CodeScaleBench showed a different pattern from SWE-bench: there, semantic retrieval helped most when the agent needed to interpret ambiguous problem descriptions and map them to implementation details that weren't obvious from an exact text or symbol search. In other words, when the requirement doesn't literally say where to change the code, a layer that understands meaning (not just strings) makes a difference.

JioHotstar's conclusion is that retrieval approaches aren't interchangeable: each one solves a particular type of problem, and choosing wrong costs tokens and agent turns without any accuracy gain. It's the opposite of treating retrieval as a universal piece that always fits.

What the agent actually uses

The report also drew attention to the design of the tools offered to the agent. In practice, agents preferred familiar developer operations, such as text search, file reading, and symbol location, over higher-level abstractions that required extra verification before trusting the result.

In short: giving the agent a more sophisticated tool doesn't mean it will use it better. If the abstraction requires an extra verification step, the agent tends to prefer the simpler, more verifiable path, even if theoretically slower.

The question that remains: code isn't just code

In a LinkedIn post commenting on the study, Siddharth Vaithinathan raised a broader architectural question: whether coding agents should understand not just the code, but also the company that produced it. According to him, indexing the repository alone doesn't capture why architectural decisions were made, what constraints shaped them, or what previous incidents taught the engineering team.

Vaithinathan suggested extending the agent's knowledge beyond code and documentation to include architecture, constraints, decisions, rationale, incidents, and outcomes. This additional context would help the agent evaluate existing design choices before proposing changes, especially in systems where historical operational experience shapes architectural decisions. It's a point that none of the six tools tested addresses: all of them index code, none indexes the team's institutional memory.

The caveat about the benchmark itself

Tarun Bhargava, also commenting on the study, raised a point relevant to interpreting any benchmark of this kind: cost and accuracy numbers would need to come from multiple runs of the same configuration, since the evaluations use limited subsets and probabilistic models. A single run can hide variance that changes the conclusion about which approach is better. Based on the available material, JioHotstar didn't publish how many repetitions it ran per configuration.

What changes for those building agents here

JioHotstar's final recommendation is pragmatic: keep filesystem tools as the baseline, use structural search for concrete, well-defined problems, and reserve semantic indexing for ambiguous requirements or ones that cut across several parts of the system. The company also recommends measuring tool-use traces alongside task accuracy, to find out whether retrieval is actually reducing unnecessary exploration or just adding noise.

For those in Brazil connecting agents like Claude Code, Cursor, or Copilot to a large monorepo, the practical takeaway is direct: before procuring or building a RAG layer over the code, it's worth measuring whether the accuracy gain is worth the extra token and turn cost, and for which type of task that holds true.

Stacking semantic indexing on top of an agent that already handles simple tasks well with grep and file reading can be money wasted; lacking semantic indexing for a team dealing with ambiguous requirements and large legacy systems can be the opposite. The choice, according to the study itself, isn't about which tool is better in the abstract, but about which task you're trying to solve.

Source: InfoQ: JioHotStar Benchmarks Knowledge Bases for AI Coding Agents