CodeRabbit says it multiplied deliveries by 20x by reviewing agent-generated code with OpenAI models

The more a team outsources code writing to agents, the more it needs someone (or something) to check what was written. That is the central argument of a report published by OpenAI about CodeRabbit, a code review platform used, according to the company, by more than 20,000 organizations and 150,000 open source projects.

The case is worth reading not for the customer count, but for what it exposes: code review has become the next bottleneck of the agent era, and solving it with agents too brings real speed gains, but at a reengineering cost that most teams have not yet measured.

The first bottleneck that every company hits as soon as you adopt coding agents is code reviews.

Harjot Gill, co-founder and CEO, CodeRabbit

What CodeRabbit does, in practice

CodeRabbit does not write code: it audits what other agents (or humans) have written, assigning risk, explaining changes, and deciding what can go to production without human review. It works as a governance layer that runs before the merge, in parallel with teams' work. For those who already use Copilot, Cursor, or their own agents in CI pipelines, this is exactly the missing piece: someone (or something) to validate at scale what agents produce at scale.

According to Harjot Gill, co-founder and CEO of the company, the product was born with GPT-3.5 Turbo and only became truly useful with the arrival of GPT-4. Since then, CodeRabbit has been an early adopter of OpenAI's reasoning models o1 and o3, taking advantage of the fact that code review runs asynchronously in the cloud: it is possible to wait for a model to think longer in exchange for deeper analysis, something that would not be tolerable in an interactive chat.

An in-house benchmark, not audited by third parties

The company maintains an internal benchmark of known bugs to evaluate each new model. In the interview with OpenAI, Gill estimated that GPT-6 Astra catches 50% of the issues in this dataset, ahead of other models tested on the same benchmark.

It is worth the counterpoint: this is a figure released by the company itself, about its own dataset, without open methodology or independent comparison. That does not invalidate the data, but it limits how much it serves as a market benchmark. What is interesting is the process described by Gill: the benchmark is alive, because the nature of bugs changes as the agents themselves evolve and make different mistakes.

The nature of bugs keeps evolving, so does our benchmark as the models get better.

Harjot Gill, co-founder and CEO, CodeRabbit

Cheaper per token, more agents per review

The gain in capacity did not come only from better models, but from cheaper models to run at scale. CodeRabbit started out using one or two subagents for code navigation; today, according to Gill, lower-cost models such as GPT-6 Luna support dozens of subagents spread across the platform, while larger models such as GPT-6 Astra handle planning and orchestration.

This division of labor, with a cheap model for repetitive tasks and an expensive model for complex decisions, is the same pattern that Brazilian teams already apply with routing between GPT-4o mini and larger models, or between local models and paid APIs. The difference is scale: Gill says this unlocked use cases that the company simply could not afford before, expanding both the validation and the explanations the system returns to the developer about a change they did not write.

The math behind "20x more software"

The number that gives the OpenAI report its name is this: according to Gill, CodeRabbit started delivering about 20 times more software over the past six months using Codex, without hiring more engineers. He estimates having submitted between 2,500 and 3,000 personal pull requests in the 30 days before the interview.

The point that deserves attention, more than the number itself (a personal, unaudited estimate), is what had to change to sustain it:

• rethinking where human approval actually adds value in the workflow, instead of keeping manual review on everything;
• letting CodeRabbit itself automatically approve and ship changes classified as low risk;
• solving infrastructure bottlenecks that emerged from running multiple agents in parallel, from local machine memory to dependencies and packaging.

We had to actually completely rethink the entire software delivery.

Harjot Gill, co-founder and CEO, CodeRabbit

The company also built a Slack bot based on Codex that triages production incidents, messages, and support tickets, returning a pull request ready for the team to review and merge. It is a pattern worth noting: the agent does not decide alone what goes to production, but it reduces the distance between identifying a problem and having a candidate fix.

Security: turning your own attack into a product

Since 2023, CodeRabbit has run sandboxed agents that clone repositories for review. More recently, the company used OpenAI's security model, called Cyber, via a layer called Daybreak, to run an internal pentest on its own platform. According to Gill, the exercise revealed flaws the team did not imagine existed.

We found multiple vulnerabilities, advanced hacks that we could not even imagine could happen.

Harjot Gill, co-founder and CEO, CodeRabbit

That work turned into a product: CodeRabbit Secure uses Daybreak with Codex models to find and fix vulnerabilities in customers' code. In a two-week promotion in which the company footed the bill for US$1 million in free deep scans, CodeRabbit says it found more than 43,000 vulnerabilities across 13,000 scans, and states that thousands of them have already been fixed by customers themselves. That is a high enough number to suggest the problem is real and systemic, not an isolated marketing exaggeration, even without external audit of the data.

Running the whole company as code

A less obvious development in the report is the use of Codex outside engineering. CodeRabbit manages Google Ads campaigns via pull request: it opens and merges a request to publish ads. It also maintains a "content studio" repository where Codex generates presentations and documents for clients.

If you can translate your marketing problems to code problems, you will run 10x faster.

Harjot Gill, co-founder and CEO, CodeRabbit

It is an idea that extends well beyond the devtools context: any workflow that can become a reviewable diff (configuration, content, even operational decisions) gains the same kind of audit trail and rollback that already exists for code.

What remains open

The report comes directly from OpenAI, with numbers provided by CodeRabbit itself without independent validation, and it serves above all as a satisfied-customer case study. There is no data on false positives from the automated review, on how much human time is still spent correcting wrong decisions by the reviewing agent, or on the total cost of running dozens of subagents per pull request.

For those evaluating whether to adopt a layer like this in their own pipeline, the practical question is not whether agents reviewing agents works (the customer volume suggests it does, up to a point), but which slice of decision-making your team is willing to hand over to an automated system, and with what audit trail behind it.