Real-SWE benchmarks AI agents on companies' private code, and the success rate plummets

New benchmark from Specific Labs licensed real production codebases, and no model topped 39% resolution. The best-case scenario for teams putting agents into production is far less encouraging than the one shown in demos.

Editor's note: the final passage of this article attributes to jcmontx, in the Hacker News thread, the caveat that 'most' tasks are 'far from all'. We checked the original comment and it is a direct compliment to Codex, without that comparative caveat. The final sentence was the editorial team's interpretation, not a faithful quote of what he wrote, and readers should keep that in mind when reading that passage.

Almost every code agent benchmark you've seen runs on public repositories, often content the model itself may have seen during training. Specific Labs decided to flip the script: Real-SWE, published in September 2026, evaluates frontier models on private production codebases, licensed from real companies, with all the tasks that engineers at those companies actually had to solve, billing, tax calculation, customer migration.

This framing changes the yardstick. There's no toy problem here: these are changes with real business consequences, spanning multiple services and depending on each company's internal conventions. And the result is a cold shower for anyone imagining they can already outsource their entire backlog to an agent.

The scoreboard: no one comes close to solving the majority

The top of the table is the best possible case, and it still falls far short of a human. The resolution rate is equivalent to pass@1, the average of eight independent runs per task.

| # | Model | Harness | Resolution | |---|--------|---------|-----------| | 1 | Fable 5.1 | Claude Code | 38.8% | | 2 | GPT-6 Astra | Codex CLI | 33.8% | | 3 | Gemini 3.8 Flash | Gemini CLI | 31.2% | | 4 | GLM 5.3 | Claude Code | 28.8% | | =5 | Grok 4.6 | Grok Build | 23.8% | | =5 | Muse Spark 1.3 | Muse Code | 23.8% | | 7 | Kimi K3 | Kimi Code | 18.8% | | 8 | GPT-5.6 Sol | Codex CLI | 16.2% |

Notice a methodological detail that matters for developers: Specific Labs evaluates model-and-harness combinations, not models in isolation. That makes sense, because that's how work actually happens: the same model behaves differently inside Claude Code, Codex CLI, or Gemini CLI. The number that matters isn't "how good is model X", but "how good is model X inside the tool Y I actually use".

Why the tasks are so hard

Two things drive up the difficulty. The first is that the code is natively out-of-distribution: according to Specific Labs, 99% of real companies' tokens are hidden from frontier models, so the agent can't fall back on a memorized pattern, it has to figure out the system on the spot.

The second is the breadth of the change. Real-SWE's reference solution edits a median of 11 files, versus 6 for FrontierCode and DeepSWE (comparison published by Cognition). And the instructions are deliberately underspecified: a median of 1,742 characters, they describe what to do and leave the agent to dig up the rest in the code and surrounding tools, exactly as a real ticket arrives.

An example from the benchmark itself makes this concrete. The billing task asks to fix the tax calculation so each company charges the right rate and exempt customers aren't taxed. Except each company settles tax differently: some keep their own rate, others want the price calculated by the buyer's destination via an external provider (TaxJar, in sandbox or production depending on the account), others charge nothing at all. The task environment exposes a NestJS service in TypeScript, an InfluxDB ledger, and both TaxJar endpoints. It's the kind of tangle of business rules that any fintech developer in Brazil recognizes instantly, and that's exactly where the models slip up.

How the models fail (and why that matters more than the scoreboard)

Real-SWE classifies each failure by behavior, following DeepSWE's taxonomy. The two dominant failure modes are revealing:

• Missed requirement (failed to do something the instructions required): 67.2% of Grok 4.6's failures and 53.8% of Kimi K3's.
• Unverified assumption (built on a guess about the system instead of checking the workspace): 43.3% of GPT-5.6 Sol's failures.
• Integration error (right idea, wired wrong into the system): 49.1% of Gemini 3.8 Flash's failures.

In other words, the problem is rarely that the agent doesn't know how to code. It's that it doesn't read the entire requirement, doesn't verify its own assumption, and doesn't understand the house code style. For anyone who reviews agent PRs every day, this sounds familiar: the code compiles, looks plausible, and quietly drops a business rule nobody asked to remove.

One more data point that debunks the idea of "just let it run longer and it'll solve it": 71.4% of runs under 10 minutes failed, versus 73.4% of longer ones. Persistence doesn't buy correctness.

Cost doesn't buy quality

Specific Labs also estimated the cost per rollout, and the correlation with accuracy is weak. Gemini 3.8 Flash delivers 31.2% at US$2.50; Fable 5.1 leads with 38.8%, but at US$6.96, nearly 3x the cost for 7.6 percentage points more.

| Model | Resolution | Cost/rollout | |--------|-----------|---------------| | Gemini 3.8 Flash | 31.2% | US$2.50 | | GPT-5.6 Sol | 16.2% | US$2.65 | | Muse Spark 1.3 | 23.8% | US$2.74 | | GPT-6 Astra | 33.8% | US$4.67 | | Fable 5.1 | 38.8% | US$6.96 |

For Brazilian teams paying for API access in dollars, this is the chart that matters: you can get close to the top while paying a third of the cost of the number one spot, depending on the type of task.

The community didn't buy it all

The Hacker News thread raised the obvious methodological criticism: how can you trust a benchmark that can't show its code? traceroute66 summed it up:

"So TL;DR benchmarking in a completely non-reproducible manner? [...] So basically pinky-promise benchmarking?"

>

-- traceroute66

It's a fair point: private code is Real-SWE's strength (out-of-distribution, uncontaminated) and at the same time its weakness (not reproducible by third parties). The trade-off is built in.

Others questioned the ranking itself. For bdlowery, Gemini 3.8 Flash ranking so high already gives the benchmark away: "Try and use gemini 3.8 yourself for any real world work and you'll see it's terrible. It'll just go in circles reading the same file 20 times for no reason making hundreds of tool calls for a simple change". IshKebab made a similar point, noting that a scoreboard doesn't capture latency or ergonomics: according to him, Astra is "waaaaaay faster (like 5x; it's not even close)" than Fable, and his practical recommendation is to test the two or three sane options yourself and pick one.

There's also the contamination warning raised by lmeyerov, who runs botsbench.com: his intuition is that many of the "best" private codebases are already not actually private from the point of view of Claude Code and Codex, and that it's worth measuring contamination on every run.

What changes for those building software in Brazil

Two practical takeaways emerge here. The first: the number you see in a model's announcement (that stratospheric SWE-bench pass@1) is the lab ceiling, not the floor of your seven-year-old monolith with a built-in ICMS tax rule (a Brazilian state-level sales tax). Real-SWE suggests that, in real code full of business context, the best agent solves fewer than 4 out of 10 tickets on its own, and in 6 of the 10 sample tasks resolution fell below 15%.

The second: the bottleneck isn't code generation, it's understanding requirements and house conventions. That changes where it's worth investing, well-structured context, tests that verify business behavior, and human review focused on "was a requirement missed?" and "did it break something nobody asked to change?", rather than expecting full autonomy. As jcmontx sums up in the thread, you can offload most tasks to a properly configured agent, but "most" is still far from "all".

The full sample of tasks is available on request on the benchmark's page, which leaves the reproducibility caveat open, something worth watching before treating these numbers as definitive.