Agents on Rails: how the official benchmark measures Claude and GLM on real Rails tasks

The Rails team published a new round of Agents on Rails: Claude Fable 5.1 ties for the top spot with 92% accuracy and costs about 40% less than Opus 5, and the stealth model ox-alpha turned out to be GLM 5.3 Flash, which solves tasks for five cents each.

The official Ruby on Rails blog published on September 2, 2026 another round of Agents on Rails, the benchmark the framework's own team maintains to measure how AI agents perform on real Rails code tasks. This time's news: Claude Fable 5.1 tied Opus 5 at the top on accuracy, but beat it on everything else (speed, cost, and even on security tasks), and the model that ran disguised under the codename ox-alpha in the previous round came out of stealth: it's GLM 5.3 Flash, from Z.ai.

For those building in Rails, what's interesting here isn't the scoreboard itself, it's the method. The benchmark gives an honest yardstick for deciding which model to put in an agent that will touch your code, and at what price.

How the benchmark works under the hood

Agents on Rails doesn't ask the model trivia questions. It runs what authors Svyatoslav Kryukov and Artur Petrov call 21 atomic tasks on Writebook (a real Rails application, from 37signals), using a harness called lemans, with default effort levels and hidden verification of the result. Each task is run 3 times, for a total of 63 runs per model. Running three times is a declared compromise between runtime and statistical reliability, and the authors themselves admit the effect of that: Fable 5.1 reached ~95% in the preliminary runs, but "got unlucky" in the official run and landed at 92%.

Hidden verification matters because it keeps the model from optimizing for the test. And there's a metric worth highlighting for anyone who takes the "Rails way" seriously: Rails API recall. Each task points to the specific Rails API the ideal solution should use. An agent can pass the test by hand-rolling a replacement, but that isn't idiomatic. As the source sums it up:

A hand-rolled replacement may pass the checks, but it isn't the Rails way.

-- Ruby on Rails Blog, Agents on Rails

The raw runs are open in the rails/ai-evals repository, so you can audit them instead of just trusting the chart.

The scoreboard: same accuracy, different bill

The central takeaway from this round is that accuracy has stopped being the differentiator at the top. Fable 5.1 and Opus 5 scored the same, 58 of 63 runs each, and both were stopped by the same task. The difference is in cost and time:

| Model | Accuracy | Cost (63 runs) | Median time per run | |---|---|---|---| | Claude Fable 5.1 | 58/63 (92%) | $75 | 5.4 min | | Claude Opus 5 | 58/63 (92%) | $120 | 9.7 min | | Claude Fable 5 | 57/63 | $146 | 6.8 min | | GLM 5.3 Flash (formerly ox-alpha) | 52/63 (83%) | $3.31 | — |

The practical takeaway: Fable 5.1 comes out ~50% cheaper than Fable 5 and ~40% cheaper than Opus 5, with the same accuracy as the leader and being the fastest in the top tier (tied with Sol). In other words, in the Fable 5.1 vs. Opus 5 comparison, choosing Opus means paying more and waiting almost twice as long for the same result.

There was also a jump in Rails API recall: Fable 5.1 reached 41%, against the historical range of 8% to 35%. And the authors make an observation worth noting for anyone who's skeptical that "a newer model just recognizes newer APIs": quote_column_name is nearly as old as Active Record, and no previous model had suggested it without being prompted. It's not recency, it's competence over the framework's surface.

Another point: Fable 5 failed security tasks phrased as a pen-test report, treating the request as if it were an attack to be blocked. 5.1 read the same report and fixed every finding. For anyone thinking about using an agent to triage security findings, that's the difference between the tool helping and the tool freezing up.

The GLM 5.3 Flash case: 83% for five cents a run

The highlight for budget-conscious devs, and that's most of them in Brazil, is GLM 5.3 Flash. It revealed the identity of the stealth ox-alpha, had its 63 runs redone under the real name and price, and confirmed the pre-release numbers: 52 of 63 (83%) for a total of $3.31. That works out to about five cents per run.

The comparison the source draws is what gives it scale: Flash matches Grok 4.6's 83% at one-fifteenth the cost. In a scenario where an agent runs many repetitive tasks, the cost difference between a top-tier model and a Flash model scales up fast, and 83% accuracy on atomic Rails tasks already covers a good chunk of the grunt work.

What this changes for builders (and when it doesn't pay off)

The benchmark isn't a product you install; it's a method for choosing the tool. The practical reading I'd make:

• Critical task, code that goes to production without heavy review: top-tier model (Fable 5.1). You pay for the accuracy ceiling and the idiomatic recall.
• High volume of atomic, repetitive tasks, with human review at the end: GLM 5.3 Flash changes the cost equation. 83% accuracy at five cents a run lets you throw the agent at a lot of things that weren't worth automating before.
• Security work: here Fable 5.1 pulled ahead of 5, so be careful about carrying over intuitions from the previous generation.

The trade-offs the source itself makes explicit: there are few runs (3 per task), so there's real statistical noise, Fable 5.1 itself showed variance between the preliminary and official rounds. And the atomic corpus is at its limit: the authors say the tasks are "starting to run out of room" and that they're preparing a Stage 2 with larger, more realistic tasks. In other words: 92% on an atomic task isn't 92% on your actual feature, which involves multiple files, business context, and side effects. The benchmark measures the floor of competence, not the ceiling of complexity you face day to day.

A usability improvement that came from the community: at Nate Berkopec's request, the charts on the Agents on Rails page were reorganized so the best models always appear in the top-right corner, whatever axis is being compared. Small detail, but it's the kind of convention that makes a leaderboard readable in seconds.

Stage 2 is worth watching: that's when the benchmark will test the stuff that actually hurts, tasks that resemble the real backlog of a Rails app in production. Until then, the current numbers and the raw runs are public for anyone who wants to ground their own AI stack decision.

Source 1: Ruby on Rails Blog (https://rubyonrails.org/2026/9/2/agents-on-rails-claude-fable-5-1-and-glm-5-3-flash)

Agents on Rails: Claude Fable 5.1 and GLM 5.3 Flash (formerly known as ox-alpha)

Wednesday, September 2, 2026

Agents on Rails: Claude Fable 5.1 and GLM 5.3 Flash (formerly known as ox-alpha)

Posted by Svyatoslav Kryukov, Artur Petrov

Claude Fable 5.1 dropped yesterday. See how it handles real Rails tasks, and where it landed on the leaderboard today. We also finally have a name for the stealth model from the last round.

Fable 5.1: We have a new best.

Fable 5.1 scored as well as Claude Opus 5 on accuracy, but surpassed it in all other marks (including price!). And it does security work now.

Fable 5.1 solved 58 of 63 runs (92%), matching Claude Opus 5 at the top of the leaderboard, and passed 20 of the 21 tasks at least once. Here is how it compares:

• Claude Fable 5.1: 58 of 63, $75 for all 63 runs, 5.4 minutes median per run.
• Claude Opus 5, the sole leader until now: 58 of 63, $120, 9.7 minutes.
• Claude Fable 5: 57 of 63, $146, 6.8 minutes.

Efficiency is the clearest difference between 5.1 and the previous generation: the bill is ~50% lower than Fable 5's and ~40% lower than Opus 5's. It's also the fastest model across the top tier, on par with Sol.

Fable 5.1 also seems less likely to mistake a security task for a security problem. In the first report, Fable 5 failed all three attempts at a task phrased like a pen-test report. Fable 5.1 read the same report and fixed every finding.

These results come from a relatively small number of runs. At this stage of the benchmark, we run each task 3 times, a compromise between runtime and accuracy. Fable performed better in the preliminary runs, completing around 95% of tasks, but got unlucky in the official run.

The best Rails recall so far

We also ranked 5.1 on Rails API recall again. Each task points to a specific Rails API that should be used by the ideal solution. (A hand-rolled replacement may pass the checks, but it isn't the Rails way.)

Until now, scores ranged from 8% to 35%. Fable 5.1 reached 41%. That improvement isn't simply a newer model recognizing newer APIs. For example, quote_column_name is nearly as old as Active Record, and no previous model had mentioned it unprompted.

One more thing…

Last round's ox-alpha has come out of stealth too: it's GLM 5.3 Flash from Z.ai. We retired the stealth slug and reran all 63 attempts under the real name and price. It matched the pre-release results with 52 of 63 (83%) scored at a total cost of $3.31. At five cents a run, Flash is a great value pick. It matches Grok 4.6's 83% score at one-fifteenth the cost.

The fine print and small updates to the Agents on Rails page

Everything ran the usual way: the lemans harness, default effort levels, three attempts per task across the 21 atomic Writebook tasks, hidden verification. The raw runs are in rails/ai-evals if you want to dig in. The current numbers live on the Agents on Rails page, as always. We added default effort levels used during the benchmark run for transparency. We also updated the charts there, thanks to an idea from Nate Berkopec: whatever you compare, the better models now appear in the top-right corner. There's a tie at the top: two models score 92%, and both were stopped by the same task. The atomic corpus is starting to run out of room, so we're preparing larger, more realistic tasks for Stage 2. Stay tuned!