OpenAI tests AI agents on raw lab data with the BioBasicsBench benchmark

OpenAI released BioBasicsBench, a 100-task benchmark created in partnership with Ligo (a biotech focused on enzyme design) to measure something quite specific: whether an agent can interpret the real mess of a wet lab, as it actually arrives, without tidying up the story to fit a neat conclusion.

OpenAI's stated motivation is a direct critique of existing scientific benchmarks. They tend to use charts and images already selected for published papers, which filters out precisely the hardest part of bench work: failed experiments, ambiguous images, contradictory observations. A benchmark built only from data that has already passed peer review doesn't test the skill that matters most in a lab's day-to-day, which is knowing when the evidence can't support any conclusion at all.

How the Benchmark Is Built

BioBasicsBench's 100 tasks cover five areas of routine bench work: protein methods, imaging and cytometry, quantification and assays, nucleic acid methods, and molecular cloning. The input data preserves the raw form in which this work actually appears in practice:

• gels and blots (agarose, SDS-PAGE, Western blot);
• protocols, notebook entries, and photographed handwritten notes;
• tables and instrument exports, including raw flow cytometry files (FCS);
• photos of the bench, plates, and reagents;
• raw microscopy, including z-stacks and time-lapse videos;
• instrument screenshots, standard curves, and exported chromatograms;
• sequencing files (FASTQ, FASTA, GenBank maps).

Each task was written by scientists using data from their own experiments and went through three layers of review: a Ligo scientist who did not write the task, an external reviewer with a PhD working in a wet lab, and a final internal review. The agent returns a structured response (conclusion, measurement, or recommendation, plus supporting reasoning when required) that is scored against a rubric with individual criteria, not a binary "right or wrong." This makes it possible to separate a correct conclusion from a partially correct analysis, something that matters far more in a lab than on a multiple-choice test.

The Numbers: GPT-6 Astra on Top, but Far From Reliable

OpenAI evaluated seven models (GPT-5.2, GPT-5.4, GPT-5.5, GPT-5.6 Sol, GPT-6.1 Sol, GPT-6 Luna, and GPT-6 Astra) across 37 model/reasoning-effort combinations, with five attempts per task, totaling 18,500 results. The main metric is mean reward, the task's average score expressed as a percentage of available credit, with 95% confidence intervals calculated by resampling whole tasks (keeping each task's five attempts together).

At the highest reasoning-effort setting (xhigh), the results were:

Two points stand out in this table. The jump from GPT-5.6 Sol to GPT-6 Astra (from 43.8% to 66.7%) is the largest in the set, and OpenAI attributes much of it to stronger native multimodal capabilities, not just more reasoning. And token cost dropped along with the quality gain: GPT-6 Astra used 4,704 output tokens on average versus 27,620 for GPT-5.2, nearly six times fewer tokens for a score more than double.

Where the Model Still Gets It Wrong, in Detail

OpenAI did not hide the failure cases of its own leading model, and they are instructive. In a molecular cloning task, Astra correctly distinguished two groups of bands on a colony PCR gel, but assumed that the larger PCR product indicated successful cloning. With that premise, it selected colonies 4, 6, 7, and 8; the reference answer, based on comparison with the experiment's controls, pointed to colonies 5, 9, and 10. The visual grouping was correct, but the experimental identity assigned to each group was reversed, which would have led to purifying the wrong plasmid.

In a second case, the model concluded that vacuoles (compartments inside the cell) are generally darker than their surroundings, even while seeing lighter examples in the same image. The cause is well known: changing the microscope's focus can invert contrast, so neither observation alone supports a general rule. Astra even noted that contrast varies with focus, but in its final answer it kept only the description of dark vacuoles, discarding the contradictory evidence it had itself noticed.

A third case is more mundane and perhaps more revealing: counting colonies on a plate. Astra's count was off by about 12% above the reference, failing to resolve colonies that were close together, leaving visible colonies unmarked, and marking empty agar as if it were a colony. Since DNA transformation efficiency is calculated from this count with fixed volume and dilution, the error propagates directly into the final estimate.

What This Means for Agent Builders

For those building agents outside the context of biology, the value of BioBasicsBench lies less in the specific results and more in the methodology. It's a well-documented example of how to build an agent evaluation for a specialized, messy domain, using real evidence instead of curated examples.

Three design decisions are worth stealing for any domain-specific agent eval:

1. A rubric with partial credit per criterion, instead of a binary pass/fail, which captures the difference between "got it right" and "got it right for the wrong reason."
2. Layered review with a specialist who didn't write the task plus an external domain reviewer, reducing the bias of whoever built the test.
3. Resampling by whole task (not by individual attempt) to calculate confidence intervals, which avoids inflating the apparent precision of the result.

The cost chart is also a direct operational data point: if one of your agents will run in production interpreting ambiguous documents (reports, support tickets, contracts), the trade-off between reasoning effort (xhigh) and tokens spent per task is exactly the kind of curve OpenAI published here, and it's worth reproducing before deciding which effort level to configure by default.

When It's Not Worth Trusting the Agent

OpenAI itself is direct about the limits of the result: a higher benchmark score doesn't mean the model can already make reliable decisions in the lab. 66.7% is the best case, at the highest reasoning effort, on a set of tasks that OpenAI itself describes as ranging from simple requests, like reading the molecular weight of a DNA sample on a gel, to detailed assay analyses, all involving routine work that bench scientists should be able to complete reliably.

The three documented errors (assuming a lab rule without checking the right control, discarding contradictory evidence the model itself noticed, getting a basic count wrong) are exactly the kind of failure that goes unnoticed if no one audits the agent's output.

For anyone thinking about using multimodal agents to triage experimental data today, the responsible path is to treat the output as a draft that needs human review at control points, not as a verdict. OpenAI also signals that it will keep monitoring this capability for biosecurity reasons, another reminder that the reliability bar here is much higher than for an agent answering email.

Source

OpenAI News: Measuring how AI agents handle everyday wet-lab science