UK AISI adopts EvalEval infrastructure to publish AI benchmarks with verifiable configuration

Anyone who has ever compared language models to decide which one to use in a product knows the problem: one vendor announces 74% on SWE-Bench, another announces 81%, and there's no way to tell whether both ran the same evaluation protocol. That's exactly the gap that the UK AI Security Institute (AISI) and the EvalEval coalition tackled in a post published on September 22, 2026, on the Hugging Face blog, co-authored by researchers from both sides, including Avijit Ghosh, Jenny Chim, and Irene Solaiman for EvalEval, and Jessica McFadyen and Lynn Tan for AISI.

The news itself is simple: AISI started using EvalEval's infrastructure to publish its evaluation results openly and reproducibly. But what's behind it matters more than the announcement: for the first time, a government body with real technical weight is exposing not just a benchmark's score, but the entire configuration that produced that score.

Where this collaboration comes from

AISI and EvalEval had already been working together since a joint workshop held alongside NeurIPS 2025, and the institute's feedback helped shape the coalition's Every Eval Ever (EEE) schema. This new phase is EEE moving from paper to practice: AISI is feeding Evaluation Cards, EvalEval's open platform, with real evaluation data from frontier models.

This adds to other projects AISI had already been building to make AI evaluation more rigorous: OptStop, focused on making evaluation more efficient (spending less compute testing the same thing), and HiBayES, which applies hierarchical Bayesian modeling to bring statistical rigor to comparisons between models, alongside work on transcript analysis and capability elicitation. The common idea is to diagnose where AI evaluation reports fail today and build shared infrastructure to close those gaps.

What exactly was published

The release covers five benchmarks used in the main experiment of an AISI paper called How Inference Compute Shapes Frontier LLM Evaluation: HealthBench, FrontierMath, Humanity's Last Exam (HLE), SWE-Bench Pro, and Terminal-Bench 2.0. The tests ran on six frontier models: Claude Opus 4, Claude Opus 4.5, Claude Opus 4.6, GPT-5, GPT-5.2, and GPT-5.4. There are also two cybersecurity evals published separately, Cyber CTFs and The Last Ones, with a partially different set of models.

The central point of the paper, and the reason this transparency matters in practice, is that reported benchmark performance depends heavily on how the evaluation protocol was set up, not just on the model itself. In the case of Humanity's Last Exam, the researchers showed that the fraction of tasks solved changes depending on the protocol: when the model gets correctness feedback from an oracle after each attempt, it keeps solving more tasks as token usage increases. In other words, the same model, tested with a protocol that's more generous in attempts and feedback, produces a noticeably higher accuracy number, even with no change to the model's weights at all. This is the kind of detail that usually disappears when someone posts an isolated benchmark card in a product comparison.

Why this changes the work of whoever evaluates models here

For someone in Brazil choosing between GPT-5.x and Claude Opus 4.x for a coding agent, for example, the bottleneck is rarely a lack of benchmarks, it's a lack of context about the benchmark. A SWE-Bench Pro or Terminal-Bench 2.0 number published by a vendor may have been obtained with a much larger compute budget per task, a different number of attempts, or the use of auxiliary tools that your production environment won't have. Without that information, comparing two models becomes comparing two photos taken with different cameras and different lighting.

With AISI's Evaluation Cards published via the EEE schema, you can open the card for, say, Terminal-Bench 2.0 and see side by side how AISI's setup compares to other reports of the same benchmark for the same models, including how much inference compute was used and which correction protocol was applied. This changes the evaluation process in three concrete ways:

• Instead of trusting a leaderboard screenshot, you can check the test's technical spec sheet before deciding it's comparable to your use case.
• Teams running their own internal evaluation suite gain a verified reference point to calibrate whether their protocol is producing inflated or deflated numbers compared to the public one.
• Meta-evaluation research (comparing benchmarks against each other, not just models against each other) becomes viable, because the data now has enough metadata for that, something that was missing when each evaluator published only the final score.

The limits of this transparency

It's worth being honest about what this change doesn't solve. First, adoption is voluntary: EEE only works as a de facto standard if other evaluation organizations and model developers also report in this format, and EvalEval's post is explicitly an invitation in that direction, not an obligation. Most of the benchmarks circulating in product marketing today still lack this level of detail.

Second, reproducing an evaluation remains expensive. Having the configuration documented doesn't mean running the same test again, with the same compute budget, is cheap or trivial for a small team. The gain here is auditability and comparability, not free reproduction.

Third, Evaluation Cards for now cover a specific slice: the benchmarks and models that went into AISI's paper, plus the two cyber evals. It's not a universal catalog yet, it's the beginning of one.

How to access and contribute

Anyone who wants to check the configuration behind these numbers can browse the Evaluation Cards on Hugging Face, filtering by benchmark or by model. EvalEval has also left the door open for model developers to report verified results and for evaluation developers to report benchmarks using the Every Eval Ever schema directly. For AI governance and policy researchers, the same repository can be used to examine the overall state of evaluation reporting across the industry, not just look at case by case.

The practical gain for those building LLM-based products in Brazil isn't having one more nice-looking number, it's having less uncertainty about what that number actually means before basing an architecture decision on it.