OpenAI publishes MultiLangBench on GitHub, exposing the COBOL bottleneck

OpenAI published coding evaluations in COBOL and Fortran with significant accuracy gains. The highlight is GPT-6 Astra, which recorded an average pass rate of 87.4% on MultiLangBench.

The benchmark covers 20 languages. Additionally, the evaluation setup and test suite are already on GitHub, with results by language.

The choice of languages stemmed from a practical reason. According to the company, feedback from developers and enterprise clients pointed to recurring difficulty from models in these cases.

However, the list also includes Visual Basic, OCaml, and 16 others.

Why COBOL and Fortran landed in OpenAI's crosshairs

The numbers explain the priority. COBOL, for instance, appears in more than 40% of online banking systems.

It also accounts for 80% of in-person credit card transactions and 95% of ATM transactions.

Fortran occupies another critical territory. Weather forecasting and nuclear reactor safety analysis depend on it.

Maintenance difficulty, however, predates any AI tool. Documentation tends to be incomplete, while the original developers have already left the company.

Additionally, in many cases, business rules exist only in the code.

Consequently, bug fixes and migration carry the same risk. Both need to preserve the behavior the system depends on.

The benchmark asks for an entire SQLite from scratch

The MultiLangBench task is impressive in scope. The model needs to build a SQLite-compatible database engine in the specified language.

Database files work as a shared interface. The candidate program receives an initial database and an SQL query, then returns a modified database.

SQLite's reference implementation checks whether the resulting content matches.

The tests were adapted from SQLite's existing test suite. The results report both the pass rate and the API cost.

This way, the exercise demands codebase organization, state management, error handling, and adding functionality without breaking what already existed.

The numbers OpenAI presented by language

With maximum reasoning effort, GPT-6 Astra scores above 83% in 19 of the 20 languages.

COBOL remains the hardest case, at 72%.

The comparison with GPT-5.6 Sol shows the leap:

COBOL went from 10.9% to 72.1%
Fortran went from 21.4% to 83.5%
Visual Basic went from 27.5% to 89.4%
OCaml went from 25.6% to 85.8%

COBOL migration demands byte-for-byte parity

The second evaluation changes the game. It requires migrating existing COBOL programs to C++, Java, Kotlin, Python, or Rust.

Behavior must be preserved. The hidden tests require byte-for-byte parity in observable outputs.

Therefore, the migration only counts as successful when all tests pass.

Small semantic details decide the outcome. COBOL's fixed-point arithmetic, for example, discards excess decimal digits unless rounding is explicitly requested.

In this evaluation, GPT-6.1 Sol reaches 86.6% accuracy with low reasoning effort. GPT-5.6 Sol scored 5.4% in the same configuration.

Cost also dropped. The run came out 39% cheaper, landing at 61% of the previous API cost.

With higher reasoning effort, accuracy rises to 97.9%. The company itself describes this level as close to the evaluation's saturation point.

Luna trades accuracy for a cheaper OpenAI bill

GPT-6 Luna occupies another tier. It records a 40.3% average pass rate on MultiLangBench with maximum reasoning effort.

Even so, it outperforms GPT-5.6 Sol across all tested languages.

By language, the numbers are as follows: COBOL at 17.1%, Fortran at 31.4%, OCaml at 34.1%, and Visual Basic at 41.6%.

The estimated average API cost reaches US$0.77 per sample. GPT-5.6 Sol, however, cost US$3.48, a difference of about 78%.

The warning that comes with the numbers

This is where the company's own caveat comes in. Production use still requires specialized review and testing based on business rules and dependencies.

It's worth separating the two clearly. The evaluations measure behavior on defined tests, rather than readiness for production systems.

The published data describe pass rate and API cost. Deployment results fall outside this scope.

For those who maintain mainframes, therefore, the practical takeaway is straightforward. Additionally, use these models as an accelerator for reading code and drafting migrations, keeping the regression test suite as the final judge.

Follow our profile on Instagram!