NVIDIA and AI4Bharat open pipeline that generated 2.8 trillion tokens to train AI in Indian languages

NVIDIA published, in partnership with AI4Bharat (the Indian-languages lab at IIT Madras) and Bodhan AI, Nemotron-pretraining-Indic-v1, a corpus of approximately 2.8 trillion tokens covering the 22 languages listed in the Indian Constitution. The announcement came out on NVIDIA's blog on Hugging Face this Friday (October 16). More important than the data volume is what comes with it: the complete generation pipeline, including prompts, the quality-evaluation flow, and the repair logic, released for anyone who wants to adapt the recipe to another language or domain.

For those developing in Portuguese or researching Brazilian indigenous languages, the value isn't in the data itself (these are Indian languages, so they won't be directly useful), but in the engineering behind it: how to turn a high-quality English corpus into reliable pretraining data for a language with little native data. It's exactly the problem any Brazilian team would face when trying to train a decent model in Guarani, Nheengatu, or even a more specialized technical Portuguese.

How the pipeline turns English into training data

The starting point is 15 subsets of Nemotron-CC-v2, covering web text, question-and-answer pairs, math, and code. The team uses NeMo Data Designer, NVIDIA's open-source framework for synthetic data generation, to run three stages: translation, quality evaluation, and repair. Each output record keeps the identifier of the original source document, which makes it possible to audit where each translated sentence came from.

The corpus was split into two groups with different workflows, because not every translation model performs well on long documents or unstable scripts:

• Twelve languages (Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Odia, Punjabi, Tamil, Telugu, and Urdu) receive both native script and romanized script (the language's text, but in the Latin alphabet).
• Ten languages (Assamese, Bodo, Dogri, Kashmiri, Konkani, Maithili, Manipuri, Sanskrit, Santali, and Sindhi) receive only the native script.

In the twelve main languages, the team translates the entire document at once, preserving context across paragraphs, tables, and equations, using Gemma-4-31B-it. The same model then evaluates its own translation against the original English text, using a rubric inspired by the Multidimensional Quality Metrics (MQM) framework, which separates errors by type and severity: meaning, fluency, terminology, style, and formatting.

Only translations scoring at least 75 out of 100 on this rubric make it into the final corpus. What falls below that isn't discarded outright: the evaluation feedback itself feeds a repair stage, and the corrected text is re-evaluated against the same threshold. This repair cycle recovered 438.4 billion tokens that would otherwise have been lost.

For the other ten languages, translation comes from a dedicated model, Indic Translate, from Bodhan AI, and goes through the same Gemma-4-31B-it screening, but without the repair cycle. Special cases get separate handling: long documents in Santali are translated in overlapping windows and then reconstructed, and Manipuri first goes through Bengali script before being converted to its native alphabet, Meitei Mayek.

Native script isn't the same thing as romanized text

A detail that often gets overlooked in discussions about multilingual data: "native script," in the project's vocabulary, doesn't mean text originally written in that language. It means the content was translated from English and rendered in the language's native alphabet. Romanized text, meanwhile, is the same Indian language, just written in Latin characters, not a translation into English.

This distinction matters when building the training mix, because native script and romanized versions can carry the same content with very different character exposure. For the twelve languages that have both versions, romanization starts from translations already accepted in native script, rather than being generated directly from English: according to the team, this two-step path produced fewer hallucinations and less English residue than going straight from the source text to the Latin alphabet.

What this changed in a real model

NVIDIA used a twelve-language subset of this corpus, combined with natural Indian text, to run continued pretraining on Nemotron 3.5 Super VL: a training budget of 100 billion tokens, of which 10% (10 billion) were synthetic and natural Indian data. Comparing the checkpoint before and after this stage, the five knowledge-and-reasoning benchmarks in Indian languages (MILU, Indic MMLU-Pro, Indic ARC-Challenge, Indic ARC-Easy, and Indic GSM8K) rose between 4.98 and 9.82 percentage points.

Mitesh Khapra, who leads AI4Bharat and made the TIME100 AI list for 2025, sums up the problem the project is trying to tackle:

We do not have enough data for Indian languages.

Mitesh Khapra, head of AI4Bharat, IIT Madras

It's worth noting the limits of this comparison: the gain measures a training stage that combines synthetic data with natural data, and doesn't isolate the contribution of the synthetic corpus alone. And the Nemotron 3.5 Super VL technical report treats these numbers as the result of a training stage, not as a definitive comparison between final versions of the model.

Testing the dataset in practice

The corpus is organized by language, script, and region tags: hi-Deva-IN is Hindi in Devanagari, hi-Latn-IN is romanized Hindi. The example NVIDIA publishes for exploring the dataset via Hugging Face's datasets is straightforward:

from datasets import get_dataset_split_names, load_dataset

dataset_id = input("Repository ID from the dataset card: ").strip()
config = "hi-Deva-IN"

splits = get_dataset_split_names(dataset_id, config_name=config)
print("Available splits:", splits)

data = load_dataset(
 dataset_id, name=config, split=splits[0], streaming=True
)
print(next(iter(data)))

Using streaming=True avoids downloading the entire corpus just to inspect a sample, which matters when you're talking about trillions of tokens. NVIDIA's own recommendation for a first study is to compare natural/synthetic and native/romanized mixes with a fixed token budget, tracking English-capability retention and per-language script quality.

What can be reused outside the Indian context

The most exportable piece of the announcement is the Nemotron Bharat plugin for NeMo Data Designer: open-source, with translation and evaluation prompts, optional feedback-driven repair, long-document windowing, and script-remapping support. A team looking to build a similar pipeline for technical Portuguese, or for a Brazilian indigenous language with little native corpus, has there a tested, at-scale architecture reference: full-document translation, automated evaluation with its own rubric, an explicit acceptance threshold, and a repair cycle guided by the evaluator's own feedback.

In short, the practical gain for those building outside India isn't the dataset, it's the process design: how to turn machine translation into auditable training data, with provenance traceable back to the source document and an explicit quality threshold (75/100) instead of accepting everything that comes out of the translation model.

Where the pipeline runs into limits

NVIDIA itself lists limitations that apply to any adaptation of the method. Quality evaluation is mostly automated, with sample-based human review whose coverage varies by language; in the twelve main languages' workflow, the same model that translates also judges its own translation, a known methodological conflict of interest. The source-identifier check removes processing duplicates, but it isn't semantic deduplication of similar content.

More importantly: translation never replaces text originally written in that language, nor does it guarantee coverage of local and cultural knowledge. For Portuguese or Brazilian indigenous languages, this means a pipeline built this way solves volume, but doesn't solve the lack of a rich native corpus; the team's own recommendation is to combine synthetic data with well-curated natural text and evaluate the resulting mix for the specific use case, not to assume that translation at scale is enough.