Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline

NVIDIA and AI4Bharat Release 2.8T Token Indic Corpus

NVIDIA, in collaboration with IIT Madras and its AI4Bharat lab, has released a massive pretraining dataset containing approximately 2.8 trillion tokens across 22 Indian languages. The release of Nemotron-pretraining-Indic-v1, along with the open, GPU-accelerated data generation pipeline used to create it, directly addresses the critical data scarcity that has impeded the development of capable AI models for a population of nearly 1.2 billion people.

A Look Inside the Data Generation Pipeline

The corpus was constructed by translating high-quality English source material, including technical and question-answer content, using NVIDIA's open-source NeMo Data Designer framework. The process employs a multi-stage workflow that includes document-level translation, quality scoring against an MQM-inspired rubric, and a targeted repair loop for outputs that initially fail to meet a 75/100 quality threshold. This full recipe is being open-sourced, enabling developers to audit, reproduce, and adapt the process for their own needs.

• Total Tokens: ~2.8 trillion
• Native-Script Tokens: ~2.6 trillion
• Romanized Tokens: ~224 billion
• Languages Covered: 22 official Indian languages (34 language-script configurations)
• Core LLM: Gemma-4-31B-it used for translation and quality assessment
• Data Sources: 15 subsets from the Nemotron-CC-v2 corpus

Ecosystem Impact and Performance Gains

By providing both the dataset and the methodology, this initiative equips the AI community to build more sophisticated models capable of reasoning in native Indian languages. The data's effectiveness was demonstrated in the development of Nemotron 3.5 Super VL, which showed score increases of up to 9.8 percentage points on Indic benchmarks after continued pretraining with just 10 billion tokens from this corpus. Opening the entire process is intended to accelerate research and foster a more robust ecosystem for Indic AI development.

The strategic value lies not just in the 2.8 trillion tokens, but in open-sourcing the entire data generation and quality assurance pipeline, effectively teaching the community how to create high-quality, synthetic data for low-resource languages at scale.