Scaling Synthetic Pre-training Data and Evaluation for Indic LLMs

Abstract

Building capable language models for India requires scalable multilingual data and robust evaluation. We introduce Nemotron-Pretraining-Indic-v1, a synthetic pretraining corpus of approximately 2.83 trillion tokens covering all 22 scheduled Indian languages in native script and twelve in romanized form. It is the largest openly released Indic pretraining corpus. Using our GPU-accelerated pipeline, we translate curated English documents, assess translation quality, and repair low-quality outputs while preserving technical content and document structure and maintaining source traceability. Through continued-pretraining experiments across twelve languages, we show that combining our synthetic corpus with natural Indic text improves knowledge and reasoning over a public-corpus baseline. Additional experiments show how data volume, script coverage, and Indic training share affect English performance. We also introduce the Indic-LLM-Eval-Suite, a multilingual benchmark suite for evaluating Indic language models across chat, instruction following, reasoning, knowledge, code, and translation. The suite combines existing Indic benchmarks with task-aware translations of English benchmarks, using automated quality assessment, targeted correction, and native-speaker review in a human-in-the-loop pipeline. We use the suite to comprehensively benchmark contemporary open-weight Indic and multilingual models across capabilities and languages, revealing substantial variation across tasks and languages. Together, these resources provide a scalable foundation for developing and evaluating more capable Indic language models.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Computation & Language / NLP

• arXiv preprint · 19 October 2026

  Cognitive Maps in Large Language Models

  Albert Qin, Jonathan Cohen and Gautam Reddy
• arXiv preprint · 20 October 2026

  The Router Already Knows

  Andrew Bistras and Marty Johnson
• arXiv preprint · 19 October 2026

  Text Dominance in Paralinguistic Conflict: Tracing Suppression Through Audio LLM Circuits

  Ankur Banerjee, Vishnu Guntakala, Danity Pike and 3 others
• arXiv preprint · 20 October 2026

  Harness Self-Evolution for Long-Horizon Terminal Agents

  Ruhan Wang, Yucheng Shi, Zongxia Li and 6 others
• arXiv preprint · 19 October 2026

  Curriculum Learning Teaches LLMs Long-Horizon Implicit Reasoning, Up To A Limit

  Jimson Huang, Maxwell Nickell, Ruowang Zhang and 1 other

All new papers in Computation & Language / NLP