Benchmarking Hindi LLMs: A New Suite of Datasets and a Comparative Analysis

• URL: http://arxiv.org/abs/2508.19831v2
• Date: Wed, 15 Oct 2025 17:12:44 GMT
• Title: Benchmarking Hindi LLMs: A New Suite of Datasets and a Comparative Analysis
• Authors: Anusha Kamath, Kanishk Singla, Rakesh Paul, Raviraj Joshi, Utkarsh Vaidya, Sanjay Singh Chauhan, Niranjan Wartikar,
• Abstract summary: We introduce a suite of five Hindi evaluation datasets: IFEval-Hi, MT-Bench-Hi, GSM8K-Hi, ChatRAG-Hi, and BFCL-Hi.<n>These were created using a methodology that combines from-scratch human annotation with a translate-and-verify process.
• Score: 3.9680588541743718
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Evaluating instruction-tuned Large Language Models (LLMs) in Hindi is challenging due to a lack of high-quality benchmarks, as direct translation of English datasets fails to capture crucial linguistic and cultural nuances. To address this, we introduce a suite of five Hindi LLM evaluation datasets: IFEval-Hi, MT-Bench-Hi, GSM8K-Hi, ChatRAG-Hi, and BFCL-Hi. These were created using a methodology that combines from-scratch human annotation with a translate-and-verify process. We leverage this suite to conduct an extensive benchmarking of open-source LLMs supporting Hindi, providing a detailed comparative analysis of their current capabilities. Our curation process also serves as a replicable methodology for developing benchmarks in other low-resource languages.

Related papers

• IndicRAGSuite: Large-Scale Datasets and a Benchmark for Indian Language RAG Systems [17.88837706307504]
  IndicMSMarco is a multilingual benchmark for evaluating retrieval quality and response generation in 13 Indian languages.<n>We build a large-scale dataset of (question, answer, relevant passage)s derived from the Wikipedias of 19 Indian languages using state-of-the-art LLMs.
  arXiv  Detail & Related papers  (2025-06-02T12:55:51Z)
• IDA-Bench: Evaluating LLMs on Interactive Guided Data Analysis [60.32962597618861]
  IDA-Bench is a novel benchmark evaluating large language models in multi-round interactive scenarios.<n>Agent performance is judged by comparing its final numerical output to the human-derived baseline.<n>Even state-of-the-art coding agents (like Claude-3.7-thinking) succeed on 50% of the tasks, highlighting limitations not evident in single-turn tests.
  arXiv  Detail & Related papers  (2025-05-23T09:37:52Z)
• Trans-Zero: Self-Play Incentivizes Large Language Models for Multilingual Translation Without Parallel Data [64.4458540273004]
  We propose a self-play framework that leverages only monolingual data and the intrinsic multilingual knowledge of Large Language Models (LLMs)<n>Experiments demonstrate that this approach not only matches the performance of models trained on large-scale parallel data but also excels in non-English translation directions.
  arXiv  Detail & Related papers  (2025-04-20T16:20:30Z)
• Regional Tiny Stories: Using Small Models to Compare Language Learning and Tokenizer Performance [1.1784026260358966]
  We focus on Hindi, Marathi, and Bengali, evaluating SLMs for regional language processing and understanding linguistic complexity.<n>Our analysis shows that language-specific tokenizers outperform general-purpose ones for Indian languages.<n>These findings advance both the practical application of SLMs to underserved languages and our theoretical understanding of neural language development.
  arXiv  Detail & Related papers  (2025-04-07T10:33:14Z)
• Instruction Tuning on Public Government and Cultural Data for Low-Resource Language: a Case Study in Kazakh [57.002807772016524]
  We introduce and open-source a large-scale (10,600 samples) instruction-following dataset, covering key institutional and cultural knowledge relevant to Kazakhstan.<n>We employ LLM-assisted data generation, comparing open-weight and closed-weight models for dataset construction, and select GPT-4o as the backbone.<n>We show that fine-tuning Qwen, Falcon, and Gemma on our dataset leads to consistent performance improvements in both multiple-choice and generative tasks.
  arXiv  Detail & Related papers  (2025-02-19T11:44:27Z)
• MMTEB: Massive Multilingual Text Embedding Benchmark [85.18187649328792]
  We introduce the Massive Multilingual Text Embedding Benchmark (MMTEB)<n>MMTEB covers over 500 quality-controlled evaluation tasks across 250+ languages.<n>We develop several highly multilingual benchmarks, which we use to evaluate a representative set of models.
  arXiv  Detail & Related papers  (2025-02-19T10:13:43Z)
• P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMs [84.24644520272835]
  We introduce P-MMEval, a large-scale benchmark covering effective fundamental and capability-specialized datasets.<n>P-MMEval delivers consistent language coverage across various datasets and provides parallel samples.<n>We conduct extensive experiments on representative multilingual model series to compare performances across models and tasks.
  arXiv  Detail & Related papers  (2024-11-14T01:29:36Z)
• Think Carefully and Check Again! Meta-Generation Unlocking LLMs for Low-Resource Cross-Lingual Summarization [108.6908427615402]
  Cross-lingual summarization ( CLS) aims to generate a summary for the source text in a different target language.<n>Currently, instruction-tuned large language models (LLMs) excel at various English tasks.<n>Recent studies have shown that LLMs' performance on CLS tasks remains unsatisfactory even with few-shot settings.
  arXiv  Detail & Related papers  (2024-10-26T00:39:44Z)
• Hindi-BEIR : A Large Scale Retrieval Benchmark in Hindi [8.21020989074456]
  Despite ongoing research, there is a lack of comprehensive benchmark for evaluating retrieval models in Hindi. We introduce the Hindi version of the BEIR benchmark, which includes a subset of English BEIR datasets translated to Hindi, existing Hindi retrieval datasets, and synthetically created datasets for retrieval. We evaluate state-of-the-art multilingual retrieval models on this benchmark to identify task and domain-specific challenges and their impact on retrieval performance.
  arXiv  Detail & Related papers  (2024-08-18T10:55:04Z)
• Suvach -- Generated Hindi QA benchmark [0.0]
  This paper proposes a new benchmark specifically designed for evaluating Hindi EQA models. This method leverages large language models (LLMs) to generate a high-quality dataset in an extractive setting.
  arXiv  Detail & Related papers  (2024-04-30T04:19:17Z)
• This is the way: designing and compiling LEPISZCZE, a comprehensive NLP benchmark for Polish [5.8090623549313944]
  We introduce LEPISZCZE, a new, comprehensive benchmark for Polish NLP. We use five datasets from the Polish benchmark and add eight novel datasets. We provide insights and experiences learned while creating the benchmark for Polish as the blueprint to design similar benchmarks for other low-resourced languages.
  arXiv  Detail & Related papers  (2022-11-23T16:51:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.