GSM-Plus-BN: A Perturbation-Based Benchmark for Bangla Mathematical Reasoning in Large Language Models

• URL: http://arxiv.org/abs/2607.13248v1
• Date: Tue, 14 Jul 2026 20:23:27 GMT
• Title: GSM-Plus-BN: A Perturbation-Based Benchmark for Bangla Mathematical Reasoning in Large Language Models
• Abstract summary: The evaluation of mathematical reasoning in large language models (LLMs) has predominantly focused on high-resource languages like English.<n>This study introduces GSM-Plus-BN, a novel perturbated Bengali mathematical dataset.<n>We evaluate six open-source LLMs Qwen3-32B, Llama-3.1-8B-Instant, Llama-3.3-70B-Versatile, Llama-4-Scout-17B-16E-Instruct, GPT-OSS-120B, and GPT-OSS-20B using a benchmark of 9,000 evaluation samples.
• Score: 0.06157382820537719
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The evaluation of mathematical reasoning in large language models (LLMs) has predominantly focused on high-resource languages like English. This has created a significant barrier to the equitable development and deployment of AI in linguistically diverse regions such as Bangladesh, where over 230 million people speak Bengali. Despite this global significance, there has been minimal prior work on mathematical reasoning in Bengali and no existing research that systematically benchmarks a perturbated Bengali mathematical dataset, leaving a critical void in assessing model robustness and true comprehension beyond pattern recognition. This study addresses this gap by introducing GSM-Plus-BN, a novel perturbated Bengali mathematical dataset derived from the English GSM-Plus benchmark and verified by human translators. We evaluate six open-source LLMs Qwen3-32B, Llama-3.1-8B-Instant, Llama-3.3-70B-Versatile, Llama-4-Scout-17B-16E-Instruct, GPT-OSS-120B, and GPT-OSS-20B using a benchmark of 9,000 evaluation samples comprising 1,000 seed questions and 8,000 perturbed variants under both Standard Prompting and Chain-of-Thought (CoT) Prompting. Experimental results show that GPT-OSS-20B achieves the highest seed question accuracy of 96.08% under Standard Prompting, while larger models such as Llama-3.3-70B and GPT-OSS-120B demonstrate superior robustness across perturbation types. Furthermore, CoT prompting substantially improves reasoning for most models compared to Standard Prompting, yet a notable performance gap persists across all models relative to their English benchmarks, underscoring the inherent difficulty of perturbed Bengali text. This research makes a foundational contribution by providing GSM-PLUS-BN as a new resource and baseline for future Bengali mathematical reasoning research.

Related papers

• Bengali Text Classification: An Evaluation of Large Language Model Approaches [0.0]
  Unlike English, Bengali faces challenges due to the lack of extensive annotated datasets and pre-trained language models.<n>This study explores the effectiveness of large language models (LLMs) in classifying Bengali newspaper articles.
  arXiv  Detail & Related papers  (2026-01-17T18:25:19Z)
• GanitLLM: Difficulty-Aware Bengali Mathematical Reasoning through Curriculum-GRPO [0.0]
  We present a Bengali mathematical reasoning model called GanitLLM.<n>We also present a new difficulty-aware Bengali math corpus and a curriculum-based GRPO pipeline.
  arXiv  Detail & Related papers  (2026-01-11T03:49:18Z)
• Structured Reasoning with Tree-of-Thoughts for Bengali Math Word Problems [0.0]
  Chain-of-Thought (CoT) prompting has shown promise, but its linear structure often propagates errors.<n>We present the a systematic study of Tree-of-Thought (ToT) reasoning for Bengali MWPs using the SOMADHAN dataset.
  arXiv  Detail & Related papers  (2025-12-05T10:07:08Z)
• Parameter-Efficient Fine-Tuning for Low-Resource Languages: A Comparative Study of LLMs for Bengali Hate Speech Detection [0.0]
  This paper presents the first application of Efficient Fine-Tuning (PEFT) for Bengali hate speech detection using LoRA and QLoRA.<n>Three instruction-tuned large language models were fine-tuned on the BD-SHS dataset of 50,281 annotated comments.<n>Llama-3.2-3B achieved the highest F1-score of 92.23%, followed by Mistral-7B at 88.94% and Gemma-3-4B at 80.25%.
  arXiv  Detail & Related papers  (2025-10-19T20:03:22Z)
• Leveraging Large Language Models for Bengali Math Word Problem Solving with Chain of Thought Reasoning [0.0]
  Solving Bengali Math Word Problems (MWPs) remains a major challenge in natural language processing (NLP)<n>No human-annotated Bengali dataset has previously addressed this task.<n>We created SOMADHAN, a dataset of 8792 complex Bengali MWPs with manually written, step-by-step solutions.
  arXiv  Detail & Related papers  (2025-05-27T15:47:10Z)
• Optimized Text Embedding Models and Benchmarks for Amharic Passage Retrieval [49.1574468325115]
  We introduce Amharic-specific dense retrieval models based on pre-trained Amharic BERT and RoBERTa backbones.<n>Our proposed RoBERTa-Base-Amharic-Embed model (110M parameters) achieves a 17.6% relative improvement in MRR@10.<n>More compact variants, such as RoBERTa-Medium-Amharic-Embed (42M) remain competitive while being over 13x smaller.
  arXiv  Detail & Related papers  (2025-05-25T23:06:20Z)
• R-Bench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation [75.33671166231096]
  We introduce a graduate-level, multi-disciplinary, EnglishChinese benchmark, dubbed as Reasoning Bench (R-Bench)<n>RBench spans 1,094 questions across 108 subjects for language model evaluation and 665 questions across 83 subjects for multimodal model testing.<n>We evaluate widely used models, including OpenAI o1, GPT-4o, DeepSeek-R1, etc.
  arXiv  Detail & Related papers  (2025-05-04T07:48:36Z)
• The Power of Question Translation Training in Multilingual Reasoning: Broadened Scope and Deepened Insights [108.40766216456413]
  We propose a question alignment framework to bridge the gap between large language models' English and non-English performance. Experiment results show it can boost multilingual performance across diverse reasoning scenarios, model families, and sizes. We analyze representation space, generated response and data scales, and reveal how question translation training strengthens language alignment within LLMs.
  arXiv  Detail & Related papers  (2024-05-02T14:49:50Z)
• Common 7B Language Models Already Possess Strong Math Capabilities [61.61442513067561]
  This paper shows that the LLaMA-2 7B model with common pre-training already exhibits strong mathematical abilities. The potential for extensive scaling is constrained by the scarcity of publicly available math questions.
  arXiv  Detail & Related papers  (2024-03-07T18:00:40Z)
• Breaking Language Barriers in Multilingual Mathematical Reasoning: Insights and Observations [59.056367787688146]
  This paper pioneers exploring and training powerful Multilingual Math Reasoning (xMR) LLMs. We construct the first multilingual math reasoning instruction dataset, MGSM8KInstruct, encompassing ten distinct languages. By utilizing translation, we construct the first multilingual math reasoning instruction dataset, MGSM8KInstruct, encompassing ten distinct languages.
  arXiv  Detail & Related papers  (2023-10-31T08:09:20Z)
• Making Large Language Models Better Reasoners with Step-Aware Verifier [49.16750018427259]
  DIVERSE (Diverse Verifier on Reasoning Step) is a novel approach that further enhances the reasoning capability of language models. We evaluate DIVERSE on the latest language model code-davinci and show that it achieves new state-of-the-art results on six of eight reasoning benchmarks.
  arXiv  Detail & Related papers  (2022-06-06T03:38:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.