IndiaFinBench: An Evaluation Benchmark for Large Language Model Performance on Indian Financial Regulatory Text

• URL: http://arxiv.org/abs/2604.19298v1
• Date: Tue, 21 Apr 2026 10:04:49 GMT
• Title: IndiaFinBench: An Evaluation Benchmark for Large Language Model Performance on Indian Financial Regulatory Text
• Abstract summary: We introduce IndiaFinBench, the first publicly available evaluation benchmark for assessing large language model (LLM) performance on Indian financial regulatory text.<n>IndiaFinBench addresses this gap with 406 expert-annotated question-answer pairs drawn from 192 documents sourced from the Securities and Exchange Board of India and the Reserve Bank of India.<n>We evaluate twelve models under zero-shot conditions, with accuracy ranging from 70.4% (Gemma 4 E4B) to 89.7% (Gemini 2.5 Flash)
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce IndiaFinBench, to our knowledge the first publicly available evaluation benchmark for assessing large language model (LLM) performance on Indian financial regulatory text. Existing financial NLP benchmarks draw exclusively from Western financial corpora (SEC filings, US earnings reports, and English-language financial news), leaving a significant gap in coverage of non-Western regulatory frameworks. IndiaFinBench addresses this gap with 406 expert-annotated question-answer pairs drawn from 192 documents sourced from the Securities and Exchange Board of India (SEBI) and the Reserve Bank of India (RBI), spanning four task types: regulatory interpretation (174 items), numerical reasoning (92 items), contradiction detection (62 items), and temporal reasoning (78 items). Annotation quality is validated through a model-based secondary pass (kappa=0.918 on contradiction detection) and a 60-item human inter-annotator agreement evaluation (kappa=0.611; 76.7% overall agreement). We evaluate twelve models under zero-shot conditions, with accuracy ranging from 70.4% (Gemma 4 E4B) to 89.7% (Gemini 2.5 Flash). All models substantially outperform a non-specialist human baseline of 60.0%. Numerical reasoning is the most discriminative task, with a 35.9 percentage-point spread across models. Bootstrap significance testing (10,000 resamples) reveals three statistically distinct performance tiers. The dataset, evaluation code, and all model outputs are available at https://github.com/rajveerpall/IndiaFinBench

Related papers

• From Financial Sentiment Classification to Return Predictability: A QLoRA Benchmark of Large Language Models [0.0]
  We construct a unified three-class benchmark from five financial text datasets.<n>We compare TF--IDF Naive Bayes, off-the-shelf FinBERT and Financial-RoBERTa encoders.<n>We evaluate economic validity on a temporally separate 2019 Benzinga sample.
  arXiv  Detail & Related papers  (2026-08-04T19:57:40Z)
• AWARE-FX: An Auditable Knowledge-Guided AI System for Measuring Corporate Foreign-Exchange Hedging Disclosure [5.667104793245957]
  Corporate annual reports contain weakly structured evidence about foreign-exchange risk management, derivative use, natural hedging, and explicit non-use.<n>This study develops AWARE-FX, an auditable AI/NLP decision-support system that converts report text into traceable firm-year hedging-disclosure measures.
  arXiv  Detail & Related papers  (2026-07-30T02:58:46Z)
• Knowledge Index of Noah's Ark [63.143852586221534]
  We introduce KINA, an 899-item benchmark across 261 fine-grained disciplines.<n>We prove a bonus-on-bar tournament weakly FOSD-dominates flat payment in released-review quality.<n>The top model, Gemini-3.1-Pro-Preview, reaches 53.17%, followed by Claude-Opus-4.6 at 49.92% and GPT-5.4 at 48.55%.
  arXiv  Detail & Related papers  (2026-06-03T17:06:49Z)
• Automated Benchmark Auditing for AI Agents and Large Language Models [46.03841647776303]
  Auto Benchmark Audit (ABA) is an agentic framework that systematically audits individual benchmark tasks.<n>We run ABA on a collection of frontier LLM benchmarks and previous NeurIPS publications, totaling 168 benchmarks across nine domains.<n>ABA identifies critical issues including ambiguous task design, execution environment conflicts, and incorrect ground truths in over 25.7% of the evaluated tasks.
  arXiv  Detail & Related papers  (2026-05-25T17:44:21Z)
• Tokenizer Fertility and Zero-Shot Performance of Foundation Models on Ukrainian Legal Text: A Comparative Study [0.0]
  Tokenizer fertility varies 1.6x across foundation models on Ukrainian legal text.<n>Qwen 3 models consume 60% more tokens than Llama-family models on identical input.<n>Few-shot degrades performance by up to 26 percentage points.
  arXiv  Detail & Related papers  (2026-05-14T14:35:05Z)
• Fact4ac at the Financial Misinformation Detection Challenge Task: Reference-Free Financial Misinformation Detection via Fine-Tuning and Few-Shot Prompting of Large Language Models [4.8239913268889]
  This paper presents our winning methodology for the "Reference-Free Financial Misinformation Detection" shared task.<n>We propose a comprehensive framework that capitalizes on the reasoning capabilities of state-of-the-art Large Language Models.<n>Our proposed system demonstrated superior efficacy, successfully securing the first-place ranking on both official leaderboards.
  arXiv  Detail & Related papers  (2026-04-16T05:38:58Z)
• A Multi-Model Approach to English-Bangla Sentiment Classification of Government Mobile Banking App Reviews [0.0]
  The study analyzed 5,652 Google Play reviews in English and Bangla for four Bangladeshi government banking apps.<n>Traditional models produced the highest accuracy (0.815), while Linear SVM produced the highest weighted F1 score (0.804)<n>DeBERTa-v3 was applied to analyze the sentiment at the aspect level across the reviews for the four apps.
  arXiv  Detail & Related papers  (2026-03-17T21:43:51Z)
• All That Glisters Is Not Gold: A Benchmark for Reference-Free Counterfactual Financial Misinformation Detection [67.89888669159899]
  RFC Bench is a benchmark for evaluating large language models on financial misinformation under realistic news.<n>The benchmark defines two complementary tasks: reference free misinformation detection and comparison based diagnosis.
  arXiv  Detail & Related papers  (2026-01-07T18:18:28Z)
• LLM Output Drift: Cross-Provider Validation & Mitigation for Financial Workflows [0.5798758080057375]
  Nondeterministic outputs (output drift) undermine auditability and trust.<n>We quantify drift across five model architectures on regulated financial tasks.<n>This finding challenges conventional assumptions that larger models are universally superior for production deployment.
  arXiv  Detail & Related papers  (2025-11-10T19:54:00Z)
• Fine-Tuning Vision-Language Models for Markdown Conversion of Financial Tables in Malaysian Audited Financial Reports [0.0]
  We propose a fine-tuned vision-language model (VLM) based on Qwen2.5-VL-7B.<n>Our approach includes a curated dataset of 2,152 image-text pairs with augmentations and a supervised fine-tuning strategy using LoRA.<n>Our model achieves a 92.20% overall accuracy on the criteria-based assessment and a 96.53% markdown TEDS score.
  arXiv  Detail & Related papers  (2025-08-04T04:54:00Z)
• BizFinBench: A Business-Driven Real-World Financial Benchmark for Evaluating LLMs [7.9458352414205295]
  Large language models excel in general tasks, yet assessing their reliability in logic-heavy, precision-critical domains like finance, law, and healthcare remains challenging.<n>We introduce BizFinBench, the first benchmark specifically designed to evaluate LLMs in real-world financial applications.<n> BizFinBench consists of 6,781 well-annotated queries in Chinese, spanning five dimensions: numerical calculation, reasoning, information extraction, prediction recognition, and knowledge-based question answering.
  arXiv  Detail & Related papers  (2025-05-26T03:23:02Z)
• Benchmarking Reasoning Robustness in Large Language Models [76.79744000300363]
  We find significant performance degradation on novel or incomplete data.<n>These findings highlight the reliance on recall over rigorous logical inference.<n>This paper introduces a novel benchmark, termed as Math-RoB, that exploits hallucinations triggered by missing information to expose reasoning gaps.
  arXiv  Detail & Related papers  (2025-03-06T15:36:06Z)
• FinTSB: A Comprehensive and Practical Benchmark for Financial Time Series Forecasting [58.70072722290475]
  Financial time series (FinTS) record the behavior of human-brain-augmented decision-making.<n>FinTSB is a comprehensive and practical benchmark for financial time series forecasting.
  arXiv  Detail & Related papers  (2025-02-26T05:19:16Z)
• Holistic Evaluation of Language Models [183.94891340168175]
  Language models (LMs) are becoming the foundation for almost all major language technologies, but their capabilities, limitations, and risks are not well understood. We present Holistic Evaluation of Language Models (HELM) to improve the transparency of language models.
  arXiv  Detail & Related papers  (2022-11-16T18:51:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.