PARSE: An Open-Domain Reasoning Question Answering Benchmark for Persian

• URL: http://arxiv.org/abs/2602.01246v1
• Date: Sun, 01 Feb 2026 14:11:51 GMT
• Title: PARSE: An Open-Domain Reasoning Question Answering Benchmark for Persian
• Abstract summary: We introduce PARSE, the first open-domain Persian reasoning QA benchmark.<n>It contains 10,800 questions across Boolean, multiple-choice, and factoid formats.<n>It is built via a controlled LLM-based generation pipeline and validated through human evaluation.
• Score: 23.04786342076179
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reasoning-focused Question Answering (QA) has advanced rapidly with Large Language Models (LLMs), yet high-quality benchmarks for low-resource languages remain scarce. Persian, spoken by roughly 130 million people, lacks a comprehensive open-domain resource for evaluating reasoning-capable QA systems. We introduce PARSE, the first open-domain Persian reasoning QA benchmark, containing 10,800 questions across Boolean, multiple-choice, and factoid formats, with diverse reasoning types, difficulty levels, and answer structures. The benchmark is built via a controlled LLM-based generation pipeline and validated through human evaluation. We also ensure linguistic and factual quality through multi-stage filtering, annotation, and consistency checks. We benchmark multilingual and Persian LLMs under multiple prompting strategies and show that Persian prompts and structured prompting (CoT for Boolean/multiple-choice; few-shot for factoid) improve performance. Fine-tuning further boosts results, especially for Persian-specialized models. These findings highlight how PARSE supports both fair comparison and practical model adaptation. PARSE fills a critical gap in Persian QA research and provides a strong foundation for developing and evaluating reasoning-capable LLMs in low-resource settings.

Related papers

• RepoReasoner: Evaluating Repository-Level Code Reasoning Ability of Long-Context Language Models [54.53236295237077]
  We introduce RepoReasoner, a benchmark for evaluating repository-level code reasoning.<n> Output Prediction measures fine-grained, stateful execution reasoning across files.<n>Call Chain Prediction evaluates high-level architectural dependency understanding under noisy context.
  arXiv  Detail & Related papers  (2026-07-28T17:12:41Z)
• DS@GT ARC at CheckThat! 2026: LLM-Based Trace Ranking and Grouped Reward Modeling for Multilingual Numerical Claim Verification [0.0]
  This paper focuses on ranking reasoning traces generated by large language models (LLMs) and predicting a final verdict for numerical claims in English and Arabic.<n>The first approach fine-tunes an LLM-based verifier using LoRA to score each reasoning trace independently as a binary classification problem.<n>The second approach uses a lightweight-IDF reward model with handcrafted numeric and temporal overlap features to score traces, and aggregates scores by verdict group to determine the final prediction.
  arXiv  Detail & Related papers  (2026-07-27T20:59:14Z)
• Language-Routed RAG and Direct Option Scoring for Multilingual Financial QA: DS@GT at FinMMEval [0.0]
  Financial certification exams such as the CFA, EFPA, and CPA demand structured domain reasoning.<n>We build a retrieval-augmented pipeline on LangGraph that detects query language and retrieves semantically relevant exemplars.<n>We show that effective multilingual financial reasoning requires language-aware retrieval, model routing, and deliberate scoring strategy selection.
  arXiv  Detail & Related papers  (2026-07-24T18:31:11Z)
• QFS-Composer: Query-focused summarization pipeline for less resourced languages [0.0]
  Large language models (LLMs) demonstrate strong performance in text summarization, yet their effectiveness drops significantly across languages with restricted training resources.<n>This work addresses the challenge of query-focused summarization (QFS) in less-resourced languages, where labeled datasets and evaluation tools are limited.<n>We present a novel QFS framework, QFS-Composer, that integrates query decomposition, question generation (QG), question answering (QA), and abstractive summarization to improve the factual alignment of a summary with user intent.
  arXiv  Detail & Related papers  (2026-04-12T15:25:39Z)
• Benchmarking Open-Source Large Language Models for Persian in Zero-Shot and Few-Shot Learning [0.0]
  This paper presents a benchmark of several open-source Large Language Models (LLMs) for Persian Natural Language Processing tasks.<n>We evaluate models across a range of tasks including sentiment analysis, named entity recognition, reading comprehension, and question answering.<n>The results reveal that Gemma 2 consistently outperforms other models across nearly all tasks in both learning paradigms.
  arXiv  Detail & Related papers  (2025-10-05T10:10:04Z)
• Agri-Query: A Case Study on RAG vs. Long-Context LLMs for Cross-Lingual Technical Question Answering [1.1458853556386799]
  We present a case study evaluating large language models (LLMs) with 128K-token context windows on a technical question answering (QA) task.<n>Our benchmark is built on a user manual for an agricultural machine, available in English, French, and German.<n>It simulates a cross-lingual information retrieval scenario where questions are posed in English against all three language versions of the manual.<n>The evaluation focuses on realistic "needle-in-a-haystack" challenges and includes unanswerable questions to test for hallucinations.
  arXiv  Detail & Related papers  (2025-08-25T14:54:46Z)
• Evaluating Large Language Model with Knowledge Oriented Language Specific Simple Question Answering [73.73820209993515]
  We introduce KoLasSimpleQA, the first benchmark evaluating the multilingual factual ability of Large Language Models (LLMs)<n>Inspired by existing research, we created the question set with features such as single knowledge point coverage, absolute objectivity, unique answers, and temporal stability.<n>Results show significant performance differences between the two domains.
  arXiv  Detail & Related papers  (2025-05-22T12:27:02Z)
• FarsEval-PKBETS: A new diverse benchmark for evaluating Persian large language models [0.5221124918965586]
  This paper introduces FarsEval-PKBETS benchmark, a subset of FarsEval project for evaluating large language models in Persian.<n>This benchmark consists of 4000 questions and answers in various formats, including multiple choice, short answer and descriptive responses.<n>It covers a wide range of domains and tasks, including medicine, law, religion, Persian language, encyclopedic knowledge, human preferences, social knowledge, ethics and bias, text generation, and respecting others' rights.
  arXiv  Detail & Related papers  (2025-04-20T17:43:47Z)
• Assessing Dialect Fairness and Robustness of Large Language Models in Reasoning Tasks [68.33068005789116]
  We introduce ReDial, a benchmark containing 1.2K+ parallel query pairs in Standardized English and AAVE.<n>We evaluate widely used models, including GPT, Claude, Llama, Mistral, and the Phi model families.<n>Our work establishes a systematic and objective framework for analyzing LLM bias in dialectal queries.
  arXiv  Detail & Related papers  (2024-10-14T18:44:23Z)
• CLAIR-A: Leveraging Large Language Models to Judge Audio Captions [73.51087998971418]
  evaluating machine-generated audio captions is a complex task that requires considering diverse factors.<n>We propose CLAIR-A, a simple and flexible method that leverages the zero-shot capabilities of large language models.<n>In our evaluations, CLAIR-A better predicts human judgements of quality compared to traditional metrics.
  arXiv  Detail & Related papers  (2024-09-19T17:59:52Z)
• From Multiple-Choice to Extractive QA: A Case Study for English and Arabic [51.13706104333848]
  We explore the feasibility of repurposing an existing multilingual dataset for a new NLP task.<n>We present annotation guidelines and a parallel EQA dataset for English and Modern Standard Arabic.<n>We aim to help others adapt our approach for the remaining 120 BELEBELE language variants, many of which are deemed under-resourced.
  arXiv  Detail & Related papers  (2024-04-26T11:46:05Z)
• Benchmarking Large Language Models for Persian: A Preliminary Study Focusing on ChatGPT [4.574416868427695]
  This paper explores the efficacy of large language models (LLMs) for Persian. We present the first comprehensive benchmarking study of LLMs across diverse Persian language tasks.
  arXiv  Detail & Related papers  (2024-04-03T02:12:29Z)
• NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens [63.7488938083696]
  We introduce NovelQA, a benchmark tailored for evaluating Large Language Models (LLMs) with complex, extended narratives.<n>NovelQA offers a unique blend of complexity, length, and narrative coherence, making it an ideal tool for assessing deep textual understanding.<n>Our evaluation of long-context LLMs on NovelQA reveals significant insights into their strengths and weaknesses.
  arXiv  Detail & Related papers  (2024-03-18T17:32:32Z)
• Generative Judge for Evaluating Alignment [84.09815387884753]
  We propose a generative judge with 13B parameters, Auto-J, designed to address these challenges. Our model is trained on user queries and LLM-generated responses under massive real-world scenarios. Experimentally, Auto-J outperforms a series of strong competitors, including both open-source and closed-source models.
  arXiv  Detail & Related papers  (2023-10-09T07:27:15Z)
• MinPrompt: Graph-based Minimal Prompt Data Augmentation for Few-shot Question Answering [64.6741991162092]
  We present MinPrompt, a minimal data augmentation framework for open-domain question answering. We transform the raw text into a graph structure to build connections between different factual sentences. We then apply graph algorithms to identify the minimal set of sentences needed to cover the most information in the raw text. We generate QA pairs based on the identified sentence subset and train the model on the selected sentences to obtain the final model.
  arXiv  Detail & Related papers  (2023-10-08T04:44:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.