PersLitEval: Fine-grained Benchmark and Evaluation of LLMs on Persian Literature Questions

• URL: http://arxiv.org/abs/2605.27015v1
• Date: Tue, 26 May 2026 13:34:19 GMT
• Title: PersLitEval: Fine-grained Benchmark and Evaluation of LLMs on Persian Literature Questions
• Abstract summary: We introduce PersLitEval, a benchmark of 4,514 Persian literature multiple-choice questions across eight fine-grained categories.<n>We evaluate models across ten prompting strategies, revealing striking category-level disparities across three tiers of task difficulty.<n>An error analysis identifies three failure modes: semantic comprehension gaps, formal linguistic knowledge gaps, and counting/enumeration errors.
• Score: 48.69228180369574
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Despite impressive multilingual capabilities, large language models (LLMs) remain poorly evaluated on literary knowledge in non-English languages. We introduce PersLitEval, a benchmark of 4,514 Persian literature multiple-choice questions across eight fine-grained categories spanning spelling, literary devices, grammar, vocabulary, word formation, and conceptual understanding, sourced from materials for the Konkur university entrance examination. We evaluate six LLMs across ten prompting strategies, revealing striking category-level disparities across three tiers of task difficulty: models reach higher accuracy on conceptual similarity tasks but struggle with formal linguistic analysis, with spelling and word formation proving the hardest across all models. Prompting strategy has a significant impact on performance, with explained few-shot examples yielding the best results, particularly on formal linguistic categories. An error analysis identifies three failure modes: semantic comprehension gaps, formal linguistic knowledge gaps, and counting/enumeration errors, suggesting that different categories require different improvement strategies.

Related papers

• Challenges in annotations by humans and LLMs: A case study of evaluative language [0.0]
  Linguists in training, a trained linguist, and annotations generated by large language models (LLMs) are compared.<n>We focus on the Appraisal theory and its Attitude subsystem, including the categories (classes) of Affect, Judgement, and Appreciation.<n>We find that models perform best compared to annotations conducted by the trained linguist, while linguists in training do not reach high agreement scores.
  arXiv  Detail & Related papers  (2026-07-30T12:28:54Z)
• LANG: Reinforcement Learning for Multilingual Reasoning with Language-Adaptive Hint Guidance [77.58408743830314]
  Reinforcement learning has proven effective for enhancing multi-step reasoning in large language models.<n>But its benefits have not fully translated to multilingual contexts.<n>We develop a novel framework that leverages language-conditioned hints to guide exploration in non-English reasoning tasks.
  arXiv  Detail & Related papers  (2026-05-21T14:47:52Z)
• IndicEval: A Bilingual Indian Educational Evaluation Framework for Large Language Models [0.0]
  This paper introduces IndicEval, a scalable benchmarking platform to assess large language models (LLMs) performance.<n>IndicEval grounds evaluation in real examination standards, enabling realistic measurement of reasoning, domain knowledge, and bilingual adaptability.<n>Experiments conducted on Gemini 2.0 Flash, GPT-4, Claude, and LLaMA 3-70B reveal three major findings.
  arXiv  Detail & Related papers  (2026-02-18T13:55:57Z)
• AraLingBench A Human-Annotated Benchmark for Evaluating Arabic Linguistic Capabilities of Large Language Models [37.79823471716066]
  AraLingBench is a fully human annotated benchmark for evaluating the Arabic linguistic competence of large language models (LLMs)<n>The benchmark spans five core categories: grammar, morphology, spelling, reading comprehension, and syntax, through 150 expert-designed multiple choice questions.
  arXiv  Detail & Related papers  (2025-11-18T09:47:01Z)
• PolyMath: Evaluating Mathematical Reasoning in Multilingual Contexts [85.78821098963607]
  PolyMath is a multilingual mathematical reasoning benchmark covering 18 languages and 4 easy-to-hard difficulty levels.<n>Our benchmark ensures difficulty comprehensiveness, language diversity, and high-quality translation.
  arXiv  Detail & Related papers  (2025-04-25T15:39:04Z)
• XIFBench: Evaluating Large Language Models on Multilingual Instruction Following [59.549015333755186]
  Large Language Models (LLMs) have demonstrated remarkable instruction-following capabilities across various applications.<n>Existing evaluations lack fine-grained constraint analysis across diverse linguistic contexts.<n>We introduce XIFBench, a comprehensive benchmark for evaluating multilingual instruction-following abilities of LLMs.
  arXiv  Detail & Related papers  (2025-03-10T17:07:52Z)
• SLAM: Towards Efficient Multilingual Reasoning via Selective Language Alignment [78.4550589538805]
  We propose an efficient multilingual reasoning alignment approach that precisely identifies and fine-tunes the layers responsible for handling multilingualism.<n> Experimental results show that our method, SLAM, only tunes 6 layers' feed-forward sub-layers including 6.5-8% of all parameters within 7B and 13B LLMs.
  arXiv  Detail & Related papers  (2025-01-07T10:29:43Z)
• Comparative Study of Multilingual Idioms and Similes in Large Language Models [4.581124233698535]
  This study explores the effectiveness of prompt engineering strategies, including chain-of-thought, few-shot, and English translation prompts.<n>We extend the language of these datasets to Persian as well by building two new evaluation sets.<n>Our findings reveal that while prompt engineering methods are generally effective, their success varies by figurative type, language, and model.
  arXiv  Detail & Related papers  (2024-10-21T19:40:05Z)
• Khayyam Challenge (PersianMMLU): Is Your LLM Truly Wise to The Persian Language? [3.4812080203308984]
  Khayyam Challenge (also known as PersianMMLU) is a collection of 20,192 four-choice questions sourced from 38 diverse tasks extracted from Persian examinations. The primary objective of the Khayyam Challenge is to facilitate the rigorous evaluation of LLMs that support the Persian language.
  arXiv  Detail & Related papers  (2024-04-09T22:38:13Z)
• Analyzing and Adapting Large Language Models for Few-Shot Multilingual NLU: Are We There Yet? [82.02076369811402]
  Supervised fine-tuning (SFT), supervised instruction tuning (SIT) and in-context learning (ICL) are three alternative, de facto standard approaches to few-shot learning. We present an extensive and systematic comparison of the three approaches, testing them on 6 high- and low-resource languages, three different NLU tasks, and a myriad of language and domain setups. Our observations show that supervised instruction tuning has the best trade-off between performance and resource requirements.
  arXiv  Detail & Related papers  (2024-03-04T10:48:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.