GhazalBench: Usage-Grounded Evaluation of LLMs on Persian Ghazals

• URL: http://arxiv.org/abs/2603.09979v1
• Date: Fri, 06 Feb 2026 13:37:27 GMT
• Title: GhazalBench: Usage-Grounded Evaluation of LLMs on Persian Ghazals
• Abstract summary: We introduce GhazalBench, a benchmark for evaluating how large language models interact with Persian ghazals.<n>We observe a consistent dissociation: models generally capture poetic meaning but struggle with exact verse recall in completion-based settings.<n>Our findings highlight the need for evaluation frameworks that jointly assess meaning, form, and cue-dependent access to culturally significant texts.
• Score: 5.695664084679044
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Persian poetry plays an active role in Iranian cultural practice, where verses by canonical poets such as Hafez are frequently quoted, paraphrased, or completed from partial cues. Supporting such interactions requires language models to engage not only with poetic meaning but also with culturally entrenched surface form. We introduce GhazalBench, a benchmark for evaluating how large language models (LLMs) interact with Persian ghazals under usage-grounded conditions. GhazalBench assesses two complementary abilities: producing faithful prose paraphrases of couplets and accessing canonical verses under varying semantic and formal cues. Across several proprietary and open-weight multilingual LLMs, we observe a consistent dissociation: models generally capture poetic meaning but struggle with exact verse recall in completion-based settings, while recognition-based tasks substantially reduce this gap. A parallel evaluation on English sonnets shows markedly higher recall performance, suggesting that these limitations are tied to differences in training exposure rather than inherent architectural constraints. Our findings highlight the need for evaluation frameworks that jointly assess meaning, form, and cue-dependent access to culturally significant texts. GhazalBench is available at https://github.com/kalhorghazal/GhazalBench.

Related papers

• When Similar Means Different: Evaluating LLMs on Arabic--Hebrew Cognates [3.5005564083285123]
  Arabic and Hebrew, as closely related Semitic languages, share a substantial lexicon of true cognates, misleading false friends, and modern loanwords.<n>This overlap poses a challenge for cross-lingual semantic understanding in large language models (LLMs)<n>We introduce SemCog Bench, a benchmark of 1,858 Arabic--Hebrew word pairs with sentence-level annotations for cognate identification and semantic reasoning.
  arXiv  Detail & Related papers  (2026-06-11T11:33:09Z)
• Constrained Semantic Decompression in LLMs through Persian Proverb-Conditioned Story Generation [5.105644262054024]
  We introduce the Proverb Aligned Narrative dataset (PAND), pairing proverbs with human-written stories and explicit meanings.<n>Our findings reveal a persistent emphdecompression gap: current language models fail to faithfully instantiate the underlying moral and causal structure encoded in proverbs.<n>Our proposed task naturally extends to other forms of compressed cultural knowledge.
  arXiv  Detail & Related papers  (2026-06-10T18:54:07Z)
• IDIOLEX: Unified and Continuous Representations for Idiolectal and Stylistic Variation [88.98544786373212]
  Existing sentence representations primarily encode what a sentence says, rather than how it is expressed.<n>We develop sentence representations that capture style and dialect, decoupled from semantic content.<n>We introduce IDIOLEX, a framework for training models that combines supervision from a sentence's provenance with linguistic features of a sentence's content.
  arXiv  Detail & Related papers  (2026-04-06T14:17:24Z)
• Mitigating Cross-Lingual Cultural Inconsistencies in LLMs via Consensus-Driven Preference Optimisation [58.01855677487771]
  We show that multilingual large language models (MLLMs) frequently exhibit inconsistent behaviour when the prompt's language changes.<n>We propose Cross-lingual Cultural Consistent Preference optimisation (C-3PO), a consensus-driven alignment framework.<n>C-3PO achieves up to a 0.10-point absolute increase in $_S$ over unaligned models, outperforming strong prompting and representation steering baselines.
  arXiv  Detail & Related papers  (2026-04-02T14:04:06Z)
• Hybrid topic modelling for computational close reading: Mapping narrative themes in Pushkin's Evgenij Onegin [0.0]
  This study presents a hybrid topic modelling framework for computational literary analysis.<n>It integrates Latent Dirichlet Allocation (LDA) with sparse Partial Least Squares Discriminant Analysis (sPLS-DA) to model thematic structure and longitudinal dynamics in narrative poetry.
  arXiv  Detail & Related papers  (2026-03-20T13:37:08Z)
• TARAZ: Persian Short-Answer Question Benchmark for Cultural Evaluation of Language Models [10.191206917633503]
  This paper presents a comprehensive evaluation framework for assessing the cultural competence of large language models (LLMs) in Persian.<n>Our framework introduces a Persian-specific short-answer evaluation that combines rule-based morphological normalization with a hybrid syntactic and semantic similarity module.
  arXiv  Detail & Related papers  (2026-02-26T10:08:02Z)
• Unmasking the Factual-Conceptual Gap in Persian Language Models [0.8180770164225369]
  We introduce DivanBench, a benchmark focused on superstitions and customs, arbitrary, context-dependent rules that resist simple logical deduction.<n>We evaluate seven Persian LLMs and reveal three critical failures: most models exhibit severe acquiescence bias, correctly identifying appropriate behaviors but failing to reject clear violations.
  arXiv  Detail & Related papers  (2026-02-19T18:42:46Z)
• AmharicStoryQA: A Multicultural Story Question Answering Benchmark in Amharic [25.361090310093072]
  We argue that evaluations overlook meaningful cultural variation that exists within a single language.<n>We introduce textbftextitAmharicStoryQA, a benchmark grounded in culturally diverse narratives from Amharic-speaking regions.
  arXiv  Detail & Related papers  (2026-02-02T20:28:19Z)
• On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluation [88.77441715819366]
  Generative spoken language models pretrained on large-scale raw audio can continue a speech prompt with appropriate content.<n>We propose a variety of likelihood- and generative-based evaluation methods that serve in place of naive global token perplexity.
  arXiv  Detail & Related papers  (2026-01-09T22:01:56Z)
• Liaozhai through the Looking-Glass: On Paratextual Explicitation of Culture-Bound Terms in Machine Translation [70.43884512651668]
  We formalize Genette's (1987) theory of paratexts from literary and translation studies to introduce the task of paratextual explicitation for machine translation.<n>We construct a dataset of 560 expert-aligned paratexts from four English translations of the classical Chinese short story collection Liaozhai.<n>Our findings demonstrate the potential of paratextual explicitation in advancing machine translation beyond linguistic equivalence.
  arXiv  Detail & Related papers  (2025-09-27T16:27:36Z)
• Fann or Flop: A Multigenre, Multiera Benchmark for Arabic Poetry Understanding in LLMs [32.247169514152425]
  We introduce emphFann or Flop, the first benchmark designed to assess the comprehension of Arabic poetry by large language models.<n>The benchmark comprises a curated corpus of poems with explanations that assess semantic understanding, metaphor interpretation, prosodic awareness, and cultural context.
  arXiv  Detail & Related papers  (2025-05-23T17:59:29Z)
• FENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction [85.26780391682894]
  We propose Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction (FENICE) FENICE leverages an NLI-based alignment between information in the source document and a set of atomic facts, referred to as claims, extracted from the summary. Our metric sets a new state of the art on AGGREFACT, the de-facto benchmark for factuality evaluation.
  arXiv  Detail & Related papers  (2024-03-04T17:57:18Z)
• Simple Linguistic Inferences of Large Language Models (LLMs): Blind Spots and Blinds [59.71218039095155]
  We evaluate language understanding capacities on simple inference tasks that most humans find trivial. We target (i) grammatically-specified entailments, (ii) premises with evidential adverbs of uncertainty, and (iii) monotonicity entailments. The models exhibit moderate to low performance on these evaluation sets.
  arXiv  Detail & Related papers  (2023-05-24T06:41:09Z)
• Natural Language Decompositions of Implicit Content Enable Better Text Representations [52.992875653864076]
  We introduce a method for the analysis of text that takes implicitly communicated content explicitly into account.<n>We use a large language model to produce sets of propositions that are inferentially related to the text that has been observed.<n>Our results suggest that modeling the meanings behind observed language, rather than the literal text alone, is a valuable direction for NLP.
  arXiv  Detail & Related papers  (2023-05-23T23:45:20Z)
• Metrical Tagging in the Wild: Building and Annotating Poetry Corpora with Rhythmic Features [0.0]
  We provide large poetry corpora for English and German, and annotate prosodic features in smaller corpora to train corpus driven neural models. We show that BiLSTM-CRF models with syllable embeddings outperform a CRF baseline and different BERT-based approaches.
  arXiv  Detail & Related papers  (2021-02-17T16:38:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.