MSQA: A Natively Sourced Multilingual and Multicultural SimpleQA Benchmark

• URL: http://arxiv.org/abs/2607.00724v2
• Date: Thu, 02 Jul 2026 08:49:19 GMT
• Title: MSQA: A Natively Sourced Multilingual and Multicultural SimpleQA Benchmark
• Abstract summary: A model that can speak a user's language must also understand the culture encoded by that language.<n>We introduce MSQA, a benchmark of 1,064 natively sourced questions across 11 language groups, five cultural dimensions, and three difficulty tiers.<n>We find substantial cultural degradation and a pronounced Locality Effect.
• Score: 31.67992649991894
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multilingual fluency often invites a stronger assumption: a model that can speak a user's language must also understand the culture encoded by that language. We call this the Illusion of Cultural Alignment. To test this assumption directly, we introduce MSQA, a benchmark of 1,064 natively sourced questions across 11 language groups, five cultural dimensions, and three difficulty tiers. Unlike translated benchmarks, MSQA targets locally grounded knowledge and reduces shortcuts from English-centric cross-lingual transfer. Evaluating 18 LLMs, we find substantial cultural degradation and a pronounced Locality Effect: cultural competence tracks pre-training exposure more closely than general reasoning ability. We further show that common inference-time remedies do not dissolve the illusion. Models remain overconfident on unfamiliar cultural questions, repeated sampling yields unstable rather than reliable correctness, and retrieval augmentation helps unevenly on long-tail facts. These findings indicate that cultural alignment cannot be inferred from multilingual ability alone and requires deeper intervention than calibration, sampling, or retrieval at inference time

Related papers

• Beyond Surface Cues: Disentangling Sociocultural Signals in Multilingual LLMs [28.912535376913436]
  The study analyzes 89,253 outputs from 12 Multilingual LLMs in English, French, and Chinese.<n>Removing direct identity cues reduces identity-label prediction in English and Chinese, but has a much smaller effect in French.<n>The ability to identify the source language drops substantially after translation and again after masking names.
  arXiv  Detail & Related papers  (2026-08-24T09:32:16Z)
• The Masked Advantage: Uncovering Local-Language Access to Cultural Knowledge in LLMs [23.21868527222738]
  We use a controlled framework built on real-world cultural questions collected from regional benchmarks and local sources.<n>We find a consistent English advantage on culture-agnostic questions, indicating stronger English proficiency.<n>Local languages show a positive knowledge-access advantage in nearly all locale-model settings.
  arXiv  Detail & Related papers  (2026-06-05T16:16:59Z)
• CultureForest: Understanding and Evaluating Cultural Norm Grounded Reasoning in LLMs [59.09971492475217]
  We introduce CultureForest, a benchmark for textitCultural Norm Grounded Reasoning<n>CultureForest comprises 5,378 examples across 8 domains and 53 countries/regions.<n>Extensive experiments reveal that even top-tier models degrade substantially in open-ended settings.
  arXiv  Detail & Related papers  (2026-06-01T08:25:12Z)
• JuICE: A Benchmark for Evaluating LLM-Judge in Identifying Cultural Errors [38.543532383605545]
  We present JuICE, a dataset of 7,470 span-level annotations of cultural and linguistic errors in long-form language responses.<n>Even the strongest LLM-judge achieves only an F1 of 0.52 in the erroneous span detection task.<n>Our findings suggest that robust cultural evaluation must move beyond surface-level detection toward frameworks that account for the depth and situatedness of cultural meaning.
  arXiv  Detail & Related papers  (2026-05-26T12:45:21Z)
• Cross-Lingual Consensus: Aligning Multilingual Cultural Knowledge via Multilingual Self-Consistency [1.338174941551702]
  Large Language Models (LLMs) exhibit significant performance discrepancies across languages.<n>We propose a novel self-supervised framework to bridge this knowledge gap.
  arXiv  Detail & Related papers  (2026-05-21T08:11:01Z)
• Mitigating Cross-Lingual Cultural Inconsistencies in LLMs via Consensus-Driven Preference Optimisation [58.01855677487771]
  We show that multilingual large language models (MLLMs) frequently exhibit inconsistent behaviour when the prompt's language changes.<n>We propose Cross-lingual Cultural Consistent Preference optimisation (C-3PO), a consensus-driven alignment framework.<n>C-3PO achieves up to a 0.10-point absolute increase in $_S$ over unaligned models, outperforming strong prompting and representation steering baselines.
  arXiv  Detail & Related papers  (2026-04-02T14:04:06Z)
• CRaFT: An Explanation-Based Framework for Evaluating Cultural Reasoning in Multilingual Language Models [0.42970700836450487]
  We introduce CRaFT, an explanation-based multilingual evaluation framework designed to assess how large language models (LLMs) reason across cultural contexts.<n>We apply the framework to 50 culturally grounded questions from the World Values Survey, translated into Arabic, Bengali, and Spanish, and evaluate three models (GPT, DeepSeek, and FANAR) across over 2,100 answer-explanation pairs.<n>Results reveal significant cross-lingual variation in reasoning: Arabic reduces fluency, Bengali enhances it, and Spanish remains largely stable.
  arXiv  Detail & Related papers  (2025-10-15T18:49:10Z)
• MMA-ASIA: A Multilingual and Multimodal Alignment Framework for Culturally-Grounded Evaluation [91.22008265721952]
  MMA-ASIA centers on a human-curated, multilingual, and multimodally aligned benchmark covering 8 Asian countries and 10 languages.<n>This is the first dataset aligned at the input level across three modalities: text, image (visual question answering), and speech.<n>We propose a five-dimensional evaluation protocol that measures: (i) cultural-awareness disparities across countries, (ii) cross-lingual consistency, (iii) cross-modal consistency, (iv) cultural knowledge generalization, and (v) grounding validity.
  arXiv  Detail & Related papers  (2025-10-07T14:12:12Z)
• Cross-Cultural Transfer of Commonsense Reasoning in LLMs: Evidence from the Arab World [68.19795061447044]
  This paper investigates cross-cultural transfer of commonsense reasoning in the Arab world.<n>Using a culturally grounded commonsense reasoning dataset covering 13 Arab countries, we evaluate lightweight alignment methods.<n>Our results show that merely 12 culture-specific examples from one country can improve performance in others by 10% on average.
  arXiv  Detail & Related papers  (2025-09-23T17:24:14Z)
• CaLMQA: Exploring culturally specific long-form question answering across 23 languages [58.18984409715615]
  CaLMQA is a dataset of 51.7K culturally specific questions across 23 different languages.<n>We evaluate factuality, relevance and surface-level quality of LLM-generated long-form answers.
  arXiv  Detail & Related papers  (2024-06-25T17:45:26Z)
• CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark [68.21939124278065]
  Culturally-diverse multilingual Visual Question Answering benchmark designed to cover a rich set of languages and cultures. CVQA includes culturally-driven images and questions from across 30 countries on four continents, covering 31 languages with 13 scripts, providing a total of 10k questions. We benchmark several Multimodal Large Language Models (MLLMs) on CVQA, and show that the dataset is challenging for the current state-of-the-art models.
  arXiv  Detail & Related papers  (2024-06-10T01:59:00Z)
• Not All Countries Celebrate Thanksgiving: On the Cultural Dominance in Large Language Models [89.94270049334479]
  This paper identifies a cultural dominance issue within large language models (LLMs) LLMs often provide inappropriate English-culture-related answers that are not relevant to the expected culture when users ask in non-English languages.
  arXiv  Detail & Related papers  (2023-10-19T05:38:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.