The Masked Advantage: Uncovering Local-Language Access to Cultural Knowledge in LLMs

• URL: http://arxiv.org/abs/2606.07422v2
• Date: Mon, 08 Jun 2026 19:53:59 GMT
• Title: The Masked Advantage: Uncovering Local-Language Access to Cultural Knowledge in LLMs
• Abstract summary: We use a controlled framework built on real-world cultural questions collected from regional benchmarks and local sources.<n>We find a consistent English advantage on culture-agnostic questions, indicating stronger English proficiency.<n>Local languages show a positive knowledge-access advantage in nearly all locale-model settings.
• Score: 23.21868527222738
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models are increasingly used to answer culturally grounded questions across languages, yet it remains unclear whether local cultural knowledge is better accessed through English or the local language. Existing evaluations face two key limitations: many rely on parallel template-based questions that may not reflect how cultural knowledge naturally appears, and raw accuracy conflates general language proficiency with language-conditioned knowledge access. We address these issues with a controlled framework built on real-world cultural questions collected from regional benchmarks and local sources. By crossing question type (culture-agnostic vs. culture-specific) with query language (English vs. local language), and estimating ability with a shared 1PL item response theory model, we separate proficiency from localized knowledge access. Across 13 locales and roughly 80 models, we find a consistent English advantage on culture-agnostic questions, indicating stronger English proficiency. However, after accounting for this proficiency gap, local languages show a positive knowledge-access advantage in nearly all locale-model settings. This advantage is often masked in raw accuracy but becomes more visible for frontier, regionally aligned, or language-adapted models. Our results suggest that weaker local-language performance does not necessarily imply weaker cultural knowledge; rather, local cultural knowledge may be more accessible through the local language but hidden by limited language proficiency.

Related papers

• BavGround: A Benchmark for Regional Cultural Grounding and Dialect Competence in Bavarian [33.0382356140632]
  We introduce BavGround, a benchmark for evaluating Bavarian regional cultural grounding and dialect competence.<n>BavGround contains 206 multiple-choice source questions across eight cultural domains per language.<n>Strong multilingual models perform best overall, but performance drops on Bavarian items and source-grounded questions.
  arXiv  Detail & Related papers  (2026-08-13T07:23:53Z)
• MSQA: A Natively Sourced Multilingual and Multicultural SimpleQA Benchmark [31.67992649991894]
  A model that can speak a user's language must also understand the culture encoded by that language.<n>We introduce MSQA, a benchmark of 1,064 natively sourced questions across 11 language groups, five cultural dimensions, and three difficulty tiers.<n>We find substantial cultural degradation and a pronounced Locality Effect.
  arXiv  Detail & Related papers  (2026-07-01T10:12:03Z)
• CultureForest: Understanding and Evaluating Cultural Norm Grounded Reasoning in LLMs [59.09971492475217]
  We introduce CultureForest, a benchmark for textitCultural Norm Grounded Reasoning<n>CultureForest comprises 5,378 examples across 8 domains and 53 countries/regions.<n>Extensive experiments reveal that even top-tier models degrade substantially in open-ended settings.
  arXiv  Detail & Related papers  (2026-06-01T08:25:12Z)
• Cross-Lingual Consensus: Aligning Multilingual Cultural Knowledge via Multilingual Self-Consistency [1.338174941551702]
  Large Language Models (LLMs) exhibit significant performance discrepancies across languages.<n>We propose a novel self-supervised framework to bridge this knowledge gap.
  arXiv  Detail & Related papers  (2026-05-21T08:11:01Z)
• Do You Know About My Nation? Investigating Multilingual Language Models' Cultural Literacy Through Factual Knowledge [68.6805229085352]
  Most multilingual question-answering benchmarks do not factor in regional diversity in the information they capture.<n>XNationQA encompasses a total of 49,280 questions on the geography, culture, and history of nine countries, presented in seven languages.<n>We benchmark eight standard multilingual LLMs on XNationQA and evaluate them using two novel transference metrics.
  arXiv  Detail & Related papers  (2025-11-01T18:41:34Z)
• CultureScope: A Dimensional Lens for Probing Cultural Understanding in LLMs [57.653830744706305]
  CultureScope is the most comprehensive evaluation framework to date for assessing cultural understanding in large language models.<n>Inspired by the cultural iceberg theory, we design a novel dimensional schema for cultural knowledge classification.<n> Experimental results demonstrate that our method can effectively evaluate cultural understanding.
  arXiv  Detail & Related papers  (2025-09-19T17:47:48Z)
• MAKIEval: A Multilingual Automatic WiKidata-based Framework for Cultural Awareness Evaluation for LLMs [37.98920430188422]
  MAKIEval is an automatic multilingual framework for evaluating cultural awareness in large language models.<n>It automatically identifies cultural entities in model outputs and links them to structured knowledge.<n>We assess 7 LLMs developed from different parts of the world, encompassing both open-source and proprietary systems.
  arXiv  Detail & Related papers  (2025-05-27T19:29:40Z)
• WorldCuisines: A Massive-Scale Benchmark for Multilingual and Multicultural Visual Question Answering on Global Cuisines [74.25764182510295]
  Vision Language Models (VLMs) often struggle with culture-specific knowledge, particularly in languages other than English.<n>We introduce World Cuisines, a massive-scale benchmark for multilingual and multicultural, visually grounded language understanding.<n>This benchmark includes a visual question answering (VQA) dataset with text-image pairs across 30 languages and dialects, spanning 9 language families and featuring over 1 million data points.
  arXiv  Detail & Related papers  (2024-10-16T16:11:49Z)
• CaLMQA: Exploring culturally specific long-form question answering across 23 languages [58.18984409715615]
  CaLMQA is a dataset of 51.7K culturally specific questions across 23 different languages.<n>We evaluate factuality, relevance and surface-level quality of LLM-generated long-form answers.
  arXiv  Detail & Related papers  (2024-06-25T17:45:26Z)
• Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsense [98.09670425244462]
  Large language models (LLMs) have demonstrated substantial commonsense understanding. This paper examines the capabilities and limitations of several state-of-the-art LLMs in the context of cultural commonsense tasks.
  arXiv  Detail & Related papers  (2024-05-07T20:28:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.