Language over Content: Tracing Cultural Understanding in Multilingual Large Language Models

• URL: http://arxiv.org/abs/2510.16565v1
• Date: Sat, 18 Oct 2025 16:19:45 GMT
• Title: Language over Content: Tracing Cultural Understanding in Multilingual Large Language Models
• Authors: Seungho Cho, Changgeon Ko, Eui Jun Hwang, Junmyeong Lee, Huije Lee, Jong C. Park,
• Abstract summary: Internal paths overlap more for same-language, cross-country questions than for cross-language, same-country questions.<n>Results show that internal paths overlap more for same-language, cross-country questions than for cross-language, same-country questions.
• Score: 10.798925500517823
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) are increasingly used across diverse cultural contexts, making accurate cultural understanding essential. Prior evaluations have mostly focused on output-level performance, obscuring the factors that drive differences in responses, while studies using circuit analysis have covered few languages and rarely focused on culture. In this work, we trace LLMs' internal cultural understanding mechanisms by measuring activation path overlaps when answering semantically equivalent questions under two conditions: varying the target country while fixing the question language, and varying the question language while fixing the country. We also use same-language country pairs to disentangle language from cultural aspects. Results show that internal paths overlap more for same-language, cross-country questions than for cross-language, same-country questions, indicating strong language-specific patterns. Notably, the South Korea-North Korea pair exhibits low overlap and high variability, showing that linguistic similarity does not guarantee aligned internal representation.

Related papers

• AmharicStoryQA: A Multicultural Story Question Answering Benchmark in Amharic [25.361090310093072]
  We argue that evaluations overlook meaningful cultural variation that exists within a single language.<n>We introduce textbftextitAmharicStoryQA, a benchmark grounded in culturally diverse narratives from Amharic-speaking regions.
  arXiv  Detail & Related papers  (2026-02-02T20:28:19Z)
• Language Models Entangle Language and Culture [1.0742675209112622]
  We create a set of real-world open-ended questions based on our analysis of the WildChat dataset.<n>We use it to evaluate whether responses vary by language, specifically, whether answer quality depends on the language used to query the model.<n>We find that language significantly impacts the cultural context used by the model.
  arXiv  Detail & Related papers  (2026-01-20T10:46:44Z)
• MyCulture: Exploring Malaysia's Diverse Culture under Low-Resource Language Constraints [7.822567458977689]
  MyCulture is a benchmark designed to comprehensively evaluate Large Language Models (LLMs) on Malaysian culture.<n>Unlike conventional benchmarks, MyCulture employs a novel open-ended multiple-choice question format without predefined options.<n>We analyze structural bias by comparing model performance on structured versus free-form outputs, and assess language bias through multilingual prompt variations.
  arXiv  Detail & Related papers  (2025-08-07T14:17:43Z)
• Nunchi-Bench: Benchmarking Language Models on Cultural Reasoning with a Focus on Korean Superstition [0.0]
  We introduce Nunchi-Bench, a benchmark designed to evaluate large language models' cultural understanding.<n>The benchmark consists of 247 questions spanning 31 topics, assessing factual knowledge, culturally appropriate advice, and situational interpretation.<n>We evaluate multilingual LLMs in both Korean and English to analyze their ability to reason about Korean cultural contexts.
  arXiv  Detail & Related papers  (2025-07-05T11:52:09Z)
• Disentangling Language and Culture for Evaluating Multilingual Large Language Models [48.06219053598005]
  This paper introduces a Dual Evaluation Framework to comprehensively assess the multilingual capabilities of LLMs.<n>By decomposing the evaluation along the dimensions of linguistic medium and cultural context, this framework enables a nuanced analysis of LLMs' ability to process questions cross-lingually.
  arXiv  Detail & Related papers  (2025-05-30T14:25:45Z)
• MAKIEval: A Multilingual Automatic WiKidata-based Framework for Cultural Awareness Evaluation for LLMs [37.98920430188422]
  MAKIEval is an automatic multilingual framework for evaluating cultural awareness in large language models.<n>It automatically identifies cultural entities in model outputs and links them to structured knowledge.<n>We assess 7 LLMs developed from different parts of the world, encompassing both open-source and proprietary systems.
  arXiv  Detail & Related papers  (2025-05-27T19:29:40Z)
• CaLMQA: Exploring culturally specific long-form question answering across 23 languages [58.18984409715615]
  CaLMQA is a dataset of 51.7K culturally specific questions across 23 different languages.<n>We evaluate factuality, relevance and surface-level quality of LLM-generated long-form answers.
  arXiv  Detail & Related papers  (2024-06-25T17:45:26Z)
• See It from My Perspective: How Language Affects Cultural Bias in Image Understanding [60.70852566256668]
  Vision-language models (VLMs) can respond to queries about images in many languages.<n>We characterize the Western bias of VLMs in image understanding and investigate the role that language plays in this disparity.
  arXiv  Detail & Related papers  (2024-06-17T15:49:51Z)
• Extrinsic Evaluation of Cultural Competence in Large Language Models [53.626808086522985]
  We focus on extrinsic evaluation of cultural competence in two text generation tasks. We evaluate model outputs when an explicit cue of culture, specifically nationality, is perturbed in the prompts. We find weak correlations between text similarity of outputs for different countries and the cultural values of these countries.
  arXiv  Detail & Related papers  (2024-06-17T14:03:27Z)
• CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark [68.21939124278065]
  Culturally-diverse multilingual Visual Question Answering benchmark designed to cover a rich set of languages and cultures. CVQA includes culturally-driven images and questions from across 30 countries on four continents, covering 31 languages with 13 scripts, providing a total of 10k questions. We benchmark several Multimodal Large Language Models (MLLMs) on CVQA, and show that the dataset is challenging for the current state-of-the-art models.
  arXiv  Detail & Related papers  (2024-06-10T01:59:00Z)
• Quantifying the Dialect Gap and its Correlates Across Languages [69.18461982439031]
  This work will lay the foundation for furthering the field of dialectal NLP by laying out evident disparities and identifying possible pathways for addressing them through mindful data collection.
  arXiv  Detail & Related papers  (2023-10-23T17:42:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.