From Words to Worlds: Benchmarking Cross-Cultural Cultural Understanding in Machine Translation

• URL: http://arxiv.org/abs/2603.17303v1
• Date: Wed, 18 Mar 2026 02:59:44 GMT
• Title: From Words to Worlds: Benchmarking Cross-Cultural Cultural Understanding in Machine Translation
• Abstract summary: We introduce CulT-Eval, a benchmark designed to evaluate how models handle different types of culturally grounded expressions.<n>CulT-Eval comprises over 7,959 carefully curated instances spanning multiple types of culturally grounded expressions.<n>We propose a complementary evaluation metric that targets culturally induced meaning deviations.
• Score: 16.809989616664605
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Culture-expressions, such as idioms, slang, and culture-specific items (CSIs), are pervasive in natural language and encode meanings that go beyond literal linguistic form. Accurately translating such expressions remains challenging for machine translation systems. Despite this, existing benchmarks remain fragmented and do not provide a systematic framework for evaluating translation performance on culture-loaded expressions. To address this gap, we introduce CulT-Eval, a benchmark designed to evaluate how models handle different types of culturally grounded expressions. CulT-Eval comprises over 7,959 carefully curated instances spanning multiple types of culturally grounded expressions, with a comprehensive error taxonomy covering culturally grounded expressions. Through extensive evaluation of large language models and detailed analysis, we identify recurring and systematic failure modes that are not adequately captured by existing automatic metrics. Accordingly, we propose a complementary evaluation metric that targets culturally induced meaning deviations overlooked by standard MT metrics. The results indicate that current models struggle to preserve culturally grounded meaning and to capture the cultural and contextual nuances essential for accurate translation. Our benchmark and code are available at https://anonymous.4open.science/r/CulT-Eval-E75D/.

Related papers

• MemeBridge: A Dataset for Benchmarking and Mitigating the Bidirectional Cultural Gap in Meme Interpretation [16.37143301773968]
  We introduce MemeBridge, a curated dataset centered on U.S.-originated memes.<n>Here, context refers to implicit cultural knowledge, including background beliefs, norms, and shared assumptions that shape meme comprehension.<n>We observe that the anticipated misunderstandings from U.S. participants are often inaccurate.
  arXiv  Detail & Related papers  (2026-08-31T23:38:23Z)
• On the Systematic Challenges of Culturally Loaded Machine Translation: Dream of the Red Chamber as the Cultural Lens [4.236324569120652]
  We investigate the challenges posed by culturally loaded translation in large language models (LLMs)<n>We construct a Chinese-Japanese bilingual dataset from the culturally representative corpus Dream of the Red Chamber.<n>Using a comprehensive evaluation protocol, we reveal three main challenges: (1) task challenges, where frontier LLMs exhibit notable performance gaps and struggle with culturally loaded content; (2) human evaluation challenges, where evaluator backgrounds lead to substantial disagreement in translation judgments; and (3) automatic evaluation challenges, where widely used metrics fail to reliably assess translation quality for this task.
  arXiv  Detail & Related papers  (2026-07-22T15:01:10Z)
• Worlds Within Words: Translating Culture in Ancient Chinese Texts with Multi-Agent Coordination [42.36051310803725]
  Large language model (LLM)-based machine translation has advanced cross-cultural communication, yet it still struggles with culture-loaded words (CLWs) in ancient Chinese texts.<n>We propose textbfMACAT, a framework that identifies culturally salient phrases and injects concise explanatory knowledge when necessary.<n>MACAT consistently outperforms both the backbone model and general-purpose MT baselines on 100 TCM documents.
  arXiv  Detail & Related papers  (2026-05-31T14:58:03Z)
• Beyond Literal Translation: Evaluating Cultural Effectiveness in Social Media UGC [39.66998905774523]
  We introduce CULTURE-MT, a benchmark for social media translation.<n>CULTURE-MT consists of 1,002 notes across 14 domains, categorized into four types based on culture-loaded symbols and linguistic style features.<n>We propose cultural effectiveness as a new evaluation criterion, focusing on expression accuracy and cultural adaptability.
  arXiv  Detail & Related papers  (2026-05-25T09:29:25Z)
• Tears or Cheers? Benchmarking LLMs via Culturally Elicited Distinct Affective Responses [28.3173238194554]
  We introduce CEDAR, a benchmark constructed entirely from scenarios capturing culturally underlinetextscElicited underlinetextscDistinct underlinetextscAffective underlinetextscResponses.<n>The resulting benchmark comprises 10,962 instances across seven languages and 14 fine-grained emotion categories, with each language including 400 multimodal and 1,166 text-only samples.
  arXiv  Detail & Related papers  (2026-01-19T13:04:26Z)
• CURE: Cultural Understanding and Reasoning Evaluation - A Framework for "Thick" Culture Alignment Evaluation in LLMs [24.598338950728234]
  Large language models (LLMs) are increasingly deployed in culturally diverse environments.<n>Existing methods focus on de-contextualized correctness or forced-choice judgments.<n>We introduce a set of benchmarks that present models with realistic situational contexts.
  arXiv  Detail & Related papers  (2025-11-15T03:39:13Z)
• CultureScope: A Dimensional Lens for Probing Cultural Understanding in LLMs [57.653830744706305]
  CultureScope is the most comprehensive evaluation framework to date for assessing cultural understanding in large language models.<n>Inspired by the cultural iceberg theory, we design a novel dimensional schema for cultural knowledge classification.<n> Experimental results demonstrate that our method can effectively evaluate cultural understanding.
  arXiv  Detail & Related papers  (2025-09-19T17:47:48Z)
• CaMMT: Benchmarking Culturally Aware Multimodal Machine Translation [47.54347964192231]
  We introduce CaMMT, a benchmark of over 5,800 triples of images along with parallel captions in English and regional languages.<n>Through automatic and human evaluations, we find that visual context generally improves translation quality.
  arXiv  Detail & Related papers  (2025-05-30T10:42:44Z)
• Not All Countries Celebrate Thanksgiving: On the Cultural Dominance in Large Language Models [89.94270049334479]
  This paper identifies a cultural dominance issue within large language models (LLMs) LLMs often provide inappropriate English-culture-related answers that are not relevant to the expected culture when users ask in non-English languages.
  arXiv  Detail & Related papers  (2023-10-19T05:38:23Z)
• On the Cultural Gap in Text-to-Image Generation [75.69755281031951]
  One challenge in text-to-image (T2I) generation is the inadvertent reflection of culture gaps present in the training data. There is no benchmark to systematically evaluate a T2I model's ability to generate cross-cultural images. We propose a Challenging Cross-Cultural (C3) benchmark with comprehensive evaluation criteria, which can assess how well-suited a model is to a target culture.
  arXiv  Detail & Related papers  (2023-07-06T13:17:55Z)
• Benchmarking Machine Translation with Cultural Awareness [50.183458829028226]
  Translating culture-related content is vital for effective cross-cultural communication. Many culture-specific items (CSIs) often lack viable translations across languages. This difficulty hinders the analysis of cultural awareness of machine translation systems.
  arXiv  Detail & Related papers  (2023-05-23T17:56:33Z)
• When Does Translation Require Context? A Data-driven, Multilingual Exploration [71.43817945875433]
  proper handling of discourse significantly contributes to the quality of machine translation (MT) Recent works in context-aware MT attempt to target a small set of discourse phenomena during evaluation. We develop the Multilingual Discourse-Aware benchmark, a series of taggers that identify and evaluate model performance on discourse phenomena.
  arXiv  Detail & Related papers  (2021-09-15T17:29:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.