Beyond Literal Translation: Evaluating Cultural Effectiveness in Social Media UGC

• URL: http://arxiv.org/abs/2605.25626v1
• Date: Mon, 25 May 2026 09:29:25 GMT
• Title: Beyond Literal Translation: Evaluating Cultural Effectiveness in Social Media UGC
• Abstract summary: We introduce CULTURE-MT, a benchmark for social media translation.<n>CULTURE-MT consists of 1,002 notes across 14 domains, categorized into four types based on culture-loaded symbols and linguistic style features.<n>We propose cultural effectiveness as a new evaluation criterion, focusing on expression accuracy and cultural adaptability.
• Score: 39.66998905774523
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Social media platforms enable large-scale cross-lingual communication, but translating user-generated content (UGC) remains challenging due to its informal style, cultural references, and interaction-based expressions. While recent LLMs have improved translation quality, existing benchmarks and metrics often fail to capture whether translations convey intended meaning and cultural resonance in real-world settings. In this work, we introduce CULTURE-MT, a benchmark for social media translation that focuses on both CULtural Transmission and UGC-specific emotion REsonance. CULTURE-MT consists of 1,002 UGC notes across 14 domains, categorized into four types based on culture-loaded symbols and linguistic style features. We also construct UGC-oriented training data to fine-tune Qwen3-8B and Qwen3-32B as baselines. We propose cultural effectiveness as a new evaluation criterion, focusing on expression accuracy and cultural adaptability. Testing 15 models, including the baselines, we find that traditional metrics fail to capture cultural effectiveness. We also observe that cultural effectiveness on base LLMs correlates with model size. Our work provides a comprehensive evaluation system for UGC translation models and will offer an open evaluation platform to advance research in this area. We release the CULTURE-MT benchmark and provide an online leaderboard where submitted translation results can be evaluated by our trained JUDGER.

Related papers

• CultureConverse: A Multilingual Multi-turn Simulation Harness for Culturally Grounded Assistance in East and Southeast Asia [72.20656906184539]
  We introduce CultureConverse, a scalable, multilingual simulation and evaluation harness for culturally grounded assistant dialogue.<n>The resulting CultureConverse-DS dataset contains 14,610 benchmark (evaluation) episodes and 274,295 oracle-guided (gold-mode) dialogues.<n>Human annotation experiments suggest that our evaluation framework is a sufficient proxy for human judgment.
  arXiv  Detail & Related papers  (2026-08-28T14:56:46Z)
• CrossCult-KIBench: A Benchmark for Cross-Cultural Knowledge Insertion in MLLMs [13.267566050457974]
  Cross-cultural knowledge insertion focuses on adapting models to specific cultural contexts while preserving original behavior in other cultures.<n> benchmark includes 9,800 image-grounded cases covering 49 culturally relevant visual scenarios across English, Chinese, and Arabic language-culture groups.<n>Memory-Conditioned Knowledge Insertion (MCKI) retrieves relevant cultural knowledge from an external memory using frozen MLLM representations.
  arXiv  Detail & Related papers  (2026-05-07T12:26:11Z)
• From Words to Worlds: Benchmarking Cross-Cultural Cultural Understanding in Machine Translation [16.809989616664605]
  We introduce CulT-Eval, a benchmark designed to evaluate how models handle different types of culturally grounded expressions.<n>CulT-Eval comprises over 7,959 carefully curated instances spanning multiple types of culturally grounded expressions.<n>We propose a complementary evaluation metric that targets culturally induced meaning deviations.
  arXiv  Detail & Related papers  (2026-03-18T02:59:44Z)
• Culturally-Aware Conversations: A Framework & Benchmark for LLMs [8.314136556868563]
  Existing benchmarks that measure cultural adaptation in LLMs are misaligned with the actual challenges these models face when interacting with users from diverse cultural backgrounds.<n>Grounded in sociocultural theory, our framework formalizes how linguistic style is shaped by situational, relational, and cultural context.<n>We construct a benchmark dataset based on this framework, annotated by culturally diverse raters, and propose a new set of desiderata for cross-cultural evaluation in NLP.
  arXiv  Detail & Related papers  (2025-10-13T16:06:14Z)
• Cross-Cultural Transfer of Commonsense Reasoning in LLMs: Evidence from the Arab World [68.19795061447044]
  This paper investigates cross-cultural transfer of commonsense reasoning in the Arab world.<n>Using a culturally grounded commonsense reasoning dataset covering 13 Arab countries, we evaluate lightweight alignment methods.<n>Our results show that merely 12 culture-specific examples from one country can improve performance in others by 10% on average.
  arXiv  Detail & Related papers  (2025-09-23T17:24:14Z)
• CultureScope: A Dimensional Lens for Probing Cultural Understanding in LLMs [57.653830744706305]
  CultureScope is the most comprehensive evaluation framework to date for assessing cultural understanding in large language models.<n>Inspired by the cultural iceberg theory, we design a novel dimensional schema for cultural knowledge classification.<n> Experimental results demonstrate that our method can effectively evaluate cultural understanding.
  arXiv  Detail & Related papers  (2025-09-19T17:47:48Z)
• CultureSynth: A Hierarchical Taxonomy-Guided and Retrieval-Augmented Framework for Cultural Question-Answer Synthesis [41.483432890962824]
  We introduce Culture Synth, a novel framework for assessing large language models' cultural competence.<n>The Culture Synth-7 benchmark contains 19,360 entries and 4,149 manually verified entries across 7 languages.
  arXiv  Detail & Related papers  (2025-09-13T16:33:56Z)
• MCEval: A Dynamic Framework for Fair Multilingual Cultural Evaluation of LLMs [25.128936333806678]
  Large language models exhibit cultural biases and limited cross-cultural understanding capabilities.<n>We propose MCEval, a novel multilingual evaluation framework that employs dynamic cultural question construction.
  arXiv  Detail & Related papers  (2025-07-13T16:24:35Z)
• From Word to World: Evaluate and Mitigate Culture Bias in LLMs via Word Association Test [50.51344198689069]
  We extend the human-centered word association test (WAT) to assess the alignment of large language models with cross-cultural cognition.<n>To address culture preference, we propose CultureSteer, an innovative approach by embedding cultural-specific semantic associations directly within the model's internal representation space.
  arXiv  Detail & Related papers  (2025-05-24T07:05:10Z)
• CulturePark: Boosting Cross-cultural Understanding in Large Language Models [63.452948673344395]
  This paper introduces CulturePark, an LLM-powered multi-agent communication framework for cultural data collection. It generates high-quality cross-cultural dialogues encapsulating human beliefs, norms, and customs. We evaluate these models across three downstream tasks: content moderation, cultural alignment, and cultural education.
  arXiv  Detail & Related papers  (2024-05-24T01:49:02Z)
• Benchmarking Machine Translation with Cultural Awareness [50.183458829028226]
  Translating culture-related content is vital for effective cross-cultural communication. Many culture-specific items (CSIs) often lack viable translations across languages. This difficulty hinders the analysis of cultural awareness of machine translation systems.
  arXiv  Detail & Related papers  (2023-05-23T17:56:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.