When Metrics Reward the Worst Translations: Internalizing Cultural Reasoning for Social Media Translation Evaluation

• URL: http://arxiv.org/abs/2609.08156v1
• Date: Tue, 08 Sep 2026 02:39:50 GMT
• Title: When Metrics Reward the Worst Translations: Internalizing Cultural Reasoning for Social Media Translation Evaluation
• Abstract summary: We show that standard metrics including COMET, XCOMET, and BERTScore exhibit near-zero or negative correlation with human cultural judgments.<n>We propose CuRIL, a reinforcement learning framework that internalizes cultural reasoning.
• Score: 40.50995079894586
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Automatic translation quality metrics trained on general-domain corpora systematically fail on social media content, where communicative intent is encoded in culturally loaded expressions (internet slang, homophonic ciphers, and platform-specific idioms) rather than surface token patterns. We conduct a systematic empirical analysis demonstrating that standard metrics including COMET, XCOMET, and BERTScore exhibit near-zero or negative correlation with human cultural judgments, and even display a severity inversion in which scores increase as translation quality deteriorates. We further show that this failure extends to large language model judges: Qwen3-235B achieves Cohen's kappa of only 0.162, revealing that the bottleneck is not reasoning capacity but cultural grounding: models lack the domain-specific cultural knowledge needed to identify which aspects of a translation require scrutiny. To address this, we propose CuRIL, a reinforcement learning framework that internalizes cultural reasoning: cultural annotations are prepended inside the model's reasoning, excluded from policy gradients via a token-level loss mask, and injected with a probability that decays to zero over training, progressively forcing autonomous cultural judgment. On a 1,444-sample human-annotated social media translation benchmark, Qwen3-8B trained with CuRIL achieves Cohen's kappa 0.370 and Exact Match accuracy of 45.22%, approaching Gemini-3.1-Pro with 30x fewer parameters and surpassing models up to 235B in scale. We further demonstrate that our judge produces reliable reward signals for downstream translation optimization, reducing the low-quality translation rate by over 20 percentage points under independent human evaluation.

Related papers

• CultureConverse: A Multilingual Multi-turn Simulation Harness for Culturally Grounded Assistance in East and Southeast Asia [72.20656906184539]
  We introduce CultureConverse, a scalable, multilingual simulation and evaluation harness for culturally grounded assistant dialogue.<n>The resulting CultureConverse-DS dataset contains 14,610 benchmark (evaluation) episodes and 274,295 oracle-guided (gold-mode) dialogues.<n>Human annotation experiments suggest that our evaluation framework is a sufficient proxy for human judgment.
  arXiv  Detail & Related papers  (2026-08-28T14:56:46Z)
• Debiasing Text-to-Image Evaluation via Implicit Cultural Alignment Reward Modeling [1.1172382217477128]
  We introduce an Implicit Cultural Alignment Reward Model for generative AI.<n>Our framework integrates an Implicit Cultural Probe with a Skip-connection Cross-Attention mechanism.<n>Our model processes each evaluation in 0.21 seconds under our local inference setup, achieving a $10times$ speedup over standard VQA-based evaluators.
  arXiv  Detail & Related papers  (2026-07-17T08:23:03Z)
• Benchmarking Frontier LLMs on Arabic Cultural and Sociolinguistic Knowledge: A Cross-Evaluation Framework with Human SME Ground Truth [0.34797121357690153]
  Cross-evaluation framework instantiated on two underrepresented Arabic dialect communities: Egyptian and Iraqi Arabic.<n>We contribute 103 validated prompt-rubric pairs authored and graded by native-speaker SMEs.<n>Across 1,307 judge evaluations: GPT-5.4 is the most reliable judge (MADj = 10.21 pp, Signed Error = -1.12%); four of five judges show systematic leniency (+2.01% to +6.56%)
  arXiv  Detail & Related papers  (2026-06-30T20:18:32Z)
• PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception [75.70273182182397]
  PerceptionRubrics is a rubric-based evaluation framework.<n>It addresses the gap between saturated benchmark scores and real-world brittleness.<n>It pairs 1,038 information-dense images with over 10,000 instance-specific rubrics.
  arXiv  Detail & Related papers  (2026-06-26T17:59:15Z)
• Benchmarking Speech-to-Speech Translation Models [55.00303727199927]
  Speech-to-speech translation (S2ST) has advanced rapidly, but offline evaluation lacks a unified protocol.<n>We introduce a benchmarking framework integrating 46 metrics across eight dimensions.<n>We deploy it on 1,248 model-language configurations from FLEURS and CVSS.
  arXiv  Detail & Related papers  (2026-06-02T07:01:33Z)
• Beyond Literal Translation: Evaluating Cultural Effectiveness in Social Media UGC [39.66998905774523]
  We introduce CULTURE-MT, a benchmark for social media translation.<n>CULTURE-MT consists of 1,002 notes across 14 domains, categorized into four types based on culture-loaded symbols and linguistic style features.<n>We propose cultural effectiveness as a new evaluation criterion, focusing on expression accuracy and cultural adaptability.
  arXiv  Detail & Related papers  (2026-05-25T09:29:25Z)
• Mitigating Cross-Lingual Cultural Inconsistencies in LLMs via Consensus-Driven Preference Optimisation [58.01855677487771]
  We show that multilingual large language models (MLLMs) frequently exhibit inconsistent behaviour when the prompt's language changes.<n>We propose Cross-lingual Cultural Consistent Preference optimisation (C-3PO), a consensus-driven alignment framework.<n>C-3PO achieves up to a 0.10-point absolute increase in $_S$ over unaligned models, outperforming strong prompting and representation steering baselines.
  arXiv  Detail & Related papers  (2026-04-02T14:04:06Z)
• From Words to Worlds: Benchmarking Cross-Cultural Cultural Understanding in Machine Translation [16.809989616664605]
  We introduce CulT-Eval, a benchmark designed to evaluate how models handle different types of culturally grounded expressions.<n>CulT-Eval comprises over 7,959 carefully curated instances spanning multiple types of culturally grounded expressions.<n>We propose a complementary evaluation metric that targets culturally induced meaning deviations.
  arXiv  Detail & Related papers  (2026-03-18T02:59:44Z)
• Distributional Open-Ended Evaluation of LLM Cultural Value Alignment Based on Value Codebook [50.10594064510559]
  DOVE is a distributional evaluation framework that directly compares human-written text with LLM-generated outputs.<n>We show that DOVE achieves superior predictive validity, attaining a 31.56% correlation with downstream tasks, while maintaining high reliability with as few as 500 samples per culture.
  arXiv  Detail & Related papers  (2026-03-16T08:33:10Z)
• "Be My Cheese?": Cultural Nuance Benchmarking for Machine Translation in Multilingual LLMs [0.0]
  We present a large-scale human evaluation benchmark for assessing cultural localisation in machine translation.<n>We evaluate 7 multilingual large language models (LLMs) across 15 target languages with 5 native-speaker raters per language.<n>Across full-text evaluations, mean overall quality is modest (1.68/3): GPT-5 (2.10/3), Claude Sonnet 3.7 (1.97/3), and Mistral Medium 3.1 (1.84/3) form the strongest tier with fewer catastrophic failures.
  arXiv  Detail & Related papers  (2026-02-04T16:35:48Z)
• CURE: Cultural Understanding and Reasoning Evaluation - A Framework for "Thick" Culture Alignment Evaluation in LLMs [24.598338950728234]
  Large language models (LLMs) are increasingly deployed in culturally diverse environments.<n>Existing methods focus on de-contextualized correctness or forced-choice judgments.<n>We introduce a set of benchmarks that present models with realistic situational contexts.
  arXiv  Detail & Related papers  (2025-11-15T03:39:13Z)
• Rethink about the Word-level Quality Estimation for Machine Translation from Human Judgement [57.72846454929923]
  We create a benchmark dataset, emphHJQE, where the expert translators directly annotate poorly translated words. We propose two tag correcting strategies, namely tag refinement strategy and tree-based annotation strategy, to make the TER-based artificial QE corpus closer to emphHJQE. The results show our proposed dataset is more consistent with human judgement and also confirm the effectiveness of the proposed tag correcting strategies.
  arXiv  Detail & Related papers  (2022-09-13T02:37:12Z)
• Understanding and Mitigating the Uncertainty in Zero-Shot Translation [92.25357943169601]
  We aim to understand and alleviate the off-target issues from the perspective of uncertainty in zero-shot translation. We propose two lightweight and complementary approaches to denoise the training data for model training. Our approaches significantly improve the performance of zero-shot translation over strong MNMT baselines.
  arXiv  Detail & Related papers  (2022-05-20T10:29:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.