Culture-Aware Machine Translation in Large Language Models: Benchmarking and Investigation

• URL: http://arxiv.org/abs/2604.24361v1
• Date: Mon, 27 Apr 2026 11:53:50 GMT
• Title: Culture-Aware Machine Translation in Large Language Models: Benchmarking and Investigation
• Abstract summary: Large language models (LLMs) have achieved strong performance in general machine translation, yet their ability in culture-aware scenarios remains poorly understood.<n>We introduce CanMT, a Culture-Aware Novel-Driven Parallel dataset for Machine Translation, together with a theoretically grounded, multi-dimensional evaluation framework for assessing cultural translation quality.
• Score: 36.27108860941823
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) have achieved strong performance in general machine translation, yet their ability in culture-aware scenarios remains poorly understood. To bridge this gap, we introduce CanMT, a Culture-Aware Novel-Driven Parallel Dataset for Machine Translation, together with a theoretically grounded, multi-dimensional evaluation framework for assessing cultural translation quality. Leveraging CanMT, we systematically evaluate a wide range of LLMs and translation systems under different translation strategy constraints. Our findings reveal substantial performance disparities across models and demonstrate that translation strategies exert a systematic influence on model behavior. Further analysis shows that translation difficulty varies across types of culture-specific items, and that a persistent gap remains between models' recognition of culture-specific knowledge and their ability to correctly operationalize it in translation outputs. In addition, incorporating reference translations is shown to substantially improve evaluation reliability in LLM-as-a-judge, underscoring their essential role in assessing culture-aware translation quality. The corpus and code are available at CanMT.

Related papers

• Beyond Literal Mapping: Benchmarking and Improving Non-Literal Translation Evaluation [57.11989521509119]
  We propose a novel agentic translation evaluation framework, centered by a reflective Core Agent that invokes specialized sub-agents.<n> Experimental results indicate the efficacy of RATE, achieving an improvement of at least 3.2 meta score compared with current metrics.
  arXiv  Detail & Related papers  (2026-01-12T09:03:42Z)
• Liaozhai through the Looking-Glass: On Paratextual Explicitation of Culture-Bound Terms in Machine Translation [70.43884512651668]
  We formalize Genette's (1987) theory of paratexts from literary and translation studies to introduce the task of paratextual explicitation for machine translation.<n>We construct a dataset of 560 expert-aligned paratexts from four English translations of the classical Chinese short story collection Liaozhai.<n>Our findings demonstrate the potential of paratextual explicitation in advancing machine translation beyond linguistic equivalence.
  arXiv  Detail & Related papers  (2025-09-27T16:27:36Z)
• How Well Do Large Reasoning Models Translate? A Comprehensive Evaluation for Multi-Domain Machine Translation [11.5400003750047]
  Large language models (LLMs) have demonstrated strong performance in general-purpose machine translation.<n>Recent advancements in LRMs, raise the question of whether structured reasoning can enhance translation quality across diverse domains.
  arXiv  Detail & Related papers  (2025-05-26T13:43:37Z)
• Bridging the Linguistic Divide: A Survey on Leveraging Large Language Models for Machine Translation [33.08089616645845]
  Large Language Models (LLMs) have reshaped the landscape of machine translation (MT)<n>We analyze techniques such as few-shot prompting, cross-lingual transfer, and parameter-efficient fine-tuning.<n>We discuss persistent challenges - such as hallucinations, evaluation inconsistencies, and inherited biases.
  arXiv  Detail & Related papers  (2025-04-02T17:26:40Z)
• LLM-based Translation Inference with Iterative Bilingual Understanding [52.46978502902928]
  We propose a novel Iterative Bilingual Understanding Translation method based on the cross-lingual capabilities of large language models (LLMs)<n>The cross-lingual capability of LLMs enables the generation of contextual understanding for both the source and target languages separately.<n>The proposed IBUT outperforms several strong comparison methods.
  arXiv  Detail & Related papers  (2024-10-16T13:21:46Z)
• The Fine-Tuning Paradox: Boosting Translation Quality Without Sacrificing LLM Abilities [18.175795328685986]
  Fine-tuning large language models (LLMs) for machine translation has shown improvements in overall translation quality. We perform an extensive translation evaluation on the LLaMA and Falcon family of models with model size ranging from 7 billion up to 65 billion parameters. We observe a decline in the ability to perform formality steering, to produce technical translations through few-shot examples, and to perform document-level translation.
  arXiv  Detail & Related papers  (2024-05-30T14:25:56Z)
• MT-PATCHER: Selective and Extendable Knowledge Distillation from Large Language Models for Machine Translation [61.65537912700187]
  Large Language Models (LLM) have demonstrated their strong ability in the field of machine translation (MT) We propose a framework called MT-Patcher, which transfers knowledge from LLMs to existing MT models in a selective, comprehensive and proactive manner.
  arXiv  Detail & Related papers  (2024-03-14T16:07:39Z)
• Lost in the Source Language: How Large Language Models Evaluate the Quality of Machine Translation [64.5862977630713]
  This study investigates how Large Language Models (LLMs) leverage source and reference data in machine translation evaluation task. We find that reference information significantly enhances the evaluation accuracy, while surprisingly, source information sometimes is counterproductive.
  arXiv  Detail & Related papers  (2024-01-12T13:23:21Z)
• Benchmarking Machine Translation with Cultural Awareness [50.183458829028226]
  Translating culture-related content is vital for effective cross-cultural communication. Many culture-specific items (CSIs) often lack viable translations across languages. This difficulty hinders the analysis of cultural awareness of machine translation systems.
  arXiv  Detail & Related papers  (2023-05-23T17:56:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.