A 2-step Framework for Automated Literary Translation Evaluation: Its Promises and Pitfalls

• URL: http://arxiv.org/abs/2412.01340v2
• Date: Thu, 02 Jan 2025 03:29:31 GMT
• Title: A 2-step Framework for Automated Literary Translation Evaluation: Its Promises and Pitfalls
• Authors: Sheikh Shafayat, Dongkeun Yoon, Woori Jang, Jiwoo Choi, Alice Oh, Seohyon Jung,
• Abstract summary: We propose and evaluate the feasibility of a two-stage pipeline to evaluate literary machine translation. Our framework provides fine-grained, interpretable metrics suited for literary translation.
• Score: 15.50296318831118
• License:
• Abstract: In this work, we propose and evaluate the feasibility of a two-stage pipeline to evaluate literary machine translation, in a fine-grained manner, from English to Korean. The results show that our framework provides fine-grained, interpretable metrics suited for literary translation and obtains a higher correlation with human judgment than traditional machine translation metrics. Nonetheless, it still fails to match inter-human agreement, especially in metrics like Korean Honorifics. We also observe that LLMs tend to favor translations generated by other LLMs, and we highlight the necessity of developing more sophisticated evaluation methods to ensure accurate and culturally sensitive machine translation of literary works.

Related papers

• DRT: Deep Reasoning Translation via Long Chain-of-Thought [89.48208612476068]
  In this paper, we introduce DRT, an attempt to bring the success of long CoT to neural machine translation (MT) We first mine sentences containing similes or metaphors from existing literature books, and then develop a multi-agent framework to translate these sentences via long thought. Using Qwen2.5 and LLama-3.1 as the backbones, DRT models can learn the thought process during machine translation.
  arXiv  Detail & Related papers  (2024-12-23T11:55:33Z)
• LLM-based Translation Inference with Iterative Bilingual Understanding [52.46978502902928]
  We propose a novel Iterative Bilingual Understanding Translation method based on the cross-lingual capabilities of large language models (LLMs) The cross-lingual capability of LLMs enables the generation of contextual understanding for both the source and target languages separately. The proposed IBUT outperforms several strong comparison methods.
  arXiv  Detail & Related papers  (2024-10-16T13:21:46Z)
• Large Language Models for Classical Chinese Poetry Translation: Benchmarking, Evaluating, and Improving [43.148203559785095]
  Large language models (LLMs) with impressive multilingual capabilities may bring a ray of hope to achieve this extreme translation demand. This paper first introduces a suitable benchmark (PoetMT) where each Chinese poetry has a recognized elegant translation. We propose a new metric based on GPT-4 to evaluate the extent to which current LLMs can meet these demands.
  arXiv  Detail & Related papers  (2024-08-19T12:34:31Z)
• Understanding and Addressing the Under-Translation Problem from the Perspective of Decoding Objective [72.83966378613238]
  Under-translation and over-translation remain two challenging problems in state-of-the-art Neural Machine Translation (NMT) systems. We conduct an in-depth analysis on the underlying cause of under-translation in NMT, providing an explanation from the perspective of decoding objective. We propose employing the confidence of predicting End Of Sentence (EOS) as a detector for under-translation, and strengthening the confidence-based penalty to penalize candidates with a high risk of under-translation.
  arXiv  Detail & Related papers  (2024-05-29T09:25:49Z)
• (Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Texts [52.18246881218829]
  We introduce a novel multi-agent framework based on large language models (LLMs) for literary translation, implemented as a company called TransAgents. To evaluate the effectiveness of our system, we propose two innovative evaluation strategies: Monolingual Human Preference (MHP) and Bilingual LLM Preference (BLP)
  arXiv  Detail & Related papers  (2024-05-20T05:55:08Z)
• Large language models effectively leverage document-level context for literary translation, but critical errors persist [32.54546652197316]
  Large language models (LLMs) are competitive with the state of the art on a wide range of sentence-level translation datasets. We show through a rigorous human evaluation that asking the Gpt-3.5 (text-davinci-003) LLM to translate an entire literary paragraph results in higher-quality translations.
  arXiv  Detail & Related papers  (2023-04-06T17:27:45Z)
• Exploring Document-Level Literary Machine Translation with Parallel Paragraphs from World Literature [35.1398797683712]
  We show that literary translators prefer reference human translations over machine-translated paragraphs at a rate of 84%. We train a post-editing model whose output is preferred over normal MT output at a rate of 69% by experts.
  arXiv  Detail & Related papers  (2022-10-25T18:03:34Z)
• On the Limitations of Cross-lingual Encoders as Exposed by Reference-Free Machine Translation Evaluation [55.02832094101173]
  Evaluation of cross-lingual encoders is usually performed either via zero-shot cross-lingual transfer in supervised downstream tasks or via unsupervised cross-lingual similarity. This paper concerns ourselves with reference-free machine translation (MT) evaluation where we directly compare source texts to (sometimes low-quality) system translations. We systematically investigate a range of metrics based on state-of-the-art cross-lingual semantic representations obtained with pretrained M-BERT and LASER. We find that they perform poorly as semantic encoders for reference-free MT evaluation and identify their two key limitations.
  arXiv  Detail & Related papers  (2020-05-03T22:10:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.