AI translation of literary texts is "fine", but readers still prefer human translations

• URL: http://arxiv.org/abs/2606.26040v1
• Date: Wed, 24 Jun 2026 17:15:14 GMT
• Title: AI translation of literary texts is "fine", but readers still prefer human translations
• Abstract summary: We ask avid readers to compare recently published human translations (HT) to machine translations (MT)<n>Readers evaluated 8K-word excerpts in two conditions: immersive reading of the whole excerpt and close reading of 386 aligned HT-MT chunk pairs.<n>Overall, readers find MT "fine", but prefer HT (slightly at excerpt-level 19/30, more clearly at chunk-level 522/772) for its ease, clarity, and immersive nature.
• Score: 7.62066411906057
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: AI translation of literary works is increasingly common. While the content may be rendered adequately, we do not know enough about how readers experience it in terms of immersiveness and literary effect, aspects poorly captured by automatic machine translation metrics or human evaluation targeting fluency and adequacy. We ask 15 avid readers to compare recently published human translations (HT) to machine translations (MT) generated with an agentic large language model (LLM)-based pipeline, for 15 recent novels in French, Polish, and Japanese and translated into English. Readers evaluated approximately 8K-word excerpts in two conditions: immersive reading of the whole excerpt (30 comparisons) and close reading of 386 aligned HT-MT chunk pairs (772 comparisons), with two readers per book and in alternating order of presentation. Overall, readers find MT "fine", but prefer HT (slightly at excerpt-level 19/30, more clearly at chunk-level 522/772) for its ease, clarity, and immersive nature. Readers' highlights show that MT's quality varies more within one book than HT's does. Crucially, readers cannot reliably tell the two apart (17/30 guess correctly) and tend to prefer the version they believe to be human. Automatic metrics, including LLM-as-a-judge approaches, fail to recover reader preferences and favor MT. We release LAIT (Literary AI Translation), a reader-centered evaluation dataset with 1K reader comments, 2K judgments and preference ratings, and 7.2K span-level annotations, along with our evaluation protocol and supporting interface.

Related papers

• Beyond Literal Mapping: Benchmarking and Improving Non-Literal Translation Evaluation [57.11989521509119]
  We propose a novel agentic translation evaluation framework, centered by a reflective Core Agent that invokes specialized sub-agents.<n> Experimental results indicate the efficacy of RATE, achieving an improvement of at least 3.2 meta score compared with current metrics.
  arXiv  Detail & Related papers  (2026-01-12T09:03:42Z)
• DITING: A Multi-Agent Evaluation Framework for Benchmarking Web Novel Translation [31.1561882673283]
  DITING is the first comprehensive evaluation framework for web novel translation.<n>AgentEval simulates expert deliberation to assess translation quality beyond lexical overlap.<n>We develop MetricAlign, a meta-evaluation dataset of 300 sentence pairs annotated with error labels and scalar quality scores.
  arXiv  Detail & Related papers  (2025-10-10T08:10:10Z)
• Do LLMs Understand Your Translations? Evaluating Paragraph-level MT with Question Answering [68.3400058037817]
  We introduce TREQA (Translation Evaluation via Question-Answering), a framework that extrinsically evaluates translation quality.<n>We show that TREQA is competitive with and, in some cases, outperforms state-of-the-art neural and LLM-based metrics in ranking alternative paragraph-level translations.
  arXiv  Detail & Related papers  (2025-04-10T09:24:54Z)
• DRT: Deep Reasoning Translation via Long Chain-of-Thought [73.21414200780171]
  We introduce DRT, an attempt to bring the success of long chain-of-thought to neural machine translation (MT)<n>We first mine sentences containing similes or metaphors from existing literature books, and then develop a multi-agent framework to translate these sentences via long thought.<n>Using Qwen2.5 and LLama-3.1 as the backbones, DRT models can learn the thought process during machine translation.
  arXiv  Detail & Related papers  (2024-12-23T11:55:33Z)
• A 2-step Framework for Automated Literary Translation Evaluation: Its Promises and Pitfalls [23.30119735386842]
  We propose and evaluate the feasibility of a two-stage pipeline to evaluate literary machine translation.<n>Our framework provides fine-grained, interpretable metrics suited for literary translation.
  arXiv  Detail & Related papers  (2024-12-02T10:07:01Z)
• (Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Texts [56.7988577327046]
  We introduce TransAgents, a novel multi-agent framework that simulates the roles and collaborative practices of a human translation company.<n>Our findings highlight the potential of multi-agent collaboration in enhancing translation quality, particularly for longer texts.
  arXiv  Detail & Related papers  (2024-05-20T05:55:08Z)
• Findings of the WMT 2023 Shared Task on Discourse-Level Literary Translation: A Fresh Orb in the Cosmos of LLMs [80.05205710881789]
  We release a copyrighted and document-level Chinese-English web novel corpus. This year, we totally received 14 submissions from 7 academia and industry teams. The official ranking of the systems is based on the overall human judgments.
  arXiv  Detail & Related papers  (2023-11-06T14:23:49Z)
• To be or not to be: a translation reception study of a literary text translated into Dutch and Catalan using machine translation [3.3453601632404073]
  This article presents the results of a study involving the reception of a fictional story by Kurt Vonnegut translated from English into Catalan and Dutch. 223 participants were recruited who rated the reading conditions using three scales: Narrative Engagement, Enjoyment and Translation Reception.
  arXiv  Detail & Related papers  (2023-07-05T15:18:52Z)
• Discourse Centric Evaluation of Machine Translation with a Densely Annotated Parallel Corpus [82.07304301996562]
  This paper presents a new dataset with rich discourse annotations, built upon the large-scale parallel corpus BWB introduced in Jiang et al. We investigate the similarities and differences between the discourse structures of source and target languages. We discover that MT outputs differ fundamentally from human translations in terms of their latent discourse structures.
  arXiv  Detail & Related papers  (2023-05-18T17:36:41Z)
• Exploring Document-Level Literary Machine Translation with Parallel Paragraphs from World Literature [35.1398797683712]
  We show that literary translators prefer reference human translations over machine-translated paragraphs at a rate of 84%. We train a post-editing model whose output is preferred over normal MT output at a rate of 69% by experts.
  arXiv  Detail & Related papers  (2022-10-25T18:03:34Z)
• ChrEnTranslate: Cherokee-English Machine Translation Demo with Quality Estimation and Corrective Feedback [70.5469946314539]
  ChrEnTranslate is an online machine translation demonstration system for translation between English and an endangered language Cherokee. It supports both statistical and neural translation models as well as provides quality estimation to inform users of reliability.
  arXiv  Detail & Related papers  (2021-07-30T17:58:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.