Creativity Bias: How Machine Evaluation Struggles with Creativity in Literary Translations

• URL: http://arxiv.org/abs/2605.13596v1
• Date: Wed, 13 May 2026 14:30:41 GMT
• Title: Creativity Bias: How Machine Evaluation Struggles with Creativity in Literary Translations
• Abstract summary: This article investigates the performance of automatic evaluation metrics (AEMs) and LLM-as-a-judge evaluation on literary translation.<n>The aim is to assess how well these tools align with professionals when evaluating translation, creativity (creative shifts & errors) and see if they can substitute laborious manual annotations.
• Score: 2.6208201746444564
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: This article investigates the performance of automatic evaluation metrics (AEMs) and LLM-as-a-judge evaluation on literary translation across multiple languages, genres, and translation modalities. The aim is to assess how well these tools align with professionals when evaluating translation, creativity (creative shifts & errors), and see if they can substitute laborious manual annotations. A dataset of literary translations across three modalities (human translation, machine translation, and post-editing), three genres and three language pairs was created and annotated in detail for creativity by experienced professional literary translators. The results show that both AEMs and LLM-as-a-judge evaluations correlate poorly with professional evaluations on creativity, with LLM-as-a-judge showing a systematic bias in favour of machine-translated texts and penalising creative and culturally appropriate solutions. Moreover, performance is consistently worse for more literary genres such as poetry. This highlights fundamental limitations of current automatic evaluation tools for literary translation and the need to create new tools that do not frequently consider out of routine translations as errors.

Related papers

• Towards a Linguistic Evaluation of Narratives: A Quantitative Stylistic Framework [51.56484100374058]
  This work proposes a quantitative approach to narrative assessment by focusing on the linguistic dimension as a primary indicator of quality.<n>The paper presents a methodology for the automatic evaluation of narrative based on the extraction of a comprehensive set of 33 quantitative linguistic features.
  arXiv  Detail & Related papers  (2026-04-21T09:21:40Z)
• Automated evaluation of LLMs for effective machine translation of Mandarin Chinese to English [8.653960569736904]
  We utilise an automated machine learning framework featuring semantic and sentiment analysis to assess Mandarin Chinese to English translation.<n>We compare original and translated texts in various classes of high-profile Chinese texts, which include novel texts that span modern and classical literature, as well as news articles.<n>Our results indicate that the LLMs perform well in news media translation, but show divergence in their performance when applied to literary texts.
  arXiv  Detail & Related papers  (2026-02-15T22:43:06Z)
• COIG-Writer: A High-Quality Dataset for Chinese Creative Writing with Thought Processes [83.84578306665976]
  Large language models exhibit systematic deficiencies in creative writing, particularly in non-English contexts.<n>We present COIG-Writer, a novel Chinese creative writing dataset that captures both diverse outputs and their underlying thought processes.
  arXiv  Detail & Related papers  (2025-10-16T15:01:19Z)
• Liaozhai through the Looking-Glass: On Paratextual Explicitation of Culture-Bound Terms in Machine Translation [70.43884512651668]
  We formalize Genette's (1987) theory of paratexts from literary and translation studies to introduce the task of paratextual explicitation for machine translation.<n>We construct a dataset of 560 expert-aligned paratexts from four English translations of the classical Chinese short story collection Liaozhai.<n>Our findings demonstrate the potential of paratextual explicitation in advancing machine translation beyond linguistic equivalence.
  arXiv  Detail & Related papers  (2025-09-27T16:27:36Z)
• Specification-Aware Machine Translation and Evaluation for Purpose Alignment [10.50113943900077]
  We provide a theoretical rationale for why specifications matter in professional translation, as well as a practical guide to implementing specification-aware machine translation (MT)<n>We compare five translation types, including official human translations and prompt-based outputs from large language models (LLMs), using expert error analysis, user preference rankings, and an automatic metric.<n>The results show that translations guided by specifications consistently outperformed official human translations in human evaluations, highlighting a gap between perceived and expected quality.
  arXiv  Detail & Related papers  (2025-09-22T10:50:37Z)
• LiTransProQA: an LLM-based Literary Translation evaluation metric with Professional Question Answering [22.464208231092176]
  We introduce LITRANSPROQA, a reference-free, LLM-based question-answering framework designed for literary translation evaluation.<n> LITRANSPROQA integrates humans in the loop to incorporate insights from professional literary translators and researchers.<n>Our evaluation shows that while literary-finetuned XCOMET-XL yields marginal gains, LITRANSPROQA substantially outperforms current metrics.
  arXiv  Detail & Related papers  (2025-05-08T17:12:56Z)
• Extending CREAMT: Leveraging Large Language Models for Literary Translation Post-Editing [0.0]
  This study evaluates the feasibility of post-editing literary translations generated by large language models (LLMs)<n>Our results indicate that post-editing LLM-generated translations significantly reduces editing time compared to human translation while maintaining a similar level of creativity.
  arXiv  Detail & Related papers  (2025-04-03T21:48:09Z)
• Lost in Literalism: How Supervised Training Shapes Translationese in LLMs [51.04435855143767]
  Large language models (LLMs) have achieved remarkable success in machine translation.<n>However, translationese, characterized by overly literal and unnatural translations, remains a persistent challenge.<n>We introduce methods to mitigate these biases, including polishing golden references and filtering unnatural training instances.
  arXiv  Detail & Related papers  (2025-03-06T12:14:45Z)
• (Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Texts [56.7988577327046]
  We introduce TransAgents, a novel multi-agent framework that simulates the roles and collaborative practices of a human translation company.<n>Our findings highlight the potential of multi-agent collaboration in enhancing translation quality, particularly for longer texts.
  arXiv  Detail & Related papers  (2024-05-20T05:55:08Z)
• Large language models effectively leverage document-level context for literary translation, but critical errors persist [32.54546652197316]
  Large language models (LLMs) are competitive with the state of the art on a wide range of sentence-level translation datasets. We show through a rigorous human evaluation that asking the Gpt-3.5 (text-davinci-003) LLM to translate an entire literary paragraph results in higher-quality translations.
  arXiv  Detail & Related papers  (2023-04-06T17:27:45Z)
• Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale [52.663117551150954]
  A few popular metrics remain as the de facto metrics to evaluate tasks such as image captioning and machine translation. This is partly due to ease of use, and partly because researchers expect to see them and know how to interpret them. In this paper, we urge the community for more careful consideration of how they automatically evaluate their models.
  arXiv  Detail & Related papers  (2020-10-26T13:57:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.