Toward Machine Translation Literacy: How Lay Users Perceive and Rely on Imperfect Translations

• URL: http://arxiv.org/abs/2510.09994v1
• Date: Sat, 11 Oct 2025 03:45:14 GMT
• Title: Toward Machine Translation Literacy: How Lay Users Perceive and Rely on Imperfect Translations
• Abstract summary: We present a human study conducted in a public museum investigating how fluency and adequacy errors impact bilingual and non-bilingual users' reliance on Machine Translation.<n>Our findings reveal that non-bilingual users often over-rely on MT due to a lack of evaluation strategies and alternatives, while experiencing the impact of errors can prompt users to reassess future reliance.
• Score: 31.543560417943898
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As Machine Translation (MT) becomes increasingly commonplace, understanding how the general public perceives and relies on imperfect MT is crucial for contextualizing MT research in real-world applications. We present a human study conducted in a public museum (n=452), investigating how fluency and adequacy errors impact bilingual and non-bilingual users' reliance on MT during casual use. Our findings reveal that non-bilingual users often over-rely on MT due to a lack of evaluation strategies and alternatives, while experiencing the impact of errors can prompt users to reassess future reliance. This highlights the need for MT evaluation and NLP explanation techniques to promote not only MT quality, but also MT literacy among its users.

Related papers

• Beyond Literal Mapping: Benchmarking and Improving Non-Literal Translation Evaluation [57.11989521509119]
  We propose a novel agentic translation evaluation framework, centered by a reflective Core Agent that invokes specialized sub-agents.<n> Experimental results indicate the efficacy of RATE, achieving an improvement of at least 3.2 meta score compared with current metrics.
  arXiv  Detail & Related papers  (2026-01-12T09:03:42Z)
• An Interdisciplinary Approach to Human-Centered Machine Translation [67.70453480427132]
  Machine Translation (MT) tools are widely used today, often in contexts where professional translators are not present.<n>Despite progress in MT technology, a gap persists between system development and real-world usage.<n>This paper advocates for a human-centered approach to MT, emphasizing the alignment of system design with diverse communicative goals.
  arXiv  Detail & Related papers  (2025-06-16T13:27:44Z)
• Low-resource Machine Translation: what for? who for? An observational study on a dedicated Tetun language translation service [7.299910666525873]
  We propose an observational study on actual usage patterns of tetun$.$org, a specialized MT service for the Tetun language in Timor-Leste.<n>Our analysis of 100,000 translation requests reveals patterns that challenge assumptions based on existing corpora.<n>Our results suggest that MT systems for institutionalized minority languages like Tetun should prioritize accuracy on domains relevant to educational contexts.
  arXiv  Detail & Related papers  (2024-11-19T06:21:51Z)
• Cyber Risks of Machine Translation Critical Errors : Arabic Mental Health Tweets as a Case Study [3.8779763612314637]
  We introduce an authentic dataset of machine translation critical errors to point to the ethical and safety issues involved in the common use of MT. The dataset comprises mistranslations of Arabic mental health postings manually annotated with critical error types. We also show how the commonly used quality metrics do not penalise critical errors and highlight this as a critical issue that merits further attention from researchers.
  arXiv  Detail & Related papers  (2024-05-19T20:24:51Z)
• Discourse Centric Evaluation of Machine Translation with a Densely Annotated Parallel Corpus [82.07304301996562]
  This paper presents a new dataset with rich discourse annotations, built upon the large-scale parallel corpus BWB introduced in Jiang et al. We investigate the similarities and differences between the discourse structures of source and target languages. We discover that MT outputs differ fundamentally from human translations in terms of their latent discourse structures.
  arXiv  Detail & Related papers  (2023-05-18T17:36:41Z)
• Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models [57.80514758695275]
  Using large language models (LLMs) for assessing the quality of machine translation (MT) achieves state-of-the-art performance at the system level. We propose a new prompting method called textbftextttError Analysis Prompting (EAPrompt) This technique emulates the commonly accepted human evaluation framework - Multidimensional Quality Metrics (MQM) and textitproduces explainable and reliable MT evaluations at both the system and segment level.
  arXiv  Detail & Related papers  (2023-03-24T05:05:03Z)
• Competency-Aware Neural Machine Translation: Can Machine Translation Know its Own Translation Quality? [61.866103154161884]
  Neural machine translation (NMT) is often criticized for failures that happen without awareness. We propose a novel competency-aware NMT by extending conventional NMT with a self-estimator. We show that the proposed method delivers outstanding performance on quality estimation.
  arXiv  Detail & Related papers  (2022-11-25T02:39:41Z)
• A Semi-supervised Approach for a Better Translation of Sentiment in Dialectical Arabic UGT [2.6763498831034034]
  We introduce a semi-supervised approach that exploits both monolingual and parallel data for training an NMT system. We will show that our proposed system can significantly help with correcting sentiment errors detected in the online translation of dialectical Arabic UGT.
  arXiv  Detail & Related papers  (2022-10-21T11:55:55Z)
• Beyond General Purpose Machine Translation: The Need for Context-specific Empirical Research to Design for Appropriate User Trust [8.539683760001573]
  We discuss research directions to support users to calibrate trust in Machine Translation systems. Based on our findings, we advocate for empirical research on how MT systems are used in practice.
  arXiv  Detail & Related papers  (2022-05-13T23:04:22Z)
• When Does Translation Require Context? A Data-driven, Multilingual Exploration [71.43817945875433]
  proper handling of discourse significantly contributes to the quality of machine translation (MT) Recent works in context-aware MT attempt to target a small set of discourse phenomena during evaluation. We develop the Multilingual Discourse-Aware benchmark, a series of taggers that identify and evaluate model performance on discourse phenomena.
  arXiv  Detail & Related papers  (2021-09-15T17:29:30Z)
• Difficulty-Aware Machine Translation Evaluation [19.973201669851626]
  We propose a novel difficulty-aware machine translation evaluation metric. A translation that fails to be predicted by most MT systems will be treated as a difficult one and assigned a large weight in the final score function. Our proposed method performs well even when all the MT systems are very competitive.
  arXiv  Detail & Related papers  (2021-07-30T02:45:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.